Skip to content
SwankyForge
SwankyForge
All insights
Tutorial11 min read

Recommendations for an online store: ship a co-view baseline first, add two-tower retrieval only when a test says it pays

A two-stage plan for product recommendations, with the business case, working pandas code for the baseline, the architecture that comes after it, and how to prove either one in an A/B test.

Product recommendations are one of the few ML systems an online store can test cleanly: show half the visitors one version, half the other, and compare revenue. That makes them a good first ML project, and also an easy one to overbuild. This note describes the order we build them in: a co-view baseline that ships in weeks, then candidate retrieval and ranking if the numbers justify it, with code for the parts that fit in a page.

For the business reader

What recommendations are worth. Salesforce reported that during Cyber Week 2025, AI and agents influenced about 20% of global orders, around $67 billion in sales, through personalised product recommendations and conversational customer service. Read that number carefully: "influenced" is attribution on Salesforce's own commerce platform, not a measured uplift, and it mixes recommendations with chat. It tells you recommendations are now standard on large stores. It does not tell you what they would add to yours. Only a test on your traffic does.

Where they sit. Each placement answers a different question and should be measured separately.

Placement Question it answers Typical source
Product page What else might fit, if not this one? Similar and co-viewed items
Cart What goes with what is already chosen? Co-purchased items, accessories
Home and category pages What is this shopper likely to want today? Personalised ranking
Search results Which relevant items is this shopper likely to buy? Ranking on top of search relevance
E-mail and push What brings this shopper back? The same models, run in batch

What to measure. Agree one primary metric before any code is written: revenue per session or orders per session across the whole site, not clicks on the widget. Clicks on recommendations can rise while revenue stays flat, because shoppers click a recommended item they would have found anyway. Widget click-through is a useful diagnostic, not a success criterion.

When it is not worth it. Two cases. A small catalogue, a few hundred products that shoppers can browse in full, rarely needs a model; a curated list and a popularity sort do the job. Low traffic is the second case, and the arithmetic is unforgiving. To detect a 5% relative lift on a 2% conversion rate (2.0% to 2.1%) with the usual settings (5% significance, 80% power), you need about 315,000 sessions in each arm. A store with 100,000 sessions a month would wait more than six months for one answer. A 10% lift needs about 81,000 sessions per arm. If your traffic cannot answer the question, a model cannot be proven to help, and you are paying for a belief.

Cost and timeline. Our band for a recommendation system is €14k–42k over 2–4 months, depending on how far past the baseline the project goes. An illustrative split is at the end of this note.

Stage 1: co-views with a popularity fallback

The baseline is item-to-item: products that appear in the same sessions are likely related. The idea is old, and a 2003 IEEE Internet Computing paper by Amazon engineers described the item-to-item approach at scale: compute similar items offline, so that the online lookup cost does not grow with the number of customers or products. It is still the right first step, because it needs nothing but the event log, it is easy to explain to a merchandiser, and it gives every later model a number to beat.

Input: one row per product view (or add-to-cart, or purchase) with session_id, item_id and ts.

import numpy as np
import pandas as pd

def co_view_scores(events, max_items_per_session=50, min_pairs=3):
    # one row per session and item; long sessions are capped to keep the join small
    s = (events.sort_values("ts")
               .drop_duplicates(["session_id", "item_id"], keep="last")
               .groupby("session_id").tail(max_items_per_session)[["session_id", "item_id"]])
    pairs = s.merge(s, on="session_id", suffixes=("_a", "_b"))
    pairs = pairs[pairs.item_id_a != pairs.item_id_b]
    co = pairs.groupby(["item_id_a", "item_id_b"]).size().rename("co").reset_index()
    co = co[co.co >= min_pairs]                     # drop pairs seen in too few sessions
    n_item = s.groupby("item_id").size()            # sessions that viewed each item
    co["score"] = co.co / np.sqrt(co.item_id_a.map(n_item) * co.item_id_b.map(n_item))
    return co

def top_k(co, k=20):
    return (co.sort_values(["item_id_a", "score"], ascending=[True, False])
              .groupby("item_id_a").head(k))

Three choices in this code matter more than the model ever will.

  • Normalisation. Raw co-occurrence counts recommend bestsellers next to everything. Dividing by the square root of both items' session counts (a cosine score) rewards pairs that appear together more often than their popularity predicts. A lift score, co * n_sessions / (n_a * n_b), pushes further towards niche pairs and needs a stricter minimum count.
  • Minimum support. Without min_pairs, two items viewed together once by one person get a perfect score. Tune the threshold on held-out data, not by eye.
  • Window and signal. Use a rolling window (60–90 days is a common start) so seasonal items fade out. Co-views find alternatives for the product page; co-purchases find complements for the cart. Build both from the same function with different event types.

For items with too few neighbours, fill the slots with popular items from the same category:

def recommend(item_id, recs, catalog, popular_by_cat, k=10):
    own = recs.loc[recs.item_id_a == item_id, "item_id_b"].tolist()[:k]
    if len(own) < k:                                # fallback: popular in the same category
        cat = catalog.set_index("item_id").at[item_id, "category"]
        own += [i for i in popular_by_cat[cat] if i != item_id and i not in own][: k - len(own)]
    return own

In production this becomes a nightly batch job that writes the top 20 neighbours per item to a key-value store, and an API that reads them. That is enough to run the first A/B test against whatever the store shows today.

Stage 2: retrieve candidates, then rank them

Item-to-item ignores who the shopper is. Personalisation needs a model that scores items for a person, and scoring a whole catalogue for every page view is too slow. The standard answer is two stages, described in the YouTube recommendations paper by Covington, Adams and Sargin (RecSys 2016): a candidate generation stage narrows a corpus of millions to hundreds of items, and a separate ranking model orders those hundreds using a much richer set of features.

Candidate generation with a two-tower model. One network embeds the shopper (recent views, cart, context such as device and entry page), another embeds the item (category, brand, price, text). Training pushes a shopper's embedding close to the items they interacted with. At serving time, item embeddings are precomputed and indexed for approximate nearest-neighbour search, so retrieval is one embedding computation and one index lookup.

The usual training trick is in-batch negatives: for each shopper in a batch, the other shoppers' items serve as negatives. That trick has a bias. Popular items appear in many batches, so they are over-used as negatives and the model learns to under-rank them. Yi et al. (RecSys 2019) showed how to correct for it by subtracting the log of each item's estimated sampling frequency from its score, and deployed the corrected model at YouTube over a corpus of tens of millions of videos. In PyTorch, the core is short:

import torch
import torch.nn as nn
import torch.nn.functional as F

class Tower(nn.Module):
    def __init__(self, n_ids, dim=64):
        super().__init__()
        self.emb = nn.Embedding(n_ids, dim)
        self.mlp = nn.Sequential(nn.Linear(dim, dim), nn.ReLU(), nn.Linear(dim, dim))

    def forward(self, ids):
        return F.normalize(self.mlp(self.emb(ids)), dim=-1)

user_tower, item_tower = Tower(n_users), Tower(n_items)

def in_batch_loss(user_ids, item_ids, log_q, temperature=0.05):
    u = user_tower(user_ids)                 # [B, d]
    v = item_tower(item_ids)                 # [B, d]
    logits = u @ v.T / temperature           # other items in the batch act as negatives
    logits = logits - log_q[item_ids]        # logQ correction for popular items
    labels = torch.arange(len(user_ids))     # the diagonal holds the true pairs
    return F.cross_entropy(logits, labels)

Here log_q is the log of each item's share of training interactions. The towers take IDs to keep the example small; in a real store the user tower takes the last N viewed items and context rather than a user ID, which also means a first-time visitor still gets an embedding after one click.

Keep the stage 1 co-view lists as a second candidate source. Two-tower retrieval and co-views fail in different ways, and the ranker can use both.

Ranking. The ranker sees a few hundred candidates per request and can afford many features: the scores from each candidate source, price relative to the shopper's usual price band, whether the item shares a category with recent views, the item's recent view trend, days since the item was added. A gradient-boosted learning-to-rank model is a common first ranker, and its feature importances are easy to inspect:

import lightgbm as lgb

# one row per (request, candidate); label 1 if the candidate was clicked or bought
features = ["covis_score", "two_tower_score", "price_vs_user_median",
            "same_category_as_viewed", "item_views_7d_trend", "days_since_added"]
train = train.sort_values("request_id")
ranker = lgb.LGBMRanker(objective="lambdarank", n_estimators=300, learning_rate=0.05)
ranker.fit(train[features], train["label"],
           group=train.groupby("request_id", sort=False).size().to_numpy())

Business rules after the model. Stock, margin and policy belong after ranking, in plain code the business can read and change: drop items that cannot ship, cap the number of items per brand in one block, exclude categories that must not be cross-sold, and if margin matters, apply a bounded boost rather than retraining the model on margin. Keeping rules outside the model means a merchandiser can change them on a Tuesday without an ML release.

Offline evaluation, and why it is not the answer

Split by time, not at random: train on everything before a cutoff date, evaluate on the week after it. A random split leaks the future into training and inflates every metric.

Measure each stage with its own metric. For candidate generation, recall@k: of the items the shopper actually engaged with next, how many were in the candidate list. For ranking, NDCG@k: did those items land near the top.

def recall_at_k(recommended, relevant, k=20):
    return len(set(recommended[:k]) & relevant) / len(relevant)

def ndcg_at_k(recommended, relevant, k=10):
    dcg = sum(1 / np.log2(i + 2) for i, item in enumerate(recommended[:k]) if item in relevant)
    ideal = sum(1 / np.log2(i + 2) for i in range(min(len(relevant), k)))
    return dcg / ideal

Offline numbers decide which model goes into the test. They do not decide whether it ships. The logs were produced by the old system, so the new model is graded on what shoppers did when shown something else, and position on the page shapes clicks regardless of relevance. Covington et al. wrote that "live A/B results are not always correlated with offline experiments", and the two reasons above are enough to expect it.

The A/B test

  • One primary metric, the one agreed at the start, such as revenue per session. Decide the minimum lift worth detecting and compute the sample size before launch, as in the arithmetic above.
  • Randomise by visitor, not by page view, so one shopper does not see both versions.
  • Run whole weeks, at least two, because weekday and weekend shoppers behave differently. Do not stop early because the curve looks good on day four.
  • Guardrail metrics that must not get worse: page response time, the share of recommended items that are out of stock when clicked, return rate on orders containing recommended items, and revenue in categories the new model shows less often.
  • Keep the old system as the control until the primary metric says otherwise, and keep the switch to roll back.

Cold start

New products have no interactions, so neither co-views nor the item tower's ID embedding know them. Place them using attributes (category, brand, price band, text and image embeddings) and reserve a small number of slots in each block for recently added items, so they collect the interactions they need. New visitors get context-based candidates (the page they landed on, the category, the device) and popular items, and move to personalised retrieval after their first few views in the session.

What it costs, stage by stage

Our band for a recommendation system is €14k–42k over 2–4 months. The table below is an illustrative split for a mid-sized project that goes through both stages, priced from our published rate card. It is not a quote and not a record of a past project.

Stage Roles and hours Cost
Data audit, metric, A/B test design Data 30 h at €45, Senior ML 20 h at €70 €2,750
Stage 1: co-views, fallback, API ML 60 h at €50, Backend 40 h at €40 €4,600
Stage 2: two-tower retrieval and ranker Senior ML 80 h at €70, ML 80 h at €50 €9,600
A/B test run and readout Senior ML 15 h at €70, Data 20 h at €45 €1,950
Pipelines, monitoring, rollback MLOps 80 h at €55 €4,400
Across all stages QA 30 h at €30, PM 50 h at €40 €2,900
Total 505 h €26,200

A project that stops after stage 1, because the first test answered the business question, sits at the lower end of the band. Catalogues with many data sources, several placements tested separately, or strict response-time budgets push towards the upper end. The recommendation systems page describes how such a project runs, and the calculator on /pricing prices your own scope from the same rate card.

Sources

Related solutionRecommendation systems for e-commerce€14k–€42k · 2–4 months

Have a process that costs too much?

Tell us about it. The first call is free, and if AI is not the answer we say so.

Discuss your task