Sequential recommendation

Predict the next item a user will interact with from their past interactions. I train four models on the same split and compare them on two datasets: MovieLens-1M (movie ratings) and RetailRocket (real e-commerce browsing sessions).

Models: two baselines (popularity, first-order Markov) and two sequence models (GRU4Rec, SASRec).

Two evaluations, answering two different questions. Ranking metrics (Recall@K / NDCG@K / MRR) answer "is the model good at guessing the next item?". Off-policy evaluation (src/offpolicy.py) answers the question a business actually has — "if we had shown this recommender's picks, would people have converted more?" — which is causal, and cannot be read off the logs directly. It is estimated from logged data alone, without deploying anything, using a self-normalised inverse-propensity-scoring (SNIPS) estimator over a contextual bandit formulation. This is the same problem structure as offline LLM evaluation: scoring a policy you have not deployed, from data generated by a different one.

Setup

pip install -r requirements.txt

Get a dataset (links in data/README.md):

• MovieLens-1M unzips to data/ml-1m/ratings.dat
• RetailRocket's events.csv goes at data/events.csv

Running

Pick the dataset with --dataset (default movielens). Baselines first, they run on CPU in seconds and check the data pipeline:

python -m src.train --model popularity --dataset retailrocket
python -m src.train --model markov --dataset retailrocket

Sequence models:

python -m src.train --model gru4rec --dataset retailrocket --epochs 15
python -m src.train --model sasrec --dataset retailrocket --epochs 15 --save model_retailrocket.pt

Training is seeded (--seed, default 42). Checkpoints store the model weights, max_len and the raw-id mapping.

Serve top-N from a saved SASRec model:

# macOS / Linux
MODEL_PATH=model_retailrocket.pt python -m uvicorn api.app:app
# Windows (cmd):  set MODEL_PATH=model_retailrocket.pt  then  python -m uvicorn api.app:app

curl -X POST localhost:8000/recommend -H "content-type: application/json"      -d '{"history": [ID1, ID2, ID3], "k": 10}'   # replace with real item ids

history holds raw dataset item ids, oldest first (MovieLens movieId or RetailRocket itemid), and the response returns raw ids too. Unknown ids, and k outside 1..100, get a 422. Checkpoints saved before the id mapping was added still load; for those, history uses internal indices. By default, items already in the history are left out of the results ("exclude_seen": false turns this off). The offline evaluation is different: there the target can be an item the user already saw.

With Docker (the checkpoint is mounted, not baked into the image):

docker build -t seqrec .
docker run -p 8000:8000 -v "$PWD/model_retailrocket.pt:/models/model.pt:ro" seqrec

Tests (no dataset or GPU needed):

pip install -r requirements-dev.txt
python -m pytest

Evaluation

Ranking quality

Leave-last-out. For each user the last interaction is the test target, the one before it is validation, and everything earlier is the history the model sees. The target is ranked against 100 sampled negatives. I report Recall@K, NDCG@K and MRR.

Off-policy / uplift (SNIPS)

Ranking metrics do not tell you whether deploying the model would change behaviour. src/offpolicy.py estimates that counterfactually from the RetailRocket logs:

• contextual bandit: context = the visitor's history, action = the item they interacted with next, reward = 1 if it converted (add-to-cart / transaction)
• logging policy π₀ modelled as popularity with add-one smoothing, because RetailRocket is organic traffic and carries no true propensities
• SNIPS value V(π) = Σ wᵢrᵢ / Σ wᵢ, wᵢ = π(aᵢ|xᵢ) / π₀(aᵢ); uplift is V(π) − V(π₀) against the logged conversion rate

Three things are reported rather than hidden, because IPS is only unbiased when π₀ has full support and is roughly right, and its variance explodes when π and π₀ disagree: weights are clipped, and the effective sample size (ESS) and clipped fraction are published alongside every estimate. π₀ is estimated, not logged — the popularity model is a proxy, and the numbers are only as good as that assumption.

Results

Provenance. The numbers below come from local runs of the commands above. The run logs and metric files were not committed, so nothing in this repo backs them yet. Training was not seeded when they were produced, so a re-run will differ slightly. Re-running python -m src.train ... and python -m src.offpolicy ... --json results.json regenerates them.

MovieLens-1M (6040 users, 3706 items)

Model	Recall@10	NDCG@10	MRR
Popularity	0.453	0.250	0.210
Markov (first-order)	0.916	0.621	0.535
GRU4Rec	0.746	0.531	0.475
SASRec	0.769	0.611	0.573

SASRec beats GRU4Rec on every metric and has the best MRR. The Markov chain is a strong baseline: on MovieLens the previous item already predicts the next one well.

RetailRocket e-commerce (22.9k visitors, 11.2k items, items with <50 events dropped)

Model	Recall@10	NDCG@10	MRR
Popularity	0.342	0.186	0.163
Markov (first-order)	0.997	0.958	0.944
GRU4Rec	0.810	0.704	0.677
SASRec	0.804	0.693	0.665

The e-commerce numbers are higher than MovieLens across the board. Browsing sessions are short and repetitive (visitors move between a handful of items), so next-item prediction against sampled negatives is easier: the first-order Markov baseline sits near the ceiling because the previous item usually predicts the next one outright. GRU4Rec and SASRec land close together (~0.80 Recall@10); neither sequence model beats the Markov baseline on this dataset, which is the honest result for such short sessions.

Correction. An earlier version of this table reported SASRec at Recall@10 0.994. That was a masking bug: for any left-padded history (almost every user) the self-attention produced nan at the final position, and nan compares false against every negative, so the ranker scored those users as a perfect rank 1. Fixing the mask (see SASRec.forward in src/models.py) removes the inflation and brings SASRec back in line with GRU4Rec. The same bug also silently broke the off-policy evaluation below until it was fixed.

Off-policy / uplift evaluation (causal)

Ranking metrics answer "is the model good at guessing the next item?". A business has a different question: if we had shown this recommender's picks, would people have converted more? You can't read that off the logs directly — the logs only record what happened under the system that was already running. It's a causal, counterfactual question, and src/offpolicy.py estimates it off-policy: from the logged data alone, without deploying anything.

I frame the RetailRocket logs as a contextual bandit:

• context = the visitor's history so far
• action = the item they interacted with next (the test target)
• reward = 1 if that interaction was an addtocart / transaction, else 0 (did it convert, not just get viewed — this is the stronger signal the ranking metrics ignore)

The estimator is self-normalised inverse propensity scoring (SNIPS): V(π) = Σ wᵢ rᵢ / Σ wᵢ, with importance weights wᵢ = π(aᵢ|xᵢ) / π₀(aᵢ). The logging policy π₀ is modelled as item popularity (add-one smoothed for full support, which IPS needs); each recommender becomes a distribution π via a softmax over its scores. Weights are clipped and I report effective sample size (ESS) and clipped-fraction so the variance is visible, not hidden.

The estimator is unit-tested and cross-checked against Open Bandit Pipeline (OBP), the reference off-policy-evaluation library: on synthetic bandit feedback our snips reproduces OBP's InverseProbabilityWeighting and SelfNormalizedInverseProbabilityWeighting exactly (tests/test_obp_agreement.py; pip install obp to run it, otherwise it skips). We keep our own implementation because OBP materialises a dense (rounds × actions) distribution, which is wasteful over an 11k-item catalogue.

python -m src.offpolicy --data data/events.csv --sasrec model_retailrocket.pt

Logging (popularity) conversion rate V(π₀) = 0.0562:

Policy	V(π) est. conversion	Uplift vs logging	ESS	Clipped
Popularity (sanity: = π₀)	0.0562	+0.0000	100%	0%
Markov (first-order)	0.0672	+0.0110	45%	44%
SASRec	0.0599	+0.0037	67%	54%

Reading it honestly: Markov shows the larger raw uplift but with a much lower ESS and heavy clipping, so that estimate is the noisier one — exactly the variance problem SNIPS is meant to expose. The popularity row is the built-in self-check: it is the logging policy, so its uplift must come out at 0, and it does.

Caveats (the numbers are only as good as these). π₀ is estimated, not logged — real off-policy evaluation uses the deployed system's true propensities, so treat the popularity model as a proxy. IPS is unbiased only under full support and a roughly-correct π₀, and its variance grows when π and π₀ disagree; the ESS and clip columns are there to keep that trade-off in view.

Layout

data/    datasets, not committed
src/     data.py, metrics.py, baselines.py, models.py, train.py, offpolicy.py
api/     FastAPI service
tests/   estimator, policy, SASRec masking and API tests, incl. an OBP cross-check
Dockerfile, .github/workflows/ci.yml

Next steps

• Off-policy estimation from logged propensities rather than a popularity proxy, and a doubly-robust estimator to cut the IPS variance seen above.
• Use addtocart / transaction events as a training signal too, not just as the evaluation reward.
• Expose latency and request metrics on the API.