Skip to content

Model Quality Test ​

This page shows the recommendation quality of SQLRec's built-in models on four public datasets, covering retrieval, binary ranking, and multitask prediction. You can compare validation results and reproduce them with benchmark/model_quality/. For request throughput and latency, see SQL Performance Test.

Results ​

All 28 dataset/model combinations were trained on the complete training splits and evaluated on the complete validation splits. The reported metrics were independently recalculated.

Ranking ​

Each cell is AUC / LogLoss. Higher AUC means better discrimination between positive and negative examples; lower LogLoss means less error in predicted probabilities. Bold values identify the best of the six single-task ranking models on each dataset. MMoE tasks are listed separately.

Model / taskMovieLensKuaiRandKuaiRecCriteo 1M
Wide & Deep0.7763 / 0.56220.6875 / 0.62590.8741 / 0.13830.7866 / 0.4601
DeepFM0.7862 / 0.55530.6884 / 0.62660.8764 / 0.13740.7881 / 0.4590
Rocket Launching0.7751 / 0.56420.6940 / 0.65270.8716 / 0.14000.7837 / 0.4662
CatBoost0.7522 / 0.58310.7072 / 0.60100.8678 / 0.14190.7759 / 0.4694
LightGBM0.7408 / 0.60400.7021 / 0.59930.8691 / 0.14040.7283 / 0.5009
XGBoost0.7357 / 0.59870.7017 / 0.59880.8649 / 0.14190.7286 / 0.5005
MMoE / click—0.6852 / 0.6578——
MMoE / long_view—0.7003 / 0.4833——

The best model can differ by metric: CatBoost has the highest KuaiRand AUC, while XGBoost has the lowest LogLoss. Consider your application's ranking and probability-prediction needs when selecting a model.

Retrieval ​

DSSM returns the Top-50 items by the inner product of user and item vectors. Recall@50 measures the fraction of positive targets retrieved. NDCG@50 also rewards placing positive items earlier in the ranking. Both are averaged over queries with positive targets; higher is better.

Dataset / DSSMRecall@50NDCG@50All / positive queriesCandidates
MovieLens34.26%19.15%926 / 873Complete 1,682-item catalog, excluding items seen in train
KuaiRand11.05%4.45%38,079 / 24,950Complete 7,583-item eligible catalog, repeat viewing allowed
KuaiRec57.82%51.47%1,411 / 1,381All observed positive/negative validation pairs for that query, excluding videos seen in train

KuaiRec retrieves from each user's observed candidates, while the other datasets use complete catalogs. Their recall scores therefore cannot be used to compare the difficulty of these three tasks directly.

Data and Test Settings ​

Datasets ​

DatasetTarget and positive labelTraining rowsValidation rowsSplit
MovieLens-100KRating preference: movie rating ≥ 480,1409,877User/item interaction holdout, approximately 80% / 10% / 10%
KuaiRand-PureClicks: is_click=1; MMoE also predicts long views1,124,319408,405Chronological
KuaiRec small_matrixWatch preference: watch_ratio > 23,741,059467,747User/video interaction holdout, approximately 80% / 10% / 10%
Criteo 1MAd clicks: target=1799,561100,266Feature-hash holdout, approximately 80% / 10% / 10%

KuaiRand trains on standard exposures and validates on both standard and random exposures. MovieLens and KuaiRec evaluate preferences between known users and items, rather than strict future-time prediction. Ranking uses observed positive and negative examples; DSSM uses training positives only. Unobserved KuaiRec user/video pairs are never treated as negatives.

Criteo uses a pinned 1M sample version. Identical feature rows stay in the same split, and labels do not affect the split. Its scores are holdout-validation results on this sample; the reserved test split was not used for tuning. Criteo tests single-task ranking only, without DSSM or MMoE.

Models and Features ​

ModelPurpose and main inputs
DSSMRetrieval using user/item IDs, categories, and related features; MovieLens and KuaiRec also use history statistics, while KuaiRand uses context and author
Wide & Deep, DeepFM, Rocket LaunchingRanking using categorical and numeric features; recommendation datasets also use IDs, history statistics, and available video duration; Rocket evaluates the Light branch
CatBoost, LightGBM, XGBoostRanking using numeric history statistics and available video duration on the first three datasets; see below for Criteo inputs
MMoEMultitask ranking using KuaiRand ranking features to predict clicks and long views together

Criteo contains 13 numeric features and 26 anonymous categorical features. CatBoost and the three neural ranking models use all 39 features. LightGBM/XGBoost use the 13 numeric features, as required by SQLRec's current training/export path. Criteo scores therefore reflect different feature configurations.

Some ranking models also use user/item smoothed positive rates from training history. See Built-in Models and the benchmark README for specific features and parameters.

Main Training Settings ​

Dataset / modelEmbedding dimensionBatch sizeEpochs
MovieLens / DSSM162563
KuaiRand / DSSM162566
KuaiRec / DSSM8641
MovieLens / three neural ranking models162566
KuaiRec / three neural ranking models162561
KuaiRand / Wide & Deep, DeepFM, MMoE8641
KuaiRand / Rocket Launching82561
Criteo / three neural ranking models82561

Criteo fills missing numeric values with zero and missing categories with a fixed placeholder. Neural ranking models apply logarithmic scaling to numeric inputs consistently during training and serving; tree models retain the original numeric values.

Tree models use 100 iterations, except MovieLens LightGBM, which uses 300. DSSM uses negative_sampling=pool, 256 negatives, and a pool capacity of 4096. Its output vectors have 32 dimensions on MovieLens/KuaiRand and 16 on KuaiRec. Training resources are 2 CPUs / 4 GiB; serving resources are 1 CPU / 2 GiB.

Reproduce the Test ​

Follow Service Deployment to prepare SQLRec, Kyuubi, shared storage, and Milvus. Build the service and model images from the current source, then run from the repository root:

bash
python3 -m venv .venv
.venv/bin/python -m pip install -r benchmark/model_quality/requirements.txt
source deploy/env.sh

# Prepare each data version once; later runs reuse it.
.venv/bin/python -m benchmark.model_quality.prepare --dataset all --upload

# Start with one dataset and model.
.venv/bin/python -m benchmark.model_quality --dataset movielens100k --model deepfm --full-eval

# Run all 28 combinations.
.venv/bin/python -m benchmark.model_quality --model all --full-eval

To test Criteo only, use --dataset criteo1m. Combined with --model all, this runs its six supported models and does not require Milvus.

These commands evaluate complete validation splits. Without --full-eval, evaluation uses up to 20,000 ranking rows or 1,000 retrieval queries; training data is unchanged.

After each run, open summary.md in the corresponding directory under benchmark/model_quality/runs/ for metrics and completion status. Each model's config.json records its resolved configuration. Add --save-predictions when you need to inspect individual predictions.

Adjust the Configuration ​

  • Data fields, labels, and feature views: benchmark/model_quality/configs/datasets/*.yaml.
  • Model and training parameters: benchmark/model_quality/config.py, with separate settings for each dataset/model combination.
  • Parameter meanings and SQL usage: Built-in Models. For complete test settings and run options, see the benchmark README.

Compare changes on the same validation data and candidate scope. Rerun data preparation when changing preparation rules or feature definitions.