There is no shortage of generic advice about recommendation systems. "Use clean data." "Test your model." Thanks. The trouble with platitudes is that they tell you what to do without telling you why, which means you cannot reason about the exceptions. This article takes the opposite approach: every practice below comes with the reasoning that justifies it, so you can decide when it applies and when it does not.
These are opinions formed by watching recommendation systems succeed and fail. Some will be uncomfortable, because the best practices in this field are often the ones that slow you down in the short term to compound value in the long term. Understanding how recommendation systems work is necessary but not sufficient; you also need a discipline about how you build and operate them.
Treat the following as a stance, not a checklist. Where I take a strong position, I will say so and explain the trade-off you are accepting if you disagree.
Make the Business Objective Explicit Before Modeling
The first practice is the one teams skip most: decide what you are actually optimizing for, in writing, before any modeling begins.
Why this matters more than the model
A recommender that maximizes clicks will happily surface clickbait. One that maximizes watch time may reward content that traps attention without satisfying anyone. The objective you choose silently shapes every recommendation. If you do not name it, the default objective, usually short-term clicks, will be chosen for you by whatever loss function is convenient. State the goal in business terms first, then translate it into a metric. The common mistakes article shows how a misaligned objective quietly degrades a product.
Earn the Right to Use Complexity
Strong opinion: do not start with deep learning. Start with the simplest thing that could work and add complexity only when a baseline proves it is needed.
A popularity baseline and item-based collaborative filtering capture a large fraction of achievable value with a fraction of the operational burden. Complex models are harder to debug, slower to serve, and more prone to silent failure. Earn the right to that complexity by demonstrating, with experiments, that a simpler approach has hit its ceiling. The step-by-step build guide deliberately sequences baselines before sophistication for this reason.
The hidden cost of complexity
The cost of a sophisticated model is not the training run; it is everything that comes after. A deep model demands a feature pipeline that stays consistent between training and serving, monitoring that can detect when its embeddings drift, and engineers who understand it well enough to debug it at 2 a.m. Each of those is a standing tax on your team. A simple model that you fully understand and can fix quickly will often deliver more value over a year than a powerful one nobody can confidently operate. Complexity should buy you a measured, meaningful lift, not just the satisfaction of having built something impressive.
Treat Evaluation as a First-Class System
Most teams pour effort into models and treat evaluation as an afterthought. Reverse that priority.
Build trustworthy offline metrics
- Use a time-based split so evaluation mirrors predicting the future from the past.
- Always compare against a popularity baseline, not against nothing.
- Report ranking-aware metrics like NDCG, not just raw accuracy.
Make online experiments routine
Offline metrics guide; online experiments decide. The gap between them is wide and frequently surprising, so an A/B test should be the standard gate for any change, not a special event reserved for big launches. A reliable evaluation harness is the difference between a recommender that improves and one that drifts.
Design for the Cold Start From Day One
New users and new items are not edge cases; they are a permanent, recurring condition. Build for them deliberately.
Have a content-based fallback for items with no interaction history, onboarding signals for new users, and a graceful handoff to personalization as data accumulates. A new user who sees irrelevant suggestions may never return, which means cold start is often where the most valuable conversions are won or lost. The guide to how recommendation systems work explains why hybrids handle this transition best.
Build Exploration Into the System
This is the practice most teams resist, because it means deliberately showing recommendations the model is unsure about. Do it anyway.
A recommender that only shows its safest bets learns nothing about the items it never surfaces, and its training data becomes a self-fulfilling prophecy. A modest exploration budget, where some traffic sees uncertain or novel items, keeps the data honest and prevents the catalog's long tail from dying. You are trading a small, measurable short-term cost for long-term health of the entire system.
Exploration is an investment, not a leak
Teams often frame exploration as lost revenue: every uncertain item shown is a "safe" recommendation forgone. That framing is wrong, because it ignores what the safe-bet-only system loses over time. Without fresh data on under-shown items, the model's view of the catalog calcifies, and you slowly lose the ability to recommend anything new. The right mental model is a research budget: you spend a little engagement now to buy knowledge that keeps the whole system viable. Keep the budget small and bounded, measure what it teaches you, and you will find it pays for itself many times over.
Keep a Human Floor Under the Algorithm
Finally, do not let the optimization run unsupervised. Maintain guardrails the model cannot override.
Practical guardrails
- Hard filters for content that should never be recommended, regardless of predicted engagement.
- Diversity constraints so a single category cannot monopolize a feed.
- Freshness rules so stale or discontinued items drop out automatically.
These guardrails encode judgment that no engagement metric captures. They are the human floor beneath the algorithm, and they are why the strongest systems feel curated rather than merely optimized. The recommendation checklist turns these guardrails into items you can verify. Crucially, guardrails should be cheap to add and impossible for the model to override, because the moment a hard rule becomes negotiable it stops protecting you. Treat them as constraints the optimization runs inside of, not as soft preferences it can trade away for a few extra clicks.
Frequently Asked Questions
Why shouldn't I start with a deep learning model?
Because complexity is a cost you should earn, not assume. Simple baselines capture much of the achievable value while being far easier to debug, serve, and trust. Only after experiments show a baseline has hit its ceiling does the operational burden of deep learning pay off.
How do I choose the right objective for my recommender?
Start from the business outcome you actually want, such as retention or revenue, and state it explicitly before modeling. Then translate it into a measurable metric. If you skip this, the model defaults to optimizing whatever is convenient, usually short-term clicks, which often works against your real goals.
Is exploration worth the short-term cost?
Yes, in almost all cases. Without exploration, the model only ever learns about items it already favors, and its training data becomes a self-fulfilling loop. A modest exploration budget keeps the data honest and preserves catalog diversity, which protects long-term performance.
How much should I trust offline metrics?
Use them to guide development and catch regressions, but never as the final word. The gap between offline and online results is wide and often surprising. Make A/B testing a routine gate so live results, not offline scores, decide what ships.
What guardrails should every recommender have?
At minimum, hard filters for content that must never appear, diversity constraints so one category cannot dominate, and freshness rules that drop stale items. These encode human judgment that engagement metrics miss and keep the system feeling curated rather than blindly optimized.
Key Takeaways
- Write down the business objective before modeling; the objective shapes every recommendation more than the model does.
- Start simple and earn the right to complexity through experiments, not assumptions.
- Treat evaluation as a first-class system, with time-based splits offline and routine A/B tests online.
- Design for cold start and build a modest exploration budget in from the start.
- Keep human guardrails, like hard filters and diversity constraints, beneath the optimization.