Making the wrong call between training a model from scratch and fine-tuning a pretrained one is one of the most expensive mistakes an AI project can make. Budgets evaporate, timelines collapse, and teams spend months building something they could have had in weeks—or, going the other direction, they settle for a generic model that will never hit the performance ceiling their use case actually demands. The decision is not always obvious, but it is always consequential.
This checklist is designed to make that decision defensible and fast. Each item includes a short justification so you understand the reasoning, not just the rule. Work through the sections in order: they move from foundational questions (data, budget, compute) to operational ones (latency, compliance, team capability). By the end, you should have a clear recommendation—and the language to explain it to a stakeholder or client.
One note on terminology before you start: "training from scratch" means initializing a model with random weights and learning everything from your data. "Fine-tuning" means starting from a pretrained model—something like a large language model, a vision transformer, or a speech encoder—and continuing training on your specific data. A third path, prompt engineering or retrieval-augmented generation (RAG), is sometimes the right answer and is noted where relevant.
Step 1: Define What You're Actually Trying to Do
This sounds obvious. Most teams skip it anyway.
âś… Checklist
- State the task in one sentence. If you can't, the problem isn't ready for model decisions yet.
- Decide whether this is a new capability or a behavior change. New capability (e.g., building a medical imaging classifier where none exists) is a stronger argument for training from scratch. Behavior change (e.g., making a general LLM respond in your brand voice) points toward fine-tuning or prompting.
- Identify whether domain-specific knowledge or general reasoning is the bottleneck. If the bottleneck is knowledge your pretrained model simply doesn't have, fine-tuning on domain data may close the gap. If the bottleneck is task structure—a completely novel output format or modality—training may be unavoidable.
- Check whether a pretrained model already exists for your modality. For text, code, images, and audio, mature pretrained models exist. For niche scientific signals, proprietary sensor formats, or unusual data types, they may not.
Why this step matters: Teams frequently jump to fine-tuning because it's trendy, or to training because it feels more rigorous. Neither instinct is reliable without a clear problem definition.
Step 2: Audit Your Data
Data quality and quantity are the single strongest predictors of which path is viable.
âś… Checklist
- Count your labeled examples. For fine-tuning a large language model, you can often get meaningful improvement with as few as 100–1,000 high-quality examples for a focused task. For training a mid-sized vision model from scratch, expect to need 50,000–500,000+ labeled images depending on class count and visual diversity.
- Assess data cleanliness. Fine-tuning amplifies noise more aggressively than training from scratch on clean data does, because the pretrained model's priors interact with corrupted signals in unpredictable ways. Rate your dataset: what percentage of labels are you confident in?
- Check for domain shift. If your data looks nothing like what the pretrained model was trained on—different language register, unusual image distributions, proprietary jargon—fine-tuning may underperform expectations. This is a point toward training from scratch or at minimum retraining more layers.
- Evaluate data collection cost. If you can only ever realistically collect 500 examples, that is a hard constraint. Training from scratch with 500 examples will almost always underfit. Fine-tuning or few-shot prompting becomes the only viable path.
- Consider data privacy restrictions. If your data cannot leave a private environment, you need a model you can run on-premises. That may rule out API-based fine-tuning services and push you toward self-hosted fine-tuning or training.
Avoiding the classic mistake of starting model development before honestly assessing data readiness is covered in detail in 7 Common Mistakes with Machine Learning Basics (and How to Avoid Them).
Step 3: Calculate the Real Compute Cost
Intuitions about cost are almost always wrong. Do the arithmetic.
âś… Checklist
- Estimate GPU-hours for each path. Training a transformer-based model with a billion parameters from scratch can cost tens of thousands of dollars in cloud compute. Fine-tuning a 7B-parameter open-source LLM on a single A100 GPU for a focused task typically runs 2–20 hours, translating to $20–$200 in cloud costs depending on provider.
- Account for iteration cycles. Neither path succeeds on the first run. Assume 3–8 training runs to get a production-ready model. Multiply accordingly.
- Include inference costs in the comparison. A fine-tuned 7B model may have higher inference costs per query than a well-prompted call to a frontier API. Training a smaller specialized model from scratch may ultimately be cheaper at scale if query volume is high enough.
- Check whether parameter-efficient fine-tuning (PEFT) methods like LoRA or QLoRA are available for your target model. These techniques reduce GPU memory requirements by 60–80% in typical cases and are now standard for LLM fine-tuning. If you're not using them, you're paying too much.
- Don't forget storage costs. Checkpoints for a multi-billion-parameter model occupy significant disk space. Budget for it.
Step 4: Assess Your Timeline
Deadlines change what's technically optimal.
âś… Checklist
- Set a hard deadline for the first working prototype. If it's four weeks away, training from scratch is almost certainly off the table for any serious model size.
- Identify the fastest path to a testable output. Prompt engineering or RAG with a frontier model gets you something testable in hours or days. Fine-tuning adds days to weeks. Training from scratch adds weeks to months.
- Plan for evaluation time. Building a proper evaluation harness—something that tells you whether the model is actually better—takes time that teams routinely underestimate. Reserve at least 20% of your project timeline for evaluation work, regardless of which training path you choose.
- Consider phased delivery. Many teams successfully ship a fine-tuned or prompted model as a first version, collect real user data, and then make a more informed decision about whether training from scratch makes sense for version two. This is often the right call. See Machine Learning Basics: Real-World Examples and Use Cases for examples of this phased approach working in practice.
Step 5: Evaluate Latency and Performance Requirements
Production requirements often override philosophical preferences.
âś… Checklist
- Define your latency ceiling. If users need a response in under 200ms, the size of model you can run—fine-tuned or trained—is constrained. A fine-tuned small model may outperform a large fine-tuned model simply because it fits within the latency budget.
- Determine whether the model must run on-device. Mobile, embedded, and edge deployments impose strict size limits. Training a compact specialized model from scratch may be necessary when no suitable pretrained small model exists for your domain.
- Set a minimum acceptable accuracy threshold before you start. "As good as possible" is not a specification. Define what failure means numerically. This prevents you from over-investing in training when fine-tuning would have crossed the threshold.
- Test the pretrained baseline before touching training data. Zero-shot and few-shot performance from a frontier model is often the most underutilized benchmark. If a well-prompted pretrained model already meets your threshold, you may not need to train or fine-tune at all.
Step 6: Check Compliance, IP, and Security Requirements
Regulatory and contractual constraints are non-negotiable and frequently overlooked until they kill a project.
âś… Checklist
- Determine whether your training data carries licensing restrictions. Data scraped from the web, licensed datasets, or proprietary corpora all have terms. Fine-tuning with data you don't have the right to use is a legal exposure, not just an ethical one.
- Check whether the pretrained model's license permits commercial fine-tuning and deployment. Many open-weight models have licenses that restrict commercial use, modify what you can charge for, or prohibit certain applications. Read the license before building on the model.
- Identify data residency and processing requirements. Healthcare, finance, and government sectors often have strict rules about where data is processed. This affects which cloud providers and which API-based fine-tuning services are permissible.
- Assess IP ownership of the final model. Who owns a fine-tuned model built on a third party's base weights? Terms vary by provider. If owning the model weights outright is a business requirement, training from scratch may be the only path that delivers clean ownership.
This is one area where reviewing Machine Learning Basics: Best Practices That Actually Work before engaging a vendor is time well spent.
Step 7: Evaluate Your Team's Capability
The best technical path is useless if the team can't execute it.
âś… Checklist
- Inventory ML expertise honestly. Fine-tuning a pretrained model requires understanding hyperparameter tuning, learning rate scheduling, overfitting diagnostics, and evaluation methodology. Training from scratch requires all of that plus architecture design, initialization strategies, and significantly more debugging experience.
- Check tooling familiarity. Hugging Face Transformers, PyTorch Lightning, and similar frameworks lower the barrier, but they don't eliminate it. Misusing these tools is a common source of training bugs that are hard to diagnose.
- Assess whether you need to hire or partner. If your team has never fine-tuned a model end-to-end, budget for learning time or external expertise. Underestimating this is one of the more expensive mistakes in AI projects.
- Consider whether managed fine-tuning services close the gap. OpenAI, Google, and others offer fine-tuning via API with minimal infrastructure burden. These are legitimate production paths, not just experimentation tools—though they come with the compliance and ownership trade-offs noted above.
For a structured view of how teams navigate these decisions in practice, Case Study: Machine Learning Basics in Practice walks through a real-world project arc from problem definition to deployment.
The Decision Matrix: Summarized
| Factor | Leans Training from Scratch | Leans Fine-Tuning | | ------------------- | ------------------------------------------ | ---------------------------------- | | Data volume | Very large (500K+ examples) | Small to medium (100–50K) | | Domain novelty | Entirely new modality or signal type | Established modality, domain shift | | Timeline | Months available | Weeks or less | | Compute budget | High | Moderate to low | | IP requirements | Full weight ownership needed | Shared or licensed base acceptable | | Team expertise | Senior ML researchers | ML practitioners or engineers | | Performance ceiling | Must exceed what pretrained models achieve | Pretrained models are competitive |
No single row in this matrix is decisive. It's the weight of the evidence across rows that drives the call.
Frequently Asked Questions
What's the most common mistake teams make when choosing between training and fine-tuning?
The most common mistake is skipping a zero-shot or few-shot baseline evaluation entirely. Teams assume a pretrained model won't be good enough and immediately invest in fine-tuning, only to discover later that a well-crafted prompt would have met the performance threshold. Always benchmark the pretrained model before committing to additional training.
Can fine-tuning make a model worse than the pretrained baseline?
Yes, and it happens more often than practitioners expect. Catastrophic forgetting—where fine-tuning on narrow data degrades the model's general capabilities—is a real failure mode. Overfitting on small datasets is another. Both produce a model that scores lower on your evaluation set than the original pretrained model did. PEFT methods like LoRA partially mitigate this by updating fewer parameters.
Is RAG ever a better choice than fine-tuning?
Frequently. If the gap between your model's current performance and your requirement is primarily about knowledge—facts, documents, policies—then retrieval-augmented generation is often faster, cheaper, and more maintainable than fine-tuning. Fine-tuning is better suited for changing how a model behaves (tone, output format, task structure) than for injecting knowledge it can look up.
How do I know if I have enough data to fine-tune?
A practical starting point: run a fine-tuning experiment on a small held-out subset, then plot performance as you add data. If performance is still climbing steeply at your data limit, you're data-constrained and should prioritize collection. If it plateaus well before your full dataset, you likely have enough. For most focused NLP tasks, 500–2,000 high-quality examples is a workable minimum for meaningful fine-tuning signal.
Does training from scratch always produce a better model if you have the resources?
Not necessarily. Pretrained models encode representations learned from vastly more data than most organizations can collect. For language, vision, and audio tasks, fine-tuning a large pretrained model typically outperforms training a smaller model from scratch even when the latter uses more of your domain-specific data. Training from scratch makes more sense when pretrained models simply don't exist for your data type or when you need full architectural control.
What should I do if my checklist produces a tie?
Default to fine-tuning unless there is a specific, concrete reason training from scratch is required. The asymmetry in cost, timeline, and risk heavily favors fine-tuning when the evidence is ambiguous. You can revisit the decision with real performance data after shipping a fine-tuned version. Committing to training from scratch without strong justification is a bet you pay for in months, not days.
Key Takeaways
- Define the task precisely before any model decision. Ambiguous problem definitions produce wrong architecture choices.
- Data volume and quality are the strongest constraints. Less than ~1,000 examples points toward fine-tuning or prompting; hundreds of thousands may justify training from scratch.
- Always run a pretrained baseline before investing in training. Zero-shot performance is frequently underestimated.
- Compute costs are not just training costs. Include iteration cycles and inference at scale in every comparison.
- Legal and IP requirements are hard constraints, not preferences. Check licenses and data rights before selecting a base model or training approach.
- When in doubt, fine-tune first, ship, collect real data, and make training-from-scratch decisions with evidence in hand.
- Use this checklist as a living document. Revisit it when your data volume changes, your performance requirements shift, or new pretrained models enter the market—which happens frequently enough that a 2025 decision may not be the right 2026 decision.