A fraud detection team at a payments company deployed what they thought was model version 4.2. A week later, they noticed a 12 percent drop in fraud catch rate. They tried to rollback to version 4.1 — but the rollback failed because version 4.1 depended on a feature pipeline that had been updated since 4.1 was deployed. The new feature pipeline produced different feature values, so model 4.1 running on the new features performed differently than model 4.1 running on the old features. It took three days to untangle which combination of model version, feature pipeline version, and configuration was running at any point in time. The root cause was that the team versioned their models but did not version their data pipelines, feature definitions, or deployment configurations. They had model versioning but not system versioning. An AI agency built them a comprehensive versioning strategy that tracked the complete state of the AI system — code, data, model, features, and configuration — as a single versioned unit. Rollbacks became instantaneous and reliable. Every prediction could be traced to the exact system version that produced it.
What Needs Versioning in AI Systems
Code Versioning
Everything that traditional software versions, plus ML-specific code:
- Pipeline code: Data ingestion, transformation, and feature engineering code
- Training code: Model architecture, training loop, hyperparameter configuration
- Serving code: Inference server, pre/post-processing, API layer
- Infrastructure code: Terraform, Kubernetes manifests, deployment configurations
- Test code: Evaluation suites, benchmark datasets, quality checks
Use Git for all code versioning. This is non-negotiable. Every code change should be a commit, every release should be a tag, and every experiment should be traceable to a specific commit.
Data Versioning
Data changes independently of code and can dramatically affect model behavior.
What to version:
- Training datasets: The exact dataset used for each training run. Hash the dataset and store the hash with the experiment metadata.
- Validation and test datasets: The exact datasets used for evaluation. These must be stable across model versions for meaningful comparison.
- Feature definitions: The transformation logic that converts raw data into model features. A change in feature definition is a data change even if the raw data has not changed.
- Reference data: Lookup tables, configuration tables, and business rules that affect pipeline behavior.
- Schema definitions: The expected schema for each dataset at each pipeline stage.
Data versioning approaches:
- Hash-based versioning: Compute a content hash of the dataset and store it as the version identifier. Simple and reliable. Works with DVC (Data Version Control) or custom implementations.
- Snapshot-based versioning: Store complete copies of the dataset at each version. Expensive but provides guaranteed reproducibility. Works well when data lakehouse time travel is available (Delta Lake, Iceberg).
- Delta-based versioning: Store only the changes between versions. More storage-efficient but more complex to implement. Works well for large datasets with incremental changes.
Model Versioning
Model artifacts need versioning with rich metadata that links them to the code and data that produced them.
What to store with each model version:
- Model artifact (weights, architecture, configuration)
- Training code commit hash
- Training data version (hash or snapshot ID)
- All hyperparameters
- Evaluation metrics on standard benchmarks
- Training infrastructure details (hardware, framework versions)
- Feature dependency list (which features does this model require?)
- Deployment requirements (resource requirements, latency expectations)
Use a model registry (MLflow, Vertex AI Model Registry, custom) that provides:
- Version numbering with semantic versioning (major.minor.patch)
- Stage management (development, staging, production, archived)
- Metadata search (find models by metric, date, owner, data version)
- Lineage tracking (which experiment produced this model?)
Configuration Versioning
Configuration includes everything that affects system behavior but is neither code nor data:
- Model serving configuration (batch size, timeout, resource allocation)
- Feature pipeline configuration (data sources, schedule, quality thresholds)
- Business rules and thresholds (prediction confidence threshold, alert thresholds)
- Infrastructure configuration (autoscaling policies, monitoring thresholds)
Version configuration in Git alongside code, or in a dedicated configuration management system with full audit trail.
System Version: Tying It All Together
The most critical concept in AI versioning is the system version — a composite version that captures the complete state of the AI system.
A system version includes:
- Code version (Git commit or tag)
- Data version (dataset hashes)
- Model version (model registry version)
- Configuration version (configuration commit or tag)
- Infrastructure version (infrastructure-as-code commit)
System version enables:
- Reproducibility: Re-create the exact state of the system at any point in time
- Debugging: When something goes wrong, identify exactly which combination of components was running
- Rollback: Revert the entire system to a known good state, not just the model
- Audit: Demonstrate to regulators exactly what system produced a specific prediction
Versioning Implementation
Version Manifest
Create a version manifest that captures the system version as a single document.
The manifest should be:
- Generated automatically by the CI/CD pipeline
- Stored with every deployment
- Queryable by timestamp (what was running at time T?)
- Linked to every prediction (which system version produced this output?)
Semantic Versioning for Models
Adopt semantic versioning (major.minor.patch) with ML-specific semantics:
- Major version: Architecture change, training data domain change, or any change that breaks backward compatibility (different input features, different output format)
- Minor version: Retraining with updated data, hyperparameter tuning, or performance improvement that maintains backward compatibility
- Patch version: Bug fix, configuration change, or minor adjustment that does not affect model behavior
Branching Strategy
- Main branch: Always reflects the current production system
- Development branch: Integration branch for changes being prepared for production
- Feature branches: Individual changes (new feature, new model version, infrastructure update)
- Release branches: Stabilization branches for preparing a new system version
Delivery Process
Phase 1: Assessment and Design (Weeks 1-3)
- Audit current versioning practices across code, data, models, and configuration
- Identify gaps (what is not versioned? what is versioned but not linked?)
- Define the versioning strategy including system version manifest
- Select tools and infrastructure for each versioning component
Phase 2: Implementation (Weeks 4-10)
- Implement code versioning standards and branching strategy
- Deploy data versioning tools and integrate with pipelines
- Deploy or configure the model registry with metadata standards
- Build the system version manifest generation
- Integrate versioning with the CI/CD pipeline
Phase 3: Migration and Adoption (Weeks 11-14)
- Migrate existing models and artifacts to the versioned model registry
- Backfill version metadata for current production systems
- Train teams on versioning practices and tools
- Integrate versioning into deployment and rollback procedures
Versioning Anti-Patterns to Avoid
Anti-pattern 1: Versioning models but not data. The most common anti-pattern. A team can reproduce their model (they know the architecture and hyperparameters) but cannot reproduce their results because they do not know which version of the data the model was trained on. Data versioning is as important as model versioning.
Anti-pattern 2: Hardcoded model paths. Applications that reference models by file path (models/fraudmodelv4.pkl) instead of by model registry version break when files move, making deployment automation and rollback impossible. Always reference models through the registry.
Anti-pattern 3: Configuration in code. Embedding configuration values (thresholds, feature lists, data sources) in application code means configuration changes require code deployments. Externalize configuration and version it independently.
Anti-pattern 4: Manual version tracking. Tracking versions in spreadsheets or wiki pages is unreliable and will fall out of sync with reality. Automate version tracking through CI/CD integration and platform-level metadata capture.
Anti-pattern 5: No garbage collection. Without a policy for archiving and deleting old versions, storage grows indefinitely. Define retention policies — keep the last N versions of each model in the active registry, archive older versions to cold storage, and delete versions older than a defined threshold unless they are tagged for permanent retention.
Versioning for Regulatory Compliance
Regulated industries have specific versioning requirements that go beyond operational best practices.
Model Risk Management (SR 11-7). Financial institutions must maintain complete documentation of model development, including the data and code used at each stage. Versioning must support this by linking every model version to its complete development history.
EU AI Act. High-risk AI systems require technical documentation including "a detailed description of the elements of the AI system and of the process for its development." Versioning provides the foundation for this documentation by capturing the system state at every point in its development.
HIPAA. AI systems processing protected health information must maintain audit trails. Versioning must capture who changed what, when, and why for every component that touches patient data.
Practical compliance requirements:
- Immutable version history (versions cannot be modified or deleted once created, only archived)
- Complete provenance chain (every version links to its predecessor and its creation context)
- Exportable audit reports (generate compliance documentation from version history)
- Access logging (who accessed which version, when, and for what purpose)
Versioning in Practice: A Worked Example
Consider a fraud detection system with these components:
- Feature pipeline (code in Git, data from 3 sources)
- Training pipeline (code in Git, training data versioned with DVC)
- Model (artifact in MLflow Model Registry)
- Serving configuration (in a configuration management system)
- Infrastructure (Terraform in Git)
System version manifest for production deployment v2.4.1:
- Feature pipeline code: Git commit abc123
- Feature pipeline data sources: transactionsv2024.12, customersv2024.12, merchants_v2024.11
- Training pipeline code: Git commit def456
- Training data: DVC hash ghi789
- Model: MLflow model fraud-detect version 24, stage production
- Serving configuration: Git commit jkl012
- Infrastructure: Git commit mno345
- Deployment timestamp: 2026-03-15T14:30:00Z
- Deployed by: CI/CD pipeline run #4721
When the model produces a suspicious prediction at timestamp 2026-03-17T09:15:00Z, the team can look up the system version that was active at that time and access every component that influenced the prediction. This is the power of comprehensive versioning.
Versioning for Regulatory Compliance
In regulated industries, versioning is not just good practice — it is a legal requirement.
Model lineage documentation. Regulators require the ability to trace any model decision back to the specific model version, training data, and code that produced it. Comprehensive versioning provides this traceability automatically. Without it, organizations spend weeks manually reconstructing lineage for regulatory reviews.
Point-in-time reproduction. A regulator may ask: "Show me the exact model that was making decisions on March 15, 2025, and demonstrate its fairness properties." With proper versioning, this is a simple lookup. Without it, it may be impossible to reproduce.
Audit trail requirements. Financial regulators (SR 11-7), healthcare regulators (FDA), and emerging AI regulations (EU AI Act) all require comprehensive audit trails of AI system changes. Versioning provides the foundation for these audit trails.
Versioning Tool Recommendations
For code versioning: Git is the universal standard. Use branching strategies (GitFlow, trunk-based development) that fit your team's workflow. Tag releases with semantic versions.
For data versioning: DVC (Data Version Control) integrates with Git and provides versioning for large datasets and model artifacts. LakeFS provides Git-like branching for data lakes. Delta Lake and Apache Iceberg provide time travel capabilities for lakehouse data.
For model versioning: MLflow Model Registry provides model versioning with stage management (staging, production, archived). Cloud-native registries (SageMaker Model Registry, Vertex AI Model Registry) provide similar capabilities with tighter cloud integration.
For system version manifests: Custom implementation is usually necessary because no off-the-shelf tool provides end-to-end system versioning. Build a simple service that creates and stores system version manifests and links them to deployments.
Versioning Migration Strategy
For organizations with existing AI systems that lack proper versioning, migrating to a versioned workflow requires careful planning.
Start with the highest-risk system. Do not try to version everything at once. Start with the system that would cause the most damage if a version could not be traced or reproduced. Implement versioning for that system first, learn from the experience, and then expand to other systems.
Version forward, not backward. Retroactively versioning historical deployments is usually impractical. Instead, establish a versioning baseline from today forward. All new deployments are versioned. Historical deployments are documented as best as possible with the information available.
Versioning for Multi-Model Systems
Modern AI applications increasingly use multiple models in concert — an orchestration model, a retrieval model, a generation model, and a safety model may all contribute to a single user interaction. Versioning multi-model systems requires additional discipline.
Composite model version. Define a composite version that captures the version of every model in the pipeline. When any individual model is updated, the composite version changes. This enables rollback of the entire multi-model pipeline to a known-good state.
Compatibility matrices. Not every combination of model versions works correctly together. Maintain a compatibility matrix that documents which versions of each model have been tested together. The deployment pipeline should enforce compatibility — preventing the deployment of a model version that has not been tested with the current versions of its upstream and downstream models.
Independent vs. coordinated updates. Some model changes can be deployed independently (a safety model update that does not change its API). Others require coordinated deployment (a retrieval model change that affects the format of context passed to the generation model). The versioning strategy must distinguish between these cases and enforce coordinated deployment when needed.
End-to-end testing per version combination. When multiple models are updated simultaneously, run end-to-end tests that exercise the full pipeline with the new version combination. Unit testing individual models is not sufficient because integration issues often emerge only when models interact.
Pricing Versioning Strategy Engagements
- Versioning assessment and strategy: $10,000 to $25,000
- Full versioning implementation: $40,000 to $100,000
- Enterprise versioning platform: $80,000 to $200,000
Versioning Governance and Enforcement
A versioning strategy that relies on developer discipline alone will eventually fail. Build enforcement into the tooling and processes.
Automated version validation. The CI/CD pipeline should validate that every deployment includes a complete system version manifest. Deployments without manifests are blocked. Manifests with missing components (no data version, no configuration version) are flagged for review.
Version audit reports. Generate monthly reports showing version coverage — what percentage of production systems have complete version manifests, what percentage of deployments were traceable, and what percentage of predictions could be linked to their system version. Track these metrics over time and set targets for improvement.
Your Next Step
This week: Attempt to answer this question for every AI system your agency manages: "What exact combination of code, data, and model is running in production right now?" If you cannot answer immediately and with confidence, you need a versioning strategy.
This month: Implement a version manifest for your most critical production system. Link every deployment to a complete system version.
This quarter: Deliver your first versioning strategy engagement as part of a broader MLOps or platform engagement.