A European bank engaged an AI agency to build a credit scoring model for small business loans. The model performed well in testing, passed internal validation, and was deployed to production. Eight months later, the European Central Bank's supervisory arm requested documentation on the model as part of a routine review. They wanted to see the complete data lineage—where the training data came from, how it was transformed, what features were derived, what data was excluded and why, and how data quality was maintained throughout the pipeline. The agency had built a good model but had not implemented systematic data lineage tracking. Reconstructing the data lineage after the fact took seven weeks, required pulling three engineers off other projects, and still produced documentation that the bank's compliance team considered inadequate. The bank incurred supervisory findings, and the agency was not invited to bid on the bank's next three AI projects. Total estimated cost: over $600,000 in lost revenue and remediation.
Data lineage is the record of where data came from, how it moved through systems, and what transformations were applied to it along the way. For AI systems, data lineage is not a nice-to-have documentation exercise—it is a compliance requirement in regulated industries and a best practice in all industries. It is also the foundation for debugging, reproducibility, and trust.
Why Data Lineage Matters for AI Systems
AI models are only as good as their data. And regulators, clients, and internal stakeholders increasingly need to understand and verify the data behind AI systems. Data lineage provides that understanding.
Regulatory compliance. Multiple regulations require organizations to demonstrate the provenance and handling of data used in automated decision-making:
- The EU AI Act requires documentation of training data for high-risk AI systems, including data collection processes and data preparation
- GDPR requires the ability to trace how personal data is processed, which includes data used in AI systems
- Financial regulators (OCC, Fed, ECB, PRA) require model risk management documentation that includes training data provenance
- Healthcare regulations (HIPAA, FDA guidance) require documentation of data used in clinical AI applications
Debugging and troubleshooting. When an AI model produces unexpected outputs, the first question is always "what data drove this result?" Without data lineage, answering that question requires manual investigation that can take days or weeks. With data lineage, you can trace from model output back to source data in minutes.
Reproducibility. If you cannot reproduce how a model was trained—including the exact data, transformations, and feature engineering—you cannot reproduce the model. Data lineage makes model training reproducible by documenting every data step.
Impact analysis. When source data changes—a vendor modifies their data schema, a data quality issue is discovered, a data source is discontinued—you need to know which downstream models and systems are affected. Data lineage provides this impact analysis capability.
Trust and transparency. Clients and stakeholders trust AI systems more when they can see where the data came from and how it was handled. Data lineage enables transparency without requiring stakeholders to understand the technical details.
What Data Lineage Captures
A complete data lineage record for an AI system includes several layers of information.
Source data documentation:
- What data sources feed the AI system (databases, APIs, files, streams)
- Who owns each data source
- The data classification and sensitivity level of each source
- The legal basis for using each data source
- The freshness and update frequency of each source
- Known quality issues or limitations of each source
Extraction documentation:
- How data is extracted from each source (queries, API calls, file transfers)
- When extractions occur (schedule, triggers)
- What filters or selections are applied during extraction
- How much data is extracted (row counts, date ranges)
- Who has access to perform extractions
Transformation documentation:
- Every transformation applied to the data, in order
- The business logic behind each transformation
- What data is added, modified, or removed at each step
- Feature engineering steps and their rationale
- Data cleaning and quality enforcement steps
- How missing values, outliers, and anomalies are handled
Storage documentation:
- Where data is stored at each stage of the pipeline
- Storage formats and schemas
- Access controls at each storage location
- Retention policies
- Encryption and security measures
Consumption documentation:
- Which models and systems consume the data
- How data is split for training, validation, and testing
- Version mapping (which data version was used to train which model version)
- Who approved the data for use in each model
Designing a Data Lineage System
Architecture Decisions
Before implementing data lineage, make several key architecture decisions.
Active versus passive lineage collection. Active lineage is captured programmatically as data flows through your pipeline—your code explicitly logs lineage events as they happen. Passive lineage is reconstructed by analyzing logs, metadata, and pipeline definitions after the fact. Active lineage is more accurate and reliable. Passive lineage is easier to retrofit but less complete. For new projects, always implement active lineage. For existing projects, start with passive lineage and migrate to active over time.
Centralized versus distributed lineage storage. Centralized storage puts all lineage data in one place (a dedicated lineage database or catalog). Distributed storage keeps lineage data alongside the data it describes. Centralized storage is better for cross-system lineage queries and compliance reporting. Distributed storage is simpler to implement initially but harder to query across systems. For most AI agencies, centralized storage is the right choice.
Granularity level. How detailed should your lineage be? Options range from coarse (dataset-to-dataset relationships) to fine (row-level or field-level lineage). For AI compliance, you typically need at minimum:
- Dataset-level lineage (which datasets feed which models)
- Transformation-level lineage (what transformations are applied between datasets)
- Schema-level lineage (which fields flow from source to model features)
- For high-risk applications, field-level lineage may be required
Implementation Approach
Step 1: Map the current data landscape. Before implementing lineage tracking, document the current state:
- Identify all data sources used in AI projects
- Map the data flow from source to model for each project
- Identify all transformation steps
- Document storage locations at each stage
- Note where lineage information currently exists (even if informal) and where gaps are
Step 2: Define lineage metadata standards. Standardize the metadata captured at each lineage point:
- Event type: Extraction, transformation, load, split, feature engineering, model training
- Timestamp: When the event occurred
- Source: Where the data came from (dataset, table, file, API)
- Destination: Where the data went
- Transformation applied: Description of what changed
- Row counts: Input and output counts (for completeness verification)
- Actor: Who or what performed the operation (person, service, script)
- Version: Data version or pipeline version
- Quality metrics: Any quality checks performed and their results
Step 3: Instrument your data pipelines. Add lineage capture to your data processing code. This means modifying your ETL/ELT pipelines, feature engineering scripts, and model training scripts to emit lineage events.
Practical approaches:
- Wrapper functions: Create wrapper functions for common data operations (read, write, transform, join) that automatically capture lineage metadata
- Pipeline framework integration: If you use a pipeline orchestration framework (Airflow, Prefect, Dagster, or similar), leverage its built-in lineage capabilities or extend them
- Logging standards: Define a standard logging format for lineage events so that all team members capture consistent information
- Automated tests: Write tests that verify lineage is being captured correctly at each pipeline step
Step 4: Build the lineage store. Implement the centralized store for lineage data:
- Choose a storage solution (graph database for complex lineage relationships, relational database for simpler lineage, or a dedicated data catalog tool)
- Define the schema for lineage records
- Implement APIs for writing and querying lineage data
- Set up retention policies (lineage data should be retained at least as long as the models it documents)
Step 5: Build lineage visualization and query capabilities. Lineage data is only useful if people can access and understand it:
- Build or configure a lineage visualization that shows data flow graphically
- Implement search and query capabilities (find all models that use a specific data source, trace a model's training data back to sources)
- Create standard reports for compliance reviews
- Enable impact analysis queries (if this data source changes, what is affected?)
Step 6: Integrate lineage into workflows. Make lineage part of normal operations:
- Include lineage review in model validation checklists
- Generate lineage reports automatically for compliance documentation
- Set up alerts for lineage anomalies (unexpected data sources, missing lineage events)
- Include lineage completeness as a gate in your deployment pipeline
Lineage for Specific AI Compliance Requirements
EU AI Act Compliance
The EU AI Act requires providers of high-risk AI systems to document training, validation, and testing datasets. Data lineage supports this by providing:
- Provenance information for all training data
- Documentation of data preparation and pre-processing
- Information about data quality measures
- Evidence that data is "relevant, representative, free of errors and complete" as the Act requires
- Traceability from model behavior back to training data characteristics
Financial Services Model Risk Management
SR 11-7 and similar guidance require banks and financial institutions to document model development, including data. Data lineage supports this by providing:
- Complete documentation of data sources and their reliability
- Transformation documentation that supports model replication
- Data quality evidence at each pipeline stage
- Audit trail for all data-related decisions during model development
GDPR Data Subject Rights
GDPR gives individuals rights over their personal data, including the right to know how it is processed. If personal data is used in AI systems, data lineage supports:
- Tracing how personal data flows through AI pipelines
- Documenting what processing is applied to personal data
- Supporting data subject access requests by showing what data is held and how it is used
- Enabling data deletion requests by identifying all locations where personal data exists
Common Data Lineage Mistakes
Starting too late. The hardest and most expensive time to implement data lineage is after systems are in production. Start lineage tracking at the beginning of every project.
Making it a separate process. If data lineage requires manual documentation effort separate from the development process, it will be incomplete and inaccurate. Integrate lineage capture into your code and pipelines so it happens automatically.
Capturing too little. Lineage that only shows "data moved from A to B" without documenting what transformations were applied is insufficient for compliance. Capture transformations, not just movement.
Capturing too much. Row-level lineage for every data point in a 100-million-row dataset creates storage and performance problems without adding proportional value. Match lineage granularity to actual compliance and operational needs.
Ignoring manual data processes. Not all data transformations happen in code. Analysts manually clean data, subject matter experts make judgment calls about data quality, and stakeholders request ad hoc data modifications. Capture these manual steps in your lineage too.
Not testing lineage accuracy. Lineage systems can have bugs just like any other system. Periodically verify that your lineage accurately reflects actual data flows by tracing several paths manually and comparing to the lineage record.
Your Next Step
Pick one active AI project in your agency. Map the complete data flow from source data to model output—every data source, every transformation, every storage location, every feature engineering step. Do this manually, on paper or in a diagram. Note where you have documentation, where you have code that could generate documentation, and where you have gaps. That gap analysis is your implementation roadmap. Start closing the gaps on your highest-risk project first, then build the processes and tooling to handle lineage systematically across all projects.