A healthcare-focused AI agency in Nashville was struggling with a hospital readmission prediction model. They had tried five different model architectures — logistic regression, random forests, XGBoost, LightGBM, and a neural network. The best AUC was 72%, well below the 85% target the hospital system required to justify the $600,000 annual contract. In desperation, they brought in a senior ML engineer who spent two weeks not touching the model but completely rebuilding the feature engineering pipeline. She created 147 new features from the same raw data — time-windowed aggregates of vital signs, interaction features between diagnosis codes and medication histories, temporal patterns in lab results, and engineered features capturing the trajectory of patient health indicators across visits. The XGBoost model that had peaked at 72% AUC hit 89% on the same test set. Same model, same data, radically different features. The contract was saved.
Feature engineering is the process of transforming raw data into inputs that make machine learning models more effective. For AI agencies delivering production ML systems, feature engineering is often the highest-leverage activity in the entire project — it frequently matters more than model architecture, hyperparameter tuning, or training technique. Yet most agencies treat feature engineering as an afterthought, spending 80% of their time on model experimentation and 20% on features. The agencies that consistently deliver high-performing models flip that ratio.
Why Feature Engineering Dominates Model Performance
The Feature Engineering Multiplier
In a landmark study of Kaggle competition winners, the top performers spent an average of 60% of their time on feature engineering and only 15% on model selection and tuning. The remaining 25% went to data cleaning and validation. This pattern holds in production ML: the gap between a mediocre model and a high-performing model is almost always closed by better features, not better architectures.
Why features matter more than models:
- Modern model architectures are mature — the accuracy difference between XGBoost, LightGBM, and a well-tuned neural network on the same features is typically 1-3%
- Feature engineering can capture domain knowledge that models cannot learn from raw data alone
- Good features reduce the need for model complexity, which improves training speed, inference speed, and interpretability
- Features that capture the right abstractions generalize better to unseen data
The Feature Audit
Before building new features, audit what you already have. Most production datasets contain features that hurt model performance.
Feature audit checklist:
- Leaky features: Features that contain information about the target that would not be available at prediction time. A "days since last purchase" feature computed using the purchase you are trying to predict is leaky and will produce inflated test metrics that collapse in production.
- Constant or near-constant features: Features with fewer than five unique values across the dataset contribute no information and add noise.
- Highly correlated features: Pairs of features with correlation above 0.95 are redundant. Keep the one with higher predictive power and remove the other.
- High-cardinality categorical features: Categorical features with thousands of unique values (like customer ID or product SKU) need encoding strategies, not raw inclusion.
- Missing value patterns: Features missing more than 50% of values may be unreliable. Features where missingness is informative (missing lab tests indicate the test was not ordered) should be encoded as a separate binary feature.
Feature Engineering Techniques for Production Systems
Temporal Feature Engineering
For any prediction task involving time-series or sequential data, temporal features are typically the most powerful feature category.
Window-based aggregations:
Compute statistics over multiple time windows to capture both recent behavior and long-term trends.
- Rolling windows: Mean, median, standard deviation, min, max, count over the last 7, 14, 30, 60, 90 days
- Expanding windows: Cumulative statistics from the beginning of the entity's history
- Exponentially weighted windows: Recent observations weighted more heavily than older ones
Temporal ratios and differences:
- Ratio of last 7 days to last 30 days (captures acceleration or deceleration)
- Difference between current period and same period last year (captures year-over-year change)
- Ratio of weekday to weekend behavior (captures behavioral patterns)
Trend features:
- Slope of a linear regression fitted to the last N observations (captures direction and magnitude of change)
- Number of consecutive increases or decreases (captures momentum)
- Time since last significant event (last purchase, last login, last complaint)
Cyclical encoding for time features:
Encode cyclical time features (hour of day, day of week, month of year) using sine and cosine transformations so that the model understands that hour 23 and hour 0 are adjacent.
- hour_sin = sin(2 pi hour / 24)
- hour_cos = cos(2 pi hour / 24)
Interaction Features
Interaction features capture relationships between features that the model might not learn automatically, especially for tree-based models with limited depth.
Arithmetic interactions:
- Ratios: featureA / featureB (price per unit, revenue per employee, clicks per impression)
- Products: featureA * featureB (captures joint effects)
- Differences: featureA - featureB (captures relative positioning)
Categorical interactions:
- Combine two categorical features into a single feature: city + productcategory = "ChicagoElectronics"
- Target encoding of interaction features: mean target value for each combination of categorical values
Domain-specific interactions:
- BMI from height and weight in healthcare
- Price-to-earnings ratio in finance
- Engagement rate from impressions and clicks in marketing
- Utilization rate from capacity and actual usage in operations
Encoding Strategies for Categorical Features
The encoding strategy for categorical features significantly affects model performance, especially for high-cardinality features.
Target encoding: Replace each categorical value with the mean target value for that category. Add Gaussian noise during training to prevent overfitting. Use leave-one-out encoding to avoid data leakage (compute the mean target value excluding the current observation).
Frequency encoding: Replace each categorical value with its frequency in the training data. Simple, effective, and leakage-free. Works well when the frequency of a category is informative (popular products behave differently from niche products).
Binary encoding: Convert each categorical value to a binary representation and create one feature per bit. More memory-efficient than one-hot encoding for high-cardinality features.
Embedding encoding: For neural network models, learn a dense embedding for each categorical value during model training. The embedding dimensionality should be roughly the fourth root of the number of unique values (so a feature with 10,000 unique values gets a 10-dimensional embedding).
Hash encoding: Apply a hash function to map categorical values to a fixed number of buckets. Handles new categorical values at inference time without retraining. Use 2-3 different hash functions and concatenate the outputs to reduce collision effects.
Text Feature Engineering
For models that use text as input alongside structured features, text feature engineering extracts structured signals from unstructured text.
Statistical text features:
- Document length (word count, character count)
- Vocabulary richness (unique words / total words)
- Average word length
- Sentence count and average sentence length
- Punctuation frequency (exclamation marks correlate with urgency or sentiment)
- Capitalization patterns (proportion of uppercase words)
Domain-specific text features:
- Keyword presence indicators for domain-important terms
- Named entity counts (number of person names, organization names, locations mentioned)
- Sentiment scores from a pre-trained sentiment model
- Topic distribution from a topic model
Embedding features:
- Sentence or document embeddings from a pre-trained language model (sentence-transformers)
- Reduce dimensionality using PCA or UMAP for use as features in tree-based models
- Use the full embedding as input for neural network models
Geospatial Feature Engineering
For applications involving location data, geospatial features capture spatial patterns.
Distance features:
- Distance to the nearest point of interest (nearest store, nearest hospital, nearest competitor)
- Distance to a reference point (city center, headquarters, port)
Density features:
- Number of entities within a radius (restaurants within 1 mile, competitors within 5 miles)
- Population density of the area
- Business density of the area
Aggregation by geographic unit:
- Average property value by zip code
- Crime rate by census tract
- Average income by metropolitan area
Geohash encoding:
- Convert latitude/longitude to geohash strings at multiple resolutions
- Treat geohashes as categorical features for encoding
Feature Store Architecture
Why Feature Stores Matter for Agencies
A feature store is a centralized repository for storing, managing, and serving ML features. For agencies delivering multiple models or maintaining long-running production systems, a feature store eliminates redundant feature computation, ensures feature consistency between training and serving, and accelerates new model development.
Benefits for agency delivery:
- Training-serving consistency: Features computed for training are identical to features computed for real-time serving, eliminating a common source of production bugs
- Feature reuse: Features built for one model can be reused by other models without recomputation
- Point-in-time correctness: Feature stores handle the complexity of computing features as of a specific historical point in time, preventing data leakage in training data
- Monitoring: Centralized feature storage enables centralized feature monitoring and drift detection
Feature Store Options
Feast (open-source): The most widely adopted open-source feature store. Supports offline (batch) and online (real-time) feature serving. Integrates with common data warehouses (BigQuery, Redshift, Snowflake) and online stores (Redis, DynamoDB). Good default choice for agencies building on cloud infrastructure.
Tecton (managed): A fully managed feature store with built-in feature transformation, monitoring, and real-time computation. Higher cost but lower operational burden. Best for agencies that want to minimize infrastructure management.
Hopsworks (open-source with managed option): Feature store with built-in feature engineering capabilities. Supports both batch and streaming feature computation. Good for agencies processing streaming data.
Custom feature stores: For simpler use cases, a feature store can be built from a data warehouse (for offline features) and a key-value store (for online features) with a thin orchestration layer. This is appropriate for agencies with one or two models and limited feature complexity.
Feature Pipeline Design
Batch feature pipelines compute features on a schedule (hourly, daily) from raw data stored in a data warehouse.
- Use SQL or PySpark for feature transformations
- Schedule with Airflow, Dagster, or Prefect
- Write computed features to both the offline store (for training) and the online store (for serving)
- Implement idempotent pipelines that can be safely re-run without creating duplicates
Streaming feature pipelines compute features in real time from streaming data sources (Kafka, Kinesis).
- Use Flink, Spark Streaming, or a serverless function for transformations
- Write computed features to the online store with low latency
- Implement exactly-once processing guarantees to prevent duplicate or missing features
- Handle late-arriving data with watermarks and grace periods
On-demand feature computation computes features at request time from the raw input.
- Use for features that depend entirely on the current request (text length, presence of keywords)
- Keep computation lightweight — complex on-demand features add inference latency
- Implement caching for features that are expensive to compute and change infrequently
Feature Selection for Production
Automated Feature Selection Methods
After generating hundreds of candidate features, select the subset that maximizes model performance while minimizing complexity.
Filter methods (fast, model-agnostic):
- Mutual information between each feature and the target
- Correlation analysis (remove features with low correlation to target or high correlation to other features)
- ANOVA F-test for categorical targets
- Chi-squared test for categorical features
Wrapper methods (slower, model-specific):
- Recursive feature elimination: Train the model, remove the least important feature, retrain, repeat
- Forward selection: Start with no features, add the feature that most improves performance, repeat
- Backward elimination: Start with all features, remove the feature whose removal least degrades performance, repeat
Embedded methods (part of model training):
- L1 regularization (Lasso): Drives unimportant feature weights to zero during training
- Tree-based feature importance: Use the feature importances from a trained tree ensemble to rank and select features
- SHAP-based selection: Use SHAP values to identify features with the highest average impact on predictions
Feature Selection Strategy for Agencies
Start broad, then narrow:
- Generate all candidate features (aim for 200-500 candidates)
- Remove features with near-zero variance or high missing rates
- Remove features with high pairwise correlation (keep the more predictive one)
- Use mutual information or tree-based importance to rank remaining features
- Train models with the top 20, 50, 100, and 200 features
- Select the feature set that achieves target accuracy with the fewest features
Fewer features is almost always better for production because:
- Fewer features mean faster inference
- Fewer features mean fewer potential drift sources to monitor
- Fewer features mean simpler data pipelines with fewer failure points
- Fewer features mean easier model interpretability for clients
Typical production feature counts:
- Simple classification (churn, fraud): 30-80 features
- Complex tabular prediction: 80-200 features
- Time-series forecasting: 50-150 features
Feature Documentation and Governance
Feature Registry
Every feature in production should be documented in a feature registry with:
- Feature name: Clear, descriptive, following a consistent naming convention
- Feature definition: Precise description of what the feature represents and how it is computed
- Data source: The raw data table(s) or stream(s) from which the feature is derived
- Computation logic: SQL query, Python function, or transformation specification
- Data type: Integer, float, categorical, boolean, embedding
- Expected range: The expected minimum and maximum values for numerical features, or the expected set of values for categorical features
- Update frequency: How often the feature is recomputed (real-time, hourly, daily)
- Owner: The team or individual responsible for maintaining the feature
- Models using this feature: List of all models that depend on this feature
Feature Versioning
Features evolve over time — computation logic changes, data sources change, business definitions change. Version features like you version code.
- Assign version numbers to feature definitions
- When a feature's computation logic changes, create a new version rather than modifying the existing version
- Maintain backward compatibility — old model versions should still work with the feature versions they were trained on
- Document the reason for each version change
Feature Quality Testing
Unit tests for feature computation:
- Test that features compute correctly on known input data
- Test edge cases: null values, empty strings, extreme values, missing data
- Test that computed values fall within expected ranges
- Test that feature computation is deterministic (same input always produces same output)
Integration tests for feature pipelines:
- Test that the pipeline reads from the correct data sources
- Test that the pipeline writes to both offline and online stores
- Test that online feature values match offline feature values for the same entity and timestamp
- Test pipeline recovery from failures (restart and produce correct output)
Data quality tests in production:
- Monitor feature value distributions for drift
- Monitor missing value rates
- Monitor feature freshness (how long since the feature was last updated)
- Alert on anomalous feature values that fall outside expected ranges
Your Next Step
Take your best-performing model and run a feature importance analysis using SHAP values. Identify the top 10 features driving predictions. For each of those features, brainstorm three derived features that capture the same signal with more granularity — temporal aggregates at multiple windows, interaction features with other high-importance features, or encoding strategies that preserve more information. Add those 30 candidate features to your training pipeline and retrain. In nearly every case, you will see a measurable accuracy improvement from features that took hours to engineer versus days of model architecture experimentation. Feature engineering is where production ML performance lives, and most agencies are underinvesting in it.