202306021623-feature_store
Where to perform feature engineering and store the resulting features in order to serve them during training and inference?
Do not do feature engineering in the preprocessing pipeline and store the features in the preprocessed data that is used for training, because then production data that comes in does not have the features.
You may add feature engineering in an inference pipeline, so that you also have the features available in production. But the downside of this approach is that you have to repeat the feature engineering every time you run the model. This is only feasible for small scale operations. Another downside is that you may end up introducing differences in the feature engineering over time between different inference pipelines.
The purpose of a feature store is to achieve development/production symmetry while avoiding these issues.
Characteristics of a feature store are:
- Centralized implementation of feature computation
- Data freshness through automatic (re-)computation of features based on certain events
- Features are discoverable and documented through a feature registry
- "Cold" batch serving
- "Hot" real-time serving