Exploratory data analysis
Why EDA?
- Maximize understanding of the data
- data types
- data ranges
- cardinality
- correlations
- outliers
- Identity and correct data problems
- Test assumptions about the problem
- Test for unwanted biases
There is no clear-cut recipee for EDA that always works. EDA is called exploratory for good reason. Nevertheless, there are some obvious ingredients:
- What do variables mean?
- Univariate analysis: histograms and other distributions
- Bivariate analysis: correlations
- Anomaly detection: outliers
- Strategies for fixing missing values
- And so on!
Tracking results in notebooks
One approach:
- Write an MLflow component that installs Jupyter and required libraries
- Execute the EDA from within this component
- Embed plots and comments into the Jupyter notebook
- Track inputs and outputs of the notebook with artifact tracking
E.g. in Weights & Biases, use save_code=True:
run = wandb.init(
project="my_exercise",
save_code=True
)
Profiling
E.g. pandas profiling:
import pandas_profiling
import pandas as pd
df = pd.read_parquet("genres_mod.parquet")
profile = pandas_profiling.ProfileReport(df)
profile.to_widgets() # runs in Jupyter notebook