Exploratory data analysis

Why EDA?

  • Maximize understanding of the data
    • data types
    • data ranges
    • cardinality
    • correlations
    • outliers
  • Identity and correct data problems
  • Test assumptions about the problem
  • Test for unwanted biases

There is no clear-cut recipee for EDA that always works. EDA is called exploratory for good reason. Nevertheless, there are some obvious ingredients:

  • What do variables mean?
  • Univariate analysis: histograms and other distributions
  • Bivariate analysis: correlations
  • Anomaly detection: outliers
  • Strategies for fixing missing values
  • And so on!

Tracking results in notebooks

One approach:

  • Write an MLflow component that installs Jupyter and required libraries
  • Execute the EDA from within this component
  • Embed plots and comments into the Jupyter notebook
  • Track inputs and outputs of the notebook with artifact tracking

E.g. in Weights & Biases, use save_code=True:

run = wandb.init(
  project="my_exercise",
  save_code=True
)

Profiling

E.g. pandas profiling:

import pandas_profiling
import pandas as pd

df = pd.read_parquet("genres_mod.parquet")
profile = pandas_profiling.ProfileReport(df)
profile.to_widgets()  # runs in Jupyter notebook