Testing and Data Validation in Data Science
Testing is very important in software engineering, but is often lacking in data science contexts. We should also test our data science solutions though, because they are increasingly used in production. We should not inform business decisions on untested code!
Testing in data science is a bit different than traditional testing though. Things to think about in data science are:
- Check for wrong encoding of features (schema check)
- Check that features aren't missing
- Check for inappropriate use of features
- Data breaks statistical assumptions of the used model
The equivalent of unit testing in software is data validation in data science, where we check the assumptions we make about our data. A notable difference with typical unit testing is that data validation tests can fail due to reasons that are external to your system, such as errors in upstream ETL pipelines or simply changes in the world!
This achieves that we:
- Catch bugs in pre-processing or data
- Notice changes in inputs, because changes in inputs result in changes in model outputs
- Big changes may invalidate your model assumptions
- Possibly a trigger for automatic retraining
Data validation checks are typically performed after data preprocessing but before data segregation in train and test sets. This way you validate the complete cleaned data as it's intended to be ingested in the training pipeline.
In some specific cases, you may also (or instead) want to validate data after data segregation. For example, if you have time series data, you may want to validate that the train set does not include future dates and the test set does not include past dates.
Deterministic vs. non-deterministic tests
A deterministic data test checks properties that can be measured without uncertainty. Examples:
- Check presence of columns
- Check length of the dataset: do we have enough samples?
- Check distinct values of a categorical variable
- Check numerical variable values are in an allowable range
- Check geographic localization of data
Non-deterministic tests instead have a statistical nature where you measure a quantity with intrinsic uncertainty.
- Check the standard deviation and the mean of columns
- Check the correlations between columns and of columns with the target label
- Check distribution of values
- Check for outliers
For non-deterministic tests you want to apply Statistical Hypothesis Testing. When performing an EDA you make modelling choices based on the EDA data set. Because the non-deterministic properties will always be somewhat difference between two data sets due to uncertainty, we want to statistically test if the test or production data set significantly deviates from the EDA data set.
We can for example achieve this with frequentist A/B testing on the mean between a null hypothesis (same mean) and an hypothesis (different mean). In this case you would for example write a t-test on samples from the EDA set and the test set.
NOTE: such a statistical test has it's own assumptions! If these do not hold, the written test is unreliable.
NOTE: if we are testing on multiple columns, consider issues of multiple-hypothesis testing! Interesting read: precision and recall trade-off and multiple-hypothesis testing.
Tests with parameters
For data validation tests it may be nice to allow the passing of different data sets to the test suite.
So this is a specific case where you may want tests with input parameters from the command line.
The note testing code contains a section on how to do this in pytest.
Tools:
- Standard test suites like
pytestwith statistical packages likescipy.stats - Great Expectations
- Pandera
Links
Also see:
Reads: