202309201300-chi-squared_test
Math
TODO
Code example
N.B. notice the difference between scipy.stats.chisquare and scipy.stats.chi2_contingency.
SE post
scipy.stats.chi2_contingency is the Pearson's chi-square test or chi-square test of association.
The null hypothesis in this case is that the two variables are independent.
You would want to use this test for example as a sort of "correlation" analysis on nominal/categorical variables.
This means that a significant p-value, e.g. p < 0.05, indicates that it's very unlikely the observed counts are true under assumption of independence, hence we conclude the variables are associated. So: significant p-value -> nominal variables are associated.
(Cf. null hypothesis testing in other statistical tests: How likely is the observed difference between two sample means under assumption of the null hypothesis? Low p? Reject null hypothesis, then we say there is a significant difference between the two groups.)
The scipy.stats.chisquare test whether frequencies in the observed categorical data have the given expected frequencies.
In other words, this tests the null hypothesis that there is no difference between the observed sample distribution and the expected distribution.
import pandas as pd
from scipy.stats import chi2_contingency
# create example dataframe
data = {'Gender': ['Male', 'Male', 'Female', 'Female', 'Male', 'Female'],
'Smoker': ['No', 'No', 'Yes', 'No', 'Yes', 'No']}
df = pd.DataFrame(data)
# create contingency table
contingency_table = pd.crosstab(df['Gender'], df['Smoker'])
Output contingency table:
Smoker No Yes
Gender
Female 2 1
Male 2 1
# perform chi-squared test
stat, p, dof, expected = chi2_contingency(contingency_table)
# print results
print("Test statistic:", stat)
print("p-value:", p)
print("Degrees of freedom:", dof)
print("Expected frequencies:\n", expected)
Nowadays the pd.corr takes a method argument with custom callables.
This works and is quite clean:
def compute_chi2(x: np.ndarray, y: np.ndarray) -> tuple:
'''
Compute chi-squared for input 1D arrays with nominal variables.
You can use this function as the method argument of pd.corr
to generate the pairwise statistics for a whole dataframe.
'''
contingency_table = pd.crosstab(x, y)
stat, p, dof, expected = chi2_contingency(contingency_table)
return stat, p, dof, expected
# In this example df only has nominal variables
chi2_p_values = df.corr(method=lambda x, y: compute_chi2(x, y)[1])