What is the question (in data analysis) | Leek & Peng, 2015 Science Vol. 347 Issue 6228
The reproducability of analyses is a hot topic right now, but ultimately even irreproducible analyses can be reproduced. Faulty analyses can be reproduced too. The more interesting question to ask then is: is this data analysis question correct?
To asses that, we need to know which type of analysis is being done. Is it the right one for the question you are trying to answer?
Conclusion:
Confusion between data analytic question types is central to the ongoing replication crisis, misconstrued press releases describing scientific results, and the controversial claim that most published research findings are false (13, 14). The solution is to ensure that data analytic education is a key component of research training. The most important step in that direction is to know the question. (1315)
Six types of data analysis
- Descriptive data analysis: "summarize the measurements in a single data set without further interpretation." (1314)
- Exploratory data analysis: "builds on a descriptive analysis by searching for discoveries, trends, correlations, or relationships between the measurements to generate ideas or hypotheses." (1314)
- "seeks to make discoveries, but can rarely confirm those discoveries" (1315)
- Inferential data analysis: "quantifies whether an observed pattern will likely hold beyond the data set in hand" (1315)
- Your common statistical analysis
- Almost always determines a correlative relation between two measurements, not the underlying (causal) mechanisms or reasons.
- Predictive data analysis: "uses a subset of measurements (the features) to predict another measurement (the outcome) on a single person or unit." (1315)
- Predict one measurement from another, but does not in itself explain why the prediction works.
- Notice the emphasis on individual. The next type of analysis is on super-individual level.
- Causal data analysis: "what happens to one measurement on average if you make another measurement change. Such an analysis identifies both the magnitude and direction of relationships between variables on average." (1315)
- E.g. smoking increases your risk of cancer; this is a causal effect expressed in terms of the average risks.
- Mechanistic data analysis: "changing one measurement always and exclusively leads to a specific, deterministic behavior in another." (1315)
- Rarely achievable outside of engineering
The crux: each type of analysis supports a type of conclusion that can be drawn from it. The cliché that you cannot infer causation from correlation is an obvious example.
| Real question type | Perceived question type | Phrase describing error |
|---|---|---|
| Inferential | Causal | "Correlation does not imply causation" |
| Exploratory | Inferential | "Data dredging" |
| Exploratory | Predictive | "Overfitting" |
| Descriptive | Inferential | "n of 1 analysis" |