50 Years of Data Science | David Donoho, 2017
Data science vs. statistics
Is data science a rebranding of statistics? What's the difference?
Is "big data" the difference? No, traditional census data was already big data even by todays standards.
"Among the core discoveries of statistics as a field were sampling and sufficiency, which allow to deal with very large datasets extremely efficiently. These ideas were discovered precisely because statisticians care about big datasets." (747)
Do data scientists have special "big data skills" that statisticians do not have?
The new skills attracting so much media attention are not skills for better solving the real problem of inference from data; they are coping skills for dealing with organizational artifacts of large-scale cluster computing. The new skills cope with severe new constraints on algorithms posed by the multiprocessor/networked world. In this highly constrained world, the range of easily constructible algorithms shrinks dramatically compared to the single-processor model, so one inevitably tends to adopt inferential approaches which would have been considered rudimentary or even inappropriate in olden times. (747)
Lol, like the writing style: "Nevertheless, the scaling cheerleaders are yelling at the top of their lungs that using more data deserves a big shout." (747)
Yanir Seroussi claims that there are few data science positions for people without work experience. Why? (Cited not because it's so important, but because it matches my personal experience):
Barlow implies that would-be data scientists may face years of further skills development post masters degree, before they can add value to their employer’s organization. In an existing big- data organization, the infrastructure of production data processing is already set in stone. The databases, software, and workflow management taught in a given data science masters program are unlikely to be the same as those used by one specific employer. Various compromises and constraints were settled upon by the hiring organizations and for a new hire, contributing to those organizations is about learning how to cope with those constraints and still accomplish something. (748)
The article will plea for a concept of data science:
Fortunately, there is a solid case for some entity called “data science” to be created, which would be a true science: facing essential questions of a lasting nature and using scientifically rigorous techniques to attack those questions. (...) The would-be notion responds to a different set of urgent trends—intellectual rather than commercial. Facing the intellectual trends needs many of the same skills as facing the commercial ones and seems just as likely to match future student training demand and future research funding trends.
The Future of Data Analysis, 1962
John Tukey published an article in 1962 where he confessed that he started thinking of himself as being interested in data analysis and everything it entails, rather than being a statistician interested in (mathematically precise) inference from the particular to the general.
In the new science of data analysis he proposed (N.B. according to his definition mathematics was not a science) formal statistics was only one of the several elements. This section provides a historical analysis of how this call to broaden the curriculum of statistics departments took at least 50 years to start to develop in the envisioned direction; even then, most statistics curricula are still restricted to formal statistics, which is only a fraction of the proposed data analysis science.
Ultimately, this section traces how advances in computational tooling for statistics, such as more recently R or SPSS, were a game changer and made the call for a new data science way more concrete.
Breiman's "Two Cultures", 2001
Breiman distinguishes two approaches to extracting value from data, either 1) prediction (what is the response for future input variables?) or 2) inference (how are input and response variables associated in "nature"?).
With "nature" it is meant here that we can view data as a black box with input and response variables, where some unknown underlying process ("nature") is responsible for input and response being associated. In other words, "nature" is the "true model generating the data" (751).
The split in cultures then is between people doing predictive modeling who want to learn and use the association to predict future variables; and people doing generative modeling who want to model this association such that the best model represents "nature" (as defined above). In generative modeling the model is used to "make inferences about the data-generating mechanism based on the structure of those models" (751) whereas predictive modeling "is effectively silent about the underlying mechanism generating the data, and allows for many different predictive algorithms, preferring to discuss only accuracy of prediction made by different algorithm on various datasets" (751).
In Breiman's estimation in 2001, 98% of academic statisticians are doing generative modeling. Contemporary machine learning today, however, is dominantly doing predictive modeling.
Generally speaking, generative modeling works best on smaller data sets and is more theory heavy. Breiman suggests that the strong emphasis of statistics on generative modeling thus prevents the statistical community to catch on to interesting problems in larger data sets, but also as a "more accurate and informative alternative to data modeling on smaller datasets." (751, Donoho citing Breiman). In other words, Breiman argues that the focus on generative modeling is a hurdle for statistics enlarging its scope to become more like a general data science. Donoho discusses some counters to this point but concedes that especially for statistics in business there is a strong case for more focus on predictive modeling. One counter I find interesting is from David Cox, who argues that "predictive success ...is not the primary basis for model choice" (751, Donoho citing Cox).
The Predictive Culture's Secret Sauce
Okay, so the predictive culture is very successful (also and mainly outside of formal statistics) and we should take it more seriously. But what is the methodology driving its succes (it's "secret sauce")?
Common Task Framework (CTF)
Mark Liberman (2010; a computational linguist) has coined the term Common Task Framework (CTF), which contains three main ingredients:
- "A publicly available training dataset involving, for each observation, a list of (possibly many) feature measurements, and a class label for that observation" (752)
- Notice the emphasis on "publicly available" interesting; I'd say the framework also applies to competitions with proprietary data which is made available within the competition.
- I also notice the mention of class labels; does this framework necessarily exclude unsupervised learning?
- Competitors whose common task is to "infer a class prediction rule from the training data" (752)
- A "scoring referee, to which competitors can submit their prediction rule." (752). The referee runs the submitted prediction rule on a testing set and reports the prediction accuracy objectively and automatically.