Black Box AI: Black box is an epistemic notion

Also see the concept black box society . I think it was Frank Pasquale who popularized the term "black box" in the field of AI.

When trying to apply the concept, I noticed there were some conceptual ambiguities around it's usage.

Typically models such as neural networks are given as an example of a "black box" model, indicating that these models are 1) opaque and 2) that if we want to try and explain their workings, we often have to reside to post-hoc and model-agnostic methods, i.e. methods that treat NNs as a "black box". But in this characterization there is already a conceptual ambiguity in the usage of the term. It seems the term "black box" is used both to refer to an opaque model (such as a NN) as well as to "black box" methods that treat a system as a black box. The former is inherently connected to opaque models, whereas you may apply the former also to transparent models.

This implies you can use "black box" methods (e.g. for generating model explanations) on "white box" i.e. transparent models. What is meant with a "black box" method is that this methods only requires knowledge about the input and output, without the need to access model internals.

This is I think a quite consistent interpretation that pertains to a level of system understanding. I'm thinking Dennet's stances here, as well as Marr's levels of explanation. A common point they have is that users can for example perfectly fine use computers without understanding their inner workings. This means that the user views the computer as a black box. His only interactions are giving input (e.g. via the keyboard ) and reading and reacting to the output (e.g. text on the screen). However, the computer is not a black box to the same degree for a computer technic. Nevertheless, it is unlikely that this computer technic knows the inner workings of the chips on a motherboard. And neither does he need to know!

TLDR; in this sense "black box" refers to taking an outside perspective on a system's functioning.

However, is this at odds with speaking of a neural network as a "black box", for example as in the phrase "opening up the black box"?

Let's again take the example of a neural network, e.g. a convolutional neural network for image recognition. The reason XAI methods often take a black box perspective - it's called 'post-hoc' and 'model-agnostic' in the literature - is that neural networks have millions of internal parameters that interact non-linearly so that we cannot readily explain anymore how they are combined to produce a particular prediction. We cannot explain the inner workings of such a neural network because we do not understand it. We don't have to explain that if we view the neural network as a black box and explain its workings purely in terms of input and output.

Nevertheless, we can analyze gradient flows in these networks, for example in order to say to which input features a certain network layer pays more "attention". These methods are model-specific and even though they "open up" the black box, they cannot be regarded as black box methods because they require access to model internals.

Now consider that the antonym "white box", which is sometimes used, suggests a box where we can look "inside" as opposed to a "black box" where we cannot. Well, model-specific XAI methods that e.g. visualize activations in intermediate neural network layers, do have access to model internals. Would this mean that a neural network can be a "white box" after all? That seems counter-intuitive. I'd say that the reason why we call neural nets black boxes is because we do not have epistemic access to their internals. We may inspect the millions of parameters

I think this means that the term "black box" in AI deviates from its general definition in that it can indicate access to internals, but acces without understanding, without knowing? That's not entirely the same as a viewpoint where you do not access the internals at all.

WIP

Conclusion: "black box" is not about accessing system internals. Typical users also often cannot access the internals of white box systems, i.e. they are often black boxes to them. Instead, "black box" is an epistemic notion that points to a level of understanding.

TODO argue/show how this is consistent with the typical definition of black box.