Explanatory Stances

Humans often use "explanatory stances" to explain events, as noted by Daniel Dennett. Related ideas are explored in this post.

Explaining via Metaphors

Here are three common metaphors humans use to explain things:

1. The mechanical stance (or the Machine Metaphor) explains behaviour how the parts of a system interact and what they do.
2. The design stance, this has different interpretations. One is of the perceived purpose of something (applied to things created by humans such as tools, but also those hypothesised to be created by universal designer or god).
3. The intentional stance (which I call "Agent Metaphor/Model") uses goals, motives, feelings, intent to explain actions and/or behaviour. Unintentional behaviour/events is usually explained using the machine metaphor (see the following paper, section [Ordinary Behavior Explanation][autonomous_intelligent_systems]).

They can be complementary when applied to the same phenomena or as Ruth Byrne puts it:

Notably, each explanatory stance can be applied to explain the same device or action, but they have different consequences for understanding it. Each stance can lead to different kinds of insights, and to different kinds of erroneous inferences. The atypical application of a particular stance, say, a mechanical stance to explain an action more typically understood from an intentional stance, such as explaining travelers in a crowded airport as like pinballs careening around a pinball machine, may be interpreted analogically to yield new inferences [Keil, 2006].

In technical fields, many complex systems are conceptualised as machines: composed of parts, each with a function, a role. Many are also conceptualised as graphs.

In everyday life we often conceptualise complex systems as humans or agents (wholly or in part). This may happen with systems using human language or behaving autonomously, but other times it is due to pragmatic reasons. They would use and expect the kind of explanation a human would give, if there were one.

What seems here most fundamental than the particular stances is the selection of a metaphor to structure thinking and obtaining insights.

Other metaphors and analogies could be proposed for specific problems.

A similar idea is expressed in "[How People Explain Action (and Autonomous Intelligent Systems Should Too)][autonomous_intelligent_systems]":

For those intentional agents, we hypothesize, people will apply the same conceptual framework of behavior explanation that they apply to humans (...) a subset of AIS that people do not regard as intentional agents; and for those, they may apply a purely mechanical explanatory framework.

And more recently, in Good Explanations in XAI:

People may tend to adopt multiple stances in their preferred explanations of an AI decision support system and its decisions, not unlike their tendencies in interacting with social robots [Clark and Fischer, 2023]. People are aware that a social robot is a machine, but interpret it as a depiction of a character, not unlike a ventriloquist dummy, and engage with it in pretense of interacting with the depicted character [Clark and Fischer, 2023]. Similarly, they may be aware that an AI decision support system is an algorithm but they may interpret its decisions as a depiction of those provided by a human, e.g., a bank loan assessor, or the organization the human represents, a bank. Hence, an intentional stance and a design stance may both be useful in different contexts for explaining how automated agents behave [Veit and Browning, 2023].

The summary is:

1. Mechanical stance (which I call "Machine Metaphor/Model"),
   • Explain outcomes by considering the parts of a system and how they interact (that is, a mechanism).
2. Design stance,
3. Intentional stance (which I call "Agent Metaphor/Model").
   • Explanation uses purposes (teleology), goals, motives, feelings, intent to explain actions and/or behaviour. It assumes a "state of mind" of the object and uses the model to predict behaviour.
   • Unintentional behaviour is usually explained using the machine metaphor.

The selection of a stance is a relevant aspect to consider when generating explanations. An example is given by byrne_human_explanations:

A potential implication for XAI is the necessity to consider when a particular stance is appropriate for explaining an AI system's decision. A question posed by a user as to why an AI system refused their loan may be a request for causal information of a mechanistic sort about how the AI system came to make that decision. Hence, it may require an explanation based on information about how the user's input features relate to a training data set, e.g., that loans have been refused for applicants of similar salary level, occupational status, credit history, as the user. Alternatively, it may be a request for functional information of a teleological sort about the purpose of the AI's decision. Accordingly, it may require instead information about the goal and consequences of the output, e.g., that decisions of this sort mitigate the risk of applicants defaulting from repayment. Each sort of explanation will lead users to develop a different understanding of aspects of an AI system, and impact their learning, satisfaction, and trust in it.

Next, a few of these stances are explored in different contexts.

Human To Human Explanations

When humans explain phenomena to each other they use language, pictures and other media. But we don't get access to each others' brain processes (to a large extent, not even our own); and even if we did, the task wouldn't be trivial.

A brain may use "communication processes" that look at "decision processes" (see Marvin Minsky, The Emotion Machine). Then the statement is exchanged and interpreted by another agent. The statement can be tested against reality, or for inconsistencies or compared to other statements.

In other words, we largely treat brains as black boxes.

Should we treat DL models the same, and accept predictions without "peeking into the black box", just as we do with humans? If so, under which circumstances?

The idea of this post is that there will be a level of complexity and utility at which this will also happen for DL models.

DL Models are Scientific Models

DL models are mathematical functions. Both the architecture and the trained model can be considered scientific models or theories. In this case, they should be explainable. Here there is an underlying metaphor of "DL models as machines" and we expect them to be explainable in terms of their parts and how they interact.

There are also societal reasons for this expectation: many models are used on decision making and high-stakes scenarios, so we expect explanations or justifications for their decisions, just as we expect from humans making decisions.

For simple or narrow-AI models, we may expect to peek into the black box, and to analyze their input-output relations to get insight into their inner workings.

They're expected to be decomposable in domain-level concepts (just as a theory is).

In other words, we want to answer: What did it learn? Which concepts is the network using for their decision making? But how can we extract their "baked in" knowledge?

But, what if the metaphor used for a complex DL model is that of a scientist, or a human, rather than a machine?

DL Models are Scientists

General-purpose models may explain decisions in a human-like way, and be as hard to inspect as the brain is. Treating these models as a scientific theory of a domain, or a narrow AI model, may be a counter-productive, and even a category error.

In this case, a detailed explanation process could be as hard to extract as it is from a brain.

So what do we do? In this case, it may be better to expect what we expect from a human: Take an explanation in natural language and test it and compare it to other established statements.

This is what the paper [The Mythos of Model Interpretability][mythos] describes, in the section about post-hoc textual explanations:

Humans often justify decisions verbally. Similarly, one model might be trained to generate predictions, and a separate model, such as a recurrent neural network language model, to generate an explanation.

In other words, we would treat it as a black box.

Summary

We have explored two metaphors "DL models as Agents / Scientists" and "DL Models as Scientific Models", and the expectations for each case:

1. As Scientific Models: We can study them as black boxes, and also internal mechanisms,
2. As Scientists: We can only study them as black boxes,
   1. This includes Agents that can output a certain explanation (e.g. humans).

As science advances, some models may become explainable. Similarly, intrinsically explainable architectures may be designed, and could be preferred over black box architectures (e.g. Capsule Networks were an effort in this direction).

Now we go back to common methods used to explain mostly narrow AI algorithms.