Explanation

This post describes what explanations are in the context of artificial intelligence.

Definitions

What is an explanation? Here is definition from "How People Explain Action (and Autonomous Intelligent Systems Should Too)" (2017):

Explanation is arguably a three-value predicate: someone, a communicator, explains something to someone, an audience. The success of an explanation therefore depends on several critical audience factors—assumptions, knowledge, and interests that an audience has when decoding the explanation.

Explaining Explanations in AI also defines of explanations and explanation in AI:

(...) "explanation" refers to numerous ways of exchanging information about a phenomenon, in this case the functionality of a model or the rationale and criteria for a decision, to different stakeholders (Lipton, 2016; Miller, 2017).

Inspired by Explanation in artificial intelligence: insights from the social sciences, this post defines explanation as:

Explanation

A two-step process involving 1. the generation of explanatory hypotheses (cognitive process) and 2. the communication to an audience (social process) including ourselves. The steps may repeat indefinitely.

Explaining can expose a lack (or illusion) of understanding; it can also show and improve understanding. Eventually, one hypothesis may be selected until it is either:

• Contradicted by experience (falsified),
• Superseded by a simpler or more general one,
• Shown to be inconsistent with prior knowledge.

So the cognitive process involves the generation of hypotheses, theories and reasons, which in turn reflect our understanding about how something works.

In the paper "From scientific theory to duality of predictive artificial intelligence models" it is stated, of scientific understanding of a phenomenon (references removed):

(...) if there is an underlying intelligible theory such that scientists can qualitatively comprehend the consequences of this theory without the need for quantitative analysis and exact calculations.

Contrastive Questions (and Explanations)

Miller et al., define contrastive questions as:

Perhaps the most important result from this work is that explanations are contrastive; or more accurately, why–questions are contrastive. That is, why–questions are of the form "Why P rather than Q?", where P is the fact that requires explanation, and Q is some foil case that was expected.

They argue that all why-questions are contrastive, since even those without explicit contrast such as "Why P?" still have an implicit one.

If we can find a useful foil, then answering the question is easier, as we only need to focus on the differences between these two cases to generate a contrastive explanation.

Counterfactuals

Notice how "What would have happened if ..." is exactly a counterfactual, and "Why P rather than Q?" involves an alternative fact: counterfactuals are useful for providing causal explanations. However, not all causal explanations require counterfactuals (this is rung 3 of Judea Pearl's Ladder of Causation)

For scientific explanation, this is more rigorous and there are also other mechanisms (e.g. Pearl's Ladder of Causation).

If we have a mathematical model, we attribute as "cause" of an output, a certain input or feature value; but in that case, it's not the cause for the real world event but for the model's output. Whether it accurately models the real world or not, is a different question.

On choosing relevant counterfactuals (or foils), the paper "Counterfactual explanations without opening the black box (...)" requires the counterfactual to be a close, possible world:

We define Counterfactual Explanations as statements taking the form: Score p was returned because variables V had values (v1, v2,...) associated with them. If V instead had values (v1', v2',...), and all other variables had remained constant, score p' would have been returned.

While many such explanations are possible, an ideal counterfactual explanation would alter values as little as possible and represent a closest world under which score p' is returned instead of p. The notion of a "closest possible world" is thus implicit in our definition.

Another kind of counterfactual seeks for the output to remain the same rather than to change (for fairness purposes), as they state:

Kusner et al.58 consider counterfactuals where the subject belongs to a different race or sex, and require that the decision made remain the same under such a counterfactual for it to be considered fair. In contrast, we consider counterfactuals in which the decision differs from its current state.

Often, multiple counterfactual explanations may be preferred, so that the subject can choose the optimal one.

How do we give an explanatory hypothesis based on counterfactuals? If the cause "X" leads to both P and Q, then it can't explain why only one occurred. So we want a case where P needs "X" to have happened, while "not X" is required for Q.

The paper Beware of Inmates Running the Asylum has an interesting example:

(...) "Why did Mr. Jones open the window?" with the response "Because he was hot" is not useful if the implied foil is Mr. Jones turning on the air conditioner, as this explains both the fact and the foil; or if the implied foil was why Ms. Smith, who was sitting closer to the window, did not open it instead, as the cited cause does not refer to a cause of Ms. Smith's lack of action.

Attribution Theory

Attribution theory is about how people assign causes to events. Assigning causes is Miller et. al. "necessary to provide explanations".

They distinguish social from general causal attribution. For the general case, they state (references removed):

Research on how people connect causes shows that they do so by undertaking a mental simulation of what would have happened had some other event turned out differently.

It's similar for social attribution, but uses a belief, desires, intents and traits (of people) as causes.

Importantly, when there are many variables, the combinatoric is large, and this would be unfeasible, so people must select a few counterfactuals to simulate, giving rise to the causal selection problem. They add:

However, simulating an entire causal chain is infeasible in most cases, so cognitive scientists and social psychologists have studied how people decide which events to 'undo' (the counterfactuals) to determine cause. For example, people tend to undo more proximal causes over more distal causes [Miller and Gunasegaram, 1990], abnormal events over normal events [Kahneman and Tversky, 1982], and events that are considered more 'controllable' [Girotto et al., 1991].

More on Contrastive Questions (and Explanations)

Hesslow states:

What I want to suggest, then, is that the explanandum should be construed as a relation which involves three things: an object a, an object of comparison b and an explanandum property E which a has and b does not have.

The complexity, of course, lies on formulating a question that makes the important difference with a reference case obvious.

In many cases, the complexity is finding a good, relevant foil. As Explaining Explanations in AI states:

However, choosing a relevant set of cases or events against which contrastive explanations are provided is not a straightforward challenge. The way in which information is transferred has a substantial impact on the quality and psychological acceptability of explanations (Hilton, 1990).

The last sentence is related to the social process of explaining, which we describe in this post (relevance from Gricean Maxims).

Researchers have pointed out many heuristics used by humans to favour some candidate causes (causal hypotheses) over others: proximal over distal events (in the causal chain of events); abnormal or unexpected events; controllable events, deviation from theoretical ideals, model, predictive power, responsibility, and so forth.

But those are taken care of by contrastive why-questions which compare the event to be explained to a reference case (particular instance or general case). In this regard, Hesslow states (bold is mine):

Many of the selection criteria listed in Section 3 can be construed as the result of choosing different objects of comparison or reference classes. Let us consider again the fire in the barn, and let us suppose that we have in the back of our minds the picture of a normal barn. (...) the normal barn has not caught fire, it follows that an explanatorily relevant condition for this barn's catching fire must be abnormal. Thus, selection of abnormal conditions can be viewed as the result of comparing the explanandum object with a normal object.

And also most other causal selections are contained:

(...) the difference between this barn now and this barn yesterday, i.e. we would be selecting a precipitating cause [proximal in the list above]. Selection of the unexpected may be viewed as the result of explaining the difference between an expected and an actual outcome. Selection according to responsibility follows from a comparison between actual and morally ideal behaviour. Selection of conditions which cause a deviation from a theoretical ideal involves a comparison between an actual and a theoretically ideal situation, and so on (cf. Hesslow, 1983).

Here is yet another illustration by Hesslow, of how contrasts cases narrow down possible causes:

For instance, if we want to explain why the fly Ml has shorter wings than Nl, then the temperature in which the flies were raised is explanatorily irrelevant, since the temperature was the same in both cases. The mutated gene on the other hand was present in one case and absent in the other.It is, therefore, explanatorily relevant.

Causal Explanations

Machine Learning Models are often doing correlation then the reasoning may use counterfactual explanations, but no causation.

However, building models where we have causal variables may be of interest and it's done in certain cases.

Explaining Explanations in AI states:

In short, contrastive theories argue that causal explanations inevitably involve appeal to a counterfactual case, be it a cause or event, which did not occur. A canonical example is provided by Lipton Lipton (1990): "To explain why P rather than Q, we must cite a causal difference between P and not-Q, consisting of a cause of P and the absence of a corresponding event in the history of not-Q".

However, it doesn't seem that all causal explanations involve counterfactuals (nor all counterfactuals need to be causal). ¹

Why-questions asking for a cause (causal explanation) can be answered via causal-counterfactual explanations. Counterfactual explanations don't use a hypothesised causal graph, so they aren't causal explanations (don't do causal attribution). In their own words:

In any case, our approach does not rely on knowledge of the causal structure of the world, (...)

Pearl has a "Ladder of Causation" for causal attribution (assigning causes):

1. Association / correlation: can't assign causes,
2. Interventions: experiments where $X$ is a cause if $P(Y|do(X_i))>P(Y)$ (other variables remain constant) i.e. $X$ increases the probability of the outcome; also observations in double blind, randomised controlled trials (RCTs),
3. Cause-finding Counterfactuals: require that we have a causal graph.

Social Process (Communication)

We have gone through the cognitive process and how contrastive questions can aid the generation and selection of a hypothesis or a cause.

The second process is that of communication. This isn't just the transfer of knowledge; in the best cases, it involves an exchange between the participants (explainer and explainee), as in a e.g. dialogue.

The communication can be aided by the gricean maxims: rules of effective communication.

• Informative (Quantity): right amount of context and details,
• Truthful (Quality, or Fidelity): the explanation should be true,
• Relevance (Relation): avoid presumed-known or superfluous details, focus on what provides insight,
  • One example given earlier is to focus on unexpected events, whilst ignoring what is presumed to be known by the listener.
• Manner (clarity): express it in elegant terms.

In some cases, humans also tend to prefer concrete over abstract explanations, so "concreteness" could be added to the list.

Relevance is primarily related to the causal selection problem in relation to an audience, as Malle et. al., state:

How do people solve this problem? They determine what exact question the audience is interested in (McClure and Hilton 1998); they take into account what their audience member already knows (Slugoski et al. 1993); and they offer elements of explanations that build bridges between presumed knowledge and novel information (Korman and Malle 2016). In short, they offer explanations that generate coherence in a knowledge structure of old and new information (Thagard 1989).

Contrastive explanations can also take care of many of these aspects automatically, by selecting a contrast that is relevant or understood by the audience.

1. As Counterfactuals Explanations Without Opening The Black Box states: "For example, S could believe that a person is inherently more trustworthy (p) because they are a Capricorn (q)." ↩