Model Explainability

The focus here is not on agents that explain themselves. Instead, the focus will be on: 1. Models that are intrinsically explainable, 2. Techniques to make the model's outputs or internals meaningful beyond the mathematical operation, 3. Use Language Models to translate model's behaviour / output to a linguistic explanation.

Explanations were defined and characterised in a previous post.

Explainable AI (XAI) is primarily about explaining the model and its output, although it may include other aspects. Model explainability can be defined as:

The degree to which we can answer questions about the model and its output. The answers are audience and context dependent. The audience, in some cases, may be ourselves.

Some systems may require that we, as operators, understand the decisions (explain to ourselves), or that the system can be explained by an expert to an audience.

As noted in the previous post, the "questions" may be implicit; and it's common that the question, implicit or explicit is a contrastive why-question.

Model explainability includes terms like interpretability and transparency (which look inside the model) but also methods to analyse black-box models. Here is one possible classification:

• Intrinsic vs Extrinsic
  • Intrinsic or Transparency: looks at the internal mechanics and roles of layers, neurons, weights; also at the complexity of the model, training process, and so forth.
  • Extrinsic or Post Hoc (aka opaque or black box): looks at input-outputs relations.
• Global (valid for all inputs) vs local (for specific inputs)

Note

Explaining and understanding can be mixed up, because the latter also involves some internal conversation or reflection. Here understanding, with or without that monologue, is equated to the cognitive process, prior to any communication. However, it certainly seems more complex.

Trade-offs

In deep learning practice, tradeoffs abound. For example, explainability tends to be harder with more accurate models, since they tend to be more complex.

Model accuracy vs Model explainability tradeoff.

Moreover, simple explanations can oversimplify its operation, or lack generality.

Explaining by Comparison

This framework is explained separately from methods to expand a bit more, but some of the methods are based on this logic behind.

Will it rain today?

Let's take an imaginary model, $y=f(u)$, the variable $u$ being the proportion of people with an umbrella, $f$ the model and $y$ the probability of rain.

It reaches low evaluation error and everyone is happy.

However, it is sometimes found that if the people don't take the umbrella it may still rain. Why? There may be different reasons:

1. The model is doing correlation / association, but there wasn't correlation data available for such an event, so the predictions bad;
   • With a large and diverse possible dataset, most questions may be answerable; but may not generalise out of distribution, restricting discoveries to certain interpolations.
   • This is useful and discoveries have been made this way, but it clearly limits them to interpolation, and low success out of distribution.
2. The model does not use causal information, like a weather forecast model would (not taking an umbrella doesn't make raining impossible).
   • Using causal models may help to overcome the problems highlighted in the previous point. It's a bit like turning it into a law or theory expected to have found some deep structure that generates the data, even the unseen data.

The model is then modified to use causal variables instead (such as pressure and temperature), implicitly turning it into a causal model. But does it have all the causal inputs?

An expert may pick known causes-effects pairs as inputs-outputs to train a model, but others may unknowingly build a correlation model instead.

A subset of causal-variables may do for a good-enough approximation, and likely be robust for generalising out of distribution. In some cases though, it may be enough to have a correlation model, but they should be distinguished.

What if...?

A model that does correlation is more likely to fail out of distribution, because it has not learnt the correlation. Causal models should in principle have less of this issue.

Counterfactuals ask What would happen if this other input (fact) was used instead of the former one, or if one feature is changed slightly? They are also similar to What ifs (as the question shows).

For a model, counterfactuals yet another prediction, but the question is helpful because that is one way humans explain things. In a similar fashion to counterfactuals, we can compare with reference inputs.

Both of these are known techniques listed in the next section.

Higher-Level Aspects of Networks

The recognition of higher level patterns in graph can also span across methods.

These can even be inspired by other networks or graphs; for example, insect colonies can be considered as graphs of insect-nodes and pheromone-edges, and certain nodes have roles and tasks they specialise on. A similar situation can be postulated to happen in human networks, and in neural (biological and artificial) networks, where the node is affected by, and also affects other nodes.

A basic description of graph and networks and how there can be transfer learning between the different areas can be found in [Siemens - Connectivism][connectivism_siemens] and particularly in [Downes - Connectivism][connectivism_downes].

Overview of methods

Within the cognitive process of explanations, model explainability benefits from methods to identify causes or relevant properties.

For all audiences, we can group these methods into more general categories, and then go into specific cases for a certain audience.

Kinds of Methods

The survey Principles and practise of explaining ML models includes a table of method kinds. A modified version of the table is below:

A method not listed there are text explanations, which can be generated from an RNN or a language model, reading the model's internal state (for example, this can generate captions).

We should remember that:

Relying on only one technique will only give us a partial picture of the whole story, possibly missing out important information. Hence, combining multiple approaches together provides for a more cautious way to explain a model. (...) At this point we would like to note that there is no established way of combining techniques (in a pipeline fashion),

In the next posts, we focus on methods that aid causal attribution (or cognitive process) with a scientific audience in mind.

Map of XAI

An interesting map of XAI is given in the survey Principles and practice of explainable ML (2021).

Most classic ML models are in the dashed area under Model types column.

Classic ML models are usually transparent (intrinsically explainable) but may benefit from post-hoc (post training) explanations, such as visualising it. When transparency is key and the predictions are accurate enough, these may be preferred over DL models.

Image from paper under CC-BY

To the visual explanations, t-SNE, PCA and other dimensionality reduction techniques can be added.

The focus here though, is explaining deep learning models. These are usually opaque ("black-box") models, and their accuracy is usually higher than classic ML models.