Notes

These are my thoughts on topics related to AI, XAI, AI for chemistry.

Organisation

• The top-left hamburger icon toggles the table of contents.

• At the bottom of some posts there is a "Sources" which includes papers I read during the write up.

License

The prose is all under CC BY 4.0, external content is linked and if I find the license it is explicitly stated (in image captions, for example).

Tools Used

• mdbook,
• Krita to edit images,
• Computer Modern font or Ubuntu > System UI > sans-serif, in that order of preference.

Explanation

This post describes what explanations are in the context of artificial intelligence.

Definitions

What is an explanation? Here is definition from "How People Explain Action (and Autonomous Intelligent Systems Should Too)" (2017):

Explanation is arguably a three-value predicate: someone, a communicator, explains something to someone, an audience. The success of an explanation therefore depends on several critical audience factors—assumptions, knowledge, and interests that an audience has when decoding the explanation.

Explaining Explanations in AI also defines of explanations and explanation in AI:

(...) "explanation" refers to numerous ways of exchanging information about a phenomenon, in this case the functionality of a model or the rationale and criteria for a decision, to different stakeholders (Lipton, 2016; Miller, 2017).

Inspired by Explanation in artificial intelligence: insights from the social sciences, this post defines explanation as:

Explanation

A two-step process involving 1. the generation of explanatory hypotheses (cognitive process) and 2. the communication to an audience (social process) including ourselves. The steps may repeat indefinitely.

Explaining can expose a lack (or illusion) of understanding; it can also show and improve understanding. Eventually, one hypothesis may be selected until it is either:

• Contradicted by experience (falsified),
• Superseded by a simpler or more general one,
• Shown to be inconsistent with prior knowledge.

So the cognitive process involves the generation of hypotheses, theories and reasons, which in turn reflect our understanding about how something works.

In the paper "From scientific theory to duality of predictive artificial intelligence models" it is stated, of scientific understanding of a phenomenon (references removed):

(...) if there is an underlying intelligible theory such that scientists can qualitatively comprehend the consequences of this theory without the need for quantitative analysis and exact calculations.

Contrastive Questions (and Explanations)

Miller et al., define contrastive questions as:

Perhaps the most important result from this work is that explanations are contrastive; or more accurately, why–questions are contrastive. That is, why–questions are of the form "Why P rather than Q?", where P is the fact that requires explanation, and Q is some foil case that was expected.

They argue that all why-questions are contrastive, since even those without explicit contrast such as "Why P?" still have an implicit one.

If we can find a useful foil, then answering the question is easier, as we only need to focus on the differences between these two cases to generate a contrastive explanation.

Counterfactuals

Notice how "What would have happened if ..." is exactly a counterfactual, and "Why P rather than Q?" involves an alternative fact: counterfactuals are useful for providing causal explanations. However, not all causal explanations require counterfactuals (this is rung 3 of Judea Pearl's Ladder of Causation)

For scientific explanation, this is more rigorous and there are also other mechanisms (e.g. Pearl's Ladder of Causation).

If we have a mathematical model, we attribute as "cause" of an output, a certain input or feature value; but in that case, it's not the cause for the real world event but for the model's output. Whether it accurately models the real world or not, is a different question.

On choosing relevant counterfactuals (or foils), the paper "Counterfactual explanations without opening the black box (...)" requires the counterfactual to be a close, possible world:

We define Counterfactual Explanations as statements taking the form: Score p was returned because variables V had values (v1, v2,...) associated with them. If V instead had values (v1', v2',...), and all other variables had remained constant, score p' would have been returned.

While many such explanations are possible, an ideal counterfactual explanation would alter values as little as possible and represent a closest world under which score p' is returned instead of p. The notion of a "closest possible world" is thus implicit in our definition.

Another kind of counterfactual seeks for the output to remain the same rather than to change (for fairness purposes), as they state:

Kusner et al.58 consider counterfactuals where the subject belongs to a different race or sex, and require that the decision made remain the same under such a counterfactual for it to be considered fair. In contrast, we consider counterfactuals in which the decision differs from its current state.

Often, multiple counterfactual explanations may be preferred, so that the subject can choose the optimal one.

How do we give an explanatory hypothesis based on counterfactuals? If the cause "X" leads to both P and Q, then it can't explain why only one occurred. So we want a case where P needs "X" to have happened, while "not X" is required for Q.

The paper Beware of Inmates Running the Asylum has an interesting example:

(...) "Why did Mr. Jones open the window?" with the response "Because he was hot" is not useful if the implied foil is Mr. Jones turning on the air conditioner, as this explains both the fact and the foil; or if the implied foil was why Ms. Smith, who was sitting closer to the window, did not open it instead, as the cited cause does not refer to a cause of Ms. Smith's lack of action.

Attribution Theory

Attribution theory is about how people assign causes to events. Assigning causes is Miller et. al. "necessary to provide explanations".

They distinguish social from general causal attribution. For the general case, they state (references removed):

Research on how people connect causes shows that they do so by undertaking a mental simulation of what would have happened had some other event turned out differently.

It's similar for social attribution, but uses a belief, desires, intents and traits (of people) as causes.

Importantly, when there are many variables, the combinatoric is large, and this would be unfeasible, so people must select a few counterfactuals to simulate, giving rise to the causal selection problem. They add:

However, simulating an entire causal chain is infeasible in most cases, so cognitive scientists and social psychologists have studied how people decide which events to 'undo' (the counterfactuals) to determine cause. For example, people tend to undo more proximal causes over more distal causes [Miller and Gunasegaram, 1990], abnormal events over normal events [Kahneman and Tversky, 1982], and events that are considered more 'controllable' [Girotto et al., 1991].

More on Contrastive Questions (and Explanations)

Hesslow states:

What I want to suggest, then, is that the explanandum should be construed as a relation which involves three things: an object a, an object of comparison b and an explanandum property E which a has and b does not have.

The complexity, of course, lies on formulating a question that makes the important difference with a reference case obvious.

In many cases, the complexity is finding a good, relevant foil. As Explaining Explanations in AI states:

However, choosing a relevant set of cases or events against which contrastive explanations are provided is not a straightforward challenge. The way in which information is transferred has a substantial impact on the quality and psychological acceptability of explanations (Hilton, 1990).

The last sentence is related to the social process of explaining, which we describe in this post (relevance from Gricean Maxims).

Researchers have pointed out many heuristics used by humans to favour some candidate causes (causal hypotheses) over others: proximal over distal events (in the causal chain of events); abnormal or unexpected events; controllable events, deviation from theoretical ideals, model, predictive power, responsibility, and so forth.

But those are taken care of by contrastive why-questions which compare the event to be explained to a reference case (particular instance or general case). In this regard, Hesslow states (bold is mine):

Many of the selection criteria listed in Section 3 can be construed as the result of choosing different objects of comparison or reference classes. Let us consider again the fire in the barn, and let us suppose that we have in the back of our minds the picture of a normal barn. (...) the normal barn has not caught fire, it follows that an explanatorily relevant condition for this barn's catching fire must be abnormal. Thus, selection of abnormal conditions can be viewed as the result of comparing the explanandum object with a normal object.

And also most other causal selections are contained:

(...) the difference between this barn now and this barn yesterday, i.e. we would be selecting a precipitating cause [proximal in the list above]. Selection of the unexpected may be viewed as the result of explaining the difference between an expected and an actual outcome. Selection according to responsibility follows from a comparison between actual and morally ideal behaviour. Selection of conditions which cause a deviation from a theoretical ideal involves a comparison between an actual and a theoretically ideal situation, and so on (cf. Hesslow, 1983).

Here is yet another illustration by Hesslow, of how contrasts cases narrow down possible causes:

For instance, if we want to explain why the fly Ml has shorter wings than Nl, then the temperature in which the flies were raised is explanatorily irrelevant, since the temperature was the same in both cases. The mutated gene on the other hand was present in one case and absent in the other.It is, therefore, explanatorily relevant.

Causal Explanations

Machine Learning Models are often doing correlation then the reasoning may use counterfactual explanations, but no causation.

However, building models where we have causal variables may be of interest and it's done in certain cases.

Explaining Explanations in AI states:

In short, contrastive theories argue that causal explanations inevitably involve appeal to a counterfactual case, be it a cause or event, which did not occur. A canonical example is provided by Lipton Lipton (1990): "To explain why P rather than Q, we must cite a causal difference between P and not-Q, consisting of a cause of P and the absence of a corresponding event in the history of not-Q".

However, it doesn't seem that all causal explanations involve counterfactuals (nor all counterfactuals need to be causal). ¹

Why-questions asking for a cause (causal explanation) can be answered via causal-counterfactual explanations. Counterfactual explanations don't use a hypothesised causal graph, so they aren't causal explanations (don't do causal attribution). In their own words:

In any case, our approach does not rely on knowledge of the causal structure of the world, (...)

Pearl has a "Ladder of Causation" for causal attribution (assigning causes):

1. Association / correlation: can't assign causes,
2. Interventions: experiments where $X$ is a cause if $P(Y|do(X_i))>P(Y)$ (other variables remain constant) i.e. $X$ increases the probability of the outcome; also observations in double blind, randomised controlled trials (RCTs),
3. Cause-finding Counterfactuals: require that we have a causal graph.

Social Process (Communication)

We have gone through the cognitive process and how contrastive questions can aid the generation and selection of a hypothesis or a cause.

The second process is that of communication. This isn't just the transfer of knowledge; in the best cases, it involves an exchange between the participants (explainer and explainee), as in a e.g. dialogue.

The communication can be aided by the gricean maxims: rules of effective communication.

• Informative (Quantity): right amount of context and details,
• Truthful (Quality, or Fidelity): the explanation should be true,
• Relevance (Relation): avoid presumed-known or superfluous details, focus on what provides insight,
  • One example given earlier is to focus on unexpected events, whilst ignoring what is presumed to be known by the listener.
• Manner (clarity): express it in elegant terms.

In some cases, humans also tend to prefer concrete over abstract explanations, so "concreteness" could be added to the list.

Relevance is primarily related to the causal selection problem in relation to an audience, as Malle et. al., state:

How do people solve this problem? They determine what exact question the audience is interested in (McClure and Hilton 1998); they take into account what their audience member already knows (Slugoski et al. 1993); and they offer elements of explanations that build bridges between presumed knowledge and novel information (Korman and Malle 2016). In short, they offer explanations that generate coherence in a knowledge structure of old and new information (Thagard 1989).

Contrastive explanations can also take care of many of these aspects automatically, by selecting a contrast that is relevant or understood by the audience.

1. As Counterfactuals Explanations Without Opening The Black Box states: "For example, S could believe that a person is inherently more trustworthy (p) because they are a Capricorn (q)." ↩

Explanatory Stances

Humans often use "explanatory stances" to explain events, as noted by Daniel Dennett. Related ideas are explored in this post.

Explaining via Metaphors

Here are three common metaphors humans use to explain things:

1. The mechanical stance (or the Machine Metaphor) explains behaviour how the parts of a system interact and what they do.
2. The design stance, this has different interpretations. One is of the perceived purpose of something (applied to things created by humans such as tools, but also those hypothesised to be created by universal designer or god).
3. The intentional stance (which I call "Agent Metaphor/Model") uses goals, motives, feelings, intent to explain actions and/or behaviour. Unintentional behaviour/events is usually explained using the machine metaphor (see the following paper, section [Ordinary Behavior Explanation][autonomous_intelligent_systems]).

They can be complementary when applied to the same phenomena or as Ruth Byrne puts it:

Notably, each explanatory stance can be applied to explain the same device or action, but they have different consequences for understanding it. Each stance can lead to different kinds of insights, and to different kinds of erroneous inferences. The atypical application of a particular stance, say, a mechanical stance to explain an action more typically understood from an intentional stance, such as explaining travelers in a crowded airport as like pinballs careening around a pinball machine, may be interpreted analogically to yield new inferences [Keil, 2006].

In technical fields, many complex systems are conceptualised as machines: composed of parts, each with a function, a role. Many are also conceptualised as graphs.

In everyday life we often conceptualise complex systems as humans or agents (wholly or in part). This may happen with systems using human language or behaving autonomously, but other times it is due to pragmatic reasons. They would use and expect the kind of explanation a human would give, if there were one.

What seems here most fundamental than the particular stances is the selection of a metaphor to structure thinking and obtaining insights.

Other metaphors and analogies could be proposed for specific problems.

A similar idea is expressed in "[How People Explain Action (and Autonomous Intelligent Systems Should Too)][autonomous_intelligent_systems]":

For those intentional agents, we hypothesize, people will apply the same conceptual framework of behavior explanation that they apply to humans (...) a subset of AIS that people do not regard as intentional agents; and for those, they may apply a purely mechanical explanatory framework.

And more recently, in Good Explanations in XAI:

People may tend to adopt multiple stances in their preferred explanations of an AI decision support system and its decisions, not unlike their tendencies in interacting with social robots [Clark and Fischer, 2023]. People are aware that a social robot is a machine, but interpret it as a depiction of a character, not unlike a ventriloquist dummy, and engage with it in pretense of interacting with the depicted character [Clark and Fischer, 2023]. Similarly, they may be aware that an AI decision support system is an algorithm but they may interpret its decisions as a depiction of those provided by a human, e.g., a bank loan assessor, or the organization the human represents, a bank. Hence, an intentional stance and a design stance may both be useful in different contexts for explaining how automated agents behave [Veit and Browning, 2023].

The summary is:

1. Mechanical stance (which I call "Machine Metaphor/Model"),
   • Explain outcomes by considering the parts of a system and how they interact (that is, a mechanism).
2. Design stance,
3. Intentional stance (which I call "Agent Metaphor/Model").
   • Explanation uses purposes (teleology), goals, motives, feelings, intent to explain actions and/or behaviour. It assumes a "state of mind" of the object and uses the model to predict behaviour.
   • Unintentional behaviour is usually explained using the machine metaphor.

The selection of a stance is a relevant aspect to consider when generating explanations. An example is given by byrne_human_explanations:

A potential implication for XAI is the necessity to consider when a particular stance is appropriate for explaining an AI system's decision. A question posed by a user as to why an AI system refused their loan may be a request for causal information of a mechanistic sort about how the AI system came to make that decision. Hence, it may require an explanation based on information about how the user's input features relate to a training data set, e.g., that loans have been refused for applicants of similar salary level, occupational status, credit history, as the user. Alternatively, it may be a request for functional information of a teleological sort about the purpose of the AI's decision. Accordingly, it may require instead information about the goal and consequences of the output, e.g., that decisions of this sort mitigate the risk of applicants defaulting from repayment. Each sort of explanation will lead users to develop a different understanding of aspects of an AI system, and impact their learning, satisfaction, and trust in it.

Next, a few of these stances are explored in different contexts.

Human To Human Explanations

When humans explain phenomena to each other they use language, pictures and other media. But we don't get access to each others' brain processes (to a large extent, not even our own); and even if we did, the task wouldn't be trivial.

A brain may use "communication processes" that look at "decision processes" (see Marvin Minsky, The Emotion Machine). Then the statement is exchanged and interpreted by another agent. The statement can be tested against reality, or for inconsistencies or compared to other statements.

In other words, we largely treat brains as black boxes.

Should we treat DL models the same, and accept predictions without "peeking into the black box", just as we do with humans? If so, under which circumstances?

The idea of this post is that there will be a level of complexity and utility at which this will also happen for DL models.

DL Models are Scientific Models

DL models are mathematical functions. Both the architecture and the trained model can be considered scientific models or theories. In this case, they should be explainable. Here there is an underlying metaphor of "DL models as machines" and we expect them to be explainable in terms of their parts and how they interact.

There are also societal reasons for this expectation: many models are used on decision making and high-stakes scenarios, so we expect explanations or justifications for their decisions, just as we expect from humans making decisions.

For simple or narrow-AI models, we may expect to peek into the black box, and to analyze their input-output relations to get insight into their inner workings.

They're expected to be decomposable in domain-level concepts (just as a theory is).

In other words, we want to answer: What did it learn? Which concepts is the network using for their decision making? But how can we extract their "baked in" knowledge?

But, what if the metaphor used for a complex DL model is that of a scientist, or a human, rather than a machine?

DL Models are Scientists

General-purpose models may explain decisions in a human-like way, and be as hard to inspect as the brain is. Treating these models as a scientific theory of a domain, or a narrow AI model, may be a counter-productive, and even a category error.

In this case, a detailed explanation process could be as hard to extract as it is from a brain.

So what do we do? In this case, it may be better to expect what we expect from a human: Take an explanation in natural language and test it and compare it to other established statements.

This is what the paper [The Mythos of Model Interpretability][mythos] describes, in the section about post-hoc textual explanations:

Humans often justify decisions verbally. Similarly, one model might be trained to generate predictions, and a separate model, such as a recurrent neural network language model, to generate an explanation.

In other words, we would treat it as a black box.

Summary

We have explored two metaphors "DL models as Agents / Scientists" and "DL Models as Scientific Models", and the expectations for each case:

1. As Scientific Models: We can study them as black boxes, and also internal mechanisms,
2. As Scientists: We can only study them as black boxes,
   1. This includes Agents that can output a certain explanation (e.g. humans).

As science advances, some models may become explainable. Similarly, intrinsically explainable architectures may be designed, and could be preferred over black box architectures (e.g. Capsule Networks were an effort in this direction).

Now we go back to common methods used to explain mostly narrow AI algorithms.

Explainable AI (XAI)

This post is an overview of the field of Explainable AI (XAI).

Why all of this?

The previous post described aspects of explanations in general.

But there are two questions that seem worth asking before moving forward:

1. Why do we need this emphasis on theories of explanations for AI, but not for Physics, Chemistry, Maths? After all, all these use formulas, models and scientific models?
2. What can theories of explanations offer, such that "(...) models of explainable AI can benefit from models of human explanation" (quote from Explainable AI: Beware of Inmates Running the Asylum (...))?

To the first question, my current answer is that XAI started because models are used in ways that affect non-experts (e.g. financial decisions such as a loan decision). Then explanations are need of such models for a general, non-expert audience to understand and challenge those decisions.

Eventually, deep learning models became more popular and complex, so the question became how can we extract insights, contextualise, or relate the internals of the models to science and other areas?

In the context of science, it seems that works needs to be done to distinguish scientific from other kinds of explanation. A starting point was recently published by Bajorath, describing a model duality (as within or outside a scientific framing).

To the second question, the previous post highlighted certain techniques which have been found useful in everyday explanation such as contrastive explanation (the contrastive nature of why-questions). Some of these are applicable or were considered for models that we will describe (here and in the next posts).

Scope of the Field of XAI

Explainable AI (XAI) aims to explain artificial intelligence models; that is, it aims to answer questions about the models' design, inner working and output, for a given audience.

An audience has certain level of expertise, information needs such as the accuracy and limitations of a model, and goals such as answering certain kinds of questions, and so forth. These aspects are part of the social process. For more detail see the post on explanations.

Broadly speaking, we find two sub-fields within XAI:

Post Training Explainability can be subdivided in:

• Of the models' input-output relations (usually termed post-hoc explainability),
• Of the models' internal representations and state, such as layers, activations, and weights (usually post training, called mechanistic interpretability).

Intrinsic Explainability: usually by constraining or conceptually grounding the model, e.g. a physical constraint that makes it more interpretable (independently of whether it's been trained or not). - An interesting analogue here are scientific models, and models that are conceptually grounded (we just learn the parameters e.g. linear regression, decision trees). - Also of interest is to explore causality and neurosymbolic computing.

Since this is a custom classification (though close to others), quotes from papers will make it clear which aspects are we mapping to it.

Explanations can also be Local or Global, referring to a single input-output (e.g. linear LIME) or to the model as a whole (e.g. combining local explanations as SP-LIME does). This post discusses the two sub-fields above, clarifying about local/global if needed.

Note

The term "black box" is commonly used to refer to complex deep learning models. It can also refer to a model unavailable for inspection (such as encrypted or proprietary models). In both cases, we can try to explain the outputs, but the internals can only be explained if the model is available.

This blog-post and most papers use "black box" in that sense: models which are available but are hard to comprehend.

A final note on terminology:

Note

In this blogpost, explainability and interpretability are considered synonyms.

Intrinsic Explainability

Intrinsic Explainability (or interpretability) refers to designing models that are constrained in form (e.g., Rudin C.) and also explaining the internals of models.

What design makes a model interpretable is domain-dependent. For example, geometric deep learning can express constraints of connectivity related to molecules and materials (e.g. GNNs);

Other strategies are: imposing physical or connectivity constraints (e.g. GNNs for molecules and materials), inductive biases (such as symmetry), CNNs for processing images (somewhat interpretable to experts), using expert-selected causal features, imposing monotonicity, sparsity or constraining model size or computational complexity.

The paper "The Mythos of Model Interpretability" suggests three components of transparency. Very briefly:

1. Simulatability i.e. can mentally run the model,
2. Decomposability i.e. each part of the model admits an intuitive explanation,
3. Algorithmic training which focuses on global vs local minimum, error and loss, guaranteed convergence;

These are important aspects that have to be considered for models, and the emphasis on each will depend on the case.

The paper states:

Sufficiently high-dimensional [linear] models, unwieldy rule lists, and deep decision trees could all be considered less transparent than comparatively compact neural networks.

Rudin's paper adds:

As discussed above, interpretability usually translates in practice to a set of application-specific constraints on the model. Solving constrained problems is generally harder than solving unconstrained problems. Domain expertise is needed to construct the definition of interpretability for the domain, and the features for machine learning. For data that are unconfounded, complete, and clean, it is much easier to use a black box machine learning method than to troubleshoot and solve computationally hard problems.

But what are examples of some algorithms for some problems? Or maybe common classes of such models?

Rudin does mention a few pointers:

1. The human-interpretable models we had before machine learning (and some of these can be machine-learnt). Some examples from Rudin's paper, though it will be extended, are:

   • Logical Models¹: use logical conditions such as "or, and, if-then" but are machine-learnt in a reasonable amount of time. The model may be large (non-simulatable) but the explaining condition may be brief and interpretable. Rudin calls these "smaller-than-global explanations" because the explanation is just a few rules of the global model which may have many (Appendix D). Examples: disjunctive normal form models and falling rule lists and CORELS.
   • Sparse Linear Models. If it uses integer coefficients is a scoring system.
   • Case-Based: they defined interpretability for a specific domain, in this case computer vision, in terms of how humans explain them (by pointing at features in regions of the image). This definition led them to create a prototype network ("prototype" in the Eleanor Rosch sense, a characteristic part of an image), and the network seems to crop the image in multiple ways (maybe just outputs prototypical regions bounding boxes) and compare these to training prototypes for each class. The interesting thing is that this output is the explanation itself. Finally it does a weighted average.

2. Adequately crafted neural networks.

   or linear fits or case-based reasoning.

It's also possible to imbue interpretable aspects into deep learning models (could be interesting to expand on this aspect, maybe through mechanistic interpretability papers). At the same time, it's important to know when a traditional ML algorithm is a better fit (for interpretability, performance, accuracy).

I would also add that, in scientific disciplines, we may want causal and interpretable models, rather than just interpretable.

Post Training Explainability

Usually this is includes extrinsic and post hoc explainability and involves explaining input-output relations, prediction(s), role of layers and neurons.., rather than elucidating precisely how a model works.

"Why Should I Trust You?" defines "explaining a prediction" as:

By "explaining a prediction", we mean presenting textual or visual artifacts that provide qualitative understanding of the relationship between the instance's components (e.g. words in text, patches in an image) and the model's prediction.

This is a visual explanation of the features' contribution to the output from the same paper, where it is used to compare models:

Image taken from paper.

This simpler, and user-friendly presentation (when correct) has some uses. For example, to spot reliance on features we wouldn't want it to use for the prediction.

If those features were missing, it would fail (it does not generalise). Related concepts are shortcut learning, Clever Hans², Right for the Wrong Reasons.

This is one reason why accuracy on a test set alone isn't a bulletproof metric, and why we need XAI. Some solutions, then, are:

• Analysis of data in database,
• Test in real world settings,
• Use intrinsically interpretable and causal models instead,
• Use XAI methods and ensure that the features responsible for the output are conceptually sound.

The "Why Should I Trust You?" paper also proposes an explainer desiderata for the explanation model: it should be 1. interpretable, by giving a qualitative understanding between inputs and outputs, making it easy to understand, 2. model agnostic and 3. locally faithful (a good fit to the original model in the vicinity of the instance being explained) and 4. globally explainable. In that paper, SP-LIME combines local explanations to provide a global explanation of the model.

Issues with Post-Training XAI Methods

XAI methods are not without problems.

One of the problems is: How can we know that the explanation describes what the model does accurately? Explanation models are sometimes misleading or wrong. It's also relevant to know their limitations and assumptions, such as which area are they supposed to be accurate on.

One interesting perspective is given by J. Bajorath, where a model needs to be explainable, then interpreted (into human, domain-specific concepts) and finally humans do causal inference (using a scientific framework/paradigm) which can be tested. In other words, by requiring that the black box prediction can be transformed into a sound causal hypothesis.

In general, they are often models (of models) and suffer problems deriving from this fact (explored by the analogy to scientific models on "Explaining Explanations in AI").

However, in the case of local attribution methods, these are simpler, and usually are not themselves black boxes.

Rudin argues that these simpler explanation models must be wrong. If it is globally accurate, then we don't need the original model. Rudin proposes calling these model-approximation techniques "summary of predictions", "summary statistics" or "trends".

Within other problems the paper states:

Even an explanation model that performs almost identically to a black box model might use completely different features, and is thus not faithful to the computation of the black box. (...)

An explanation model depending on race could construe "This person is predicted to be arrested because they are black." (as the paper states) even if the original model did not depend on this feature directly (it could by proxy features). This would mislead a user very badly.

Within these limitations, methods such as SHAP, LIME, Counterfactuals (even though these don't use an extra model), can provide some understanding of what the model is doing (though as stated above, they can also be misleading). Explaining Explanations in AI makes an analogy between local approximation and scientific models, they key part being that both may approximate a more detailed model within a narrow domain (in which they are valid).

A an issue with this analogy was noted by Rudin:

Note that the term "explanation" here refers to an understanding of how a model works, as opposed to an explanation of how the world works. The terminology "explanation" will be discussed later; it is misleading.

In Appendix C, Rudin argues that choosing from one of many possible counterfactual explanations (which apparently are also called inverse classification!) is often undecidable:

Some have argued that counterfactual explanations [e.g., see 37] are a way for black boxes to provide useful information while preserving secrecy of the global model. Counterfactual explanations, also called inverse classification, state a change in features that is sufficient (but not necessary) for the prediction to switch to another class (e.g., "If you reduced your debt by $5000andincreasedyoursavingsby$50% then you would have qualified for the loan you applied for"). This is important for recourse in certain types of decisions, meaning that the user could take an ac"ion to reverse a decision [61].

In other words, let us say that there is more than one counterfactual explanation available (e.g., the first explanation is "If you reduced your debt by $5000andincreasedyoursavingsby$50% then you would have qualified for the loan you applied for" and the second explanation is "If you had gotten a job that pays $500 more per week, then you would have qualified for the loan"). In that case, the explanation shown to the user should be the easiest one for the user to actually accomplish. However, it is unclear in advance which explanation would be easier for the user to accomplish. In the credit example, perhaps it is easier for the user to save money rather than get a job or vice versa.

Post Hoc Explanation Classes

In general, all methods are like a toolkit to produce explanations.

Many papers describe a few types of post training explainability methods and classes (such as 1, 1, 2 and 3 ).

Below a custom set of classes is given usually named after a distinct XAI concept they are based off.

• Textual e.g. using RNNs or a language model to translate the network state into text (trained with descriptions, similar to captioning images),
• Similarity (or Case-Based): Representative items for each class provide insights about the model's internal reasoning. Can be automated with distance KNNs but specific examples may require human selection.
  • They do not explicitly state what parts of the example influence the model. How do inputs from different classes compare? And same?
• Visualizations: of learned representations. Altering certain input features to maximise activation of a particular neuron (then looking back at the modified image to see what this neuron is responding to). Easier to communicate to non-technical audiences. Most approaches are intuitive and not hard to implement.
  • There is an upper bound on how many features can be considered at once. Humans must inspect plots to derive explanations. Class boundaries?
• Contrastive and Counterfactual Methods look for similar cases that lead to a different (or the same) decision, or that help clarifying the decision in some way. They are a separate category because they may use the original model, and the key characteristic is the type of question they ask. But, as Explaining Explanations in AI says:

  (...) choosing a relevant set of cases or events against which contrastive explanations are provided is not a straightforward challenge.

• Approximations: Simple, surrogate, interpretable models explain opaque ones, locally and/or globally. They also allow us to ask contrastive or what-if questions (see previous method) such as "What if we change X by X'?" Or "How could I get Y' rather than Y?" However, these approximation models are usually valid over a narrow domain, making many of these questions unfeasible.
  • Explaining Explanations in AI states:

  These models can be understood as a "do it yourself kit" for explanations, allowing a practitioner to directly answer "what if questions" or generate contrastive explanations without external assistance.

  • Feature relevance can be considered a linear approximation model. They operate locally or globally (e.g. LIME and SP-LIME). These Methods may make assumptions which do not hold (e.g. feature independence, linearity).
  • As local and approximations they often have a domain of applicability.

An interesting approach could be to use and compare the results of several different these methods, which often appears as a suggestion. For example, in this paper:

For all practical purposes, it is advisable to apply at least two distinct explanation approaches and compare the results for consistency.

Causal Explanations from ML models

An audience of scientists may be interested in generating causal, testable hypotheses from ML models (via intervention in Pearl's language), as "From scientific theory to duality of predictive artificial intelligence models" states (refs removed):

If an ML model is derived to predict the results of a natural process, then model interpretation might lead to the formulation of testable hypotheses of causal relationships. A causal relationship exists if interpretable features determining the prediction of a test instance are directly responsible for the outcome of the underlying natural process. Accordingly, causal ML involves the incorporation of knowledge or inference of causal relationships.

Causal inference (or attribution) is a hypothesis about the responsibility of variables to either change or produce the output.

1. related names are: rule lists, expert systems, decision trees, disjunctive normal form models, associative classifiers. ↩

2. The description above is sometimes called the "Clever Hans" effect for AI. This derives from a horse ("Hans") believed to know arithmetic, but instead it was guessing through queues in the owner's face and behaviour. ↩

Topics in XAI

Let's now discuss three key topics which stand on their own:

• Local and Counterfatctual explanations which are popular post-hoc explainability classes,
• Modelling trade-offs,
• Generalisation out of distribution.

Finally, an interesting map of XAI extracted from one paper is shown.

Local Explanations

These models can be understood as a "do it yourself kit" for explanations, allowing a practitioner to directly answer "what if questions" or generate contrastive explanations without external assistance.

Linear, gradient-based and decision trees are used to explain particular predictions, as local explanation models. They can be extended to be global (as we will discuss).

Local explanation models can be defined as simpler and interpretable models used to approximate and explain particular predictions of the original model.

Explaining Explanations in AI adds:

Explainable AI generates approximate simple models and calls them 'explanations', suggesting reliable knowledge of how a complex model functions.

To my interpretation, the paper also argues that this isn't an explanation in itself. Rather, they are models we can use to generate local explanations.

It must still be complemented by the areas where the model is accurate, breaks down, or its fit to the prediction model is unknown. And where the explanation model is accurate, they say:

Over the domain for which the model accurately maps onto the phenomena we are interested in, it can be used to answer 'what if' questions, for example "What would the outcome be if the data looked like this instead?" and to search for contrastive explanations, for example "How could I alter the data to get outcome X?"

Connecting this post with the one on explanations, particularly contrastive ones.

These models operates on instance-level explanations. Explanations do not generalize on a global scale, unless something else is done (e.g. SP-LIME, explained in this post).

Hence, contrastive explanations may only be possible for neighbouring points, where the local explanation models are accurate, but restricting its usefulness. They also fail to model highly non-linear models (property of curvature) and interdependency between variables.

Besides models like SHAP and LIME which fit a simpler model, local explanations can be provided by instance gradients (saliency maps,$\frac{y_j}{/}\mathbf{x}$), fitting simpler local models (e.g. Linear LIME). In this case small perturbations might result in very different explanations.

Trade-offs?

We may expect model explainability to be inversely correlated with model complexity or accuracy. Graphically:

Hypothesis: Model explainability v. Accuracy tradeoff.

And in "Why Should I Trust You?" (refs removed):

Recognizing the utility of explanations in assessing trust, many have proposed using interpretable models, especially for the medical domain. While such models may be appropriate for some domains, they may not apply equally well to others (...). Interpretability, in these cases, comes at the cost of flexibility, accuracy, or efficiency.

And in [SHAP][shap]:

However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability.

And in Explaining Explanations in AI:

(...) three-way trade-off between the simplicity of the approximated model, the size of the domain it describes, and the accuracy of this description.

And also:

A trade-off inherently occurs between the insightfulness of the approximated model, the simplicity of the presented function, and the size of the domain to which is applies and remains valid (Bastani et al., 2017; Lakkaraju et al., 2017).

Other researchers such as [Rudin][interpretable_ml] disagree (references were removed, and bold is mine):

Two obstacles to using interpretable models are that they are harder to optimize because they require extra constraints, and there is an incorrect perception that they are less accurate than black boxes. On the first point, the community is getting quite good at building interpretable sparse models and interpretable neural networks. On the second point, there is no scientific evidence that accuracy must be sacrificed when adding interpretability constraints.

Rudin's more detailed paper restates the first point:

There is a widespread belief that more complex models are more accurate, meaning that a complicated black box is necessary for top predictive performance. However, this is often not true, particularly when the data are structured, with a good representation in terms of naturally meaningful features.

And the second point:

The researcher needs to create a model that has the capability of uncovering the types of patterns that the user would find interpretable, but also the model needs to be flexible enough to fit the data accurately. This, and the optimization challenges discussed above, are where the difficulty lies with constructing interpretable models.

So there is the:

• Problem of optimisation (under constraints) and
• The problem of designing such transparent models (including neural networks), which require expertise, while black boxes may not,
  • We should add that black boxes have plenty of issues with accountability, reliability, accuracy, and value of the explanations.
• There is also the problem of input representation, that is of finding or creating a good representation in terms of naturally meaningful features.

But in terms of the trade-off there is no clear scientific evidence that interpretable models are less accurate, though they can be harder to design and optimise.

Within the class of NNs though, they do tend to perform better as we scale them up until eventually plateau or decrease its performance. But there doesn't seem to be any cross-algorithm evidence or formal argument of the complexity-accuracy tradeoff.

Rudin also considers the Rashomon set argument: if there are several models with similar high-accuracy, then there may also be some within the set that are interpretable.

An interesting, related question is: is the three-way trade-off of domain of applicability, ease of understanding and accuracy also valid in general, for any model? This may still support Rashomon sets, but the ease of understanding may be quite low (requiring very high expertise). But here we can ask the question more generally about models.

Counterfactual Explanations in AI

"[Counterfactual explanations without opening the black box][without_opening_bbox]" defines these explanations as the minimum change in the input (in some distance metric) to change the output (or keep it the same depending on the goal).

In other words, counterfactual explanations are contrastive in the sense that they compare two situations, and non-causal, unless the use a causal model.

Explaining Explanations in AI argues for the use contrastive explanations for the original model, which has less limitations than linear, local explanations¹:

Rather than explicitly generating a model that approximates functional values over a restrictive domain, and relying on the user to interpret this, contrastive explanations directly offer an alternative data point: "If your data had looked like this, you would have been given this classification score instead." These alternative data points can be computed exactly. As such, many of the challenges facing 'modelling' approaches to generating explanations, such as the quality of the approximation or the limits of a chosen domain, do not arise to a comparable degree.

And the selection of the alternative data point is of high importance (must be relevant, and similar enough to background other causes). This is achieved by a particular Lagrange-style constrained optimisation which helps select that counterfactual (one that is both close to the data point of interest and to a certain desired output value).

As an example, they use:

$${\mathrm{argmin}}_{x^{\prime }}{\mathrm{argmax}}_{\lambda }\lambda (f(x^{\prime })-y{)}^2+d(x,x^{\prime })$$

with an $L_1$ norm (absolute distance between the given $x$ a close value to find, which is $x^{\prime }$). The important part here is that the $L_1$ norm can usually find a resulting vector that contains several zeros ($x^{\prime }=x$ for many features), this makes counterfactuals easier to explain (less differences).

• There may also be local minima, which can all be provided as a cluster of explanations to look at,
• $x^{\prime }$ must also be a "possible world" or possible point,
• This optimisation is harder if variables are discrete (assumed continuous here).

Their other paper [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR][without_opening_bbox] states why and for whom these explanations may be most useful:

Principally, counterfactuals bypass the substantial challenge of explaining the internal workings of complex machine learning systems.70 Even if technically feasible, such explanations may be of little practical value to data subjects. In contrast, counterfactuals provide information to the data subject that is both easily digestible and practically useful for understanding the reasons for a decision, challenging them, and altering future behaviour for a better result.

Counterfactual explanations explain automated decisions (partly or fully), don't require opening the black box (explaining inner workings) and are easier to generate and to understand as well than the internals of the model. Not disclosing the model protects companies' trade secret and (as they state) "the privacy of individuals whose data is contained in the training dataset (...) Assuming reasonable limitations are set on the number of counterfactuals that must be provided, counterfactuals are also less likely to provide information that reveals trade secrets or allows gaming of decision-making systems."

Advantage of counterfactual over simpler model

Note also that local fitting of a prediction model may be faithful, but both models could be inaccurate. It's also not enough to have an interpretable model, as they continue:

It is not enough to simply offer a human interpretable model as an explanation. For an individual to be able to trust such a model as an approximation, they must know over which domain a model is reliable and accurate, where it breaks down, and where its behaviour is uncertain. If the recipient of a local approximation does not understand its limitations, at best it is not comprehensible, and at worst misleading

That is, the domain where the models (both) are accurate, inaccurate or unknown should be characterised, and understood by the recipients.

• Couldn't we train local explainable models from scratch, and throw away the complex one? Usually no, explanation models are trained with predictions of the complex model, that may not exist in the training data.

When are counterfactual explanations insufficient?

The authors note that their counterfactual explanations don't require causal graphs or causal models, which in some cases can be needed.

Also, as they state:

As a minimal form of explanation, counterfactuals are not appropriate in all scenarios. In particular, where it is important to understand system functionality, or the rationale of an automated decision, counterfactuals may be insufficient in themselves. Further, counterfactuals do not provide the statistical evidence needed to assess algorithms for fairness or racial bias. Given these limitations, more general forms of explanations and interpretability should still be pursued to increase accountability and better validate the fairness and functionality of systems.

Out of Distribution

Consider an imaginary model $y=f(u)$, $f$ being the model, $u$ being the proportion of people with an umbrella and $y$ the probability of rain. The model reaches low evaluation error and everyone is happy.

However, the model consistently fails to predict rains when people didn't take the umbrella. Why could this happen? Some of the reasons below were inspired by the paper "The Mythos of Model Interpretability":

1. The model undefitted the data, and we may need a better model.
2. The dataset is not representative the deployment environment, and the model can't generalise out of training distribution. Can it be fixed if we don't have those datapoints? Were there simply wrong datapoints, that led the model in the wrong direction? Can we create synthetic data?
3. The approach itself was incorrect: we use variables that promote association rather than causation.

Selecting possible causal variables, such as pressure and temperature, rather than the fraction of humans carrying out an umbrella, could help to make it more accurate, and even more explainable. But does it have all the causal inputs? Why do we expect it to work out of distribution, though?²

A subset of causal-variables may do for a good-enough approximation, and even generale well out of distribution. In some cases though, it may be enough to have a correlation model, but they should be distinguished.

Selecting those variables is not very easy, though. An expert must pick known causes-effects pairs as inputs-outputs to train a model, but others may unknowingly build a correlation model instead.

It is hard to predict whether a model will work out of distribution without knowing what it has learnt. Knowing what a model has learnt is part of the XAI discipline, both opening the box, or carefully comparing its outputs.

Similarly, [this two-page comment][interpretable_ml] by Cynthia Rudin highlights the preference for interpretable (transparent) models in high stakes scenarios.

In Deep Learning Models, the problem constraints can be used to add inductive biases or priors to architectures, such as symmetry constraints, connectivity (say through graph networks). This may also reduce the amount of training data needed, improve generalisation and improve interpretability.

An idea related to "Out Of Distribution" inference is that of "Transfer Learning": If a model has learnt "essential, compact features" then they should generalise to other task, as stated in Scientific discovery in the age of artificial intelligence (references where removed):

Self-supervised learning (Box 1) has enabled neural networks trained on labelled or unlabelled data to transfer learned representations to a different domain with few labelled examples, for example, by pre-training large foundation models and adapting them to solve diverse tasks across different domains.

This is especially useful when models can leverage large amount of data, which is usually in the form of unlabelled data (there are also mechanisms to label data semi-reliably).

Another promising path towards better generalisation is that of Causal AI. As "Scientific discovery in the age of artificial intelligence" puts it:

Although many scientific laws are not universal, their applicability is generally broad. Compared with state-of-the-art AI, human brains can better and faster generalize to modified settings. An attractive hypothesis is that this is because humans build not just a statistical model of what they observe but a causal model, that is, a family of statistical models indexed by all possible interventions (for example, different initial states, actions of agents or different regimes). Incorporating causality in AI is still a young field

Map of XAI

An interesting map of XAI is given in the survey Principles and practice of explainable ML (2021).

Most classic ML models are in the dashed area under Model types column.

Classic ML models are usually transparent (intrinsically explainable) but may benefit from post-hoc (post training) explanations, such as visualising it. When transparency is key and the predictions are accurate enough, these may be preferred over DL models.

Image from paper under CC-BY

To the visual explanations, t-SNE, PCA and other dimensionality reduction techniques can be added.

The focus here though, is explaining deep learning models which are often, but not always, more accurate than classic ML models.

1. Only valid for a small region, can't fit curvature / non-linearity, struggles with variable interdependency modelling. ↩

2. Could metaphors and analogies (from experience) be the missing ingredient of this to succeed? Could using causal models help to overcome these problems? How can we make a model that uses analogies? ↩

Additive Feature Attribution Methods

"Additive Feature Attribution Methods" (AFAMs) are a class of extrinsic explainability methods (where the reference model's internals aren't analysed). This post explores a few of these methods.

What are AFAMs?

The Additive Feature Attribution Methods (AFAMs) class was identified in the paper A Unified Approach To Interpreting Model Predictions. The methods in this class approximate a prediction of the original model ($f$) with an explanation model ($g$) which is a linear addition of binary features, making it simpler and interpretable.

How do explanation models relate to out general idea of explanations? Here is what Explaining Explanations in AI says:

These [explanation] models can be understood as a "do it yourself kit" for explanations, allowing a practitioner to directly answer "what if questions" or generate contrastive explanations without external assistance.

Mathematically, the explanation model ($g$) is described as:

$$f(x)\approx g(z^{\prime })={\varphi }_0+\sum _{i=1}^M{\varphi }_i{z}_i^{\prime }$$

Each ${\varphi }_i\in R$ is an effects of a binary feature $z_i^{\prime }\in \{0,1{\}}^M$ in the output. The different methods in the class estimate ${\varphi }_i$ differently.

A Unified Approach To Interpreting Model Predictions defines Explanation Model as:

We introduce the perspective of viewing any explanation of a model's prediction as a model itself, which we term the explanation model." and also "Instead, we must use a simpler explanation model, which we define as any interpretable approximation of the original model.".

The paper Explaining Explanations in AI relates such models to approximate scientific models (which are all of them as reminded by Box's maxim "All models are wrong but some are useful"). So it is important for the recipients or users to know where models are reliable, where they break down or where they have unknown behaviour. In their words:

For an individual to be able to trust such a model as an approximation, they must know over which domain a model is reliable and accurate, where it breaks down, and where its behaviour is uncertain. If the recipient of a local approximation does not understand its limitations, at best it is not comprehensible, and at worst misleading.

This is not to say that local approximations are without merit, but rather that they can only reliably have explanatory power if their limitations are clearly documented and understood by recipients.

The paper also highlights a link between explanation models (and would also apply to transparent models) to contrastive and what-if explanations:

Over the domain for which the model accurately maps onto the phenomena we are interested in, it can be used to answer "what if" questions, for example "What would the outcome be if the data looked like this instead?" and to search for contrastive explanations, for example "How could I alter the data to get outcome X?"

However (as they note) local explanation models won't be accurate outside of the local domain (by definition). Also, the explanation models may be faithful to the prediction model, but if the latter is unreliable, the explanation model will also be (though on the positive side, this may be more obvious in the simpler model).

SHAP

Linear LIME, DeepLIFT and other methods calculate ${\varphi }_i$s differently, in turn yielding different coefficients.

The Unified Approach to Interpret Model Predictions proposes that models should have local accuracy, missingness, consistency which, they argue, lead to coefficients that are more intuitive for humans.

The terms are defined as:

• Local Accuracy: There must be equality when the input is the original one ($x$), that is $f(x)=g(x^{\prime })$.
• Missingness: If the reference vector ($x^{\prime }$) has a "missing" component ($x_i^{\prime }=0$) then the feature must have no impact, that is ${\varphi }_i=0$.
• Consistency: if one of two models is larger just turning feature $i$ on and off, then it must have a larger ${\varphi }_i$.

This how they bring in a result from game-theory by Shapley, which they call Shapley values.

Their Theorem 1 guarantees that a linear explanation model plus the 3 requirements leave Shapley Values as the best (and unique) coefficients. Other methods violate some of these 3 properties (so the authors modify them to comply).

SHAP (SHapley Additive Explanations) Values are the Shapley Values of a conditional expectation function of the original model: $f(h_x(z^{\prime }))=\mathbb{E}[f(z)|z_S]$ (Section 4, see Figure 1). $S$ are non-zero indices.

• For example, with $\vec{z}=⟨v_1,0,v_2⟩$ then ${\varphi }_3=\mathbb{E}[f(z)|z_{1,3}]$. So the Shapley values are the change in the expected model prediction when conditioning on a feature.

Note

The most accurate Shapley Values are expensive to calculate. Approximations can be used in some cases to speed this up.

Approximating SHAP Values

The approximations can be model agnostic: Shapley Sampling Values, Quantitative Input Influence, Kernel SHAP; or they can be model-specific: Max SHAP, Deep SHAP.

The actual approximations are:

Approximation 1: Feature independence (implies non-multicollinearity).

• Shapley sampling values method,
• Quantitative Input Influence,
• Kernel SHAP (requires both assumptions).

Approximation 2, model linearity: Shapley regression values.

• SHAP provides both global (average across inputs) and local (for a given input).

Kernel SHAP

Can we modify Linear LIME's Loss function so that the values of coefficients found are Shapley values? Yes! These are also more intuitive to humans, and remove some of the heuristics (kernel selection and complexity metric) of it.

The Linear LIME quantities of proximity kernel (${\pi }_x$), complexity penalty ($\Omega (g)$) and Loss ($L$) are turned into a shapley kernel, $\Omega (g)=0$, and the same weighted loss.

The SHAP values / coefficients for this linear model with assumed-uncorrelated features can be estimated by weighted linear regression.

This method is called Kernel SHAP, and it's fast to compute.

LIME and SP-LIME

The paper "Why Should I Trust You?": Explaining the Predictions of Any Classifier proposes the Local Interpretable Model-Agnostic eXplanation (LIME) and the Submodule Picking LIME.

LIME isn't a particular model but rather a set of characteristics that explainable methods should have, according to the authors (more on this later).

Let's first look at LIME as implemented in a concrete case.

Why would this be useful? A concrete example

A linear model with few-ish, interpretable features, locally fit to a complex one is in line with LIMEs' requirements.

Local fit means fitting only in the vicinity of the input of interest. The input features may differ from those in the original model.

The original paper shows an example comparing two different models that were fit by linear ones:

Image taken from paper.

The image makes clear some of the uses of it:

• The interpretable features, alongside their contributions (weights) to the prediction, can help decide whether to trust the prediction or not.
• Feature engineering such as removing features (or certain data) that the model uses but harm generalisation,
• Comparing models is easy (through the linear proxies). It's especially useful if the original models' accuracy (and other metrics) are similar, and their features non-interpretable.
• Here, one of them is untrustworhy (right hand side), giving high weight to meaningless features.

LIME / Desiderata

The authors consider 4 properties to be desirable in an explanation model:

• Local: the interpretable or explanation model approximates the original model in the vicinity of a particular prediction. In contrast, global explanations explain the full model.
• Model-agnostic: any model can in principle be explained by this method.
• Interpretable Explanation: In this paper, "interpretable" is a desired characteristic of "explanation", and provides qualititative understanding (a simple answer to "Why was this prediction made?"). In their own words:

  An essential criterion for explanations is that they must be interpretable, i.e., provide qualitative understanding between the input variables and the response. We note that interpretability must take into account the user's limitations.

Additionally, they include 4. A global perspective, a mechanism to get a sense of the full model's behaviour.

The first 3 can be called LIME; the Submodule Picking LIME (SP-LIME), which selects LIME explanations to give a global explanation of the model.

LIME is a slightly more explicit version of the first 3 desired properties. Let's explore this.

• Which specific representations does the LIME framework consider interpretable?

  An example of an interpretable representation is a binary vector with components indicating presence / absence of a feature (e.g. a word) in the explanation model.

• Which models does the LIME paper consider interpretable?

  (...) interpretable models, such as linear models, decision trees, or falling rule lists [27], i.e. a model $g\in G$ can be readily presented to the user with visual or textual artifacts.

• Complexity is the opposite of interpretable, so the loss (we skip it here) to train the explainable model accounts for it, and gives local samples more importance than remote ones. Here $G$ being the model class of $g$:

  As not every $g\in G$ may be simple enough to be interpretable thus we let $\Omega (g)$ be a measure of complexity (as opposed to interpretability) of the explanation $g\in G$.

  The definition of complexity depends on $G$. For linear models it may be the number of weights.

• They also define a sampling procedure to create the training set. This is detailed in the concrete algorithm in the next section.

Linear LIME: An Algorithm

The paper implements LIME using the class $G$ of sparse linear models as explanation model, which we could call Linear LIME (as A Unified Approach to Interpreting Model Predictions does). Here is my interpretation of the algorithm (the primed variables denote binary vectors):

1. A model $f$ and an input vector $x\in R^n$ needs explaining,
2. Start an interpretable, binary vector $x^{\prime }\in \{0,1{\}}^{n^{\prime }}$ with only the dimensions of interest of $x$ (it may be all-ones often),
3. Generate perturbed binary variants of $x^{\prime }$ called $z_i^{\prime }$,
4. Use different $z^{\prime }$ to make variants of $x$ called $z\in R^n$.
   • Written in [A Unified Approach to Interpreting Model Predictions][unified_approach_lcobf] paper as $z=h_x(z^{\prime })$.
5. Now we have training tuples $(f(z),z^{\prime },{\pi }_x(z))$.
6. Use the dataset to fit the linear model $g$ using $K-LASSO$.
   • To select $K$ interpretable features they apply $K$-LASSO.
   • LASSO is like a sparse linear regression, some coefficients are pushed to $0$ and correspond to variables we can ignore. How many are are pushed to $0$ may be controlled by the complexity penalty in the loss function.
   • This is why the number of input features for the interpretable model ($x^{\prime }$, $z^{\prime }$) may be much smaller than that of $x$, $z$.
   • Then the non-zero variables are used to fit $g$ to $f$ but now using least-squares.

SP-LIME is defined as:

[SP-LIME] a global understanding of the model by explaining a set of individual instances.

The complex part of SP-LIME is selecting instances that add the maximum insight, and avoiding repeated ones. The algorithm is briefly described later on.

Linear LIME: A problem

Explaining Explanations in AI has both an interesting explanation and also criticism of LIME, so it is pasted below without changes:

The other option, and most influentially proposed by Ribeiro et al. (2016) in their Locally Interpretable Model-Agnostic Explanations (LIME) approach, is to binarize the problem. Rather than trying to fit a linear classifier to a large range of values, the authors consider a binary problem, where for each feature they attempt to switch it on and off, allowing them to answer the question “What is the contribution of feature f to the classifier response, given the data it currently sees?” This leaves open the question of “the contribution compared to what?” For unstructured data, such as a count of how many times particular words occur in a document, it makes sense to compare against a baseline created by setting the count to 0. For structured data this is more problematic. For example, how can we evaluate the importance of someone’s salary to a loan decision, if the classifier can only evaluate people with valid salaries? The answer is to compare it against a different valid salary, but it is unclear how this valid salary should be chosen.

Particularly it highlights that we are looking at $g(z_i^{\prime }=1,...)-g(z_i^{\prime }=0)$ rather than anything else (the ${\varphi }_0$ can be meaningless, since that can be very far away from the value on $f$).

And here it matters to which state of $z_i$ each $z_i^{\prime }$ value maps to. It is also important that the corresponding $z$ vectors near $x$! Otherwise, the difference is meaningless, because one of them is likely inaccurate.

The last part of the paper explores which state should be $0$ which is can be actual $0$ or a reference value. As said above, the $z$ points must be near $x$ for best results.

There are many interesting caveats with LIME which are worth exploring in detail.

SP-LIME: The Algorithm

The goal here is picking the most informative instances, and without repetition.

A coverage metric is defined: $c(V,W,I)={\sum }_j{1}_{[}\exists i\in V:W_{ij}>0]I_j$.

Here, $W$ is a matrix of weights (columns) for each instance (row).

$I$ is the global importance of a component, defined as $I_j=\sqrt{{\sum }_i{W}_{ij}}$. The larger the sum of weights is, the more important.

A marginal coverage for each candidate instance ($i$) $c_i-c$ is performed, then the instance that increases $c$ of $V$ the most is appended to it.

Note

This approximates the $V$ with the largest coverage, but it's not exact because the total max could be one that does not max it on each step!

To increase the coverage at all, it must add some non-zero value to a column of zeros.

LIME: Final Comments

Explanation models do not replace but complement accuracy or other evaluation metrics.

Methods II

Visual Explainability Methods

• Saliency Maps: visually show which features are most important in a particular prediction. They can give a false sense of confidence unless also tested with the "wrong label". It does not give any information on what processing is done with the regions. They can be generated for 1D, 2D and ND inputs. For example, here is for radiology:

  

  Left-most: input image; next: input + saliency map; right-most: doctor's annotation (top) and RNN-model generated annotation (bottom). Image taken from paper.

• Variations: Individual Conditional Expectation, Partial Dependence Plots, can help visualise decision boundaries; they only vary 1 or 2 variables. Quotes below are snippets from original:

  • [ICE] operates on instance level, depicting the model's decision boundary as a function of a single feature, with the rest of them staying fixed.

  • (...) employ [ICE] plots to inspect the model's behaviour for a specific instance, where everything except salary is held constant, fixed to their observed values, while salary is free to attain different values.

  • PDPs are a similar idea, but the remaining features are average values over the dataset points, rather than particular values of an instance.

• Validity Interval Analysis: another technique fitting the NN behaviour to try to extract explanations.

Linear Combination Of Features

• SHAP,
• Influence Functions,
• LIME (previous post)

Other methods

• Anchors: the authors of LIME also proposed this nice method described by Principles and practice of explainability in ML:

  • A similar technique, called anchors, can be found in (Ribeiro et al., 2018). Here the objective is again to approximate a model locally, but this time not by using a linear model. Instead, easy to understand "if-then" rules that anchor the model's decision are employed. The rules aim at capturing the essential features, omitting the rest, so it results in more sparse explanations.

  • (...) decides to use anchors in order to achieve just that, generate easy-to-understand "if-then" rules that approximate the opaque model's behaviour in a local area (Figure 9). The resulting rules would now look something like "if salary is greater than 20 k£ and there are no missed payment, then the loan is approved.

• Dimensionality Reduction: Principal Component Analysis, t-SNE, Dimensionality Reduction, Independent Component Analysis, Non-negative Matrix Factorisation.
• Counterfactuals: Rudin states "A counterfactual explanation describes a minimal change to the input that would result in the opposite prediction." We replace the problem by a hypothetical opposite:
  • Rudin's paper has concretes example as well: "your loan application was denied, but if you had $1000 less debt, you would have qualified for the loan." and "If you reduced your debt by $5000 and increased your savings by $50% then you would have qualified for the loan you applied for".
  • A is a cause of B if, in an imaginary situation, A not happening implies B not happening,
  • Change the instance slightly, but such that the model classifies the new instance in a different category,

  • (...) the applicant had missed one payment that led to this outcome, and that had he/she missed none the application would had been accepted.

• Contrastive: Is about comparing carefully selected instances: Why P rather than Q?.

  • In such cases, people expect to observe a particular event, but then observe another, with the observed event being the fact and the expected event being the foil.

  • It's a good "question generator". What do you expect if X is done, rather than Y?

Explanation-producing Architectures

Architectures designed to make explaining part of their operation easier.

• Using Explicit Attention: An attention layer/mask learns how parts of an input embedding pay attention to other parts. The layer is somewhat interpretable. In chemistry, it could learn which atoms connect (or pay attention to) other atoms.

• Dissentangled Representations: Disentangled representations have individual dimensions that describe meaningful and independent factors of variation. —Explaining Explainability (2018). Examples of architectures are $\beta$-VAE, INFOGan, capsule networks.

XAI4Chem: Contrastive explanations

Previous posts have discussed explanations and model explainability.

One area of interest, both in academia and the pharmaceutical industries, is the explainability of neural networks predicting structures or properties of molecules.

For example the paper "Enhancing preclinical drug discovery with artificial intelligence" (the references were edited out, and can be found in the original article) states:

(...) many successful applications of QSAR-based VS workflows for hit identification have been reported. Zhang et al. described the successful implementation of an ML-based QSAR workflow for VS that led to the discovery of novel antimalarial agents.

The ML algorithms were trained on a small dataset of antimalarial drugs. The authors continue:

The QSAR models were used to carry out VS against the ChemBridge database and resulted in the selection of 174 compounds for a follow-up screening in Plasmodium falciparum growth inhibition and cellular assays. Experimental validation revealed 25 of the selected compounds to be active, yielding a hit rate of 14.2%, with the most potent hit having an EC50 value of 95.6 nM. Subsequently, many studies have reported the application of ML and DL-based QSAR workflows as promising VS tools.

That sound cool and useful. A human expert may be able to detect a pattern on those structures.

• But can we figure out an evaluation mechanism that singles out which fragments are most important?

• At the same time, isolated fragments may be misleading, because the shape of the molecule is also as important. Maybe a "fragments vs shape" could make up a directionally useful plot?

Paper Idea

Feature Attribution Methods, such as SHAP and LIME help answer the question "Why was P obtained?".

On the other hand, contrastive explanations answer the question "Why P rather than Q?".

Here, "Q" is the foil (an alternative event), and it's an important tool to make explanations simpler: the aspects that are similar do not need to be explained, it is the differences that count towards explaining "Why P rather than Q".

The paper "Contrastive explanations for machine learning predictions in chemistry" (2025) develops a framework called MolCE for:

deriving contrastive explanations for machine learning models in chemistry to systematically generate intuitive explanations of predictions in high-dimensional feature spaces.

Machine Understanding

This is a work in progress, that explores the topic of "Machine Understanding", "Machine Thinking" and related ideas.

Strong AI and the Turing test

The Turing test (1950, by Alan Turing), is a remote chat where a human is tasked to tell who of the other two members of the chat is a computer.

In "Is the Brain's Mind a Computer Program?" (1990) Searle states that Strong-AI advocates take the Turing test as a test for machine thinking:

The Turing test, as currently understood, is simply this: if a computer can perform in such a way that an expert cannot distinguish its per formance from that of a human who has a certain cognitive ability—say, the ability to do addition or to understand Chinese—then the computer also has that ability.

But, we could ask the Strong AI advocates:

• Does such machine experience the world the same than us?
• What about the internal mechanisms?
• Why only care about overt behaviour?
• Is the Turing test the best one for this purpose?

There are a few reasons that make this interpretation untrustworthy.

• First, we may be bad at evaluating this difference.
• Second, even if we evaluate correctly, it may be that the test does not provide conclusive evidence of thinking or another cognitive capacity.

First: Are we good evaluators?

Humans anthropomorphise language easily (assigning a mind behind it), as the ELIZA effect has shown.

Humans have also been wrong about Clever Hans; Hans was a horse assumed to understand arithmetic, and later on shown to guess the answer social queues (which is still impressive, but a different capability).

This has happened in recent times as well, with LLMs.

The take-away seems to be that we should evaluate behaviour more carefully, and that looking at the internal mechanisms of a network may help us do precisely that.

Second: Turing test's usefulness

Searle's Chinese Room Argument (CRA for short; from "Brains, Minds and Programs", 1980) is a thought experiment aiming to show the absurdity of the Strong AI view. Searle summarises his result as:

Now let me add the point that the Chinese Room demonstrated. Having the symbols by themselves—just having the syntax—is not sufficient for having the semantics. Merely manipulating symbols is not enough to guarantee knowledge of what they mean.

Searle concludes that programs restricted to symbol manipulation can't think. A program may have the right input-output behaviour (e.g. passes the Turing test), but the system wouldn't yet know the meaning of the Chinese symbols/words it is manipulating.

A last statement, equivalent to all of the above, of his:

You can't get semantically loaded thought contents from formal computations alone, (...)

General Ideas about Meaning

Before considering details about meaning let's ask: What is meaning (in broad terms)? Searle states that meaning is what the symbols refer to:

If, for example, I am thinking about the last presidential election, certain words will go through my mind, but the words are about the election only because I attach specific meanings to these words, in accordance with my knowledge of English. In this respect they are unlike Chinese symbols for me.

But this seems a bit mysterious and one may want to ask Searle: What if all we need are symbols, processing, and grounded symbols?¹

What about researchers other than Searle? Koller and Bender, in Climbing Towards NLU (2021) define meaning as the set of (expressions, intents) pairs. Expressions are symbols, gestures and intent is what's not in the expression itself (this is the intent).

They hold that machines can't learn intents from form alone (symbols) though they may if trained on meaning pairs, or a similar task.

Climbing Towards NLU talks about all the of the above: the Turing test, Searle's Chinese Room, Stevan Harnad's grounded symbols, and they describe their own Octopus test for language understanding. They say:

Searle's thought experiment begins from the premise that it is possible to manipulate forms well enough to be indistinguishable from a system that understands the meaning of the forms, reasons about it, and responds appropriately. We observe that much recent work in NLP claims to be building systems where not only the runtime system but in fact also the process for building it only has access to form.

(Which is precisely what bare-LLMs do.) They continue:

But language is used for communication about the speakers' actual (physical, social, and mental) world, and so the reasoning behind producing meaningful responses must connect the meanings of perceived inputs to information about that world. This in turn means that for a human or a machine to learn a language, they must solve what Harnad (1990) calls the symbol grounding problem. Harnad encapsulates this by pointing to the impossibility for a non-speaker of Chinese to learn the meanings of Chinese words from Chinese dictionary definitions alone.

Hardware Thoughts

Searle asks interesting questions:

why couldn't we give semantics to other computers?

(...)

What, in fact, are the differences between animal brains and computer systems that enable the Chinese room argument to work against computers but not against brains?

In other words: Where does meaning (or semantics) come from, that we can't just add it in modern computers, but brains have it? Searle continues:

The most obvious difference is that the processes that define something as a computer—computational processes—are completely independent of any reference to a specific type of hardware implementation. One could in principle make a computer out of old beer cans strung together with wires and powered by windmills.

But when it comes to brains, although science is largely ignorant of how brains function to produce mental states, one is struck by the extreme specificity of the anatomy (...)

And later on, in the replies to comments states:

(...) formal symbols have no physical, causal powers. The only power that symbols have, qua symbols, is the power to cause the next step in the program when the machine is running. And there is no question of waiting on further research to reveal the physical, causal properties of 0's and 1's. The only relevant properties of 0's and 1's are abstract computational properties, and they are already well known.

So even if we consider "the processing of a rich representation of interlinked symbols themselves linked to input (e.g. perceptions) and outputs (e.g. an action, decision or conclusion)." Searle would say no. Searle explains that the difference is that

(...) the brain does not merely instantiate a formal pattern or program (it does that, too), but it also causes mental events by virtue of specific neurobiological processes.

• Why would Searle consider that physical, causal links are required, rather than just the computation? He says:

But the simulation should not be confused with duplication. The computational model of mental processes is no more real than the computational model of any other natural phenomenon.

For example (again Searle's words):

No one expects to get wet in a pool filled with Ping-Pong-ball models of water molecules. So why would anyone think a computer model of thought process es would actually think ?

He says that, to think computers must "duplicate the specific causal powers of brains, and it could not do that just by running a formal program."

So overall his view is that a running computer program, on a standard computer, is symbol processing without meaning (exemplified by the CRA), and the difference (which also adds meaning in) is to be found in the physical and causal structure of the brain which computers (as currently designed) lack.

Also, the causal structure isn't that of wrapping current computers in a robot body, but about the kind of causation in the neural network.

Fascinating. But it's worth pondering here:

• Why exactly does meaning require a particular physico-causal organisation of matter?
• And, what if semantics / meaning are not what Searle thinks (and require certain causal structure), and all that is needed is rich representations of symbols (included grounded ones), links between them, and their processing? Note that I am not saying form alone since this was robustly refuted by Koller and Bender but e.g. rich representations of symbols connected to multimodal perceptions. This is what they suggest in Section 7 (among other options):

One approach to providing grounding is to train distributional models on corpora augmented with perceptual data, such as photos (...)

Searle does consider this question, once in the quote at the beginning of this section, another time below in relation to the Churchlands' "Could a Machine Think?" (1990):

(...) could not later investigation also show that semantics are entirely constituted of syntax ?

It's an interesting dilemma. It seems likely to me that Searle is right, and that the mind is physico-biological and, just as digestion, an emergent property of matter.

Simulating digestion simply isn't digestion.

But...is it relevant?

Let's assume that we are not interested in a machine that develops a mind with an actual self, understanding, and thoughts, but that carries out the computations of those processes, solves problems, represents things internally in a rich, and understandable (by us) way and so forth.

It may even do research and discover things.

What would we call what these machines do? What are the limits of it? What moral status would these systems have (maybe none)? Would they be more than just useful-ish computers?

1. Harnad's grounded symbols are those linked to perceptions, memories, emotions, and so forth. And those in turn are just some input representation. ↩

Connectivism

Personal comment: I disagree the theory's definition of learning, but agree with their view of knowledge in the digital era and the consequences of it.

Also, their idea of relating learning with a network/graph is quite useful (even though they may have done it for other reasons).

Lastly, connectivism is distinct but related to connectionism. The latter is a AI/DL framework inspired in biological neural networks.

Connectivism (2004) is "a learning theory for the digital age".

Origin and Definitions

As the digital era unfolds a few important changes become clear:

• Knowledge grows faster and faster;
• Computers and algorithms role changes, and can now store, retrieve, analyse information and follow instructions.

So the environment requires that we:

• Select and discriminate what to (and not to) learn;
• Develop ways to search the best knowledge-sources;
• Understand how knowledge is distributed in human networks, how it grows, changes...

Definitions of Learning

Connectivism defines learning as (bold is mine):

Learning is a process that occurs within nebulous environments of shifting core elements – not entirely under the control of the individual. Learning (defined as actionable knowledge) can reside outside of ourselves (within an organization or a database), is focused on connecting specialized information sets, and the connections that enable us to learn more are more important than our current state of knowing.

It also states:

Learning is a process of connecting specialized nodes or information sources.

So if learning is redefined as knowledge that we can use, then indeed it is being conceptualised like a recipe in a database.

This does not seem a definition most people subscribe to.

What seems clear though is that there may be less need to know (how & what) and more need of knowing-where the knowledge needed is (plus select and delegate).

If instead we define learning as a persistent change in the object's interaction with the world, as a result of a previous interaction then it isn't learning what has changed. What changed are the specific tasks. For example, learning how to search may become more and more important.

By this definition, one may argue that the aspects of learning that matter remain in place: making sense of a new problem.

Also, at a lower pace, we also had to know how to search before the rapid informational changes: find the right expert for a job, record information on a piece of paper and so forth.

But this insight into the dynamics of knowledge-rich era is still a useful idea:

When knowledge, however, is needed, but not known, the ability to plug into sources to meet the requirements becomes a vital skill. As knowledge continues to grow and evolve, access to what is needed is more important than what the learner currently possesses.

Systems That Learn

In their view, not only individuals can learn but also other objects, such as organisations and human (or social) networks.

And, as mentioned earlier, the digital era involves even faster knowledge growth, and what we know will become proportionally less:

When knowledge, however, is needed, but not known, the ability to plug into sources to meet the requirements becomes a vital skill. As knowledge continues to grow and evolve, access to what is needed is more important than what the learner currently possesses.

Either way, representing learning as "CRUD" operations on a graph / network is a useful idea.

They define connectivism as:

Connectivism is the integration of principles explored by chaos, network, and complexity and self-organization theories. (...) Connectivism is driven by the understanding that decisions are based on rapidly altering foundations. New information is continually being acquired. The ability to draw distinctions between important and unimportant information is vital. The ability to recognize when new information alters the landscape based on decisions made yesterday is also critical.

Notice that all the above fits well with a "behaviourist" view of learning but adding an interpretation of some aspects of learning via a graph (e.g. how the brain or an organisation learns via modifications of the graph) and considering that now information changes and grows at a faster rate. Also, considering a network of people as the "knowledge storage device" is quite insightful, as it is also meaningful to consider it from the perspective of a node.

I'd also consider that the process of "digitally storing and updating information" isn't by itself organisational learning. But how the organisation responds to and prepares for new problems and events does fit the behaviourist definition of learning quite well.

Networks

A central object of the theory are networks; as they state:

A network can simply be defined as connections between entities. Computer networks, power grids, and social networks all function on the simple principle that people, groups, systems, nodes, entities can be connected to create an integrated whole. Alterations within the network have ripple effects on the whole.

And also:

The starting point of connectivism is the individual. Personal knowledge is comprised of a network, which feeds into organizations and institutions, which in turn feed back into the network, and then continue to provide learning to individual. This cycle of knowledge development (personal to network to organization) allows learners to remain current in their field through the connections they have formed.

(...)

This amplification of learning, knowledge and understanding through the extension of a personal network is the epitome of connectivism.

Hence structure of interconnected nodes (a network or graph) is an important part of the naming connectivism. The aspects that the links and the nodes represent, how it grows or shrinks, what the link-pattern and architecture look like and so forth will change from problem to problem.

A set of concrete instances of the idea of a graph seems to inspire connectionism, which appears in the area of human and organisations learning.

Examples of Networks

1. The Internet, and later on the World Wide Web
   • Seems to have inspired this theory the most: nodes/documents becomes more useful, by being accessible and interlinked; suddenly, the individual is connected via a "fast"-link to a world wide network of documents; this idea also shines light on what an organisation is and how to improve it (more accessible, diverse knowledge for example)
   • These both were from the start problems about networking; heterogeneous devices or resources linked together,
   • These technologies opened up access to a network of resources and people.
   • The world changed towards quicker decision-making, but I argue pattern-matching, and thinking strategies have not changed much.
2. Neural Networks, both biological and artificial;
3. Certain aspects of organisms and organisations. The paper linked above states:

   The organization and the individual are both learning organisms. Increased attention to knowledge management highlights the need for a theory that attempts to explain the link between individual and organizational learning.

As said, networks are the main object of the theory, so let's briefly define them:

Networks: collections of linked nodes or items.

What is considered a node is defined by us. It could be an individual, a group, a government, countries that trade, libraries, and so forth.

There are types of nodes and edges (or links) with different properties. For example, a node may have restricted access to a resource. There are also topologies with specific wiring pattern.

Knowledge and Learning

The theory views knowledge and learning as properties of networks, and defines them as:

• Knowledge

  • Network view: it is stored distributed across the net (in nodes, edges) in a latent manner. Distributed knowledge means that no single node or edge is uniquely responsible for a concept, rather, many parts of the whole network are. There will be nodes that are more relevant in some situation than other, but the knowledge is still distributed across nodes and links.
  • Behavioural view: It is evidenced as an appropriate response to an input signal or stimuli.

• Learning

  • Network view: a change of network connections by exposure to experience and reasoning.
  • Behavioural view: a persisting change in knowledge. Learning is evidenced through a change towards more appropriate behavioural responses (to a signal or stimuli.)

In a network, one node or link changing means much more has changed due to the ripple effects given by the links.

Discussion on whether all networks really learn are left for the next post. This one focuses on learning-human-networks and how the theory can help understand and design them.

A human in the net

A promising area for connectivism is as a paradigm to think of human networks (or human-and-resources networks), how to organise them (or letting them self-organise), which properties make nodes and the net grow and so forth.

It appears of especial important in the digital, interconnected world; but also classrooms, organisations, and social networks in general.

Let's consider the case of a human or agent embedded in a wider network such as a classroom, a group, community, organisation.

This networks can be seen as an metaphorically organism, or just as a whole: when the connections and nodes change and so does the behaviour in response to input signals, which in certain cases will imply learning, and formation of knowledge.

Example

Think of the response of an organisation in the event of a fire or an emergency.

The input (fire alarm) ripples through the network as the nodes act and propagate activity: learning isn't only intra-personal, the whole network can be said to have learnt in this example.

In the case of a classroom, this framework may aid us with questions such as:

• Centralised human-learning-network guided by teacher or a decentralised network? Should the teacher be more like a routing node?
• Are all items considered part of it, such as resources, or agents such as humans? Is it useful to split or join different networks?
• An open network where agents can change resources, or closed networks where they only have read permissions (if even so)?

A human is a net (theory of mind)

We also want to focus on the individuals. Can connectivism help?

Their general claim is that any network can learn by modifying connections and nodes of the network. And stores distributed knowledge.

So can parts of the brain, which are networks of neurons, and learning is later on evidenced on the behaviour of individuals.

Refining Connectivism

Do all networks learn?

The notion that all networks learn, just by virtue of being networks seems off. It does not correspond to what most of us would call learning.

Is there some way to look at this, which makes more intuitive sense? Admittedly, there will be grey areas, but my view is that it is possible to conceptualise learning more clearly.

Different networks vary on:

• The breath of signals they are responsive to,
• The intensity of signals they are responsive to,
• How varied their response is,
• How much directionality there is to the process,
• Possibly others (still thinking on it).

The hypothesis is that, what we ordinarily call a learning system maps to:

High-complexity systems that respond to a wide variety of input signals in a wide variety of ways, and change in a way that makes future reactions appropriate (which is left to intuition for the moment).

Usually, networks of complex systems or of hybrid systems also change complex ways and will also be learning systems.

On the other hand, low-complexity systems change in narrow and even predictable ways whichare not considered learning systems.

In a sense learning is not the mechanism, but it's just the label. Complex systems that change directionally and in various ways learn; simple systems that change --by the same mechanism-- but do not behave in complex ways are not learning systems.

Just reinforce it: the proposal is that learning mainly points to high-complexity, but the actual kind of change could be the same in both systems.

We could say that there is a threshold of a binary function $f(Complexity,Degre{e}_{A}dapt,Store{s}_{C}hanges)$ were systems are considered at learners. There will also be, certainly, grey areas.

With that, which one learns?

We can apply the previous criteria to a large number of cases, and see whether we are satisfied:

Do biological networks learn?

It is generally agreed that learning involves changes in the wiring pattern, or degree of wiring, and this change can happen in different ways, and it is built into the system (brain) e.g. by Hebbian learning. We can take this case as our paradigmatic learning system.

Do artificial neural networks (ANNs) learn?

During training phase, connections (weights) are updated towards a goal through backpropagation, and the network behaviour (predictions) change. General-purpose process a wide variety of signals are are closer to learning systems than narrow AIs. Both only learn during training phase.

Does a colony of insects learn?

Complex organisms communicate and change their behaviour: Ants propagate pheromones which adapt their behaviour; humans similarly can do it by imitation of some kind. This "node update" propagates out, and may return to update the node further. Finally the network is seen as reacting appropriately.

But is this "reaction", or emergent behaviour, the same as learning? Only if the change is permanent. The variety of signals they process and the behaviours is quite varied and certainly are learning systems.

Does the world wide web, or a library learn?

The system will learn only if it is designed to do so. For example, after a certain event (like a query) a program may change the system to respond more appropriately in the future.

When humans have designed them to adapt (say with software) these have limited learning ability and change in response to a narrow variety of events, and are not quite learning systems. Considered without the programs, these wouldn't have any learning at all.

Hybrid systems such as humans-web, or humans-library systems certainly are learning systems.

The structure, interface, reliability that they have though, matters a lot in terms of how usable they are to humans, and so forth. These properties are still important across most network systems, be it learning networks or not. In other words, all non-learning networks can become learning networks when associated with other systems that can change them

Does the electric grid learns? Same as above.

Does a molecule or a material, which can be represented as a graph, learn?

Signals can change them. But even if we define appropriate as increasing the likelihood to persist / survive, the ways in which it reacts and the number of stimuli it responds to are narrow, and won't be learning systems.

Do mixes of nodes made of proved-learning-systems (say humans) and non-learning systems (say a book), learn?

Yes, especially if we consider a company, a community and so forth.

So all the different kinds may be considered to learning, if they permanently change in response to a stimuli and this adequately changes future responses.

Atom vectors

The quest for machine representations of objects is a long-standing research theme. For example, we have vector representations:

flowchart TB
A(Machine representations
of objects) --> B(Vector Representations)
B --> C(Descriptors)
B --> D(Embeddings)

Descriptors are expert-designed vectors; embeddings are machine-learnt vectors $\in {\mathbb{R}}^N$.

This post focuses on embeddings because they require less human effort, and produce more general-purpose vectors.

Characteristics of Embeddings

Embeddings are usually real-valued, dense rather than sparse, non-human readable. They also form a structured vector-space, with semantically similar vectors close together, and meaningful vector-arithmetic.

It is desirable, but not always possible that they:

• Are interpretable,
• Can be generated with data scarce environments,
  • Or are data-hungry but it is easily available,

The paper Scientific discovery in the age of artificial intelligence proposes a different list of characteristics (references removed):

Scientifically meaningful representations are compact, discriminative, disentangle underlying factors of variation and encode underlying mechanisms that generalize across numerous tasks.

Though here representation doesn't necessarily refer to generated embeddings (my reading of it, at least), but both to generated embeddings and ways to encode and represent the input.

They suggest that promising approaches to achieve this task are: 1. Geometric deep learning, 2. Self-supervised learning (pre-trained on unlabelled data, then fine-tuned), 3. Language modelling, here they proposed traditional next-token learning, and also masked learning; and the language part refers to learning sequences be it language or molecular strings etc.

Embeddings for Atoms

Embeddings for atoms were inspired by NLP models from the 2010s.

One such example was learning continuous vector representations of words (2013). They proposed an automated mechanism to generate word-vectors by absorbing information from that word's environment (neighbouring words).

In chemistry, embeddings can be used for downstream machine-learning tasks such as materials' property prediction, so they became popular in the field. Materials science has exploited the same ideas, for example:

• properties of an atom can be inferred from the environments it lives in (Atom2Vec, 2018),
• atoms are to compounds as words are to sentences (SkipAtom, 2022),

The surprise was that similar words (or atoms) end up with similar vectors. The vectors also support semantically meaningful arithmetic operations, and became useful for downstream tasks. A classic example was:

vector("Queen") = vector("King") - vector("Man") + vector("Woman")

Both Atom2Vec (2018) and SkipAtom (2022) are unsupervised algorithms that obtain their atom vectors from databases of compounds. Atom vectors can be combined into compound vectors, and used for downstream tasks like property-prediction.

Classifications and Featurisers

The method used to generate our vectors is called a featuriser (we can use a featuriser or create our own). There are many common approaches:

• Simple: like hot-encoded, random;
• Human-designed: Composition-Based Feature Vector (CBFV) which are expert-curated vectors as in Jarvis, Magpie;
• Machine-learnt: embeddings, like SkipAtom.

Atom-vectors can be combined to describe compounds. Examples of combination methods are concatenation into a long vector and pooling of vectors (e.g. summing them up).

Comparing representations

A performance-comparison of vector representations is carried out in "Is domain knowledge necessary for machine learning materials properties?" (2020).

Their conclusion is: human-designed Composition Based Feature Vectors (CBFV like Jarvis and Olyinyk) outperform other methods if there isn't much data. This was prior to SkipAtom, but does include Atom2Vec.

Otherwise, performance in downstream tasks is similar to hot-encoded or random vectors.

(...) Although new, data-driven approaches are of interest, those studied here have yet to surpass CBFVs in terms of material property prediction with small data.

However, "Domain Independent XAI for Material Science" (2025) challenges that conclusion:

Our method challenges this perception: we obtain excellent classifiers that are interpretable and based on a small amount of training data without using any domain knowledge: (...)

They assert that one-hot encoded vectors can still achieve good results using small datasets, as long as the network is designed in the way they specify.

Thoughts

Human-designed vectors are easier to interpret; machine-learnt vectors require less effort but more data to train them.

Can we design machine-learnt interpretable vectors that are intrinsically interpretable? Attention-masks and disentangled representations are closer to this.

Should the representation be just the simplest, and the network learn all that is needed for the given tasks?

Atom2Vec

A popular representation of atoms as vectors appeared in (2018): Atom2Vec.

They take compounds from a database, and build a matrix like the one below:¹

Let's describe using the compound $\mathrm{B}{\mathrm{i}}_2\mathrm{S}{\mathrm{b}}_3$ as an example:

• When $\mathrm{Bi}$ is the target, it generates $(2)\mathrm{S}{\mathrm{b}}_3$, placed in the first column. (2) is the stoichiometry of the element $\mathrm{Bi}$ in the compound.
• When $\mathrm{Sb}$ is the target it generates $(3)\mathrm{B}{\mathrm{i}}_2$, placed in the fourth column. (3) is the stoichiometry of the element $\mathrm{Sb}$ in the compound.

Since a particular atom binds to a very small fraction of all groups, each row is very sparse (high fraction of zeros). The same is valid for columns.

SVD Method

A normalised matrix $X_u$ is obtained by normalising each row vector independently. Using the euclidean norm (2-norm) allows for an intuitive similarity metric:

$$\mathrm{dist}(\overrightarrow{u_1},\overrightarrow{u_2})=1-\overrightarrow{u_1}\cdot \overrightarrow{u_2}=1-\mathrm{similarity}$$

In their best-performing model, they compute $SVD(X_u)=UD{V}^T$, collect the $d$-rows with the largest singular values, and compute $F=U^{\prime }{D}^{\prime }$ where $D^{\prime }$ is the slice of rows of D with the $d$ largest singular values, and $U^{\prime }$ the corresponding columns.

Note

The strategy has certain beauty to it: the new f-vectors retain the inner product similarity but are denser. Though now, the columns have no explicit meaning.

Findings

• Similar atoms have similar vectors,
• Increasing the distance threshold in stages, vectors can be clustered hierarchically, from the leaf-nodes (atoms) downwards (groups).
  • At some level, groups match the periodic table groups. (I don't know how the grouping is made unambiguous).
  • At a very large distance, all atoms merge into a single group. The result is called dendogram.

Image (modified) from Original Paper under CC-BY-SA 4.0. The atoms are rotated to make the image fit (rotated).

• Looking at the variation of some dimensions in the vectors, we can assign meaning to some of them.

Benches

Then, they compared to "empirical features" —a vector (group, period,...), padded to match their $d$— with the task of predicting the DFT-found formation-energies of $\approx 10^4$ elpasolite crystals ($\mathrm{AB}{\mathrm{C}}_2{\mathrm{D}}_6$).

Each solid was represented as a concatenation of atom vectors, and feed it to a hidden layer. (They also do other tasks.)

The paper ends with an interesting insight:

Structural information has to be taken into account to accurately model how atoms are bound together to form either environment or compound, where the recent development on recursive and graph-based neural networks might help.

1. It would be a binary matrix but the database contains some compounds multiple times and those are left duplicated (for some strange reason). ↩

SkipAtom

Atom2Vec was already described; now it's time for SkipAtom, another algorithm to learn atom embeddings.

Method

First, compounds are downloaded. Then, the Voronoi Decomposition is used to derive graphs from unit-cells, and from the graphs generate training-pairs. As they show in the paper:

Image from Original Paper under CC-BY-SA 4.0

Finally, each pair X-Y is used to train a shallow network to predict the target (Y) from the reference (X).

Image from Original Paper (slightly modified) under CC-BY-SA 4.0

• The resulting representation is dense and structured/semantic. This can be shown using dimensionality reduction techniques (PCA, t-SNE,..).
• The architecture is described as:

  (...) single hidden layer with linear activation, whose size depended on the desired dimensionality of the learned embeddings, and an output layer with 86 neurons (one for each of the utilized atom types) with softmax activation. (...) minimizing the cross-entropy loss between the predicted context atom probabilities and the one-hot vector representing the context atom, given the one-vector representing the target atom as input.

Representations of Compounds (Pooling)

The analogy to NLP is that words are like atoms, and sentences are like compounds. Hence, distributed representations of atoms can be combined (pooled) into a vector representing a compound.

Vector-pooling options are:

• sum: $\sum s_i{\vec{a}}_i$ where $s_i$ is the stoichiometry (can be fractional),
• mean: $\frac{\sum s_i{\vec{a}}_i}{\sum s_i}$, i.e. divided by total number of atoms (can be fractional too).
• max: $\max (M_i)$, reduces material matrix $\mathrm{M}$ to vector. Selects max value of each column, each row being an atom in the compound.

The resulting compound representation is then used for training a feed-forward NN on different tasks. Also benchmarked using MatBench.

The pooling can also be done with hot-encoded vectors for atoms. This is done in ElemNet (mean pooling), and in Bag-of-atoms (sum pooling). The advantage: no training required, the disadvantage: the result is a sparse vector, and can be less accurate.

Results

They run two groups of tests:

• Embeddings' quality through elpasolite task. Where the atom vectors are concatenated into a compound vector (no pooling). The vectors train a network for property prediction. SkipAtom performs best here.
• Embeddings quality and pooling methods, through 9 prediction tasks. The results are:
  • Pooling: sum and mean-pooling outperform max-pooling,
  • Kind: Mat2Vec does best, and second SkipAtom,
  • Bag-of-Atoms (sum pool of hot enc) does best in one task.

They conclude that these methods are most useful when no structural info is available.

However, there isn't a simple answer to which representation is best, it depends in the task. This is discussed in more detailed in Results.

Reference-based Coordinate Assignment

RCA is a dimensionality reduction method. Importantly, this implies we already have some representation available, which we may want to reduce.

The main idea could be conceptually represented as:

$${\mathbf{x}}_d=distance({\mathbf{x}}_n;{\mathbf{a}}_n,{\mathbf{b}}_n,\dots ,{\mathbf{d}}_n)$$

On the right hand side, the distance of each $n$ dimensional vector $\mathbf{a}\dots \mathbf{d}$ to $x_n$ is calculated. Each of these vectors is a centroid calculated from a cluster of embeddings.

The result, on the left hand side, is a reduced $d$-dimensional sample vector $x_d$.

An important point is raised in the paper:

To avoid degeneracies arising from equidistant configurations, the reference points must span the structure of the dataset.

Which means the reference vectors (centroids) should be able to reach any point in the original dataset, and not lose expressivity.

The method has useful properties:

1. Compared to a large vector, it is cheaper to use to train a model (and can still be accurate),
2. Compared to other dimensionality reduction methods, it is easier to extend,
3. With certain care, the dimensions may be interpretable.

Comparison

SkipVec and Atom2Vec were previously discussed.

The paper Is domain knowledge necessary for machine learning materials properties? compared descriptors (same as vector representations) generated in different ways for downstream tasks.

They find hand-crafted descriptors useful for small and large datasets, but these are cumbersome to create —expert knowledge is required. One-hot and random-vectors perform similar to hand-crafted descriptors in large datasets.

With that, it seems wise to use hand-crafted descriptors for small datasets, and learnt, one-hot or random for larger ones.

SkipAtom evaluates on different approaches and tasks, and finds their method outperforms one-hot and random-vector, but does not test hand-crafted ones.

SkipAtom's comparison of representations is discussed below.

Simple classification

We have a simple classification of the available methods:

1. Human-engineered vectors;
2. Low effort vectors: one-hot encoded (ElemNet), random, Atom2Vec;
3. Machine-learnt vectors (SkipAtom).

Atom2Vec may be in category 3 above; however, it is not an optimisation that creates them, it is a matrix factorisation.

Quality of Atom Representations

ElemNet (One-hot), Random, Atom2Vec, Mat2Vec and SkipAtom compared.

The atom-vectors were concatenated into compound representations, and these used to predict elpasolites (compounds) formation-energy.

SkipAtom outperformed other methods.

Pooling approaches

Vector pooling strategies were compared through 9 prediction tasks; 5 regressions, 3 classifications and OQMD Formation Energy prediction (also a regression).

• OQDM: bag-of-atoms, which is sum-pooling hot-enc atom vectors, is best.

The rest is summarised well in the paper:

(...) the models described in this report outperform the existing benchmarks on tasks where only composition is available (namely, the Experimental Band Gap, Bulk Metallic Glass Formation, and Experimental Metallicity tasks). Also, on the Theoretical Metallicity task and the Refractive Index task, the pooled SkipAtom, Mat2Vec and one-hot vector representations perform comparably [to the SOTA], despite making use of composition information only.

And an interesting observation:

The ElemNet architecture demonstrated (...) Perhaps surprisingly, the combination of a deep feed-forward neural network with compound representations consisting of composition information alone results in competitive performance when comparing to approaches that make use of structural information.

Use cases and limitations

Training does not rely on labelled data (unsupervised learning).

The model just needs the formula at inference time, and does fine with non-stoichiometric solids. So having the material's composition —but no structural information— we can still calculate some properties.

Similar compounds have similar vectors, which is useful. But without structural information, all isomers have the same vector, which is a limitation.

It is computationally cheap, and can help screen large number of compounds as a first selection step.

Best Practices

TL;DR from the paper "Best practices in machine learning for chemistry" (2021), a very similar paper by the same authors is "Machine learning for molecular and materials science".

• For Datasets
  • Ensure dataset remains available, and is version-tagged (they change)
  • For home-made or mixes, explain the process of generation
  • Describe any data curation, balancing, augmentation, and so on.

In the Best Practices paper they say:

For reasons of reproducibility, it is crucial that these databases use some mechanism for version control (e.g. release numbers, Git versioning, or timestamps) as part of the metadata and maintain long-term availability to previous versions of the database.

• For Representations

  • Try more than one, compare
  • Use very basic ones as baseline representation to compare (example random or one-hot)

• Justify Model

  • More complex isn't always better
  • Compare to baselines (mean for regression, most common class for classification)
  • Compare to very simple models and to SOTA
  • Any interpretability we can offer?

• Evaluate Model

  • Have 3 separate datasets: for training and optimising, for evaluating during training and detect overfitting, and testing for testing which should represent where it will be applied (should test what we want it to succeed on).
  • Test extrapolative learning: leave out some class entirely, or train until a cutoff date and evaluate with dates after that.
  • Test intrapolative learning: with varied test sets
  • Mindful of shorcut learning (have varied test dataset).

• Reproducibility: Results and code must be made available and reproducible

They also state:

In all reports, remember to cite the methods and packages employed to ensure that the development community receive the recognition they deserve.

They provide a great checklist but since the license is a mess I am not including it here.

Example

Take ElemNet as an example and go through the checklist.

• Database:
  • They provide a link, but no timestamped or git version,
  • No info on curation or preprocessing (we may assume none was performed).
• Representations:
  • They justify and compare the results to baselines.
• Model:
  • Describe why is new and useful idea,
  • Describe architecture.
• Evaluation:
  • Show training and different hyperparameters,
  • Studied which compounds model is accurate vs not.
• Reproducibility: Results and code are available.

The comparison would be improved if they also ran a deep learning model with human-made descriptors.

They also included other useful statistics like inference time.

Machine learning for molecular and materials science

Very compressed bits from the paper Machine learning for molecular and materials science (2018).

Representations of atoms, molecules, materials

The process of converting raw data into a format more suitable for an algorithm is called feature engineering.

The more suitable the representation of the input data, the more accurately can an algorithm map it to the output data.

Selecting how best to represent the data may require insight into both the underlying scientific problem and the operation of the learning algorithm, since it is not always obvious which choice of representation will give the best performance.

Examples

• Coulomb Matrix: atomic nuclear repulsion information.
• Graphs: Connectivity of molecules.
• String representations: SMILES, SELFIES,..
• Solid-state unit-cells: Representations based on radial distribution functions, Voronoi tessellations, and property-labelled materials fragments (...)

  In the solid-state, the conventional description of crystal structures by translation vectors and fractional coordinates of the atoms is not appropriate for ML, since a lattice can be represented in an infinite number of ways by choosing a different coordinate system.

List of a few ML algorithms

Naive Bayes, Nearest Neighbour, Decision Trees, Kernel methods, a family including support vector machine (SVM) and kernel ridge regression (KRR). Artificial neural networks (ANNs) and deep neural networks (DNNs).

Research and Data Objects

FAIR Principles

The FAIR Guiding Principles are suggestions for improving data reusability. Also reproducibility and transparency. Here the essence of it is described.

The idea

The paper states:

[good data management] is the key conduit leading to knowledge discovery and innovation.

We can imagine science as an activity of "solving the puzzle of knowledge together". Theories, observations, workflows, processing pipelines and algorithms, software and other data are the puzzle pieces. If the piece is (each term only briefly summarised, oversimplified):

• Findable: there is some place to query for it,
• Accessible: the object is explorable, downloadable,
• Interoperable: uses a common format rather than private,
• Reusable: contains permissive usage policies / license such that it can be reused in future research.

Benefits

Sharing data objects has many benefits:

• Researchers would earn citations through publishing data-objects.
• Software developers can also get recognition for their published data-objects;
• Peer-researchers can then test and reuse the results, while enhancing data-mining and have other areas.

And so forth.

For humans and machines

Different from other frameworks, FAIR is concerned about how non-human agents and tools would discover, retrieve and be able to use the data.

Sharing and finding data objects

Common repositories used for scientific objects are of two kinds:

• Special-purpose: World Wide Protein Data Bank (wwPDB), NOMAD and Materials Cloud (for materials)
• General-purpose: FigShare, Zenodo (CERN), GitHub.

Important

GitHub does not produce DOIs nor guarantees that the item is preserved (it can easily be deleted).

Tip

1. It's important to either inspect or create (for each case above) labels, description, license and other metadata.

2. There is a registry of research data repositories—re3data which is a useful starting point for finding not datasets or software, but a repository (and metadata such as where it's located, who controls it and so forth)

Citing Objects

The recommended practice is to link a dynamic DOI (that tracks versions), especially for software or datasets.

Zenodo handily combines a concept and version DOIs, mapping to the software project and version, respectively.

There are two issues, one on each side of the citation problem:

1. Developers are burdened to update the DOI's version, say in the README.
2. Researchers cite a paper DOI rather than dataset / software appears, or simply name the software.

To solve 1. a 2019 paper analysing citations of Zenodo objects suggests:

[recommended identifier] ideally represents the whole software project, which is less prone to impermanence. This recommendation does not require maintenance effort on the developer (...)

This is not the "recommended practice" of citing the version as well, but it is a first step. The next step would be to use and update the software DOI version.

Thoughts on non-paper citations

A clear limitation is that software dependencies don't get any credits, only the top level.

It seems common to also add a citation file format (file.cff) to GitHub repos which help when linking repository to Zenodo. Then we must add the Zenodo recommended citation choosing version or concept DOI, as described earlier.

Visualising High Dimensional Data

• PCA
• Dimensionality Reduction
• t-SNE

Reference of some of the techniques, book cited in some papers: 18. Hastie T, Tibshirani R, Friedman J (2001) The Elements of Statistical Learning, Springer Series in Statistics (Springer, New York).

Queries

• How to build useful machine-representations of atoms?
  • $R^1$ would just be a 1D vector embedding, likely of little use. But are there taxonomies of representations (including a matrix per atom?)

Missing

• Splits for datasets, validation / testing.
• Balancing datasets.
• Evaluation metrics, loss functions,
• ROC classification performance metric

Ideas

• Train and show the results of atoms for vectors in a website? With some button to load each dataset?

• List of examples of successful applications of ML in chemistry

  1. Anything for which there are useful datasets from experiments or from calculation:
     • orbital energies,
     • geometries,
     • formation energy,
     • energy band-gap

• Which models are related to "Causal Inference"? Are they useful? For example The Mythos of Model Interpretability states (removed a reference which is to Judea Pearl's Causality book, and may be worth checking):

The task of inferring causal relationships from observational data has been extensively studied. Causal inference methods, however, tend to rely on strong assumptions and are not widely used by practitioners, especially on large, complex data sets.