Notes

These are my thoughts on topics related to AI, XAI, AI for chemistry.

Organisation

• The top-left hamburger icon toggles the table of contents.

• At the bottom of posts there is a "Sources" which includes papers I read during the write up.

License

The prose is all under CC BY 4.0, external content is linked and if I find the license it is explicitly stated (in image captions, for example).

Tools Used

• mdbook,
• Krita to edit images,
• Computer Modern font or Ubuntu > System UI > sans-serif, in that order of preference.

Explanations

This post describes what explanations are, and forms a basis for understanding how to explain artificial intelligent systems.

Definition

What is an explanation? There are many definitions. Here is one, from "How People Explain Action (and Autonomous Intelligent Systems Should Too)" (2017):

Explanation is arguably a three-value predicate: someone, a communicator, explains something to someone, an audience. The success of an explanation therefore depends on several critical audience factors—assumptions, knowledge, and interests that an audience has when decoding the explanation.

The definition of "Explanation" given above is unclear regarding what "explains" means. We could just as well define "Interview" as "Someone interviews someone about something". Though even in this vague form, it still highlights an exchange between two agents.

Inspired by Explanation in artificial intelligence: insights from the social sciences, this post defines "explaining" broadly as:

Explaining

A two-step process involving 1. the generation of explanatory hypotheses (cognitive process) and 2. the communication to an audience (social process).

Usually, one best hypothesis may be selected until contradicted by experience, superseded by a simpler one, or shown to be inconsistent with prior knowledge.

The process may repeat and update during the interaction. Sometimes it's during an explanation that we find errors in the understanding. Hence, explanations can provide understanding! Also, the explanandum (that which is to be explained) may be refined just as a photo-camera may gain focus with increased exposure.

Our understanding is reflected in the hypothesis formed in the cognitive process: Understanding is having a theory, hypothesis or model about how something works (cognitive process). It's also frequent that there is an illusion of understanding, and explaining or forcing predictions may make the illusion more evident.

Note

This post is mostly jargon-free. Technical articles about topics such as abductive inference are linked in the "Sources" at the bottom of this post. A brief discussion of logic inference in AI is given in this article by Gordon Brander.

Cognitive Process

Contrastive Questions

Research has shown that why-questions are usually contrastive. That is, they are phrased as Why P rather than Q? instead of simply Why P? Usually P is the real case (or fact) and Q the expected case (or foil), which may also be implicit.

As the paper Beware of Inmates Running the Asylum states:

For example, explaining "Why did Mr. Jones open the window?" with the response "Because he was hot" is not useful if the implied foil is Mr. Jones turning on the air conditioner, as this explains both the fact and the foil; or if the implied foil was why Ms. Smith, who was sitting closer to the window, did not open it instead, as the cited cause does not refer to a cause of Ms. Smith's lack of action.

The foil focuses the explanation on the differences between the two cases (ignoring similarities). This is usually easier to explain than the standalone fact. It can also reduce confusion.

Hesslow states this idea in a concise way:

What I want to suggest, then, is that the explanandum should be construed as a relation which involves three things: an object a, an object of comparison b and an explanandum property E which a has and b does not have.

The complexity, of course, lies on knowing which differences matter.

Attributing Causes

As Miller et al. state:

Attribution theory is the study of how people attribute causes to events; something that is necessary to provide explanations.

We never provide a full causal chain (it is endless), but a short-enough one that explains the event in question (this is the causal selection problem).

Researchers have pointed out many heuristics used by humans to favour some candidate causes (causal hypotheses) over others: proximal over distal events (in the causal chain of events); abnormal or unexpected events; controllable events, deviation from theoretical ideals, model, predictive power, responsibility, and so forth.

But those are taken care of by contrastive why-questions which compare the event to be explained to a reference case (particular instance or general case). In this regard, Hesslow states (bold is mine):

Many of the selection criteria listed in Section 3 can be construed as the result of choosing different objects of comparison or reference classes. Let us consider again the fire in the barn, and let us suppose that we have in the back of our minds the picture of a normal barn. (...) the normal barn has not caught fire, it follows that an explanatorily relevant condition for this barn's catching fire must be abnormal. Thus, selection of abnormal conditions can be viewed as the result of comparing the explanandum object with a normal object.

And also most other causal selections are contained:

(...) the difference between this barn now and this barn yesterday, i.e. we would be selecting a precipitating cause [proximal in the list above]. Selection of the unexpected may be viewed as the result of explaining the difference between an expected and an actual outcome. Selection according to responsibility follows from a comparison between actual and morally ideal behaviour. Selection of conditions which cause a deviation from a theoretical ideal involves a comparison between an actual and a theoretically ideal situation, and so on (cf. Hesslow, 1983).

Here is yet another illustration by Hesslow, of how contrasts cases narrow down possible causes:

For instance, if we want to explain why the fly Ml has shorter wings than Nl, then the temperature in which the flies were raised is explanatorily irrelevant, since the temperature was the same in both cases. The mutated gene on the other hand was present in one case and absent in the other.It is, therefore, explanatorily relevant.

Pragmatism

Notably, accuracy may not be preferred in an explanation; rather, usefulness, simplicity, generality and consistency with prior knowledge are.

Many of these results come from work by Tania Lombrozo. (This section will eventually be expanded.)

Social Process (Communication)

We have gone through the cognitive process and how contrastive questions can aid the generation and selection of a hypothesis or a cause. The second process is that of commucation.

The communication can be aided by the gricean maxims: rules of effective communication.

• Informative (Quantity): right amount of context and details,
• Truthful (Quality, or Fidelity): the explanation should be true,
• Relevance (Relation): avoid presumed-known or superfluous details, focus on what provides insight,
  • One example given earlier is to focus on unexpected events, whilst ignoring what is presumed to be known by the listener.
• Manner (clarity): express it in elegant terms.

In some cases, humans also tend to prefer concrete over abstract explanations, so "concreteness" could be added to the list.

Relevance is primarily related to the causal selection problem in relation to an audience, as Malle et. al., state:

How do people solve this problem? They determine what exact question the audience is interested in (McClure and Hilton 1998); they take into account what their audience member already knows (Slugoski et al. 1993); and they offer elements of explanations that build bridges between presumed knowledge and novel information (Korman and Malle 2016). In short, they offer explanations that generate coherence in a knowledge structure of old and new information (Thagard 1989).

Contrastive explanations can also take care of many of these aspects automatically, by selecting a contrast that is relevant or understood by the audience.

Metaphors: The Machine and The Agent

Humans often use "explanatory stances" to explain events, as noted by Daniel Dennett. There are three common ones:

1. Mechanical stance (which I call "Machine Metaphor/Model"),
   • Explain outcomes by considering the parts of a system, what they do and how they interact (that is, a mechanism).
2. Design stance, this has different interpretations. One is of the perceived purpose of something (applied to things created by humans such as tools, but also those hypothesised to be created by universal designer or god).
3. Intentional stance (which I call "Agent Metaphor/Model").
   • Explanation uses goals, motives, feelings, intent to explain actions and/or behaviour.
   • Unintentional behaviour/events is usually explained using the machine metaphor (see the following paper, section Ordinary Behavior Explanation).

They can be complementary when applied to the same phenomena or as Ruth Byrne puts it:

Notably, each explanatory stance can be applied to explain the same device or action, but they have different consequences for understanding it. Each stance can lead to different kinds of insights, and to different kinds of erroneous inferences. The atypical application of a particular stance, say, a mechanical stance to explain an action more typically understood from an intentional stance, such as explaining travelers in a crowded airport as like pinballs careening around a pinball machine, may be interpreted analogically to yield new inferences [Keil, 2006].

In technical fields, many complex systems are conceptualised as machines: composed of parts, each with a function, a role. Many are also conceptualised as graphs.

Ordinary people conceptualise certain kinds of complex systems as humans or agents (wholly or in part). This may happen with systems using human language or behaving autonomously, but other times it is due to pragmatic reasons. They would use and expect the kind of explanation a human would give, if there were one.

What seems here most fundamental than the particular stances is the selection of a metaphor to structure thinking and obtaining insights.

Other metaphors and analogies could be proposed for specific problems.

Similar ideas can be found in "How People Explain Action (and Autonomous Intelligent Systems Should Too)":

For those intentional agents, we hypothesize, people will apply the same conceptual framework of behavior explanation that they apply to humans (...) a subset of AIS that people do not regard as intentional agents; and for those, they may apply a purely mechanical explanatory framework.

And more recently, in Good Explanations in XAI:

People may tend to adopt multiple stances in their preferred explanations of an AI decision support system and its decisions, not unlike their tendencies in interacting with social robots [Clark and Fischer, 2023]. People are aware that a social robot is a machine, but interpret it as a depiction of a character, not unlike a ventriloquist dummy, and engage with it in pretense of interacting with the depicted character [Clark and Fischer, 2023]. Similarly, they may be aware that an AI decision support system is an algorithm but they may interpret its decisions as a depiction of those provided by a human, e.g., a bank loan assessor, or the organization the human represents, a bank. Hence, an intentional stance and a design stance may both be useful in different contexts for explaining how automated agents behave [Veit and Browning, 2023].

We can summarise some of these ideas (including a standard audience) in a brief table:

The post on explanatory stances continues this line of reasoning and connects them with how we explain humans and deep learning models.

Explanatory Stances

This post explores the question: How should Deep Learning Models be explained?

In the previous post explanatory metaphors, or as Dennett calls them "stances" were described. The summary is:

1. Mechanical stance (which I call "Machine Metaphor/Model"),
   • Explain outcomes by considering the parts of a system and how they interact (that is, a mechanism).
2. Design stance,
3. Intentional stance (which I call "Agent Metaphor/Model").
   • Explanation uses purposes (teleology), goals, motives, feelings, intent to explain actions and/or behaviour. It assumes a "state of mind" of the object and uses the model to predict behaviour.
   • Unintentional behaviour is usually explained using the machine metaphor.

The selection of a stance is a relevant aspect to consider when generating explanations. An example is given by byrne_human_explanations:

A potential implication for XAI is the necessity to consider when a particular stance is appropriate for explaining an AI system's decision. A question posed by a user as to why an AI system refused their loan may be a request for causal information of a mechanistic sort about how the AI system came to make that decision. Hence, it may require an explanation based on information about how the user's input features relate to a training data set, e.g., that loans have been refused for applicants of similar salary level, occupational status, credit history, as the user. Alternatively, it may be a request for functional information of a teleological sort about the purpose of the AI's decision. Accordingly, it may require instead information about the goal and consequences of the output, e.g., that decisions of this sort mitigate the risk of applicants defaulting from repayment. Each sort of explanation will lead users to develop a different understanding of aspects of an AI system, and impact their learning, satisfaction, and trust in it.

Next, a few of these stances are explored in different contexts.

Human To Human Explanations

When humans explain phenomena to each other they use language, pictures and other media. But we don't get access to each others' brain processes (to a large extent, not even our own); and even if we did, the task wouldn't be trivial.

A brain may use "communication processes" that look at "decision processes" (see Marvin Minsky, The Emotion Machine). Then the statement is exchanged and interpreted by another agent. The statement can be tested against reality, or for inconsistencies or compared to other statements.

In other words, we largely treat brains as black boxes.

Should we treat DL models the same, and accept predictions without "peeking into the black box", just as we do with humans? If so, under which circumstances?

The idea of this post is that there will be a level of complexity and utility at which this will also happen for DL models.

DL Models are Scientific Models

DL models are mathematical functions. Both the architecture and the trained model can be considered scientific models or theories. In this case, they should be explainable. Here there is an underlying metaphor of "DL models as machines" and we expect them to be explainable in terms of their parts and how they interact.

There are also societal reasons for this expectation: many models are used on decision making and high-stakes scenarios, so we expect explanations or justifications for their decisions, just as we expect from humans making decisions.

For simple or narrow-AI models, we may expect to peek into the black box, and to analyze their input-output relations to get insight into their inner workings.

They're expected to be decomposable in domain-level concepts (just as a theory is).

In other words, we want to answer: What did it learn? Which concepts is the network using for their decision making? But how can we extract their "baked in" knowledge?

But, what if the metaphor used for a complex DL model is that of a scientist, or a human, rather than a machine?

DL Models are Scientists

General-purpose models may explain decisions in a human-like way, and be as hard to inspect as the brain is. Treating these models as a scientific theory of a domain, or a narrow AI model, may be a counter-productive, and even a category error.

In this case, a detailed explanation process could be as hard to extract as it is from a brain.

So what do we do? In this case, it may be better to expect what we expect from a human: Take an explanation in natural language and test it and compare it to other established statements.

This is what the paper [The Mythos of Model Interpretability][mythos] describes, in the section about post-hoc textual explanations:

Humans often justify decisions verbally. Similarly, one model might be trained to generate predictions, and a separate model, such as a recurrent neural network language model, to generate an explanation.

In other words, we would treat it as a black box.

Summary

We have explored two metaphors "DL models as Agents / Scientists" and "DL Models as Scientific Models", and the expectations for each case:

1. As Scientific Models: We can study them as black boxes, and also internal mechanisms,
2. As Scientists: We can only study them as black boxes,
   1. This includes Agents that can output a certain explanation (e.g. humans).

As science advances, some models may become explainable. Similarly, intrinsically explainable architectures may be designed, and could be preferred over black box architectures (e.g. Capsule Networks were an effort in this direction).

Now we go back to common methods used to explain mostly narrow AI algorithms.

Explainable AI

Explanations were defined and characterised in explanations. This post explores the connection of explanations to deep learning models.

Model Explainability

Explainable AI (XAI) is primarily about explaining machine and deep learning models and their outputs. In this blogpost, explainability and interpretability are considered synonyms. Model explainability can be defined as:

The degree to which we can answer questions a model's predictions and inner workings. The answers are context and audience (including ourselves).

Another definition is found in "Why Should I Trust You?":

By "explaining a prediction", we mean presenting textual or visual artifacts that provide qualitative understanding of the relationship between the instance's components (e.g. words in text, patches in an image) and the model's prediction.

Below, a few types of model explainability, namely Intrinsic and Extrinsic¹, Local and Global, are explained. The quote just above is closer to our "Extrinsic Explainability".

Global and Local Explanations

• Global: valid for all inputs (explains the model) e.g. by combining local explanations.
• Local: for specific inputs-outputs.

Intrinsic Explainability

Looks at the internal mechanics, at the roles of layers, neurons, weights; it may also relate to constraining the model in form (e.g., Rudin C. or Zachary C.) —that is, imposing physical constraints, inductive biases, causal inputs selected by experts, monotonicity, sparsity, constraining model size or computational complexity. Or as Zachary C. puts it:

Sufficiently high-dimensional [linear] models, unwieldy rule lists, and deep decision trees could all be considered less transparent than comparatively compact neural networks.

Transparency is domain-dependent. For example, the field of geometric deep learning can express constraints (or inductive biases) of connectivity related to molecules and materials (e.g. GNNs), making them more interpretable than other network architectures for representing this particular input / physical problem. This can be further extended to symmetry and other aspects.

Extrinsic Explainability

From Zachary's paper:

(...) opaque models can be interpreted after the fact, without sacrificing predictive performance.

This is what this post calls Extrinsic Explainability.

On top of the previous quote, the paper "Why should I trust you?" has an explainer desiderata: it should give a qualitative understanding between inputs and outputs, be easy to understand, and be model agnostic and locally faithful (a good fit to the original model in the vicinity of the instance being explained). In that paper, SP-LIME combines local explanations to provide a global explanation of the model.

Rudin argues that these simpler explanation models² must be wrong. If it is perfectly accurate, then we don't need the original model. Rudin proposes calling these model-approximation techniques "summary of predictions", "summary statistics" or "trends".

On the other hand, methods such as SHAP, LIME, t-SNE, can provide some understanding of the model, even if using approximations. Some of those popular methods are explained in strategies.

Brief Aside: Neural Netwoks

This post assumes a working idea of what deep learning models or neural networks are. A simple definition is provided in the paper Can we open the black box of AI? (Section "Good Trip").

But what exactly do these networks learn? "Scientific discovery in the age of artificial intelligence" states that:

[AI methods] includes deep representation learning (Box 1), particularly multilayered neural networks capable of identifying essential, compact features that can simultaneously solve many tasks that underlie a scientific problem.

So a key aspect of understanding and explaining will be to decode those "essential, compact features" into domain concepts. What concepts, if any, are stored there, in the synapses?

It's also useful to have in mind a general idea of where are neural networks models being used, and how:

1. Domain-specific (Narrow AI): these are small or large models but trained on a specific domain (protein folding, generating new molecules, predicting spectra and so forth). These models benefit from XAI, inductive biases and constrains, and would ideally be interpretable.
2. Domain-general (Foundation Models): there is a spectrum between networks trained for a task in a domain, and for a whole domain (e.g. chemistry). These models are usually very large, pre-trained in some unsupervised way and then need to be fine tuned to specific tasks, where they reuse the learnt building blocks.
3. AI agents: These are clusters of models working together to carry out many parts of the scientific process of discovery (hypothesis generation, reading literature, suggesting experiments and running code simulations etc.) In some cases they may also have access to robotics platforms and run real world experiments. The difference to other approaches is that these models are reasoning, and to some extent work like a team of scientists.

Here we are concerned with 1. primarily, and with the possibility to explain them, design them such that they are interpretable and finally understand them better.

Trade-offs?

We may expect model explainability to be inversely correlated with model complexity or accuracy. Graphically:

Hypothesis: Model explainability v. Accuracy tradeoff.

And in "Why Should I Trust You?" (refs removed):

Recognizing the utility of explanations in assessing trust, many have proposed using interpretable models, especially for the medical domain. While such models may be appropriate for some domains, they may not apply equally well to others (...). Interpretability, in these cases, comes at the cost of flexibility, accuracy, or efficiency.

And in [SHAP][shap]:

However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability.

Other researchers such as Rudin disagre (bold is mine, references were removed):

Two obstacles to using interpretable models are that they are harder to optimize because they require extra constraints, and there is an incorrect perception that they are less accurate than black boxes. On the first point, the community is getting quite good at building interpretable sparse models and interpretable neural networks. On the second point, there is no scientific evidence that accuracy must be sacrificed when adding interpretability constraints.

Rudin's more detailed paper states something similar:

There is a widespread belief that more complex models are more accurate, meaning that a complicated black box is necessary for top predictive performance. However, this is often not true, particularly when the data are structured, with a good representation in terms of naturally meaningful features.

I'd make two comments to the quote above. First, good representation in terms of naturally meaningful features may be hard to obtain or create. Second, NNs tend to perform better as we scale them up. Though there is some "optimal-size region" and going beyond could plateau or even decrease its performance.

For complex tasks (Natural Language Processing, Computer Vision), DL models surpass most other algorithms. For narrower tasks, it is sometimes possible to find interpretable models that are also very accurate (benchmarks?), but they can be very hard to design, making the time-risk-benefit tradeoff worth considering:

Interpretable models can entail significant effort to construct, in terms of both computation and domain expertise. (...) for high-stakes decisions, analyst time and computational time are less expensive than the cost of having a flawed or overly complicated model. (...) The researcher needs to create a model that has the capability of uncovering the types of patterns that the user would find interpretable, but also the model needs to be flexible enough to fit the data accurately. This, and the optimization challenges discussed above, are where the difficulty lies with constructing interpretable models.

Out of Distribution

Consider an imaginary model $y=f(u)$, $f$ being the model, $u$ being the proportion of people with an umbrella and $y$ the probability of rain. The model reaches low evaluation error and everyone is happy.

However, the model consistently fails to predict rains when people didn't take the umbrella. Why could this happen? Some of the reasons below were inspired by the paper "The Mythods of Model Interpretability":

1. The model undefitted the data, and we may need a better model.
2. The dataset is not representative the deployment environment, and the model can't generalise out of training distribution. Can it be fixed if we don't have those datapoints? Were there simply wrong datapoints, that led the model in the wrong direction? Can we create synthetic data?
3. The approach itself was incorrect: we use variables that promote association rather than causation.

Selecting possible causal variables, such as pressure and temperature, rather than the fraction of humans carrying out an umbrella, could help to make it more accurate, and even more explainable. But does it have all the causal inputs? Why do we expect it to work out of distribution, though?³

A subset of causal-variables may do for a good-enough approximation, and even generale well out of distribution. In some cases though, it may be enough to have a correlation model, but they should be distinguished.

Selecting those variables is not very easy, though. An expert must pick known causes-effects pairs as inputs-outputs to train a model, but others may unknowingly build a correlation model instead.

It is hard to predict whether a model will work out of distribution without knowing what it has learnt. Knowing what a model has learnt is part of the XAI discipline, both opening the box, or carefully comparing its outputs.

Similarly, this two-page comment by Cynthia Rudin highlights the preference for interpretable (transparent) models in high stakes scenarios.

In Deep Learning Models, the problem constraints can be used to add inductive biases or priors to architectures, such as symmetry constraints, connectivity (say through graph networks). This may also reduce the amount of training data needed, improve generalisation and improve interpretability.

An idea related to "Out Of Distribution" inference is that of "Transfer Learning": If a model has learnt "essential, compact features" then they should generalise to other task, as stated in Scientific discovery in the age of artificial intelligence (references where removed):

Self-supervised learning (Box 1) has enabled neural networks trained on labelled or unlabelled data to transfer learned representations to a different domain with few labelled examples, for example, by pre-training large foundation models and adapting them to solve diverse tasks across different domains.

This is especially useful when models can leverage large amount of data, which is usually in the form of unlabelled data (there are also mechanisms to label data semi-reliably).

Another promising path towards better generalisation is that of Causal AI. As "Scientific discovery in the age of artificial intelligence" puts it:

Although many scientific laws are not universal, their applicability is generally broad. Compared with state-of-the-art AI, human brains can better and faster generalize to modified settings. An attractive hypothesis is that this is because humans build not just a statistical model of what they observe but a causal model, that is, a family of statistical models indexed by all possible interventions (for example, different initial states, actions of agents or different regimes). Incorporating causality in AI is still a young field

Model Insights from Comparisons

How many ways do we have to make comparisons? Probably dozens. Analogies, metaphors, counterfactuals, a reference case (opposite or similar), a prototype or class-assignment (generalisation uses comparison).

Counterfactuals What would have happened with an alternative input (a hypothetical case counter to the fact). It's most informative to use the minimum changes that change an output class. They are also similar to What ifs (as the question shows).

Counterfacturals and other comparisons can help to explain models without opening the box.

For a model, counterfactuals are yet another inference from another input, but the comparison is helpful because that is one way humans understand things. We can use them as a proxy to "understand how the model is thinking" (that is, by comparing results or inferences).

In a similar fashion to counterfactuals, we can compare with reference inputs.

Overview of methods

There are many methods to identify causes or relevant properties on models, that help explain how they work. Some of them include counterfactuals and comparison, in the same sense as used in our previous section.

For all audiences, we can group these methods into more general categories, and then go into specific cases for a certain audience.

Kinds of Methods

The survey Principles and practise of explaining ML models includes a table of method kinds. A modified version of the table is below:

A method not listed there are text explanations, which can be generated from an RNN or a language model, reading the model's internal state (for example, this can generate captions).

We should remember that:

Relying on only one technique will only give us a partial picture of the whole story, possibly missing out important information. Hence, combining multiple approaches together provides for a more cautious way to explain a model. (...) At this point we would like to note that there is no established way of combining techniques (in a pipeline fashion),

In the next posts, we focus on methods that aid causal attribution (or cognitive process) with a scientific audience in mind.

Map of XAI

An interesting map of XAI is given in the survey Principles and practice of explainable ML (2021).

Most classic ML models are in the dashed area under Model types column.

Classic ML models are usually transparent (intrinsically explainable) but may benefit from post-hoc (post training) explanations, such as visualising it. When transparency is key and the predictions are accurate enough, these may be preferred over DL models.

Image from paper under CC-BY

To the visual explanations, t-SNE, PCA and other dimensionality reduction techniques can be added.

The focus here though, is explaining deep learning models which are often, but not always, more accurate than classic ML models.

1. Intrinsic explainability is also called "Transparency", "Inherently interpretable models"; Extrinsic explainability is also called "black boxedness", post-hoc explainability, opaqueness. ↩

2. In their words : "We introduce the perspective of viewing any explanation of a model’s prediction as a model itself, which we term the explanation model." and also "Instead, we must use a simpler explanation model, which we define as any interpretable approximation of the original model.". ↩

3. Could metaphors and analogies (from experience) be the missing ingredient of this to succeed? Could using causal models help to overcome these problems? How can we make a model that uses analogies? ↩

Additive Feature Attribution Methods

This post explores the "Additive Feature Attribution Methods" class of extrinsic explainability methods (where the reference model's internals aren't analysed).

What are AFAMs?

Additive Feature Attribution Methods (AFAMs) approximate a prediction of the original model ($f$) with an explanation model ($g$) which is a linear addition of binary features (making it simpler and interpretable).

Mathematically:

$$f(x)\approx g(z^{\prime })={\varphi }_0+\sum _{i=1}^M{\varphi }_i{z}_i^{\prime }$$

Each ${\varphi }_i\in R$ is an effects of a binary feature $z_i^{\prime }\in \{0,1{\}}^M$ in the output. The different methods in the class estimate ${\varphi }_i$ differently.

Note

Explanation Model

[In their words][shap_values] : "We introduce the perspective of viewing any explanation of a model’s prediction as a model itself, which we term the explanation model." and also "Instead, we must use a simpler explanation model, which we define as any interpretable approximation of the original model.".

Finally, just for completeness:

Note

The AFAM class was identified in the paper A Unified Approach To Interpreting Model Predictions.

SHAP

Linear LIME, DeepLIFT and other methods calculate ${\varphi }_i$s differently, in turn yielding different coefficients.

The Unified Approach to Interpret Model Predictions proposes that models should have local accuracy, missingness, consistency which, they argue, lead to coefficients that are more intuitive for humans.

The terms are defined as:

• Local Accuracy: There must be equality when the input is the original one ($x$), that is $f(x)=g(x^{\prime })$.
• Missingness: If the reference vector ($x^{\prime }$) has a "missing" component ($x_i^{\prime }=0$) then the feature must have no impact, that is ${\varphi }_i=0$.
• Consistency: if one of two models is larger just turning feature $i$ on and off, then it must have a larger ${\varphi }_i$.

Their Theorem (Theorem 1) guarantees ahat a linear explanation model plus the 3 requirements leave Shapley Values (a result from game-theory found by Shapley) as the best (and unique) coefficients. Other methods violate some of these 3 properties (so the authors modify them to comply).

SHAP (SHapley Additive Explanations) Values are the Shapley Values of a conditional expectation function of the original model: $f(h_x(z^{\prime }))=\mathbb{E}[f(z)|z_S]$ (Section 4, see Figure 1). $S$ are non-zero indices.

• For example, with $\vec{z}=⟨v_1,0,v_2⟩$ then ${\varphi }_3=\mathbb{E}[f(z)|z_{1,3}]$. So the Shapley values are the change in the expected model prediction when conditioning on a feature.

Note

The most accurate Shapley Values are expensive to calculate. Approximations can be used in some cases to speed this up.

Approximating SHAP Values

The approximations can be model agnostic: Shapley Sampling Values, Quantitative Input Influence, Kernel SHAP; or they can be model-specific: Max SHAP, Deep SHAP.

The actual approximations are:

Approximation 1: Feature independence (implies non-multicollinearity).

• Shapley sampling values method,
• Quantitative Input Influence,
• Kernel SHAP (requires both assumptions).

Approximation 2, model linearity: Shapley regression values.

• SHAP provides both global (average across inputs) and local (for a given input).

Kernel SHAP

Can we modify Linear LIME's Loss function so that the values of coefficients found are Shapley values? Yes! These are also more intuitive to humans, and remove some of the heuristics (kernel selection and complexity metric) of it.

The Linear LIME quantities of proximity kernel (${\pi }_x$), complexity penalty ($\Omega (g)$) and Loss ($L$) are turned into a shapley kernel, $\Omega (g)=0$, and the same weighted loss.

The SHAP values / coefficients for this linear model with assumed-uncorrelated features can be estimated by weighted linear regression.

This method is called Kernel SHAP, and it's fast to compute.

LIME and SP-LIME

The paper "Why Should I Trust You?": Explaining the Predictions of Any Classifier proposes the Local Interpretable Model-Agnostic eXplanation (LIME) and the Submodule Picking LIME.

LIME isn't a particular model but rather a set of characteristics that explainable methods should have, according to the authors (more on this later).

Let's first look at LIME as implemented in a concrete case.

Why would this be useful? A concrete example

A linear model with few-ish, interpretable features, locally fit to a complex one is in line with LIMEs' requirements.

Local fit means fitting only in the vicinity of the input of interest. The input features may differ from those in the original model.

The original paper shows an example comparing two different models that were fit by linear ones:

Image taken from paper.

The image makes clear some of the uses of it:

• The interpretable features, alongside their contributions (weights) to the prediction, can help decide whether to trust the prediction or not.
• Feature engineering such as removing features (or certain data) that the model uses but harm generalisation,
• Comparing models is easy (through the linear proxies). It's especially useful if the original models' accuracy (and other metrics) are similar, and their features non-interpretable.
• Here, one of them is untrustworhy (right hand side), giving high weight to meaningless features.

LIME / Desiderata

The authors consider 4 properties to be desirable in an explanation model:

• Local: the interpretable or explanation model approximates the original model in the vicinity of a particular prediction. In contrast, global explanations explain the full model.
• Model-agnostic: any model can in principle be explained by this method.
• Interpretable Explanation: In this paper, "interpretable" is a desired characteristic of "explanation", and provides qualititative understanding (a simple answer to "Why was this prediction made?"). In their own words:

  An essential criterion for explanations is that they must be interpretable, i.e., provide qualitative understanding between the input variables and the response. We note that interpretability must take into account the user's limitations.

Additionally, they include 4. A global perspective, a mechanism to get a sense of the full model's behaviour.

The first 3 can be called LIME; the Submodule Picking LIME (SP-LIME), which selects LIME explanations to give a global explanation of the model.

LIME is a slightly more explicit version of the first 3 desired properties. Let's explore this.

• Which specific representations does the LIME framework consider interpretable?

  An example of an interpretable representation is a binary vector with components indicating presence / absence of a feature (e.g. a word) in the explanation model.

• Which models does the LIME paper consider interpretable?

  (...) interpretable models, such as linear models, decision trees, or falling rule lists [27], i.e. a model $g\in G$ can be readily presented to the user with visual or textual artifacts.

• Complexity is the opposite of interpretable, so the loss (we skip it here) to train the explainable model accounts for it, and gives local samples more importance than remote ones. Here $G$ being the model class of $g$:

  As not every $g\in G$ may be simple enough to be interpretable thus we let $\Omega (g)$ be a measure of complexity (as opposed to interpretability) of the explanation $g\in G$.

  The definition of complexity depends on $G$. For linear models it may be the number of weights.

• They also define a sampling procedure to create the training set. This is detailed in the concrete algorithm in the next section.

Linear LIME: An Algorithm

The paper implements LIME using the class $G$ of sparse linear models as explanation model, which we could call Linear LIME (as A Unified Approach to Interpreting Model Predictions does). Here is my interpretation of the algorithm (the primed variables denote binary vectors):

1. A model $f$ and an input vector $x\in R^n$ needs explaining,
2. Start an interpretable, binary vector $x^{\prime }\in \{0,1{\}}^{n^{\prime }}$ with only the dimensions of interest of $x$ (it may be all-ones often),
3. Generate perturbed binary variants of $x^{\prime }$ called $z_i^{\prime }$,
4. Use different $z^{\prime }$ to make variants of $x$ called $z\in R^n$.
   • Written in A Unified Approach to Interpreting Model Predictions paper as $z=h_x(z^{\prime })$.
5. Now we have training tuples $(f(z),z^{\prime },{\pi }_x(z))$.
6. Use the dataset to fit the linear model $g$ using $K-LASSO$.
   • To select $K$ interpretable features they apply $K$-LASSO.
   • LASSO is like a sparse linear regression, some coefficients are pushed to $0$ and correspond to variables we can ignore. How many are are pushed to $0$ may be controlled by the complexity penalty in the loss function.
   • This is why the number of input features for the interpretable model ($x^{\prime }$, $z^{\prime }$) may be much smaller than that of $x$, $z$.
   • Then the non-zero variables are used to fit $g$ to $f$ but now using least-squares.

SP-LIME is defined as:

[SP-LIME] a global understanding of the model by explaining a set of individual instances.

The complex part of SP-LIME is selecting instances that add the maximum insight, and avoiding repeated ones. The algorithm is briefly described later on.

SP-LIME: The Algorithm

The goal here is picking the most informative instances, and without repetition.

A coverage metric is defined: $c(V,W,I)={\sum }_j{1}_{[}\exists i\in V:W_{ij}>0]I_j$.

Here, $W$ is a matrix of weights (columns) for each instance (row).

$I$ is the global importance of a component, defined as $I_j=\sqrt{{\sum }_i{W}_{ij}}$. The larger the sum of weights is, the more important.

A marginal coverage for each candidate instance ($i$) $c_i-c$ is performed, then the instance that increases $c$ of $V$ the most is appended to it.

Note

This approximates the $V$ with the largest coverage, but it's not exact because the total max could be one that does not max it on each step!

To increase the coverage at all, it must add some non-zero value to a column of zeros.

LIME: Final Comments

Explanation models do not replace but complement accuracy or other evaluation metrics.

Robustness Fixes

• Normalised Moving Rate (NMR): tests the stability of the list against the collinearity. Smaller NMR means more stable ordering.
• Modified Index Position, in the paper's words:

  [MIP] works similarly to NMR by iteratively removing the top feature and retraining and testing the model. Thereafter, it examines how the features are reordered in the model which implies the effect of collinearity.

These two methods (MIP, NMR) can be useful both in having a reliable sorting of features, and on selecting one —most stable— of several methods.

Definition of a few concepts

Let's now look at other methods.

Methods II

Visual Explainability Methods

• Saliency Maps: visually show which features are most important in a particular prediction. They can give a false sense of confidence unless also tested with the "wrong label". It does not give any information on what processing is done with the regions. They can be generated for 1D, 2D and ND inputs. For example, here is for radiology:

  

  Left-most: input image; next: input + saliency map; right-most: doctor's annotation (top) and RNN-model generated annotation (bottom). Image taken from paper.

• Variations: Individual Conditional Expectation, Partial Dependence Plots, can help visualise decision boundaries; they only vary 1 or 2 variables. Quotes below are snippets from original:

  • [ICE] operates on instance level, depicting the model's decision boundary as a function of a single feature, with the rest of them staying fixed.

  • (...) employ [ICE] plots to inspect the model's behaviour for a specific instance, where everything except salary is held constant, fixed to their observed values, while salary is free to attain different values.

  • PDPs are a similar idea, but the remaining features are average values over the dataset points, rather than particular values of an instance.

• Validity Interval Analysis: another technique fitting the NN behaviour to try to extract explanations.

Linear Combination Of Features

• SHAP,
• Influence Functions,
• LIME (previous post)

Other methods

• Anchors: the authors of LIME also proposed this nice method described by Principles and practice of explainability in ML:

  • A similar technique, called anchors, can be found in (Ribeiro et al., 2018). Here the objective is again to approximate a model locally, but this time not by using a linear model. Instead, easy to understand "if-then" rules that anchor the model's decision are employed. The rules aim at capturing the essential features, omitting the rest, so it results in more sparse explanations.

  • (...) decides to use anchors in order to achieve just that, generate easy-to-understand "if-then" rules that approximate the opaque model's behaviour in a local area (Figure 9). The resulting rules would now look something like "if salary is greater than 20 k£ and there are no missed payment, then the loan is approved.

• Dimensionality Reduction: Principal Component Analysis, t-SNE, Dimensionality Reduction, Independent Component Analysis, Non-negative Matrix Factorisation.
• Counterfactuals: Rudin states "A counterfactual explanation describes a minimal change to the input that would result in the opposite prediction." We replace the problem by a hypothetical opposite:
  • Rudin's paper has concretes example as well: "your loan application was denied, but if you had $1000 less debt, you would have qualified for the loan." and "If you reduced your debt by $5000 and increased your savings by $50% then you would have qualified for the loan you applied for".
  • A is a cause of B if, in an imaginary situation, A not happening implies B not happening,
  • Change the instance slightly, but such that the model classifies the new instance in a different category,

  • (...) the applicant had missed one payment that led to this outcome, and that had he/she missed none the application would had been accepted.

• Contrastive: Is about comparing carefully selected instances: Why P rather than Q?.

  • In such cases, people expect to observe a particular event, but then observe another, with the observed event being the fact and the expected event being the foil.

  • It's a good "question generator". What do you expect if X is done, rather than Y?

Explanation-producing Architectures

Architectures designed to make explaining part of their operation easier.

• Using Explicit Attention: An attention layer/mask learns how parts of an input embedding pay attention to other parts. The layer is somewhat interpretable. In chemistry, it could learn which atoms connect (or pay attention to) other atoms.

• Dissentangled Representations: Disentangled representations have individual dimensions that describe meaningful and independent factors of variation. —Explaining Explainability (2018). Examples of architectures are $\beta$-VAE, INFOGan, capsule networks.

XAI4Chem: Contrastive explanations

Previous posts have discussed explanations and model explainability.

One area of interest, both in academia and the pharmaceutical industries, is the explainability of neural networks predicting structures or properties.

Here is an example, from the paper "Enhancing preclinical drug discovery with artificial intelligence" (the references were edited out, and can be found in the original article):

(...) many successful applications of QSAR-based VS workflows for hit identification have been reported. Zhang et al. described the successful implementation of an ML-based QSAR workflow for VS that led to the discovery of novel antimalarial agents.

The ML algorithms were trained on a small dataset of antimalarial drugs. The authors continue:

The QSAR models were used to carry out VS against the ChemBridge database and resulted in the selection of 174 compounds for a follow-up screening in Plasmodium falciparum growth inhibition and cellular assays. Experimental validation revealed 25 of the selected compounds to be active, yielding a hit rate of 14.2%, with the most potent hit having an EC50 value of 95.6 nM. Subsequently, many studies have reported the application of ML and DL-based QSAR workflows as promising VS tools.

That sound cool and useful. A human expert may be able to detect a pattern on those structures.

• But can we figure out an evaluation mechanism that singles out which fragments are most important?

• At the same time, isolated fragments may be misleading, because the shape of the molecule is also as important. Maybe a "fragments vs shape" could make up a directionally useful plot?

Paper Idea

Feature Attribution Methods, such as SHAP and LIME help answer the question "Why was P obtained?".

On the other hand, contrastive explanations answer the question "Why P rather than Q?".

Here, "Q" is the foil (an alternative event), and it's an important tool to make explanations simpler: the aspects that are similar do not need to be explained, it is the differences that count towards explaining "Why P rather than Q".

The paper "Contrastive explanations for machine learning predictions in chemistry" (2025) develops a framework called MolCE for:

deriving contrastive explanations for machine learning models in chemistry to systematically generate intuitive explanations of predictions in high-dimensional feature spaces.

Atom vectors

The quest for machine representations of objects is a long-standing research theme. For example, we have vector representations:

flowchart TB
A(Machine representations
of objects) --> B(Vector Representations)
B --> C(Descriptors)
B --> D(Embeddings)

Descriptors are expert-designed vectors; embeddings are machine-learnt vectors $\in {\mathbb{R}}^N$.

This post focuses on embeddings because they require less human effort, and produce more general-purpose vectors.

Characteristics of Embeddings

Embeddings are usually real-valued, dense rather than sparse, non-human readable. They also form a structured vector-space, with semantically similar vectors close together, and meaningful vector-arithmetic.

It is desirable, but not always possible that they:

• Are interpretable,
• Can be generated with data scarce environments,
  • Or are data-hungry but it is easily available,

The paper Scientific discovery in the age of artificial intelligence proposes a different list of characteristics (references removed):

Scientifically meaningful representations are compact, discriminative, disentangle underlying factors of variation and encode underlying mechanisms that generalize across numerous tasks.

Though here representation doesn't necessarily refer to generated embeddings (my reading of it, at least), but both to generated embeddings and ways to encode and represent the input.

They suggest that promising approaches to achieve this task are: 1. Geometric deep learning, 2. Self-supervised learning (pre-trained on unlabelled data, then fine-tuned), 3. Language modelling, here they proposed traditional next-token learning, and also masked learning; and the language part refers to learning sequences be it language or molecular strings etc.

Embeddings for Atoms

Embeddings for atoms were inspired by NLP models from the 2010s.

One such example was learning continuous vector representations of words (2013). They proposed an automated mechanism to generate word-vectors by absorbing information from that word's environment (neighbouring words).

In chemistry, embeddings can be used for downstream machine-learning tasks such as materials' property prediction, so they became popular in the field. Materials science has exploited the same ideas, for example:

• properties of an atom can be inferred from the environments it lives in (Atom2Vec, 2018),
• atoms are to compounds as words are to sentences (SkipAtom, 2022),

The surprise was that similar words (or atoms) end up with similar vectors. The vectors also support semantically meaningful arithmetic operations, and became useful for downstream tasks. A classic example was:

vector("Queen") = vector("King") - vector("Man") + vector("Woman")

Both Atom2Vec (2018) and SkipAtom (2022) are unsupervised algorithms that obtain their atom vectors from databases of compounds. Atom vectors can be combined into compound vectors, and used for downstream tasks like property-prediction.

Classifications and Featurisers

The method used to generate our vectors is called a featuriser (we can use a featuriser or create our own). There are many common approaches:

• Simple: like hot-encoded, random;
• Human-designed: Composition-Based Feature Vector (CBFV) which are expert-curated vectors as in Jarvis, Magpie;
• Machine-learnt: embeddings, like SkipAtom.

Atom-vectors can be combined to describe compounds. Examples of combination methods are concatenation into a long vector and pooling of vectors (e.g. summing them up).

Comparing representations

A performance-comparison of vector representations is carried out in "Is domain knowledge necessary for machine learning materials properties?" (2020).

Their conclusion is: human-designed Composition Based Feature Vectors (CBFV like Jarvis and Olyinyk) outperform other methods if there isn't much data. This was prior to SkipAtom, but does include Atom2Vec.

Otherwise, performance in downstream tasks is similar to hot-encoded or random vectors.

(...) Although new, data-driven approaches are of interest, those studied here have yet to surpass CBFVs in terms of material property prediction with small data.

However, "Domain Independent XAI for Material Science" (2025) challenges that conclusion:

Our method challenges this perception: we obtain excellent classifiers that are interpretable and based on a small amount of training data without using any domain knowledge: (...)

They assert that one-hot encoded vectors can still achieve good results using small datasets, as long as the network is designed in the way they specify.

Thoughts

Human-designed vectors are easier to interpret; machine-learnt vectors require less effort but more data to train them.

Can we design machine-learnt interpretable vectors that are intrinsically interpretable? Attention-masks and disentangled representations are closer to this.

Should the representation be just the simplest, and the network learn all that is needed for the given tasks?

Atom2Vec

A popular representation of atoms as vectors appeared in (2018): Atom2Vec.

They take compounds from a database, and build a matrix like the one below:¹

Let's describe using the compound $\mathrm{B}{\mathrm{i}}_2\mathrm{S}{\mathrm{b}}_3$ as an example:

• When $\mathrm{Bi}$ is the target, it generates $(2)\mathrm{S}{\mathrm{b}}_3$, placed in the first column. (2) is the stoichiometry of the element $\mathrm{Bi}$ in the compound.
• When $\mathrm{Sb}$ is the target it generates $(3)\mathrm{B}{\mathrm{i}}_2$, placed in the fourth column. (3) is the stoichiometry of the element $\mathrm{Sb}$ in the compound.

Since a particular atom binds to a very small fraction of all groups, each row is very sparse (high fraction of zeros). The same is valid for columns.

SVD Method

A normalised matrix $X_u$ is obtained by normalising each row vector independently. Using the euclidean norm (2-norm) allows for an intuitive similarity metric:

$$\mathrm{dist}(\overrightarrow{u_1},\overrightarrow{u_2})=1-\overrightarrow{u_1}\cdot \overrightarrow{u_2}=1-\mathrm{similarity}$$

In their best-performing model, they compute $SVD(X_u)=UD{V}^T$, collect the $d$-rows with the largest singular values, and compute $F=U^{\prime }{D}^{\prime }$ where $D^{\prime }$ is the slice of rows of D with the $d$ largest singular values, and $U^{\prime }$ the corresponding columns.

Note

The strategy has certain beauty to it: the new f-vectors retain the inner product similarity but are denser. Though now, the columns have no explicit meaning.

Findings

• Similar atoms have similar vectors,
• Increasing the distance threshold in stages, vectors can be clustered hierarchically, from the leaf-nodes (atoms) downwards (groups).
  • At some level, groups match the periodic table groups. (I don't know how the grouping is made unambiguous).
  • At a very large distance, all atoms merge into a single group. The result is called dendogram.

Image (modified) from Original Paper under CC-BY-SA 4.0. The atoms are rotated to make the image fit (rotated).

• Looking at the variation of some dimensions in the vectors, we can assign meaning to some of them.

Benches

Then, they compared to "empirical features" —a vector (group, period,...), padded to match their $d$— with the task of predicting the DFT-found formation-energies of $\approx 10^4$ elpasolite crystals ($\mathrm{AB}{\mathrm{C}}_2{\mathrm{D}}_6$).

Each solid was represented as a concatenation of atom vectors, and feed it to a hidden layer. (They also do other tasks.)

The paper ends with an interesting insight:

Structural information has to be taken into account to accurately model how atoms are bound together to form either environment or compound, where the recent development on recursive and graph-based neural networks might help.

1. It would be a binary matrix but the database contains some compounds multiple times and those are left duplicated (for some strange reason). ↩

SkipAtom

Atom2Vec was already described; now it's time for SkipAtom, another algorithm to learn atom embeddings.

Method

First, compounds are downloaded. Then, the Voronoi Decomposition is used to derive graphs from unit-cells, and from the graphs generate training-pairs. As they show in the paper:

Image from Original Paper under CC-BY-SA 4.0

Finally, each pair X-Y is used to train a shallow network to predict the target (Y) from the reference (X).

Image from Original Paper (slightly modified) under CC-BY-SA 4.0

• The resulting representation is dense and structured/semantic. This can be shown using dimensionality reduction techniques (PCA, t-SNE,..).
• The architecture is described as:

  (...) single hidden layer with linear activation, whose size depended on the desired dimensionality of the learned embeddings, and an output layer with 86 neurons (one for each of the utilized atom types) with softmax activation. (...) minimizing the cross-entropy loss between the predicted context atom probabilities and the one-hot vector representing the context atom, given the one-vector representing the target atom as input.

Representations of Compounds (Pooling)

The analogy to NLP is that words are like atoms, and sentences are like compounds. Hence, distributed representations of atoms can be combined (pooled) into a vector representing a compound.

Vector-pooling options are:

• sum: $\sum s_i{\vec{a}}_i$ where $s_i$ is the stoichiometry (can be fractional),
• mean: $\frac{\sum s_i{\vec{a}}_i}{\sum s_i}$, i.e. divided by total number of atoms (can be fractional too).
• max: $\max (M_i)$, reduces material matrix $\mathrm{M}$ to vector. Selects max value of each column, each row being an atom in the compound.

The resulting compound representation is then used for training a feed-forward NN on different tasks. Also benchmarked using MatBench.

The pooling can also be done with hot-encoded vectors for atoms. This is done in ElemNet (mean pooling), and in Bag-of-atoms (sum pooling). The advantage: no training required, the disadvantage: the result is a sparse vector, and can be less accurate.

Results

They run two groups of tests:

• Embeddings' quality through elpasolite task. Where the atom vectors are concatenated into a compound vector (no pooling). The vectors train a network for property prediction. SkipAtom performs best here.
• Embeddings quality and pooling methods, through 9 prediction tasks. The results are:
  • Pooling: sum and mean-pooling outperform max-pooling,
  • Kind: Mat2Vec does best, and second SkipAtom,
  • Bag-of-Atoms (sum pool of hot enc) does best in one task.

They conclude that these methods are most useful when no structural info is available.

However, there isn't a simple answer to which representation is best, it depends in the task. This is discussed in more detailed in Results.

Reference-based Coordinate Assignment

RCA is a dimensionality reduction method. Importantly, this implies we already have some representation available, which we may want to reduce.

The main idea could be conceptually represented as:

$${\mathbf{x}}_d=distance({\mathbf{x}}_n;{\mathbf{a}}_n,{\mathbf{b}}_n,\dots ,{\mathbf{d}}_n)$$

On the right hand side, the distance of each $n$ dimensional vector $\mathbf{a}\dots \mathbf{d}$ to $x_n$ is calculated. Each of these vectors is a centroid calculated from a cluster of embeddings.

The result, on the left hand side, is a reduced $d$-dimensional sample vector $x_d$.

An important point is raised in the paper:

To avoid degeneracies arising from equidistant configurations, the reference points must span the structure of the dataset.

Which means the reference vectors (centroids) should be able to reach any point in the original dataset, and not lose expressivity.

The method has useful properties:

1. Compared to a large vector, it is cheaper to use to train a model (and can still be accurate),
2. Compared to other dimensionality reduction methods, it is easier to extend,
3. With certain care, the dimensions may be interpretable.

Comparison

SkipVec and Atom2Vec were previously discussed.

The paper Is domain knowledge necessary for machine learning materials properties? compared descriptors (same as vector representations) generated in different ways for downstream tasks.

They find hand-crafted descriptors useful for small and large datasets, but these are cumbersome to create —expert knowledge is required. One-hot and random-vectors perform similar to hand-crafted descriptors in large datasets.

With that, it seems wise to use hand-crafted descriptors for small datasets, and learnt, one-hot or random for larger ones.

SkipAtom evaluates on different approaches and tasks, and finds their method outperforms one-hot and random-vector, but does not test hand-crafted ones.

SkipAtom's comparison of representations is discussed below.

Simple classification

We have a simple classification of the available methods:

1. Human-engineered vectors;
2. Low effort vectors: one-hot encoded (ElemNet), random, Atom2Vec;
3. Machine-learnt vectors (SkipAtom).

Atom2Vec may be in category 3 above; however, it is not an optimisation that creates them, it is a matrix factorisation.

Quality of Atom Representations

ElemNet (One-hot), Random, Atom2Vec, Mat2Vec and SkipAtom compared.

The atom-vectors were concatenated into compound representations, and these used to predict elpasolites (compounds) formation-energy.

SkipAtom outperformed other methods.

Pooling approaches

Vector pooling strategies were compared through 9 prediction tasks; 5 regressions, 3 classifications and OQMD Formation Energy prediction (also a regression).

• OQDM: bag-of-atoms, which is sum-pooling hot-enc atom vectors, is best.

The rest is summarised well in the paper:

(...) the models described in this report outperform the existing benchmarks on tasks where only composition is available (namely, the Experimental Band Gap, Bulk Metallic Glass Formation, and Experimental Metallicity tasks). Also, on the Theoretical Metallicity task and the Refractive Index task, the pooled SkipAtom, Mat2Vec and one-hot vector representations perform comparably [to the SOTA], despite making use of composition information only.

And an interesting observation:

The ElemNet architecture demonstrated (...) Perhaps surprisingly, the combination of a deep feed-forward neural network with compound representations consisting of composition information alone results in competitive performance when comparing to approaches that make use of structural information.

Use cases and limitations

Training does not rely on labelled data (unsupervised learning).

The model just needs the formula at inference time, and does fine with non-stoichiometric solids. So having the material's composition —but no structural information— we can still calculate some properties.

Similar compounds have similar vectors, which is useful. But without structural information, all isomers have the same vector, which is a limitation.

It is computationally cheap, and can help screen large number of compounds as a first selection step.

Discovering Inorganic Solids

These are some of my opinions and ideas after reading two papers by Rosseinsky group:

1. Discovery of Crystalline Inorganic Solids in the Digital Age (2025).
2. Element selection for crystalline inorganic solid discovery guided by unsupervised machine learning of experimentally explored chemistry (2021)

Introduction

In solid-state chemistry, some elemental compositions (phase fields) are more likely to lead to isolable compounds than others.

Deep learning models can help differentiate between these two groups, and lead researchers to the promising areas. The models can be trained for this task with data from ICSD, the Inorganic Crystal Structure Database.

Such models would improve the allocation of resources when exploring new phase fields.

Searching for new compounds

Some definitions will be used:

• Phase field: the elements selected. Can be thought as the labels for cartesian axes.
• Composition: the values or ranges of values in each axes. Once we have the axes' labels we can explore values computationally.

We can search for compounds by analogy and by exploration, characterised in the table below:

Analogy Based Search

The analogy-based search involves:

1. Starts from a naturally occuring mineral, or previously discovered structures,
2. Change its composition retaining the crystalline structure. For example, $\mathrm{L}{\mathrm{i}}_7\mathrm{S}{\mathrm{i}}_2{\mathrm{S}}_7\mathrm{I}$ can be expanded by analogy to $\mathrm{L}{\mathrm{i}}_7\mathrm{S}{\mathrm{i}}_{2-\mathrm{x}}\mathrm{G}{\mathrm{e}}_{\mathrm{x}}{\mathrm{S}}_7\mathrm{I}$, conserving the crystalline structure.

With respect to analogy-based search, the paper notes:

(...) it is straightforward to expand known structures by analogy through substitution, but the initial identification of such structures, which cannot be by analogy, is an entirely different question (...)

And usefully,

The properties of the analogy-based materials can be superior to those of the initial discovery (...)

Exploratory Search

The ML-aided exploratory-search involves:

1. Human selects elements or phase field e.g. $\mathrm{Y}-\mathrm{Sr}-\mathrm{Ca}-\mathrm{Ga}-\mathrm{O}$, $\mathrm{LiSiX}{\mathrm{X}}^{\prime }$,...
   • A VAE decodes the seed-input into similar compounds (nearby in latent space).
   • The reconstruction loss is used as a ranking metric for the generated compounds.
2. Computationally search in composition-space (Crystal Structure Prediction, CSP), find low-energy probe structures, e.g. $\mathrm{L}{\mathrm{i}}_3\mathrm{Sn}{\mathrm{S}}_3\mathrm{Cl}$.
   • Can use physical constraints (like max n of atoms).
   • Calculate thermodynamically stable¹ probe structure (this step is complex). Hints experimentalists of promising region.
3. Try synthesis, and find somewhat similar structures to the computationally suggested one.

We can describe the exploration steps as a flow as well:

---
config:
  flowchart:
    htmlLabels: true
---
flowchart LR
A(("`**Input Phase**
(e.g Na-Ca-O)`")) --> B(VAE)
B -- "`**Ranked Phases**`" --> D(Distance Metric)
D -- "`**Compositions**`" --> F(Thermodynamics)
F -- "`**Probe**`" --> H(Try synthesis)

style A fill:#123456,color:#f4f4f4,stroke-width:0px

1. With respect to the convex hull. ↩

Discovering Inorganic Solids

In depth workflow

Dataset

The atom descriptors are taken from a atom-property database and include atomic weight, valence, ionic radius, and others.

The 4-element crystals are selected from ICSD, and only the elements are retained. For example, $\mathrm{CaNaLi}{\mathrm{O}}_2$ would become $\mathrm{CaNaLiO}$ for training.

The data is scaled 24 fold by performing all possible permutations of 4 elements, i.e. 4! (factorial). This enhances learning, reduces overfitting.

Architecture: Variational Autoencoder (VAE)

Here the emphasis is on exploiting a pattern and not on interpretability, the human expert evaluates the compounds afterwards.

An autoencoder consists of two parts, an encoder, and a decoder. The overall task is to reconstruct the original vector from the compressed representation.

The encoder compresses the 148 vector into a 4D vector (latent vector), and the decoder decompresses it into 148D. The euclidean distance is then computed as a measure of error, and the gradient is used to correct the weights.

Since the model is trained only on phase fields that lead to isolable materials, it is biased towards those compounds.

Just like a single-class classifier using cat-only images, the VAE only sees positive instances, and no learning comes from predicting negatives.

Inference Stage

Input structures are passed with a bit of noise each time and the reconstruction loss is used to rank them for synthetic exploration.

A larger reconstruction loss means the phase is less likely to be synthesizable, since it learn to reconstruct only synthesizable regions.

The rank will also tell how different the compound is to the original.

Example of Results

After VAE ranking, the decision to explore Li-Sn-S-Cl phase field was based on the high conductivity of a related ternary field Li-Sn-S.

The following image shows calculations performed, each a tripod, in a red background. Dark red represents little enery barrier from the convex hull, bright red the opposite.

Most solids found by the group or by others are in dark areas of the plot with tripods overlaying.

The magenta point A in the image is the new phase found, not far from the probe structure which was $\mathrm{L}{\mathrm{i}}_3\mathrm{Sn}{\mathrm{S}}_3\mathrm{Cl}$.

Image from Original Paper under CC-BY-SA 4.0

Best Practices

TL;DR from the paper "Best practices in machine learning for chemistry" (2021), a very similar paper by the same authors is "Machine learning for molecular and materials science".

• For Datasets
  • Ensure dataset remains available, and is version-tagged (they change)
  • For home-made or mixes, explain the process of generation
  • Describe any data curation, balancing, augmentation, and so on.

In the Best Practices paper they say:

For reasons of reproducibility, it is crucial that these databases use some mechanism for version control (e.g. release numbers, Git versioning, or timestamps) as part of the metadata and maintain long-term availability to previous versions of the database.

• For Representations

  • Try more than one, compare
  • Use very basic ones as baseline representation to compare (example random or one-hot)

• Justify Model

  • More complex isn't always better
  • Compare to baselines (mean for regression, most common class for classification)
  • Compare to very simple models and to SOTA
  • Any interpretability we can offer?

• Evaluate Model

  • Have 3 separate datasets: for training and optimising, for evaluating during training and detect overfitting, and testing for testing which should represent where it will be applied (should test what we want it to succeed on).
  • Test extrapolative learning: leave out some class entirely, or train until a cutoff date and evaluate with dates after that.
  • Test intrapolative learning: with varied test sets
  • Mindful of shorcut learning (have varied test dataset).

• Reproducibility: Results and code must be made available and reproducible

They also state:

In all reports, remember to cite the methods and packages employed to ensure that the development community receive the recognition they deserve.

They provide a great checklist but since the license is a mess I am not including it here.

Example

Take ElemNet as an example and go through the checklist.

• Database:
  • They provide a link, but no timestamped or git version,
  • No info on curation or preprocessing (we may assume none was performed).
• Representations:
  • They justify and compare the results to baselines.
• Model:
  • Describe why is new and useful idea,
  • Describe architecture.
• Evaluation:
  • Show training and different hyperparameters,
  • Studied which compounds model is accurate vs not.
• Reproducibility: Results and code are available.

The comparison would be improved if they also ran a deep learning model with human-made descriptors.

They also included other useful statistics like inference time.

Machine learning for molecular and materials science

Very compressed bits from the paper Machine learning for molecular and materials science (2018).

Representations of atoms, molecules, materials

The process of converting raw data into a format more suitable for an algorithm is called feature engineering.

The more suitable the representation of the input data, the more accurately can an algorithm map it to the output data.

Selecting how best to represent the data may require insight into both the underlying scientific problem and the operation of the learning algorithm, since it is not always obvious which choice of representation will give the best performance.

Examples

• Coulomb Matrix: atomic nuclear repulsion information.
• Graphs: Connectivity of molecules.
• String representations: SMILES, SELFIES,..
• Solid-state unit-cells: Representations based on radial distribution functions, Voronoi tessellations, and property-labelled materials fragments (...)

  In the solid-state, the conventional description of crystal structures by translation vectors and fractional coordinates of the atoms is not appropriate for ML, since a lattice can be represented in an infinite number of ways by choosing a different coordinate system.

List of a few ML algorithms

Naive Bayes, Nearest Neighbour, Decision Trees, Kernel methods, a family including support vector machine (SVM) and kernel ridge regression (KRR). Artificial neural networks (ANNs) and deep neural networks (DNNs).

Research and Data Objects

FAIR Principles

The FAIR Guiding Principles are suggestions for improving data reusability. Also reproducibility and transparency. Here the essence of it is described.

The idea

The paper states:

[good data management] is the key conduit leading to knowledge discovery and innovation.

We can imagine science as an activity of "solving the puzzle of knowledge together". Theories, observations, workflows, processing pipelines and algorithms, software and other data are the puzzle pieces. If the piece is (each term only briefly summarised, oversimplified):

• Findable: there is some place to query for it,
• Accessible: the object is explorable, downloadable,
• Interoperable: uses a common format rather than private,
• Reusable: contains permissive usage policies / license such that it can be reused in future research.

Benefits

Sharing data objects has many benefits:

• Researchers would earn citations through publishing data-objects.
• Software developers can also get recognition for their published data-objects;
• Peer-researchers can then test and reuse the results, while enhancing data-mining and have other areas.

And so forth.

For humans and machines

Different from other frameworks, FAIR is concerned about how non-human agents and tools would discover, retrieve and be able to use the data.

Sharing and finding data objects

Common repositories used for scientific objects are of two kinds:

• Special-purpose: World Wide Protein Data Bank (wwPDB), NOMAD and Materials Cloud (for materials)
• General-purpose: FigShare, Zenodo (CERN), GitHub.

Important

GitHub does not produce DOIs nor guarantees that the item is preserved (it can easily be deleted).

Tip

1. It's important to either inspect or create (for each case above) labels, description, license and other metadata.

2. There is a registry of research data repositories—re3data which is a useful starting point for finding not datasets or software, but a repository (and metadata such as where it's located, who controls it and so forth)

Citing Objects

The recommended practice is to link a dynamic DOI (that tracks versions), especially for software or datasets.

Zenodo handily combines a concept and version DOIs, mapping to the software project and version, respectively.

There are two issues, one on each side of the citation problem:

1. Developers are burdened to update the DOI's version, say in the README.
2. Researchers cite a paper DOI rather than dataset / software appears, or simply name the software.

To solve 1. a 2019 paper analysing citations of Zenodo objects suggests:

[recommended identifier] ideally represents the whole software project, which is less prone to impermanence. This recommendation does not require maintenance effort on the developer (...)

This is not the "recommended practice" of citing the version as well, but it is a first step. The next step would be to use and update the software DOI version.

Thoughts on non-paper citations

A clear limitation is that software dependencies don't get any credits, only the top level.

It seems common to also add a citation file format (file.cff) to GitHub repos which help when linking repository to Zenodo. Then we must add the Zenodo recommended citation choosing version or concept DOI, as described earlier.

Connectivism

Connectivism¹ is a learning theory where the central object of the theory are networks.

On this post, a few key ideas are presented; in the next post, a different interpretation of the theory is proposed.

Origins

Connectivism seems inspired by:

1. The Internet, and later on the World Wide Web,
   • These both were from the start problems about networking; heterogeneous devices or resources linked together,
   • These technologies opened up access to a network of resources and people.
   • The world changed towards quicker decision-making, but I argue pattern-matching, and thinking strategies have not changed much.
2. The view of an individual as a network of neurons,
3. The view of other systems as organisms, like organisations. The paper linked above states:

   The organization and the individual are both learning organisms. Increased attention to knowledge management highlights the need for a theory that attempts to explain the link between individual and organizational learning.

As said, networks are the main object of the theory, so let's briefly define them:

Networks: collections of linked nodes or items.

What is considered a node is defined by us. It could be an individual, a group, a government, countries that trade, libraries, and so forth.

There are types of nodes and edges (or links) with different properties. For example, a node may have restricted access to a resource. There are also topologies with specific wiring pattern.

Knowledge and Learning

The theory views knowledge and learning as properties of networks, and defines them as:

• Knowledge

  • Network view: it is stored distributed across the net (in nodes, edges) in a latent manner. Distributed knowledge means that no single node or edge is uniquely responsible for a concept, rather, many parts of the whole network are. There will be nodes that are more relevant in some situation than other, but the knowledge is still distributed across nodes and links.
  • Behavioural view: It is evidenced as an appropriate response to an input signal or stimuli.

• Learning

  • Network view: a change of network connections by exposure to experience and reasoning.
  • Behavioural view: a persisting change in knowledge. Learning is evidenced through a change towards more appropriate behavioural responses (to a signal or stimuli.)

In a network, one node or link changing means much more has changed due to the ripple effects given by the links.

Discussion on whether all networks really learn are left for the next post. This one focuses on learning-human-networks and how the theory can help understand and design them.

A human in the net

A promising area for connectivism is as a paradigm to think of human networks (or human-and-resources networks), how to organise them (or letting them self-organise), which properties make nodes and the net grow and so forth.

It appears of especial important in the digital, interconnected world; but also classrooms, organisations, and social networks in general.

Let's consider the case of a human or agent embedded in a wider network such as a classroom, a group, community, organisation.

This networks can be seen as an metaphorically organism, or just as a whole: when the connections and nodes change and so does the behaviour in response to input signals, which in certain cases will imply learning, and formation of knowledge.

Example

Think of the response of an organisation in the event of a fire or an emergency.

The input (fire alarm) ripples through the network as the nodes act and propagate activity: learning isn't only intra-personal, the whole network can be said to have learnt in this example.

In the case of a classroom, this framework may aid us with questions such as:

• Centralised human-learning-network guided by teacher or a decentralised network? Should the teacher be more like a routing node?
• Are all items considered part of it, such as resources, or agents such as humans? Is it useful to split or join different networks?
• An open network where agents can change resources, or closed networks where they only have read permissions (if even so)?

A human is a net (theory of mind)

We also want to focus on the individuals. Can connectivism help?

Their general claim is that any network can learn by modifying connections and nodes of the network. And stores distributed knowledge.

So can parts of the brain, which are networks of neurons, and learning is later on evidenced on the behaviour of individuals.

1. https://www.scispace.com/pdf/elearnspace-connectivism-a-learning-theory-for-the-digital-4dh6aurogw.pdf ↩

Refining Connectivism

Do all networks learn?

The notion that all networks learn, just by virtue of being networks seems off. It does not correspond to what most of us would call learning.

Is there some way to look at this, which makes more intuitive sense? Admittedly, there will be grey areas, but my view is that it is possible to conceptualise learning more clearly.

Different networks vary on:

• The breath of signals they are responsive to,
• The intensity of signals they are responsive to,
• How varied their response is,
• How much directionality there is to the process,
• Possibly others (still thinking on it).

The hypothesis is that, what we ordinarily call a learning system maps to:

High-complexity systems that respond to a wide variety of input signals in a wide variety of ways, and change in a way that makes future reactions appropriate (which is left to intuition for the moment).

Usually, networks of complex systems or of hybrid systems also change complex ways and will also be learning systems.

On the other hand, low-complexity systems change in narrow and even predictable ways whichare not considered learning systems.

In a sense learning is not the mechanism, but it's just the label. Complex systems that change directionally and in various ways learn; simple systems that change --by the same mechanism-- but do not behave in complex ways are not learning systems.

Just reinforce it: the proposal is that learning mainly points to high-complexity, but the actual kind of change could be the same in both systems.

We could say that there is a threshold of a binary function $f(Complexity,Degre{e}_{A}dapt,Store{s}_{C}hanges)$ were systems are considered at learners. There will also be, certainly, grey areas.

With that, which one learns?

We can apply the previous criteria to a large number of cases, and see whether we are satisfied:

Do biological networks learn?

It is generally agreed that learning involves changes in the wiring pattern, or degree of wiring, and this change can happen in different ways, and it is built into the system (brain) e.g. by Hebbian learning. We can take this case as our paradigmatic learning system.

Do artificial neural networks (ANNs) learn?

During training phase, connections (weights) are updated towards a goal through backpropagation, and the network behaviour (predictions) change. General-purpose process a wide variety of signals are are closer to learning systems than narrow AIs. Both only learn during training phase.

Does a colony of insects learn?

Complex organisms communicate and change their behaviour: Ants propagate pheromones which adapt their behaviour; humans similarly can do it by imitation of some kind. This "node update" propagates out, and may return to update the node further. Finally the network is seen as reacting appropriately.

But is this "reaction", or emergent behaviour, the same as learning? Only if the change is permanent. The variety of signals they process and the behaviours is quite varied and certainly are learning systems.

Does the world wide web, or a library learn?

The system will learn only if it is designed to do so. For example, after a certain event (like a query) a program may change the system to respond more appropriately in the future.

When humans have designed them to adapt (say with software) these have limited learning ability and change in response to a narrow variety of events, and are not quite learning systems. Considered without the programs, these wouldn't have any learning at all.

Hybrid systems such as humans-web, or humans-library systems certainly are learning systems.

The structure, interface, reliability that they have though, matters a lot in terms of how usable they are to humans, and so forth. These properties are still important across most network systems, be it learning networks or not. In other words, all non-learning networks can become learning networks when associated with other systems that can change them

Does the electric grid learns? Same as above.

Does a molecule or a material, which can be represented as a graph, learn?

Signals can change them. But even if we define appropriate as increasing the likelihood to persist / survive, the ways in which it reacts and the number of stimuli it responds to are narrow, and won't be learning systems.

Do mixes of nodes made of proved-learning-systems (say humans) and non-learning systems (say a book), learn?

Yes, especially if we consider a company, a community and so forth.

So all the different kinds may be considered to learning, if they permanently change in response to a stimuli and this adequately changes future responses.

Visualising High Dimensional Data

• PCA
• Dimensionality Reduction
• t-SNE

Reference of some of the techniques, book cited in some papers: 18. Hastie T, Tibshirani R, Friedman J (2001) The Elements of Statistical Learning, Springer Series in Statistics (Springer, New York).

CompChem Map

This is a draft of areas I'd like to organise in some taxonomy.

Finding Useful Molecules

• Get a Materials Database(s), either method

Method 1: Direct (Compounds to Properties)

• Use DFT to guide towards one that fits the requirements (slow if we have billions of compounds.),
• Or use the DB to train a NN to make predictions (needs labelled data for training)
• Or similarity metrics to find new (similar) molecules.
• ...

Method 2: Inverse (Properties to Compounds)

1. Use the gradient to update embedding.
2. Maximise or Minimise the needed properties.

The paper's approach is more towards a Direct method. It is a method to generate embeddings that can then be used to train a neural network to predict properties.

This can be arranged (not very tidily) in a chart:

---
config:
    flowchart:
        htmlLabels:false
---

flowchart TB

A[("Compounds Database")]

subgraph Direct["`**Direct**`"]
    direction LR
    B("`Electronic Structure
    Predictions`")
    C("`Train NN on DB
    (with labelled data)`")
    D("`Use similarity metrics to
        find nearby candidates`")
end

subgraph Inverse["`**Inverse**`"]
    direction TB
    E("`Train VAE to create
    smooth surface`")
    E --> F("`Link MLP to
        latent vector`")
    F --> G("`Minimise or Maximise pps.
            by changing vector`")
end

A --> Inverse
A --> Direct

Unsupervised-Learning of Representations of Atoms

Other investigations of unsupervised learning of machine representation of atoms are:

• Zhou, Q. et al. Learning atoms for materials discovery. (2018).
• Tshitoyan, V. et al. Unsupervised word embeddings capture latent knowledge from materials science literature. (2019).
• Chakravarti, S. K. Distributed representation of chemical fragments. (2018).
• Butler K. et al. Distributed Representations of Atoms and Materials for Machine Learning. (2022).

Supervised-Learning of Representations of Atoms

• Jha, D. et al. ElemNet: deep learning the chemistry of materials from only elemental composition. (2018).
• Goodall, R. E. & Lee, A. A. Predicting materials properties without crystal structure: deep representation learning from stoichiometry. (2020).

Databases and Benchmarks

Bear in mind when using databases what this Machine learning for molecular and materials science states:

Data may require initial pre-processing, during which missing or spurious elements are identified and handled.

Identifying and removing such errors is essential if ML algorithms are not to be misled by their presence.

• The Machine Learning for molecular and materials science aggregates many DBs and such in tables at the end
• Pillong, M. et al. A publicly available crystallisation data set and its application in machine learning. CrystEngComm (2017).
• ICSD: Inorganic Crystal Structure Dataset
• Jain, A. et al. The materials project: a materials genome approach to accelerating materials innovation.
  • The materials project database https://materialsproject.org/
• Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm.
  • Matbench benchmark: https://hackingmaterials.lbl.gov/automatminer/datasets.html
• Materials design and discovery with high-throughput density functional theory: the open quantum materials database (OQMD)

Also ElemNet lists materials-and-properties' databases (experimentally observed and hypothetical):

DFT calculations have offered opportunities for large-scale data collection such as the Open Quantum Materials Database (OQMD), the Automatic Flow of Materials Discovery Library (AFLOWLIB), the Materials Project, and the Novel Materials Discovery (NoMaD); they contain DFT computed properties of $\sim 10^4-10^6$ of experimentally-observed and hypothetical materials. In the past few decades, such materials datasets have led to the new data-driven paradigm of materials informatics

ElemNet describes OQDM as well (bold is mine):

OQMD is an extensive high-throughput DFT database, consisting of DFT computed crystallographic parameters and formation enthalpies of experimentally observed compounds taken from the Inorganic Crystal Structure Database (ICSD) and hypothetical structures created by decorating prototype structures from the ICSD with different compositions.

AI in Pharma

The paper Enhancing preclinical drug discovery with artificial intelligence discusses all the areas in which Artificial Intelligence models can improve the Drug Discovery Pipeline.

The steps in which AI can help include:

1. Target Identification,
2. Hit Identification,
3. Lead Optimisation,
4. Addressing the low preclinical-to-clinical success rate.

Queries

• How to build useful machine-representations of atoms?
  • $R^1$ would just be a 1D vector embedding, likely of little use. But are there taxonomies of representations (including a matrix per atom?)

Missing

• Splits for datasets, validation / testing.
• Balancing datasets.
• Evaluation metrics, loss functions,
• ROC classification performance metric

Ideas

• Train and show the results of atoms for vectors in a website? With some button to load each dataset?

• List of examples of successful applications of ML in chemistry

  1. Anything for which there are useful datasets from experiments or from calculation:
     • orbital energies,
     • geometries,
     • formation energy,
     • energy band-gap

• Which models are related to "Causal Inference"? Are they useful? For example The Mythos of Model Interpretability states (removed a reference which is to Judea Pearl's Causality book, and may be worth checking):

The task of inferring causal relationships from observational data has been extensively studied. Causal inference methods, however, tend to rely on strong assumptions and are not widely used by practitioners, especially on large, complex data sets.