Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Atom vectors

The quest for machine representations of objects is a long-standing research theme. For example, we have vector representations:

flowchart TB
A(Machine representations
of objects) --> B(Vector Representations)
B --> C(Descriptors)
B --> D(Embeddings)

Descriptors are expert-designed vectors; embeddings are machine-learnt vectors .

This post focuses on embeddings because they require less human effort, and produce more general-purpose vectors.

Characteristics of Embeddings

Embeddings are usually real-valued, dense rather than sparse, non-human readable. They also form a structured vector-space, with semantically similar vectors close together, and meaningful vector-arithmetic.

It is desirable, but not always possible that they:

  • Are interpretable,
  • Can be generated with data scarce environments,
    • Or are data-hungry but it is easily available,

The paper Scientific discovery in the age of artificial intelligence proposes a different list of characteristics (references removed):

Scientifically meaningful representations are compact, discriminative, disentangle underlying factors of variation and encode underlying mechanisms that generalize across numerous tasks.

Though here representation doesn't necessarily refer to generated embeddings (my reading of it, at least), but both to generated embeddings and ways to encode and represent the input.

They suggest that promising approaches to achieve this task are: 1. Geometric deep learning, 2. Self-supervised learning (pre-trained on unlabelled data, then fine-tuned), 3. Language modelling, here they proposed traditional next-token learning, and also masked learning; and the language part refers to learning sequences be it language or molecular strings etc.

Embeddings for Atoms

Embeddings for atoms were inspired by NLP models from the 2010s.

One such example was learning continuous vector representations of words (2013). They proposed an automated mechanism to generate word-vectors by absorbing information from that word's environment (neighbouring words).

In chemistry, embeddings can be used for downstream machine-learning tasks such as materials' property prediction, so they became popular in the field. Materials science has exploited the same ideas, for example:

  • properties of an atom can be inferred from the environments it lives in (Atom2Vec, 2018),
  • atoms are to compounds as words are to sentences (SkipAtom, 2022),

The surprise was that similar words (or atoms) end up with similar vectors. The vectors also support semantically meaningful arithmetic operations, and became useful for downstream tasks. A classic example was:

vector("Queen") = vector("King") - vector("Man") + vector("Woman")

Both Atom2Vec (2018) and SkipAtom (2022) are unsupervised algorithms that obtain their atom vectors from databases of compounds. Atom vectors can be combined into compound vectors, and used for downstream tasks like property-prediction.

Classifications and Featurisers

The method used to generate our vectors is called a featuriser (we can use a featuriser or create our own). There are many common approaches:

  • Simple: like hot-encoded, random;
  • Human-designed: Composition-Based Feature Vector (CBFV) which are expert-curated vectors as in Jarvis, Magpie;
  • Machine-learnt: embeddings, like SkipAtom.

Atom-vectors can be combined to describe compounds. Examples of combination methods are concatenation into a long vector and pooling of vectors (e.g. summing them up).

Comparing representations

A performance-comparison of vector representations is carried out in "Is domain knowledge necessary for machine learning materials properties?" (2020).

Their conclusion is: human-designed Composition Based Feature Vectors (CBFV like Jarvis and Olyinyk) outperform other methods if there isn't much data. This was prior to SkipAtom, but does include Atom2Vec.

Otherwise, performance in downstream tasks is similar to hot-encoded or random vectors.

(...) Although new, data-driven approaches are of interest, those studied here have yet to surpass CBFVs in terms of material property prediction with small data.

However, "Domain Independent XAI for Material Science" (2025) challenges that conclusion:

Our method challenges this perception: we obtain excellent classifiers that are interpretable and based on a small amount of training data without using any domain knowledge: (...)

They assert that one-hot encoded vectors can still achieve good results using small datasets, as long as the network is designed in the way they specify.

Thoughts

Human-designed vectors are easier to interpret; machine-learnt vectors require less effort but more data to train them.

Can we design machine-learnt interpretable vectors that are intrinsically interpretable? Attention-masks and disentangled representations are closer to this.

Should the representation be just the simplest, and the network learn all that is needed for the given tasks?