Matbench Benchmarking Suite

Matbench is a curated set of 13 supervised, pre-cleaned, ready-to-use machine learning (ML) tasks designed for benchmarking and fair comparison in materials science. The tasks cover a wide domain of inorganic materials science applications, including electronic, thermodynamic, mechanical, and thermal properties across crystals, 2D materials, disordered metals, and more.

Matbench provides:

• Standardized Tasks: Each task includes a set of inputs (crystal structures or chemical compositions), a set of outputs (target properties like formation energy), and a standardized test procedure to score algorithms.
• Leaderboards: Results for state-of-the-art materials ML algorithms on standardized test problems.
• Reproducibility: Every result is backed by peer-reviewed publications or Jupyter notebooks.
• Python Package: A programmatic way to access datasets and run benchmarks.

Matformer is a Periodic Graph Transformer architecture designed for crystal material property prediction. It is specifically engineered for periodic graph representation learning, ensuring invariance to periodicity and the ability to capture repeating patterns by utilizing geometric distances between identical atoms in neighboring cells.

Note: The version provided in Matbench v0.1 uses a modified training script to ensure compatibility with official Matbench datasets, as the original implementation was not capable of training on them directly. The model weights and parameters are sourced from the original Matformer repository.

MODNet (Materials Optimal Descriptor Network, v0.1.12) is a feed-forward neural network designed for materials property prediction.

Key characteristics include:

• Feature Engineering: Uses all compatible matminer features.
• Feature Selection: Employs a relevance-redundancy based feature selection algorithm.
• Hyperparameter Optimization:
  • For the 9 smaller tasks: Uses a nested grid search.
  • For the 4 larger tasks (matbench_perovskites, matbench_mp_gap, matbench_mp_is_metal, matbench_mp_eform): Uses a genetic algorithm (GA). This specific benchmark version utilizes an improved GA-based optimization.

Raw data and example notebooks can be found in the matbench repository.

The coNGN (Connectivity optimized Nested Graph Network) is a model optimized for both crystal preprocessing and GNN architectures. It specifically incorporates nested line graphs to improve performance on crystal structures.

For raw data downloads and example notebooks, refer to the matbench repository.

The coGN (Connectivity optimized Graph Network) algorithm is a model optimized for both crystal preprocessing for Graph Neural Networks (GNNs) and the GNN architectures themselves. It addresses the strong interdependency between these two aspects.

For raw data downloads and example notebooks, refer to the matbench repository.

The DimeNet++ implementation is a fast and uncertainty-aware directional message passing model for non-equilibrium molecules, adapted from kgcnn.

Key Implementation Details:

• Original Codebase: Based on https://github.com/gasteigerjo/dimenet.
• Memory Optimizations: To accommodate memory constraints, the batch size was reduced to 16 and the maximum number of edges/neighbors was limited to 17 (using a 5Å cutoff).
• Features: Uses geometry and atom type. No additional features were introduced.
• Preprocessing: Includes a standard scaler for regression tasks.
• Handling Geometry: Angles, multi-edges, and correct images are taken into account.
• Hardware used for training: A100-SXM with 41 GB of memory.

Matbench uses Mean Absolute Error (MAE) as the primary ranking metric for regression tasks for the following reasons:

1. Interpretability: The meaning of MAE is easily inferred.
2. Relative Error Handling: For targets that require relative error analysis (like bulk moduli), Matbench transforms the targets to order-of-magnitude form (e.g., log10) before calculating MAE.
3. Robustness: MAE is valid for all target values, whereas Mean Absolute Percentage Error (MAPE) fails when true values are zero.

While MAE is the primary metric, Matbench provides multiple error metrics to help assess generalization error.

The matbench_v0.1: AMMExpress v2020 benchmark uses the Automatminer express v1.03.20200727 algorithm.

Core Workflow:

• Automatic Featurization: Generates features for the materials data.
• Tree-based Feature Reduction: Reduces the feature space using tree-based methods.
• AutoML: Uses a genetic-algorithm based approach via the TPOT package to optimize the machine learning pipeline.

Configuration Details: All data in this benchmark was generated using the express preset with default settings.

Raw Data & Examples: Raw data and example notebooks are available in the matbench repository.

The MegNet implementation in this benchmark is an adaptation of kgcnn.

Model Architecture & Hyperparameters:

• Model Depth: 3
• MegNet Block Units: [64, 32, 32]
• Encoder: Set2Set encoder for node and edge embeddings.
• Feed-forward Blocks Units: [64, 32]
• Activation: softplus
• Distance Expansion: Gauss distance expansion with a cutoff of 5A and 25 bins (0.4 sigma).
• Input Embeddings: Atom species embedding vector of size [64]. Charge is added as an input graph attribute.
• Loss Function: MAE loss.
• Learning Rate Scheduler: Linear scheduler from 5e-4 to 5e-6 over 1000 epochs using Adam.
• Scaling: Standard scaler used for regression tasks.
• Hardware: Trained on A100-SXM with 41 GB memory.

Note: Hyperparameters were copied from training on QM9/QM7 datasets and were not optimized for this specific benchmark.