AiiDA Core

AiiDA is a workflow manager for computational science designed for provenance, performance, and extensibility. Key capabilities include:

• Workflows: Write complex, auto-documenting workflows in Python. The event-based engine supports high throughput (tens of thousands of processes per hour) with full checkpointing and can link to executables on local or remote computers.
• Data Provenance: Automatically tracks inputs, outputs, and metadata in a provenance graph to ensure full reproducibility. The system supports fast queries on graphs with millions of nodes.
• HPC Interface: Seamlessly move calculations from local to remote environments. AiiDA supports major schedulers out of the box, including SLURM, PBS Pro, torque, SGE, and LSF.
• Plugin Interface: Extend functionality via plugins for new simulation codes (input generation/parsing), data types, schedulers, and transport modes.
• Open Science: Facilitates sharing by allowing users to export subsets of provenance graphs to peers or platforms like Materials Cloud.

AiiDA is an open-source Python infrastructure designed for automating, managing, persisting, sharing, and reproducing complex computational science workflows. It is built around the ADES model (Automation, Data, Environment, and Sharing).

Key capabilities include:

• Workflows: Build and execute auto-documenting workflows that link multiple codes across local and remote computers.
• High-throughput: An event-based engine supporting tens of thousands of processes per hour with full check-pointing.
• Data provenance: Automatic tracking of inputs, outputs, and metadata via extensive provenance graphs to preserve full data lineage.
• Advanced queries: A query language capable of performing fast graph queries on millions of nodes.
• Plugin interface: Extensible support for computational codes, data analytics tools, data types, schedulers, and connection modes via a plugin system.
• HPC interface: Seamless integration with heterogeneous and remote computing resources, including out-of-the-box support for schedulers like SLURM, PBS Pro, torque, SGE, and LSF.
• Open science: Ability to export full databases or subsets for sharing with collaborators or publishing to platforms like Materials Cloud (Archive and Explore).

AiiDA supports several batch job schedulers to manage job queues and execution on compute resources. Depending on your cluster, you should configure the appropriate scheduler plugin for your computer.

Supported schedulers include:

• PBSPro: Full feature support. Use NodeNumberJobResource for job resources.
• SLURM: Full feature support. Use NodeNumberJobResource for job resources.
• SGE (Sun Grid Engine / Oracle Grid Engine): Full feature support. Use ParEnvJobResource for job resources.
• LSF: Supported (tested on CERN lxplus).
• Torque: Full feature support. Use NodeNumberJobResource for job resources.
• direct: A plugin that executes commands in a new bash shell and monitors the PID.

Warning on direct scheduler:

• It is intended for local debugging only.
• It lacks queueing (all jobs run in parallel).
• Do not use on supercomputers; it runs on the login node, which is typically forbidden and can cause AiiDA to lose track of jobs if SSH connections land on different login nodes.

AiiDA provides several core plugins that extend its functionality across various domains, such as workflow management, shell execution, and project isolation. These plugins are not specific to a single research domain but serve as general-purpose extensions to the AiiDA ecosystem.

Key Core Plugins

• AiiDA WorkGraph: Used to efficiently design and manage flexible workflows.
• AiiDA shell: Simplifies the execution of shell commands within AiiDA.
• AiiDA submission controller: Provides classes to manage large numbers of submissions.
• AiiDA hyperqueue: A plugin for the HyperQueue metascheduler that enables sub-node jobs.
• AiiDA project: A project manager that provides custom Python environments and isolated project directories.
• AiiDA code registry: A registry of simulation codes and computers to facilitate easy setup in AiiDA.
• AiiDA plugin cutter: A cookiecutter recipe used to scaffold new AiiDA plugins.
• AiiDA PythonJob (pre-alpha): Allows running non-AiiDA Python functions on a remote computer.

In AiiDA, a code defines what executable should run and how it should be executed. Codes are stored in the provenance graph, which ensures reproducibility and allows you to query all calculations performed with a specific code.

There are three primary code types:

1. Installed: The executable is already present on the target computer. This is recommended for using local compilers and libraries.
2. Portable: The executable and its associated files are stored by AiiDA and automatically copied to the target computer for every execution. This is ideal for small, machine-independent scripts (e.g., Python or bash).
3. Containerized: The executable is part of a container image (e.g., Docker or Singularity). This is the best way to encapsulate complex environments.

Each type is implemented via a code plugin that stores the necessary configuration.

# Example of the mental model: A code is a Node in the provenance graph
# that links a calculation to a specific executable environment.

While the overall structure is flexible, following these conventions helps organize different types of extensions:

• calculations/: Contains CalcJob implementations (e.g., MycodeCalculation).
• parsers/: Contains parser implementations (e.g., BasicMycodeParser).
• data/: Contains code-specific Data formats.
• workflows/: Contains WorkChain implementations.
• commands.py: Contains verdi subcommands for interacting with your plugin.

A minimal package requires only the top-level package and a pyproject.toml file.

aiida-mycode/
   aiida_mycode/
      __init__.py
      calculations/
         __init__.py
         mycode.py      # MycodeCalculation
      parsers/
         __init__.py
         basic.py       # BasicMycodeParser
      data/
         __init__.py    # MyData format
      commands.py       # verdi subcommand
      workflows/
         __init__.py
         basic.py       # basic workflow
   LICENSE
   README.md
   pyproject.toml       # metadata, requirements, entry points