OmegaConf Documentation

repository·main·Indexed 25 days ago

https://github.com/omry/omegaconf

A hierarchical configuration system providing a consistent API for managing configurations merged from YAML files, Python dataclasses, and CLI arguments. Features include dotlist serialization, node protection (readonly and protected), custom resolver annotation validation, and an optional pydevd plugin for enhanced debugger integration.

Tokens
29.8K
Snippets
57
Records
137
Agent score
78%

What's inside OmegaConf

  1. Overview of OmegaConf hierarchical configuration

    main

    OmegaConf is a hierarchical configuration system. It allows you to merge configurations from multiple disparate sources into a single, consistent API. Supported sources include:

    • YAML configuration files
    • Python dataclasses and objects
    • Command Line Interface (CLI) arguments
  2. Overview of OmegaConf

    main
    OmegaConf is a hierarchical configuration system based on YAML. It allows you to merge configurations from multiple sources—including YAML files, CLI arguments, and environment variables—into a single unified object. It provides a consistent API for accessing configuration values regardless of their origin and supports runtime type safety through the use of Structured Configs.
  3. Interpolated values are validated and converted to annotated types

    main

    When accessing an interpolated field in a Structured config, OmegaConf validates the value and attempts to convert it to the annotated type. For example, if an int field interpolates a str that contains digits, it will be automatically converted to an integer.

    Note: This validation step is currently skipped for container node interpolations (e.g., interpolating an int into a Dict).

    from omegaconf import II, OmegaConf
    from dataclasses import dataclass
    
    @dataclass
    class Interpolation:
        str_key: str = "string"
        int_key: int = II("str_key")
    
    cfg = OmegaConf.structured(Interpolation)
    
    # Accessing cfg.int_key when str_key is 'string' raises InterpolationValidationError
    # But if we update the source:
    cfg.str_key = "1234"
    assert cfg.int_key == 1234  # Automatically converted to int
  4. Understand the differences between `readonly` and `protected`

    main

    When securing configuration, choose between readonly and protected based on the required scope and type of protection:

    Featurereadonlyprotected
    InheritanceCascades to all descendantsNon-inheriting (only the specific node is protected)
    Write-throughBlocks writes; raises ReadonlyConfigErrorBlocks writes; raises ProtectedNodeError
    Parent DeletionDoes not prevent del or pop via parentPrevents del or pop via parent; raises ProtectedNodeError
    ReplacementDoes not prevent replacing a container via parentPrevents replacing a container via parent; raises ProtectedNodeError
    RemedyOmegaConf.read_write(cfg)unprotect(cfg, ...) context manager
  5. Understand TupleConfig immutability and mutation rules

    main

    A TupleConfig is a structurally immutable sequence. Unlike ListConfig, it does not support in-place modifications.

    What is forbidden:

    • Item assignment (e.g., cfg[0] = value)
    • Slice assignment
    • Deletion, insertion, append(), extend(), or sort()
    • In-place concatenation
    • Using OmegaConf.update() to target an index (e.g., OmegaConf.update(cfg, "field.0", value))

    What is allowed:

    • Reading: Indexing, iteration, and slicing are supported. Slicing returns a new TupleConfig.
    • Sequence Operations: + (concatenation), * (repetition), count(), and index() are supported and return new TupleConfig instances.
    • Nested Mutability: While the TupleConfig structure itself is immutable, any nested containers inside the tuple remain mutable (matching standard Python tuple behavior).
    • Immutability vs Read-only: Tuple immutability is intrinsic and independent of the readonly flag. Calling read_write() or changing the readonly flag will not allow structural mutation of a TupleConfig.
  6. Understand the proposed Provenance Tracking model

    main

    OmegaConf is proposing a general per-node provenance model to allow users to map a configuration node back to its origin. This is intended to support IDE features like 'go to definition', diagnostics, and tracking which merged source 'won' during configuration composition.

    Instead of just tracking YAML line/column numbers, the model is designed to be abstract to support various sources including:

    • Filesystem: Local YAML files.
    • Hydra Sources: Packaged configs, plugin-backed sources, or composed configs.
    • Python: Nodes created via OmegaConf.create().
    • Synthetic/Merge: Nodes created through operations like merging or manual mutation.

    Each node may optionally carry metadata including kind (e.g., file, python, merge), source (path or URI), line, column, and span.

  7. Understand OmegaConf interpolation syntax

    main

    OmegaConf uses an ANTLR-based grammar to parse string expressions, primarily for interpolations. An interpolation string is any string containing the ${ character sequence. These strings can be a single interpolation or a concatenation of multiple fragments (interpolations and regular strings).

    Examples of interpolation strings:

    • ${foo.bar}
    • https://${host}:${port}
    • Hello ${name}
    • ${a}${oc.env:B}${c}
    from omegaconf import OmegaConf
  8. Core concept: How TypeAdapters work

    main

    A TypeAdapter acts as a boundary between external Python values and OmegaConf nodes. It allows OmegaConf to represent external types (like numpy.ndarray or torch.Tensor) as structured configuration nodes while preserving the ability to materialize them back into native Python objects.

    Key Behaviors

    • Scalar Adapters: When accessing a scalar (e.g., np.float32), the native type is preserved. type(cfg.lr) is np.float32 will be True rather than being coerced to a standard Python float.
    • Composite Adapters: When accessing a composite object (e.g., an array), it is accessed as an OmegaConf container (ListConfig or DictConfig). To get the original object back, use OmegaConf.to_object(cfg).

    Lifecycle

    1. Node Creation: When a structured config is built, OmegaConf validates the type annotation against registered adapters. For untyped assignment, OmegaConf uses the value's MRO to find a matching adapter.
    2. Value Assignment: Subsequent values are passed through adapter.convert() for validation and coercion.
    3. Materialization: adapter.from_node() is called to turn the stored OmegaConf nodes back into the original Python type.
    cfg.lr = np.float32(0.01)
    type(cfg.lr) is np.float32           # True — scalar, not coerced to float
    
    cfg.weights = np.array([1.0, 2.0, 3.0])
    type(cfg.weights) is ListConfig      # True — composite, accessed as OmegaConf container
    OmegaConf.to_object(cfg).weights     # np.array([1.0, 2.0, 3.0])  — materialized
  9. Manage adapter versioning and fallbacks

    main

    Versioning

    Use HandledType.version to track the representation version of a specific type. You must bump this version if to_node() changes its output in a way that breaks compatibility with older stored representations (e.g., renaming fields, changing types, or adding required fields). Adding optional fields with defaults does not require a bump.

    Fallbacks

    Fallbacks allow a specialized adapter to store a representation that a more general adapter can read.

    • Use compatible_versions to provide an exact list of working versions.
    • Use compatible_version_range to provide a policy (e.g., " >=3,<5 ").
    • The special range "*" explicitly accepts future compatibility risks.
    • Note: Fallback is only one level deep; chained fallback is not supported.
  10. Use variable interpolation in configurations

    main

    OmegaConf supports lazy variable interpolation using the ${} syntax. Interpolations are evaluated when the node is accessed.

    Config Node Interpolation: Interpolated values can point to other nodes in the configuration using dot-notation (foo.bar), brackets ([foo][1]), or a mix (foo[1]).

    • Absolute Interpolation: ${path.to.node}
    • Relative Interpolation: Prefixed with dots. ${..foo} points to the foo sibling of the current node's parent.

    Nested Interpolation: You can nest interpolations to dynamically select sub-configs, e.g., ${plans[${selected_plan}]}.

    Custom Resolvers: You can add new interpolation types by registering custom functions using OmegaConf.register_resolver(name, function).

  11. How `to_object()` handles adapter-derived nodes

    main

    When calling to_object() on a configuration containing adapter-derived nodes, OmegaConf prioritizes the adapter's materialization logic over the standard SCMode.INSTANTIATE path. The behavior depends on the node type:

    • Scalar adapter node: Returns the stored value directly, as it is already considered the external type.
    • Composite adapter node: Calls adapter.from_node(node), which returns the external object (e.g., a class instance) rather than the internal OmegaConf representation (e.g., a DictConfig).

    Note: For to_object() to work on adapted nodes, the corresponding adapter must be registered in the current process.

  12. Use Optional fields in Structured configs

    main

    To allow a field to hold a None value, use Python's typing.Optional annotation. Regular fields (without Optional) will raise a ValidationError if you attempt to assign None to them.

    from typing import Optional, Dict, List
    from omegaconf import OmegaConf
    from dataclasses import dataclass, field
    
    @dataclass
    class Modifiers:
        num: int = 10
        optional_num: Optional[int] = 10
        optional_dict: Optional[Dict[str, int]] = None
        list_optional: List[Optional[int]] = field(default_factory=lambda: [10, MISSING, None])
    
    conf = OmegaConf.structured(Modifiers)
    
    # This is allowed because of Optional annotation
    conf.optional_num = None
    assert conf.optional_num is None