jsonargparse

repository·main·Indexed 19 days ago

https://github.com/omni-us/jsonargparse

A library for creating command-line interfaces (CLIs) and making Python applications configurable using type hints and docstrings. It allows for minimal effort CLI derivation from type hints and supports parsing from the command line, config files, and environment variables. Key features include the `auto_cli` function for automated CLI generation, `ArgumentParser` for low-level control, and support for complex nested structures via Python type hints.

Tokens
48.3K
Snippets
151
Records
194
Agent score
65%

What's inside jsonargparse

  1. Key features of jsonargparse

    main

    jsonargparse provides several advanced capabilities for CLI and configuration management:

    • Type Hint Support: Extensive support for nested types (Union, Optional), containers (list, dict), protocols, user-defined generics, restricted types (regex, numbers), paths, URLs, and PEP 563/604 annotations.
    • Configuration Files: Supports json, yaml, toml, and jsonnet. It handles structured (non-flat) hierarchies and relative paths within configs.
    • Variable Interpolation: Powered by OmegaConf.
    • Dependency Injection: Supports types that expect a class instance or callables that return a class instance.
    • Introspection: Resolves parameters used via **kwargs.
    • Argument Linking: Directs parsed values to multiple parameters to prevent unnecessary interpolation.
    • Tab Completion: Supported via shtab or argcomplete.
  2. Use Jsonnet for configuration files

    main

    To use Jsonnet files as configuration, you must install the jsonnet and jsonschema packages (e.g., pip install jsonargparse[jsonnet]).

    There are two ways to use Jsonnet:

    1. Global Parser Mode: Set parser_mode='jsonnet' when instantiating ArgumentParser. This makes all parsing methods (parse_args, parse_path, parse_string) expect Jsonnet format instead of YAML.

    2. Specific Argument Action: Use ActionJsonnet to allow specific arguments to be Jsonnet files or strings, while keeping the rest of the parser in YAML mode. This is useful for parametrized Jsonnet files that require external variables.

    When using ActionJsonnet, you can specify an argument to serve as the source for external variables. Note: The external variables argument must be provided in the command line before the Jsonnet path so the dictionary is available during parsing.

    from jsonargparse import ArgumentParser, ActionJsonnet
    
    # Method 1: Global Jsonnet mode
    parser = ArgumentParser(parser_mode="jsonnet")
    parser.add_argument("--config", action="config")
    cfg = parser.parse_args(["--config", "example.jsonnet"])
    
    # Method 2: Using ActionJsonnet for parametrized files
    parser = ArgumentParser()
    parser.add_argument("--in_ext_vars", type=dict)
    parser.add_argument("--in_jsonnet", action=ActionJsonnet(ext_vars="in_ext_vars"))
    
    # Usage: external vars must come before the jsonnet path
    cfg = parser.parse_args(["--in_ext_vars", '{"param": 123}', "--in_jsonnet", "example.jsonnet"])
  3. Handle Callable types

    main

    When an argument is typed as Callable, you have three primary options:

    1. Import Path: Provide a string representing the import path of a callable (e.g., --callable=time.sleep).
    2. Instantiable Class: Provide a configuration object (as a JSON string) containing a class_path and init_args. This creates a class instance that is itself callable.
    3. Instance Factories: (See dependency injection documentation) Use a callable that returns class instances.
    # Option 2: Instantiable Class
    # Using a JSON string to define class_path and init_args
    value = {
        "class_path": "__main__.OffsetSum",
        "init_args": {
            "offset": 3,
        },
    }
    cfg = parser.parse_args(["--callable", str(value)])
    init = parser.instantiate(cfg)
    init.callable(5) # Returns 8
  4. How parameter resolvers work

    main

    To determine the correct names and types for a parser, jsonargparse uses three layers of resolvers in a specific order:

    1. AST Resolver: Analyzes Python source code using the ast library to trace how *args and **kwargs are used (e.g., passed to other functions, stored in self, or used in conditional calls). This is the primary method.
    2. Assumptions Resolver: A fallback used only if AST fails. It assumes that if __init__ has *args or **kwargs, they are being forwarded to a parent class via super().__init__(*args, **kwargs). It then collects parameters from parent classes.
    3. Stubs Resolver: An optional layer that uses *.pyi stub files (via the typeshed-client package) to identify parameters and type hints. This is particularly useful for the Python standard library.

    To debug resolver failures or unsupported cases, set the environment variable JSONARGPARSE_DEBUG=true.

  5. Define nested namespaces using dot notation

    main

    Unlike standard argparse, jsonargparse allows you to create hierarchical namespaces using dot notation in argument names (e.g., --lev1.opt1).

    Using Dataclasses for Groups

    You can group nested options by using a dataclass as the type for a parent argument. This promotes reusability.

    Accessing Values in Namespace

    The returned Namespace object allows accessing nested keys in two ways:

    1. Dictionary-style: cfg['lev1']['opt1'] or cfg['lev1.opt1'].
    2. Attribute-style: cfg.lev1.opt1.
    3. Conversion: Use cfg.as_dict() to convert the nested namespace into a standard nested dictionary.
    from dataclasses import dataclass
    
    @dataclass
    class Level1Options:
        opt1: str = "from default 1"
        opt2: str = "from default 2"
    
    parser = ArgumentParser()
    parser.add_argument("--lev1", type=Level1Options, default=Level1Options())
    
    cfg = parser.parse_args([])
    print(cfg.lev1.opt1) # 'from default 1'
    print(cfg['lev1.opt1']) # 'from default 1'
  6. Identify public APIs in jsonargparse

    main

    While many objects in the package may be importable, they are not considered public. Truly public objects are strictly those listed in:

    • jsonargparse.__all__
    • jsonargparse.typing.__all__

    Modules starting with _ are private implementation details, and objects within them prefixed with _ are internal to that specific module.

  7. Implement Dependency Injection in jsonargparse

    main

    Dependency injection is supported by using specific type hints in your class or function signatures. This allows the parser to handle object instantiation automatically.

    Two primary patterns are supported:

    1. Class Type Hints: Use a class type to accept an instance of that class or any subclass.
      • Example: module: ModuleBaseClass (accepts an instance of ModuleBaseClass).
    2. Callable Type Hints: Use a callable that returns an instance of a class.
      • Example: module: Callable[[int], ModuleBaseClass] or using a Protocol like module: ModuleFactoryProtocol.

    This decoupos the usage of an object from its specific instantiation logic.

  8. Use Instance Factories for dependency injection

    main

    Instance factories are callables that return class instances. This is useful for dependency injection when a class requires parameters that are only available after injection (e.g., an optimizer that needs model parameters).

    When instantiate() is called, a partial function is provided. You can define these factories using two approaches:

    1. Using Callable

    Use Callable[[ArgType1, ArgType2], ReturnType] in the type hint.

    • Limitation: Only supports positional and unnamed parameters.
    • Example: type=Callable[[Iterable], Optimizer]

    2. Using Protocol

    Define a Protocol with a __call__ method. This is the preferred method if you need to support keyword arguments during the injection phase.

    • Example:
      class OptimizerFactory(Protocol):
          def __call__(self, params: Iterable) -> Optimizer: ...

    Default Values: You can provide a default factory using a lambda in a class signature. Note that add_argument does not support AST resolving for lambdas; in that case, use a dictionary with class_path and init_args as the default value.

    from typing import Callable, Iterable, Protocol
    
    class Optimizer:
        def __init__(self, params: Iterable):
            self.params = params
    
    class SGD(Optimizer):
        def __init__(self, params: Iterable, lr: float):
            super().__init__(params)
            self.lr = lr
    
    class OptimizerFactory(Protocol):
        def __call__(self, params: Iterable) -> Optimizer: ...
    
    parser = ArgumentParser()
    # Using Protocol to allow keyword arguments like params=[1, 2]
    parser.add_argument("--optimizer", type=OptimizerFactory)
    
    value = {
        "class_path": "__main__.SGD",
        "init_args": {"lr": 0.02},
    }
    cfg = parser.parse_args(["--optimizer", str(value)])
    init = parser.instantiate(cfg)
    
    optimizer = init.optimizer(params=[6, 5])
    print(optimizer.lr) # 0.02
  9. Configure `auto_cli` with Classes and Subcommands

    main

    When using auto_cli with a class, the CLI structure follows the class hierarchy.

    • Subcommands: Methods of the class become subcommands.
    • as_positional=False: Use this flag to make required arguments non-positional (i.e., they must be passed as --key=value).
    • return_instance=True: Forces the parser to only look at __init__ arguments and return a class instance, ignoring other methods as subcommands.
    • Nested Dicts: Passing a dictionary allows you to define custom subcommand names and help text.
    from random import randint
    from jsonargparse import auto_cli
    
    class Main:
        def __init__(self, max_prize: int = 100):
            """Args:
                max_prize: Maximum prize.
            """
            self.max_prize = max_prize
    
        def person(self, name: str):
            """Args:
                name: Name of winner.
            """
            return f"{name} won {randint(0, self.max_prize)}€!"
    
    if __name__ == "__main__":
        # This will create a subcommand 'person'
        print(auto_cli(Main))
  10. Distinguish between omitted and null values using Unset

    main

    By default, jsonargparse (following argparse behavior) assigns None to arguments not provided on the command line. To distinguish between an argument that was omitted and one that was explicitly set to null, use the Unset sentinel.

    Setup: Enable the sentinel using set_parsing_settings(unset_sentinel=True).

    Argument States:

    1. Unset: The argument was not provided and no default was specified in add_argument.
    2. None: The argument was explicitly set to null (e.g., --opt=null) OR add_argument included default=None.
    3. Any other value: The argument was provided with a specific value.

    Filtering Unset values: When using dump(), save(), or validate(), you can use the skip_unset parameter to exclude Unset entries. For CLI usage, use the --print_config=skip_unset flag.

    from jsonargparse import ArgumentParser, Unset, set_parsing_settings
    
    set_parsing_settings(unset_sentinel=True)
    
    parser = ArgumentParser()
    parser.add_argument("--num", type=int | None)                 # no default given
    parser.add_argument("--flag", type=int | None, default=None)  # explicit None
    
    # Case 1: Omitted
    cfg = parser.parse_args([])
    assert cfg.num is Unset
    assert cfg.flag is None
    
    # Case 2: Explicitly set to null
    cfg = parser.parse_args(["--num=null"])
    assert cfg.num is None
    
    # Case 3: Provided value
    cfg = parser.parse_args(["--num=5"])
    assert cfg.num == 5
  11. Parse URLs and remote filesystems

    main

    By using the u (URL via requests) or s (filesystem via fsspec) flags with path_type, you can parse URLs as if they were local paths. The .read_text() method will automatically perform a GET request to retrieve content.

    Configuration:

    • URL support: Requires the urls extra (pip install jsonargparse[urls]).
    • fsspec support: Requires the fsspec extra (pip install jsonargparse[fsspec]).
    • Global Settings: To allow loading entire configuration files from URLs (e.g., my_tool.py --config http://example.com/config.yaml), enable the following settings:
    from jsonargparse import set_parsing_settings
    set_parsing_settings(config_read_mode_urls_enabled=True)

    Remote Path Resolution:

    Relative paths found inside a remote config are resolved relative to the remote location. For example, a relative path model/state_dict.pt inside s3://bucket/config.yaml resolves to s3://bucket/model/state_dict.pt.

    from jsonargparse import ArgumentParser, path_type
    
    # 'fur' = file (f), url (u), readable (r)
    Path_fur = path_type('fur')
    
    parser = ArgumentParser()
    parser.add_argument("--data", type=Path_fur)
    
    # This works for both local files and URLs
    cfg = parser.parse_args(["--data", "http://example.com/data.txt"])
    print(cfg.data.read_text())
  12. Manipulate dictionary arguments

    main

    Arguments with dict type can be managed in two ways:

    1. Full Replacement: Provide a JSON-formatted dictionary string. This replaces any existing value.
    2. Individual Item Updates: Use dot notation (--dict.key=value) to set or update specific keys without replacing the entire dictionary.
    parser = ArgumentParser()
    parser.add_argument("--dict", type=dict)
    # Full replacement
    parser.parse_args(['--dict={"key1": "val1", "key2": "val2"}'])
    # Individual item updates
    parser.parse_args(["--dict.key1=val1", "--dict.key2=val2"])