simple-parsing

repository·master·Indexed 19 days ago

https://github.com/lebrice/simpleparsing

A utility library that extends Python's argparse to support strongly typed, structured command-line arguments using dataclasses. It enables modular, reusable, and nested argument configurations, supporting container types, Python Enums, and class inheritance for hierarchical parsing. Key features include the ArgumentParser class for typed parsing, simple_parsing.field() for custom aliases and value restrictions, and automatic help text generation from docstrings and comments.

Tokens
17K
Snippets
44
Records
58
Agent score
68%

What's inside simple-parsing

  1. How type annotations drive conversion in simple-parsing

    master
    In simple-parsing, the type annotation provided for an argument acts as a conversion function. If the attribute type is not a built-in type or a dataclass, the library attempts to use the type annotation to convert the input value. This mechanism allows for seamless integration with custom types like enum.Enum.
  2. Creating subcommands with subparsers

    master

    In simple-parsing, you can create subcommands (similar to git pull vs git commit) by using a Union type annotation on a dataclass attribute.

    When the types within the Union are all dataclasses, simple-parsing automatically generates subparsers for each type. By default, the command name used in the CLI is the lowercased name of the dataclass.

    To customize command names (e.g., mapping multiple names to the same dataclass), use the subparsers function with a dictionary mapping command names to the corresponding types.

    from dataclasses import dataclass
    from typing import Union
    from simple_parsing import ArgumentParser
    
    @dataclass
    class Train:
        train_dir: str = "~/train"
    
    @dataclass
    class Test:
        test_dir: str = "~/test"
    
    @dataclass
    class Program:
        command: Union[Train, Test]
    
    parser = ArgumentParser()
    parser.add_arguments(Program, dest="prog")
    args = parser.parse_args()
    prog: Program = args.prog
    prog.command.execute()
  3. Enable dash/underscore variants in `ArgumentParser`

    master

    By default, simple_parsing.ArgumentParser uses the exact attribute names. However, you can enable add_option_string_dash_variants=True during initialization to automatically allow both dashes (-) and underscores (_) when referring to arguments.

    When this is enabled:

    • If an attribute name contains underscores (e.g., some_value), the parser will automatically accept the dash variant (e.g., --some-value).
    • This behavior extends to custom aliases as well.
    • If an alias contains leading dashes, the parser preserves the number of dashes even when a prefix is added (e.g., a prefix train and an alias -d results in --train.d and -train.d).
    from dataclasses import dataclass
    from simple_parsing import ArgumentParser, field
    
    @dataclass
    class RunSettings:
        debug: bool = field(alias=["-d"], default=False)
        some_value: int = field(alias=["-v"], default=123)
    
    # Enable dash/underscore variants
    parser = ArgumentParser(add_option_string_dash_variants=True)
    parser.add_arguments(RunSettings, dest="train")
    parser.add_arguments(RunSettings, dest="valid")
  4. Access nested arguments via dot notation in CLI

    master

    When using nested dataclasses, simple-parsing generates CLI flags using dot notation to represent the hierarchy. If you register a top-level dataclass with a dest name (e.g., dest="hparams"), the nested fields will be prefixed with that destination name and the field names of the nested classes.

    Example hierarchy:

    • HyperParameters (registered as hparams)
      • gender (a TaskHyperParameters instance)
        • name

    CLI flag: --hparams.gender.name

  5. Automatic help string generation

    master

    The library automatically generates --help documentation using:

    1. The dataclass docstring for the group description.
    2. Inline comments for individual field descriptions.
    3. Type annotations to determine the expected type (e.g., str, float).
  6. Merge multiple dataclass instances using ConflictResolution.ALWAYS_MERGE

    master

    When parsing multiple dataclass instances from the command line, you may want to avoid using distinct prefixes for each instance's arguments. By passing ConflictResolution.ALWAYS_MERGE to the argument parser constructor, simple-parsing will create a single argument for each attribute. This attribute will collect values from all instances into a list (e.g., an attribute of type str will become a list[str], containing one value for each class instance provided).

    # Concept: Passing ConflictResolution.ALWAYS_MERGE to the parser constructor
    # to merge attributes from multiple dataclass instances into lists.
    parser = ArgumentParser(conflict_resolution=ConflictResolution.ALWAYS_MERGE)
  7. Reuse argument groups with prefixing

    master

    You can reuse the same dataclass multiple times by calling add_arguments with different dest values. simple-parsing will automatically prefix the command-line arguments with the destination name to avoid collisions. For example, if dest="train", the field log_dir becomes --train.log_dir on the CLI.

    parser.add_arguments(Options, dest="train")
    parser.add_arguments(Options, dest="valid")
    args = parser.parse_args()
    
    train_options: Options = args.train
    valid_options: Options = args.valid
  8. Use `subgroups` to switch between argument sets

    master

    The subgroups function allows you to define a choice between different configuration dataclasses for a single field. This is useful when you want to switch between different sets of parameters (e.g., different model architectures or dataset configurations) without the complexity and errors often associated with standard argparse subparsers.

    To use subgroups:

    1. Define your specific configuration dataclasses (e.g., ModelAConfig, ModelBConfig).
    2. Define a base/parent class if needed for type hinting.
    3. In your main Config dataclass, assign the field using subgroups(mapping, default=...), where mapping is a dictionary of {name: Class} and default is an instance of the default class.

    When running the CLI, you can select the subgroup using a flag (e.g., --model model_a), and the --help output will dynamically update to show only the arguments relevant to the selected subgroup.

    from dataclasses import dataclass
    from simple_parsing import ArgumentParser, subgroups
    
    @dataclass
    class ModelAConfig:
        lr: float = 3e-4
        optimizer: str = "Adam"
    
    @dataclass
    class ModelBConfig:
        lr: float = 1e-3
        momentum: float = 1.234
    
    @dataclass
    class Config:
        # Select between ModelAConfig and ModelBConfig
        model: ModelAConfig = subgroups(
            {"model_a": ModelAConfig, "model_b": ModelBConfig},
            default=ModelAConfig(),
        )
    
    parser = ArgumentParser()
    parser.add_arguments(Config, dest="config")
    args = parser.parse_args()
    config: Config = args.config
  9. Use ConflictResolution.AUTO for prefixed arguments

    master

    If you want each instance of a dataclass to have a distinct prefix for its arguments when parsing multiple instances, use the ConflictResolution.AUTO option in the argument parser constructor.

    # Concept: Using AUTO to allow distinct prefixes for different instances
    parser = ArgumentParser(conflict_resolution=ConflictResolution.AUTO)
  10. Use dataclasses to define argument groups

    master
    With simple-parsing, you can define groups of related parameters by using Python's @dataclass. This approach allows you to group attributes together and even attach methods to the dataclass, promoting the 'Separation of Concerns' principle by keeping argument-related logic alongside the arguments themselves.
  11. Configure argument conflict resolution in SimpleParsing

    master

    When using multiple dataclasses or nesting them, simple-parsing must decide how to handle overlapping argument names, as argparse requires a flat list of arguments. You can control this behavior by setting the conflict_resolution argument in the simple_parsing.ArgumentParser constructor using the ConflictResolution enum.

    Available strategies:

    • Prefixing (Default): Each individual argument is given a differentiating prefix to allow reuse of the same class multiple times. Use ConflictResolution.AUTO or ConflictResolution.EXPLICIT.
    • Disallow Reuse: Prevents the reuse of arguments. Use ConflictResolution.NONE.
    • List Merging: Parses a list of values instead of a single value and redistributes them to the instances later. Use ConflictResolution.ALWAYS_MERGE.
    import simple_parsing
    from simple_parsing import ConflictResolution
    
    # Example of setting conflict resolution
    parser = simple_parsing.ArgumentParser(conflict_resolution=ConflictResolution.AUTO)
    # Or
    parser = simple_parsing.ArgumentParser(conflict_resolution=ConflictResolution.NONE)
    # Or
    parser = simple_parsing.ArgumentParser(conflict_resolution=ConflictResolution.ALWAYS_MERGE)
  12. Use docstrings and comments for help text

    master

    Simple-parsing allows you to define help text for command-line arguments using three different styles. When multiple styles are used for the same attribute, the library selects the help text based on a specific priority order.

    Help Text Priority Order

    If an attribute has multiple documentation markers, the following order determines which one is used in the --help output:

    1. Docstring below: A multi-line string (""" or ''') on the lines following the attribute.
    2. Comment above: A single or multi-line comment (#) on the line(s) preceding the attribute.
    3. Inline comment: A comment on the same line as the attribute definition.

    Supported Styles

    • Docstring below:
      attr: float = 1.0
      """Docstring below"""
    • Comment above:
      # Comment above
      attr: float = 1.0
    • Inline comment:
      attr: float = 1.0 # inline comment

    Note: For clarity, it is recommended to add blank lines between consecutive attribute assignments when using the 'comment above' or 'docstring below' styles, though this does not affect the --help output.

    from dataclasses import dataclass
    from simple_parsing import ArgumentParser
    
    parser = ArgumentParser()
    
    @dataclass
    class DocStringsExample:
        """Class docstring appearing in the help group."""
    
        attribute1: float = 1.0
        """docstring below, takes highest priority"""
    
        # Comment above, takes second priority
        attribute2: float = 1.0
    
        attribute3: float = 1.0 # inline comment, takes lowest priority
    
    parser.add_arguments(DocStringsExample, "example")
    args = parser.parse_args()