Uproot

repository·main·Indexed 43 days ago

https://github.com/scikit-hep/uproot5

A pure Python and NumPy library for reading and writing ROOT files. Designed as an I/O library to stream data efficiently into Python machine learning libraries without requiring a C++ ROOT installation. It provides tools for navigating ROOT files via uproot.open, handling custom class interpretations through CustomInterpretation and the Reader interface, and integrating with libraries like Awkward Array, Pandas, and Dask.

Tokens
24.3K
Snippets
76
Records
99
Agent score
54%

What's inside Uproot

  1. What is Uproot?

    main

    Uproot is a pure Python and NumPy library for reading and writing ROOT files.

    Unlike the standard C++ ROOT implementation or PyROOT, Uproot is an I/O library that does not depend on the C++ ROOT toolkit. It uses NumPy to cast blocks of data from ROOT files directly into NumPy arrays, making it ideal for streaming data into Python-based machine learning libraries.

  2. Understand ReadOnlyDirectory vs ReadOnlyFile

    main

    Uproot 4 distinguishes between two main types for navigating files:

    1. uproot.ReadOnlyDirectory: Returned by uproot.open(). It is used to look up objects and implements the Mapping protocol. It handles directory separators (/) and cycle numbers (;).
    2. uproot.ReadOnlyFile: Accessible via directory.file. It is responsible for the physical Source, ROOT file headers, and class definitions (streamers).

    Navigating a directory:

    directory = uproot.open("https://scikit-hep.org/uproot3/examples/nesteddirs.root")
    # Accessing objects
    tree = directory["one/two/tree"]
    
    # Accessing the underlying file
    file_obj = directory.file
    print(file_obj.uuid)
    print(file_obj.class_named("TTree"))
    directory = uproot.open("https://scikit-hep.org/uproot3/examples/nesteddirs.root")
    tree = directory["one/two/tree"]
  3. Manage caching and memory usage

    main

    Uproot uses two internal caches to streamline interactive use and improve performance:

    1. object_cache: Stores objects like TDirectories, histograms, and TTrees. This allows repeated access to the same object via different names or paths without re-instantiating it.
    2. array_cache: Stores array outputs (up to a maximum number of bytes). It relies on the array having an nbytes or memory_usage attribute (supported by NumPy, Awkward Array, and Pandas).

    Default Settings:

    • object_cache: 100 objects.
    • array_cache: 100 MB.

    Customization: You can override these by passing object_cache or array_cache arguments to uproot.open() or by setting the object_cache and array_cache properties on the ReadOnlyFile object. To disable caching, set these to None. Any MutableMapping (like a standard dict) can be used as a cache.

    # Example of object cache behavior
    file = uproot.open("https://scikit-hep.org/uproot3/examples/hepdata-example.root")
    histogram = file["hpx"]
    # Both references point to the exact same object in memory
    print(histogram, file["hpx"])
  4. Choose an entry point for custom class reading

    main

    Depending on the complexity of your custom data format, you can choose from three different entry points to read custom classes in uproot:

    1. Pre-defined interpretation AsCustom: This is the easiest method. It uses pre-defined Reader classes to read custom classes without requiring extra work from the user.
    2. Define custom Reader: A relatively simple way to implement custom reading rules. You do not need to implement a new interpretation, only a few custom Reader classes.
    3. Define custom Interpretation: The most general and powerful method, but also the most complex. It involves implementing the CustomInterpretation interface.
  5. Find and navigate objects in a ROOT file

    main

    The object returned by uproot.open represents a ReadOnlyDirectory (the root directory /). This object behaves like a Python Mapping (dictionary-like).

    Listing contents

    Use .keys() to see the available objects and paths in the file.

    Accessing objects

    Use square brackets [] to extract objects. You can access nested objects using slashes (/) or by chaining brackets. Note that objects are not read from disk until they are explicitly requested via square brackets or mapping methods like .values() or .items().

    Cycle numbers (e.g., ;1) can be included in the key or omitted.

    Inspecting class names

    To see the class types of objects without actually reading the data from disk, use .classnames().

    Shortcut: Open a specific object directly

    You can jump straight to a specific object by appending a colon (:) followed by the object path to the file path string.

    Note: If your filename contains a colon, wrap the path in a pathlib.Path object to avoid ambiguity.

    import uproot
    
    # Open a file
    file = uproot.open("https://scikit-hep.org/uproot3/examples/nesteddirs.root")
    
    # List keys
    print(file.keys())
    # Output: ['one;1', 'one/two;1', 'one/two/tree;1', 'one/tree;1', 'three;1', 'three/tree;1']
    
    # Access nested objects
    dir = file["one"]
    dir_two = file["one/two"]
    tree = file["one/two/tree"]
    
    # Get class names without reading data
    print(file.classnames())
    
    # Shortcut: Open a specific TTree directly
    events = uproot.open("https://scikit-hep.org/uproot3/examples/Zmumu.root:events")
  6. How the Reader interface works to convert data to awkward.Array

    main

    The Reader interface acts as a node in a 'data tree' that mirrors the structure of an awkward array. The process of converting custom classes to awkward.Array follows five distinct procedures:

    1. Tree Configuration Generation (Python): Recursively traverses streamer information to create a nested dict (the 'tree configuration') describing the structure. Each node in the dict contains a reader type, a name, and necessary metadata (like ctype for CTypeReader or sub_readers for ObjectReader).
    2. Reader Tree Building (Python): Instantiates a tree of Reader objects based on the configuration. Children readers are instantiated first and passed to parent readers.
    3. Data Reading (C++): A BinaryParser is used to read data from the binary stream. The root reader's read method is called for every event, recursively calling children readers.
    4. Raw Data Retrieval (C++): The get_data method is called on the root reader. Leaf readers return numpy arrays; internal/wrapper readers return tuples packaging their own data with their children's data.
    5. Array Reconstruction (Python): The reconstruct_array method uses the original tree configuration and the retrieved raw data to build the final awkward.Array. It uses operations like awkward.unflatten (for sequences) or awkward.zip (for objects) to reconstruct the structure.
    ---
    title: How Reader Works
    ---
    
    flowchart TD
      Z[(All Streamers Information)] -.-> A[Tree Configuration Generating]
        A --> B[(Tree Configuration)]
        A --> C[Reader Tree Building]
        B -.-> C
        C --> D[Data Reading]
        D --> E[Get Raw Data]
        E --> F[Array Reconstruction]
        E --> G[(Raw Data)]
         F -.-> H[(awkward.Array)]
        G -.-> F
        B -.-> F
  7. Handling nested data structures with Awkward Arrays

    main

    In particle physics datasets, entries often have varying numbers of values (e.g., different numbers of particles per collision event). These 'jagged' datasets are most naturally expressed as Awkward Arrays. Using uproot.arrays() returns these structures, allowing you to use Awkward Array's NumPy-like functions for efficient analysis instead of Python loops.

    To filter specific branches while loading, use the filter_name argument with a regular expression.

    >>> events = uproot.open("https://scikit-hep.org/uproot3/examples/HZZ.root:events")
    >>> ak_arrays = events.arrays(filter_name="/(Jet|Muon)_P[xyz]/")
    >>> ak_arrays[:2].tolist()
    [{'Jet_Px': [], 'Jet_Py': [], 'Jet_Pz': [], 'Muon_Px': [-52.89945602416992, 37.7377815246582], ...}, ...]
  8. Implement a custom interpretation using CustomInterpretation

    main

    To implement a fully custom interpretation, you must inherit from the uproot.interpretation.custom.CustomInterpretation base class.

    While the base class provides several methods like cache_key, typename, and final_array, you are responsible for implementing core logic such as match_branch and basket_array.

    Identification via match_branch

    The match_branch method is critical for identification. It is called by uproot.interpretation.identify.interpretation_of for every registered interpretation class. It receives the same arguments as interpretation_of and must return True if the branch should be handled by your custom interpretation, or False otherwise.

  9. Use the AsCustom interpretation for custom classes

    main

    The AsCustom interpretation class and the Reader interface are designed to help users convert custom classes into awkward.Array objects. Instead of implementing a full reading process from scratch, users can focus on implementing custom rules for specific nodes in a data tree.

    Key components of AsCustom:

    • target_branches: set[str]: A class property used to specify which branches should be subject to custom interpretation. You can set this to a set of full branch paths.
    • regularize_object_path: A method used to remove the ;N suffix from TTree object paths, helping to resolve correct object paths during interpretation.
  10. Configure parallel processing for reading

    main

    Uproot supports parallelism in three distinct stages:

    1. Byte Reading: Handled by the Source type. You can influence this via uproot.open options, specifically num_workers and num_fallback_workers.
    2. Decompression: Decompressing TBasket data depends on the uproot.decompression_executor. By default, this is a ThreadPoolExecutor with workers matching your CPU core count. Decompression runs in compiled extensions with the Python GIL released.
    3. Interpretation: Interpreting decompressed data depends on the uproot.interpretation_executor. By default, this is a TrivialExecutor which runs sequentially.

    You can modify these executors locally via function parameters or globally by replacing the uproot.decompression_executor or uproot.interpretation_executor objects.

  11. Navigate TTree branches using Mapping protocols

    main

    Both TTree and TBranch objects implement the Mapping protocol. You can navigate the hierarchy using standard dictionary-like methods:

    • .keys(): Returns branch names.
    • .values(): Returns branch objects.
    • .items(): Returns (name, branch) pairs.
    • .typenames(recursive=True, ...): Returns a mapping of branch names to their C++ type strings.

    Filtering options for .keys(), .values(), .items(), and .typenames():

    • recursive=True (default): Searches directories recursively.
    • filter_name: Select branches by name using a string, glob, regex (e.g., "/pattern/i"), or a function str -> bool.
    • filter_typename: Select branches by their C++ type using the same options as filter_name.
    • filter_branch: Select branches using a function TBranch -> bool (e.g., to filter by compression_ratio).
    >>> tree.keys()
    ['evt', 'evt/Beg', 'evt/I16', ...]
    >>> tree["evt"].keys()
    ['Beg', 'I16', 'I32', ...]
    
    # Filtering examples
    >>> events.keys(filter_name="px*")
    ['px1', 'px2']
    >>> events.keys(filter_name="/p[xyz][0-9]/i")
    ['px1', 'py1', 'pz1', ...]
    >>> events.keys(filter_branch=lambda b: b.compression_ratio > 10)
    ['Run', 'Q1', 'Q2']
  12. Inspect TBranches of a TTree

    main

    A TTree in Uproot behaves like a Mapping, where keys are TBranch names and values are TBranch objects. Since branches can be nested, both TTree and TBranch inherit from HasBranches.

    To inspect the structure of a TTree without reading data:

    • Use .keys() to see available branch names.
    • Use .values() to see the TBranch objects.
    • Use .typenames() to see the data types of each branch.
    • Use .show() for a human-readable summary of names, types, and how the data is interpreted (e.g., how it maps to arrays).
    >>> events = uproot.open("https://scikit-hep.org/uproot3/examples/Zmumu.root:events")
    >>> events.keys()
    ['Type', 'Run', 'Event', 'E1', 'px1', 'py1', 'pz1', 'pt1', 'eta1', 'phi1', 'Q1', 'E2', 'px2', 'py2', 'pz2', 'pt2', 'eta2', 'phi2', 'Q2', 'M']
    >>> events.typenames()
    {'Type': 'char*', 'Run': 'int32_t', 'Event': 'int32_t', 'E1': 'double', ...}
    >>> events.show()
    name                 | typename                 | interpretation
    ---------------------+--------------------------+-------------------------------
    Type                 | char*                    | AsStrings()
    Run                  | int32_t                   | AsDtype('>i4')
    ...