DOCSEARCHbyAlgolia
Submit Docs

Awkward Array

repository·main·Indexed 21 days ago

https://github.com/scikit-hep/awkward

A library for manipulating JSON-like, nested, and variable-sized data using NumPy-like idioms and high-performance compiled operations. It supports arbitrary-length lists, records, and mixed types, providing a dynamically typed extension to NumPy for non-rectangular data structures. The package includes awkward-cpp for optimized C++ performance and provides tools for Apache Parquet integration, CUDA testing, and header-only C++ libraries for direct array construction.

Tokens
89K
Snippets
301
Records
338
Agent score
75%

What's inside Awkward Array

  1. Overview of Awkward Array

    main
    Awkward Array is a library designed for handling nested, variable-sized data. It supports arbitrary-length lists, records, mixed types, and missing data using NumPy-like idioms. This makes it suitable for complex data structures common in scientific computing that do not fit into standard fixed-size arrays.

  2. What is Awkward Array?

    main

    Awkward Array is a library designed for handling nested, variable-sized data. It supports arbitrary-length lists, records, mixed types, and missing data using NumPy-like idioms.

    Key characteristics:

    • Dynamically typed: Arrays can hold complex, nested structures.
    • Fast performance: Operations are compiled and highly efficient.
    • NumPy compatibility: Behavior coincides with NumPy when array dimensions are regular and generalizes when they are not.

  3. Commonly asked topics in Awkward Array

    main

    Based on user surveys, the most frequent areas where developers seek guidance include:

    • Data Integration: Interfacing with Pandas, NumPy, Arrow, and Histogramming.
    • Persistence: Saving and loading data using HDF5 and Parquet, and working with lazy arrays.
    • Physics: Handling Lorentz vectors and TVector3.
    • Restructuring: Using functions like ak.reduce, ak.concatenate, ak.stack, ak.getitem, ak.cross, ak.num, ak.with_field, ak.where, ak.zip, and ak.sort.
    • Data Types: Working with jagged arrays and strings.
    • Performance: Using Numba for fast code.

    If you are looking for specific solutions, many of these topics have documented discussions on GitHub issues and Stack Overflow.

  4. What is awkward indexing?

    main

    Awkward indexing is an extension of NumPy-style indexing designed for ragged arrays. While standard slicing (e.g., array[..., :2]) applies the same slice to every sublist, awkward indexing allows you to pull out a different number of items for each sublist by providing an index array that matches the structure of the target array.

    To use awkward indexing, the index array must satisfy two requirements:

    1. Structural Match: The index array must have a structure matching the array being sliced up to (but not including) the final dimension. You can verify the number of items in each dimension using ak.num(array, axis=n).
    2. Raggedness: The index array must have at least one ragged (var) dimension or contain missing values.
    import awkward as ak
import numpy as np

array = ak.Array(
    [
        [[0.0, 1.1, 2.2], [3.3, 4.4, 5.5, 6.6], [7.7]],
        [],
        [[8.8, 9.9, 10.10, 11.11, 12.12]],
    ]
)

# Example of an awkward index that pulls specific items from the final dimension
index = ak.Array(
    [
        [[], [0], [0]],
        [],
        [[2, 3, 4]],
    ]
)

# Resulting array will have the structure of the index, but values from the original array
result = array[index]

  5. What is AwkwardForth?

    main

    AwkwardForth is a domain-specific language (DSL) and a subset of standard Forth used to create columnar Awkward Arrays from record-oriented data sources. It is specifically designed for scenarios where the data schema or type is discovered at runtime, requiring a dynamic deserialization procedure.

    Use Cases

    • Record-oriented sources: Ideal for ProtoBuf, Avro, and complex ROOT TTrees (e.g., std::vector<std::vector<int>> or unsplit classes).
    • When to avoid: It is not intended for columnar data sources like Apache Arrow, Parquet, or simple ROOT numerical types (e.g., int, float, or std::vector<int>), as these do not require the same runtime deserialization logic.

  6. What is an Awkward Array?

    main

    An Awkward Array is a general tree-like data structure (similar to JSON) that is contiguous in memory and operated upon with compiled, vectorized code. Unlike standard NumPy arrays, Awkward Arrays support:

    • Ragged/Jagged dimensions: Dimensions can have varying lengths (e.g., a list of lists where each sub-list has a different size).
    • Missing values: Dimensions can contain None or missing values.
    • Complex structures: They can store primitive types (numbers, dates, strings), nested records (dictionaries), and unions (mixing different types in the same array).
    • High performance: Computations are performed using optimized kernels, making them significantly faster than pure-Python loops over jagged data.
    import numpy as np
import awkward as ak

# Example of a ragged array with varying lengths
ragged = ak.Array([
    [1, 2, 3],
    [4],
    [5, 6]
])

# Example of an array with records (nested dictionaries)
records = ak.Array([
    [
        {"name": "Benjamin List", "age": 53},
        {"name": "David MacMillan", "age": 53},
    ]
])

  7. What is Awkward Array and how does it compare to other libraries?

    main

    Awkward Array is a data analyst-friendly extension of NumPy-like idioms designed for arbitrary, non-rectangular data structures.

    Key comparisons:

    • Python builtins (lists, dicts): Can handle arbitrary structures but are slow and memory-intensive for large datasets.
    • NumPy: Ideal for rectangular arrays of numbers, but lacks support for arbitrary/nested structures.
    • Pandas/Polars/cuDF/Dask: Well-suited for tabular data and relational indexes, but not for arbitrary nested structures.
    • Apache Arrow: Manages arrays of arbitrary structures with great interoperability, but lacks manipulation functions oriented toward data analysts.

    Awkward Array is intended to be used interchangeably with NumPy and can share data with Arrow and DataFrames. Computations are compiled for speed, and imperative-style computations can be accelerated with Numba.

  8. Create UnionArrays for mixed-type data

    main

    An ak.contents.UnionArray represents a "sum type" (data that is type X OR type Y). It is the most complex node type and contains multiple contents.

    It uses two index-typed attributes:

    • tags: Specifies which content array to use for each element.
    • index: Specifies which element from that content array to use.

    The element at index i is accessed as: contents[tags[i]][index[i]].

    Note: Not all operations (like Numba iteration) support UnionArrays. For small tests, ak.from_iter is the easiest way to create them.

    import awkward as ak
import numpy as np

# UnionArray with tags and indices
layout = ak.contents.UnionArray(
    ak.index.Index8(np.array([0, 1, 2, 0, 0, 1, 1, 2, 2, 0], np.int8)),
    ak.index.Index64(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])),
    [
        ak.contents.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 9.9])),
        ak.from_iter([[], [1], [1, 2], [1, 2, 3], [1, 2, 3, 4], [1, 2, 3, 4, 5], [6], [6, 7], [6, 7, 8], [6, 7, 8, 9]], highlevel=False),
        ak.from_iter(["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine"], highlevel=False),
    ],
)
print(ak.to_list(layout))

  9. How dimension-reducer functions work

    main

    Dimension-reducer (or aggregation) functions replace a list of numbers with a single scalar value. Common operations include adding, multiplying, minimizing, maximizing, or performing logical operations (ak.any and ak.all).

    In Awkward Array, lists are already considered 'grouped', so these functions are applied directly to the nested structures. Unlike relational databases that require a 'group by' operation, Awkward reducers operate on the existing list dimensions.

    import awkward as ak
import numpy as np

array = ak.Array([[1, 2, 3], [4, 5], [], [6]])
# Reduces all values in the nested lists to a single scalar
result = ak.sum(array)

  10. Understand Regular vs Ragged dimensions

    main

    In Awkward, dimensions can be either regular (fixed size) or ragged (variable size).

    • Regular dimensions: Expressed as an integer representing their size in Datashape. The type object contains size information.
    • Ragged dimensions: Expressed as var in Datashape. The corresponding ragged type object does not contain size information because the size is no longer a constant part of the type.

    Example of the difference:

    import numpy as np
import awkward as ak

regular = ak.from_numpy(np.arange(8).reshape(2, 4))
ragged = ak.from_regular(regular)

regular.type.show()  # Shows fixed dimensions
ragged.type.show()  # Shows 'var' for ragged dimensions
    import numpy as np
import awkward as ak

regular = ak.from_numpy(np.arange(8).reshape(2, 4))
ragged = ak.from_regular(regular)

regular.type.show()
ragged.type.show()

  11. What are named axes in Awkward Array?

    main

    Named axes allow you to assign names to the dimensions (axes) of an array. This improves code readability, documentation, and robustness against changes in data structure. Awkward Array automatically propagates these names through high-level operations, indexing, and broadcasting.

    Named axes are inspired by hist and PyTorch Named Tensors and are supported by other libraries like xarray and haliax.

  12. Iterate over GPU arrays in Numba CUDA JIT functions

    main

    Awkward Arrays on GPUs can be used within functions JIT-compiled by @numba.cuda.jit.

    Constraints:

    • You can only perform iteration within these functions.
    • You cannot call ak.* functions inside a Numba CUDA JIT-compiled function.

    For more details, see the how-to-use-in-numba-cuda.md guide.