python-hcl2: HCL2 Parser for Python

HCL blocks often have multiple labels (e.g., resource "aws_instance" "main"). Normally, you must traverse each label individually: resource.aws_instance.main.

To skip all name labels and go directly to the block body, append a tilde (~) to the segment. This is useful for querying all blocks of a certain type regardless of their specific names or labels.

Examples:

# Without ~: must name every label
hq 'resource.aws_instance.main.ami' main.tf

# With ~: skip all labels, access body directly
hq 'resource~.ami' main.tf

# All resource blocks regardless of labels
hq 'resource~[*]' main.tf

# Filter blocks by body content
hq 'resource~[select(.ami)]' main.tf

v8 exposes the HCL processing lifecycle as a series of composable functions. This allows for more granular control over the transformation stages.

Forward Pipeline (HCL $\rightarrow$ Dict):

1. parses_to_tree(text): HCL string $\rightarrow$ raw lark.Tree
2. transform(lark_tree): raw lark.Tree $\rightarrow$ StartRule (typed IR)
3. serialize(tree): StartRule $\rightarrow$ dict

Reverse Pipeline (Dict $\rightarrow$ HCL):

1. from_dict(data): dict $\rightarrow$ StartRule
2. reconstruct(tree): StartRule $\rightarrow$ HCL string

The python-hcl2 library operates through a multi-stage pipeline that can be used in its entirety or accessed at individual stages for advanced manipulation.

Forward Pipeline (Parsing): HCL2 Text $\rightarrow$ Lark Parse Tree $\rightarrow$ LarkElement Tree $\rightarrow$ Python Dict

Reverse Pipeline (Serialization): Python Dict $\rightarrow$ LarkElement Tree $\rightarrow$ HCL2 Text

Output Formats

• --ndjson: Use this for directory queries. It streams results, making it compatible with tools like head, grep, and jq. Recommended for large datasets.
• --json: Use this for single files or when you require a single parseable JSON array.

Parallelization

hq automatically parallelizes queries across CPU cores when processing 20+ files. You can control this behavior with the --jobs flag:

• --jobs 0: Force serial execution.
• --jobs N: Set an explicit worker count (where N is the number of workers).

Hybrid mode allows you to combine structural HCL navigation with Python expressions. Use the :: separator to split the structural path (left) from the Python expression (right). In the expression, _ is bound to each result from the structural path.

Expression Normalization: You don't need to prefix attributes with _. For example, name_labels is automatically treated as _.name_labels.

# Get name_labels for all variables
hq 'variable[*]::name_labels' variables.tf

# Get block_type
hq 'variable[*]::block_type' variables.tf --value

# Call methods
hq 'resource.aws_instance[*].tags::entries()' main.tf

# Use builtins
hq 'variable[*]::len(_.name_labels)' variables.tf --value

When with_comments=True (the default), comments are captured in __comments__ (standalone line comments) and __inline_comments__ (comments inside expressions). Each comment is an object with a "value" key containing the text.

Note: Comments are read-only; they are captured during parsing but are not restored when converting a dict back to HCL2 via dump/dumps.

from hcl2 import loads, SerializationOptions

data = loads(
    "# Configure the provider\nx = 1\n",
    serialization_options=SerializationOptions(with_comments=True),
)

print(data["__comments__"])
# Output: [{"value": "Configure the provider"}]

For large datasets (20+ files), hq automatically uses multiprocessing when using --json or --ndjson output modes. You can control the worker count using the --jobs flag.

# Auto-parallel (default for 20+ files with --json/--ndjson)
hq 'resource[*]' large-monorepo/ --ndjson

# Force serial processing
hq 'resource[*]' large-monorepo/ --ndjson --jobs 0

# Explicit worker count
hq 'resource[*]' large-monorepo/ --ndjson --jobs 8

You can extract multiple fields per result in a single query by constructing an object {key: .field}. This is highly effective when using --ndjson for downstream processing.

Example: Instance types for cost analysis

hq 'resource.aws_instance~[*] | {name: .name_labels, type: .instance_type}' infra/ --ndjson

Example: CPU and memory for ECS services

hq 'module~[select(.cpu)] | {cpu, memory}' infra/ --ndjson

Example: Resource inventory with tags

hq 'resource~[*] | {type: .name_labels, tags}' infra/prod/ --ndjson

# Extract name and type
hq 'resource.aws_instance~[*] | {name: .name_labels, type: .instance_type}' infra/ --ndjson

# Extract cpu and memory from modules
hq 'module~[select(.cpu)] | {cpu, memory}' infra/ --ndjson

# Extract type and tags
hq 'resource~[*] | {type: .name_labels, tags}' infra/prod/ --ndjson

When querying large monorepos, use the --ndjson flag to emit one JSON object per line. This is ideal for streaming results to other tools like jq without loading the entire result set into memory.

# One JSON per result, compact
hq 'resource[*]' dir/ --ndjson

# With source location
hq 'resource[*]' dir/ --ndjson --with-location

# Pipe to jq for further processing
hq 'resource[*]' dir/ --ndjson | jq '.tags'