libfyaml

The generic subsystem uses fy_generic, a compact sum-type representation of YAML/JSON values.

Key benefits:

• Efficiency: Small scalars can be stored directly in machine words; uses arena-based allocators and deduplication.
• Ergonomics: Maps well to sum-type models in languages like Python, Rust, or TypeScript.
• Concurrency: Uses an immutable/functional model where new values are produced from old ones, making multithreaded read-sharing safe without locks.
• Error Handling: Uses a fy_invalid no-op carrier. Operations propagate this state, allowing for straight-line code where errors are checked at the final conversion/use site rather than after every step.

#include <libfyaml.h>
#include <libfyaml/libfyaml-generic.h>

fy_generic doc, updated, emitted;

// Parse file into a generic value
doc = fy_parse_file(FYOPPF_DISABLE_DIRECTORY, "intro-config.yaml");

// Transform values (immutable style)
updated = fy_assoc(doc, "port", 8080);

// Emit the result
emitted = fy_emit(
        updated,
        FYOPEF_DISABLE_DIRECTORY |
        FYOPEF_MODE_YAML_1_2 |
        FYOPEF_STYLE_BLOCK,
        NULL);

// Use the result (e.g., print to stdout)
fputs(fy_cast(emitted, ""), stdout);

libfyaml provides several mechanisms to optimize memory usage:

Deduplication

By default, dedup=True uses a deduplication allocator that stores only one copy of repeated strings or scalars. This is highly efficient for large documents with repeated keys or values. Set dedup=False to use a standard allocator, which may be faster for small documents with little repetition.

Trimming

• trim=True (default): Releases unused allocator pages immediately after parsing is complete.
• trim=False: Prevents automatic releasing of pages. Use this if you plan to modify the document immediately after parsing to avoid reallocation overhead.
• Manual Trim: You can manually trigger memory release using doc.trim().

Cloning

To decouple a part of a document from the original allocator (e.g., to allow the original large document to be garbage collected), use .clone() to create an independent copy.

# Using trim and manual trim
doc = fy.loads(large_yaml, trim=False)
# ... do some work ...
doc.trim()   # release unused memory now

# Cloning a sub-section to keep it alive after the root is deleted
original = fy.load("big.yaml")
part = original.get_at_unix_path("/config/server").clone()
del original   # 'part' is now independent and safe

While libfyaml shares some naming conventions with PyYAML, there are critical differences in behavior and return types:

Key Differences

• Return Type: fy.loads() returns a FyGeneric object, not a native Python dict or list. You must call .to_python() to convert it to a standard Python type.
• Configuration: Instead of Loader classes (e.g., SafeLoader), libfyaml uses a mode string. Use mode='yaml1.1-pyyaml' to approximate PyYAML's SafeLoader behavior.
• YAML Version: libfyaml defaults to YAML 1.2, whereas PyYAML uses YAML 1.1. This affects how types like yes/no, octal integers (0755), and sexagesimal numbers are resolved.
• Strictness: libfyaml follows the YAML specification more strictly (e.g., accepting duplicate anchors or specific characters in anchor names that PyYAML might reject).

Unique libfyaml Features

libfyaml provides several features not available in PyYAML:

• Source markers: Byte/line/column positions via create_markers=True.
• Preservation: Options to keep_comments=True and keep_style=True (for round-tripping).
• Path navigation: Direct document surgery using get_at_unix_path and set_at_unix_path.
• Deduplication: Significant memory savings for repetitive documents.

# PyYAML style
import yaml
result = yaml.safe_load("x: 1")
type(result)          # dict

# libfyaml style
import libfyaml as fy
result = fy.loads("x: 1")
type(result)          # FyGeneric
type(result.to_python())  # dict

# Emulating PyYAML behavior in libfyaml
result = fy.loads("x: 1", mode='yaml1.1-pyyaml')

Transparent Parse Caching allows libfyaml to cache parsed YAML or JSON inputs in a generic arena and reuse that representation on subsequent runs. This is designed to speed up process startup by avoiding re-parsing stable, large inputs.

Key Characteristics

• Opt-in: The cache is disabled by default and must be explicitly enabled.
• Content-Addressed: The cache key is derived from the input bytes and the parser/decoder options. Changing the input or the relevant configuration will result in a cache miss.
• Generic-Based: The cache stores a serialized generic arena. While the core parser API remains unchanged, cache hits are most efficient for consumers using the generic subsystem. For non-generic consumers, libfyaml may need to reconstruct higher-level structures from the cached generic data.
• Performance Paths:
  • Zero-copy fast path: Uses fixed-address mapping of the stored arena.
  • Relocation fallback: Used if fixed-address mapping is not possible.

The generic API uses specific storage types to handle YAML data that doesn't fit into simple scalar types. These types manage how sequences, mappings, and metadata-heavy values are laid out in memory:

• fy_generic_indirect: A wrapper used to attach YAML metadata to a generic value. It is allocated out-of-place and referenced by a fy_generic word with the type tag FY_INDIRECT_V. It stores the actual value and optional metadata (controlled by FYGIF_* flags). Aliases are represented as an indirect type where the value is set to fy_invalid and the anchor holds the target name.
• fy_generic_sequence: Out-of-place storage for a sequence. It consists of a header followed by a contiguous block of count fy_generic items. Note that allocations must remain 16-byte aligned.
• fy_generic_mapping: Out-of-place storage for a mapping. It stores count key/value pairs as a contiguous flexible array of fy_generic_map_pair elements. Allocations must remain 16-byte aligned.
• fy_generic_collection: A generic view over either a sequence or mapping buffer. It shares the memory layout of fy_generic_sequence. For mappings, count represents the number of pairs, and items contains 2 * count interleaved key/value fy_generic elements.

libfyaml aims for round-trip fidelity, meaning comments should appear at approximately the same position relative to the content they annotate, even after operations like key/sequence sorting.

Sort-stability Contract

A fycp_top comment is attached to the token that moves during a sort operation:

• Standalone mapping keys: The fycp_top comment is attached to the key scalar token. When fy_node_mapping_sort moves the key, the comment follows.
• Sequence items starting with a mapping: The fycp_top comment is moved from the key scalar to the BLOCK_MAPPING_START token. When fy_node_sequence_sort moves the sequence item, the comment follows.

Emitter Contract

To prevent comments from shifting left or right after a sort, the emitter reconstructs the comment's column using the formula: block_indent + indent_delta.

• indent_delta must be computed at the time of attachment using the indent level of the block the comment belongs to.
• The emitter must pass the correct old_indent (the block's own indent) when emitting fycp_bottom comments.