asdf

An ASDF schema is a YAML document that follows specific structural conventions. Below is a breakdown of a typical schema component:

• %YAML 1.1 and ---: Header indicating YAML 1.1 version and the start of the document.
• $schema: The URI of the metaschema (e.g., http://stsci.edu/schemas/yaml-schema/draft-01).
• id: A unique URI identifying the schema (e.g., asdf://asdf-format.org/core/schemas/quantity-2.0.0). This is how the schema is referenced in the ASDF library.
• title / description: Optional documentation strings.
• type: Defines the expected data type (e.g., object, number).
• properties: Defines named properties of a mapping to be validated.
• anyOf, allOf, oneOf, not: JSON Schema combiners used to create complex validation logic.
• tag: An ASDF-specific validator used to assert the YAML tag of an object. Supports wildcards (e.g., tag:stsci.edu:asdf/core/ndarray-1.*).
• required: A list of property names that must be present.
• ...: The YAML document end indicator.

%YAML 1.1
---
$schema: http://stsci.edu/schemas/yaml-schema/draft-01
id: asdf://asdf-format.org/core/schemas/quantity-2.0.0

title: Quantity object containing numeric value and unit
description: >-
  An object with a numeric value, which may be a scalar
  or an array, and associated unit.

type: object
properties:
  value:
    description: A vector of one or more values
    anyOf:
      - type: number
      - tag: tag:stsci.edu:asdf/core/ndarray-1.*
  unit:
    description: The unit corresponding to the values
    tag: tag:stsci.edu:asdf/unit/unit-1.*
  required: [value, unit]
...

ASDF files can reference items in other ASDF files using the JSON Pointer syntax. You can create these references in two ways:

1. Using the Python API: Use the AsdfFile.make_reference(path_list) method. The path_list should be a list of keys representing the path to the target item in the source file.
2. Manual JSON Pointer: Manually write a dictionary with the key '$ref' and a value following the pattern 'filename.asdf#path/to/item'.

Once references are in the tree, you can manage them using:

• AsdfFile.find_references(): Looks up all references so they can be used as if they were local to the tree. This does not move data; it keeps them as references.
• AsdfFile.resolve_references(): Replaces all external references with their actual content directly in the tree. When you write the file back to disk, the references are gone and replaced by the literal data.

import asdf
from asdf import AsdfFile

# Method 1: Using make_reference
ff = AsdfFile()
with asdf.open('target.asdf') as target:
    ff.tree['my_ref_a'] = target.make_reference(['a'])

# Method 2: Manual JSON Pointer
ff.tree['my_ref_b'] = {'$ref': 'target.asdf#b'}
ff.write_to("source.asdf")

# Managing references
with asdf.open('source.asdf') as ff:
    ff.find_references()  # Access data via references
    ff.resolve_references()  # Replace references with actual data

ASDF uses URIs to identify four specific types of entities. For all types, it is recommended to use the asdf:// scheme with a pattern that includes a domain you control, a project name, the entity type, and the name/version.

Schemas

Schemas must include an id property containing their identifying URI. This URI is used when referring to the schema in asdf library functions. Recommended pattern: asdf://<domain>/<project>/schemas/<name>-<version>

Tags

Tags annotate typed objects in an ASDF YAML tree. Unlike schemas, tags are not associated with a downloadable resource; the URI itself communicates the type of the YAML object. Recommended pattern: asdf://<domain>/<project>/tags/<name>-<version>

Manifests

Manifest documents are language-independent definitions of ASDF extensions. They must include an id property containing their identifying URI. Recommended pattern: asdf://<domain>/<project>/manifests/<name>-<version>

Extensions

Extensions URIs are included in an ASDF file's metadata to advertise that additional software support is required to interpret the file. Like tags, these are not associated with a specific resource. Recommended pattern: asdf://<domain>/<project>/extensions/<name>-<version>

# Examples of recommended URI patterns:

Schema:      asdf://example.com/example-project/schemas/foo-1.2.3
Tag:         asdf://example.com/example-project/tags/foo-1.2.3
Manifest:    asdf://example.com/example-project/manifests/foo-1.2.3
Extension:   asdf://example.com/example-project/extensions/foo-1.2.3

Because ASDF is an archival format, extension authors must ensure backwards compatibility so that files created with older extensions remain readable.

Versioning Principles

• Semantic Versioning: Use SemVer for versioning tags, schemas, and extensions.
• Independence: Versions for tags and schemas do not need to move in lock-step with each other.
• Breaking Changes: Any change that breaks backwards compatibility must increment the major version.
• Schema Evolution: To ensure archival compatibility, never modify an existing schema file. When a schema version is increased, create a new schema file (e.g., xyz-1.1.0) that exists in parallel with the old one (xyz-1.0.0).

The Update Workflow

When upgrading a schema (e.g., from 1.0.0 to 1.1.0), you typically need to:

1. Create a new schema file for the new version.
2. Create a new tag (e.g., tag/xyz-1.1.0).
3. Update the Converter to support both the old and new tags (unless using wildcards).
4. Create a new manifest version that lists the new tag and schema.
5. Create a new Extension using the new manifest. This extension should appear earlier in the registered extensions list than the old version.

How ASDF handles multiple versions

• Reading: When opening an old file, ASDF checks extensions in order. It will skip the new extension if the manifest doesn't contain the old tag, eventually finding the old extension that supports it.
• Writing: When writing a file, ASDF checks the list of extensions. It will select the first (newest) extension that supports the current type, resulting in the new tag being used.

When building complex schemas, you can reference other schemas using either the ASDF tag validator or the standard JSON Schema $ref keyword. Each has distinct trade-offs.

Using the tag validator

This is the recommended way to reference schemas in ASDF because it allows for flexible version matching using wildcards (e.g., tag:example/schema-1.*).

• Pros:
  • Supports wildcards for minor/bugfix version updates without breaking the referring schema.
  • Avoids duplicate validation: The schema associated with the tag is not re-validated during the referring schema's process.
• Cons:
  • It is a custom ASDF validator; non-ASDF tools may not support it.
  • Requires the target object to have a specific tag.

Using $ref

This is the standard JSON Schema way to reference other schemas.

• Pros:
  • Standardized and widely supported by most JSON Schema validators.
• Cons:
  • Duplicate Validation: If a tagged object is checked with a $ref, it is validated twice (once by the tag and once by the $ref).
  • No Wildcards: Requires an exact version match (down to the bugfix version). Any update to the referenced schema requires an update to the referring schema.