DOCSEARCHbyAlgolia
Submit Docs

Overture Schema

repository·main·Indexed 42 days ago

https://github.com/overturemaps/schema

Defines the data structures for Overture Maps using a Pydantic-based Python model system. Includes tools for schema validation via overture-schema-cli and overture-validate, documentation generation with overture-codegen, and PySpark integration for validating large-scale DataFrames.

Tokens
83.1K
Snippets
197
Records
376
Agent score
53%

What's inside Overture Schema

  1. Overview of Overture Schema Base Theme

    main

    The overture-schema-theme-base package provides the foundational building blocks for Overture Maps' theme-based schema architecture. It is designed to be used as a dependency by other theme packages (such as theme-addresses, theme-buildings, etc.) to ensure consistency across the ecosystem.

    Key responsibilities include:

    • Shared Structures: Defining common data structures used across multiple themes.
    • Validation Logic: Providing centralized validation utilities to ensure data integrity.
    • Foundational Patterns: Establishing the core patterns that all Overture Maps themes must follow.

  2. Overview of the Overture Schema Divisions Theme

    main

    The overture-schema-divisions package provides shared structures and validation logic specifically for the Overture Maps divisions theme. It is designed to handle geographic and administrative boundaries by providing:

    • Common place types: Standardized definitions for various types of divisions.
    • Administrative hierarchy utilities: Tools to manage and navigate the relationships between different levels of administrative entities (e.g., country $\rightarrow$ state $\rightarrow$ county).
    • Validation patterns: Reusable logic to ensure data conforms to the expected schema for divisions.

  6. Overview of the Overture Schema Code Generator

    main

    The Overture Schema Code Generator is a tool designed to produce documentation and code from Overture Maps Pydantic schema definitions. It solves the problem of losing semantic information (like NewType names, custom constraints, and domain-specific primitives) when using standard JSON-schema extraction.

    Instead of relying on Pydantic's flattened JSON schema, the generator introspects the Python type system to extract a tree-shaped Intermediate Representation (IR) called FieldShape. This preserves the full domain language of the schema, including nested Annotated constraints and semantic NewType wrappers.

    Key Outputs:

    • Markdown Documentation: Pages containing field tables, cross-page links, constraint descriptions, and examples.
    • PySpark Validation Modules: Per-model expression builders, StructType schemas, a feature registry, and conformance test modules.

  7. Overview of Overture Schema packages

    main

    The Overture schema is implemented as Pydantic models and distributed via several independent Python packages.

    Core Packages

    • overture-schema: The umbrella package. This is the primary package consumers should pin to their dependencies to get a coherent set of all themes and support packages.

    Themes

    Each theme package contains specific feature types and shared structures:

    • theme-addresses
    • theme-base
    • theme-buildings
    • theme-divisions
    • theme-places
    • theme-transportation

    Foundation and Tooling

    • Foundation: system (base types) and common (shared structures like names and sources).
    • Tooling: cli (validation and JSON Schema generation), codegen (docs and code generation), validation (union-based validation), and pyspark (Spark validation expressions).

  8. Use the Overture Schema Places Theme

    main

    The overture-schema-theme-places package provides shared structures and validation logic specifically for the Overture Maps places theme. It is designed to ensure consistency across different implementations of place-related data by providing:

    • Common place categories: Standardized classifications for different types of places.
    • Contact structures: Uniform schemas for contact information associated with places.
    • Validation utilities: Tools to validate that place data adheres to the defined schema requirements.

  10. Understand the Overture Schema purpose

    main

    The Overture schema provides type-safe Python models designed to validate and work with Overture data. Developers use these schemas to:

    • Validate Overture data: Ensure incoming data conforms to expected structures and types.
    • Build data processing pipelines: Implement pipelines with built-in type safety.
    • Extend schemas: Add custom fields and validation rules to the existing framework.

  11. How model extraction and field ordering works

    main

    The extract_model function converts a Pydantic BaseModel into a RecordSpec. It follows specific rules for ordering and resolution:

    Field Ordering

    Documentation order is determined by _class_order and _field_order to ensure a logical reading flow:

    1. MRO-aware sequence: For single inheritance, base class fields appear first. For multiple inheritance, the primary chain (the first base class) is processed first, followed by the class's own fields, and finally mixin fields.
    2. Deduplication: _field_order walks this hierarchy and collects __annotations__ keys, ensuring each field is listed only once.

    Field Extraction Logic

    • Alias Resolution: For every field, the extractor resolves the alias chain in order: validation_alias > alias > Python name.
    • Type Analysis: It uses field_info.annotation (rather than get_type_hints()) to avoid issues with unresolved TypeVars in generic base classes.
    • Metadata Handling: If Pydantic moves Annotated metadata to field_info.metadata, attach_field_metadata ensures constraints (like length limits) are still captured via attach_constraints.
    • Model Constraints: Constraints from decorators like @require_any_of or @require_if are extracted via ModelConstraint.get_model_constraints.

  12. Use Field Aliases for non-Python identifiers

    main

    Some data fields use names that are reserved keywords in Python (e.g., class). The schema uses Pydantic Field(alias=...) to map these to valid Python attributes.

    • Attribute Name: The name used in your Python code (e.g., Building.class_).
    • Alias Name: The actual name used in the data (e.g., class).

    To export data using the original data field names instead of the Python attribute names, you must dump the model with by_alias=True.

    # Exporting data with original field names
data = building_instance.model_dump(by_alias=True)