DocLang Reference Toolkit

DocLang is a universal, AI-native markup language designed to represent structured document content. Unlike traditional formats like HTML or LaTeX, DocLang is optimized for Large Language Model (LLM) and Vision-Language Model (VLM) workflows by focusing on:

• Token Efficiency: Uses a controlled, minimalist vocabulary of tags and attributes to map naturally to LLM tokenizers.
• Semantic & Geometric Preservation: Maintains document structure, semantics, and layout/geometric information (like bounding boxes) simultaneously.
• Complex Component Support: Provides native primitives for tables (including spanned cells and headings), formulas, code, nested lists, and charts.
• Lossless Round-tripping: Enables conversion between formats while preserving the original content's meaning and structure.

It is specifically designed to bridge the gap between OCR-centric formats (like PageXML or ALTO XML) and generative AI requirements, prioritizing semantic context over mere geometric location.

DocLang suggests two governance profiles to help producers implement consistent control surfaces:

Minimal Profile (Recommended Baseline)

Use this for a standard baseline of compliance and access control:

• Licensing & compliance: licenses, compliance_requirements
• Classification & access: data_classification, access_policy (or access_control_level), retention_policy
• PII: pii_status, pii_sensitivity_level
• Extraction: extraction_permitted, pii_extraction_allowed
• RAG: rag_permitted, rag_indexing_allowed, rag_pii_exposure_allowed
• Training: training_permitted, training_pii_included

Full Profile (Expanded Control Surface)

Includes everything in the Minimal profile plus advanced controls for:

• Purpose limitation and lawful basis
• Component-level scope constraints
• Transformation requirements (redaction/masking/pseudonymization)
• Caching/index retention and auditability
• Model scope restrictions and provenance requirements

DocLang allows for fine-grained control over how document data is handled by AI systems through specific XML-based control elements. These controls are grouped into three main categories:

1. Data Extraction Controls: Define what can be extracted (e.g., extraction_scope), the purpose, granularity, and whether PII or sensitive data is allowed. It also manages audit requirements and human-in-the-loop constraints.
2. RAG (Retrieval-Augmented Generation) and Retrieval Controls: Manage how documents are used in RAG pipelines, including indexing permissions, embedding scopes, chunking constraints, and caching policies.
3. Document Training Controls: Govern whether a document can be used for model training, defining the scope, purpose, and model types, as well as provenance and audit retention requirements.

Implementation Note: Implementations should define controlled vocabularies for enums such as extraction_scope, rag_embedding_scope, and training_model_type. When training_permitted is set to false, training-related fields should be omitted or set to none.

<!-- Data extraction controls -->
<extraction_permitted>true</extraction_permitted>
<extraction_scope>tables_and_forms_only</extraction_scope>
<extraction_purpose>case_metrics</extraction_purpose>
<extraction_granularity>field_level</extraction_granularity>
<pii_extraction_allowed>false</pii_extraction_allowed>
<sensitive_data_extraction_allowed>false</sensitive_data_extraction_allowed>
<extraction_transformation_required>redact</extraction_transformation_required>
<extraction_output_constraints>internal_systems_only</extraction_output_constraints>
<downstream_sharing_permitted>false</downstream_sharing_permitted>
<extraction_audit_required>true</extraction_audit_required>
<extraction_audit_retention unit="day">90</extraction_audit_retention>
<human_in_the_loop_required>true</human_in_the_loop_required>

<!-- RAG and retrieval controls -->
<rag_permitted>true</rag_permitted>
<rag_indexing_allowed>true</rag_indexing_allowed>
<rag_embedding_scope>summaries_only</rag_embedding_scope>
<rag_chunking_constraints>max_512_tokens</rag_chunking_constraints>
<rag_query_restrictions>support_intent_only</rag_query_restrictions>
<rag_output_attribution_required>true</rag_output_attribution_required>
<rag_output_transformation_required>summarize_and_redact</rag_output_transformation_required>
<rag_pii_exposure_allowed>false</rag_pii_exposure_allowed>
<rag_sensitive_data_exposure_allowed>false</rag_sensitive_data_exposure_allowed>
<rag_downstream_sharing_permitted>false</rag_downstream_sharing_permitted>
<rag_caching_allowed>true</rag_caching_allowed>
<rag_cache_retention unit="day">30</rag_cache_retention>
<rag_audit_required>true</rag_audit_required>
<rag_audit_retention unit="day">90</rag_audit_retention>
<rag_model_scope>enterprise_internal_models</rag_model_scope>

<!-- Document training controls -->
<training_permitted>false</training_permitted>
<training_scope>none</training_scope>
<training_purpose>none</training_purpose>
<training_model_type>none</training_model_type>
<training_dataset_reuse_allowed>false</training_dataset_reuse_allowed>
<training_derivative_sharing_permitted>false</training_derivative_sharing_permitted>
<training_pii_included>false</training_pii_included>
<training_sensitive_data_included>false</training_sensitive_data_included>
<training_provenance_required>true</training_provenance_required>
<training_audit_required>true</training_audit_required>
<training_audit_retention unit="day">365</training_audit_retention>
<model_output_usage_constraints>internal_only</model_output_usage_constraints>
<right_to_be_forgotten_applicability>true</right_to_be_forgotten_applicability>

DocLang uses <field_region> to group related form elements. Within a region, you can define individual data points using <field_item>.

Common components of a <field_item> include:

• <key>: The label or name of the field.
• <value>: The data content (can include <marker> for indexing or <hint> for supplementary text).
• <checkbox>: Represents a boolean or selection state, often with a class attribute (e.g., unselected).
• <marker>: A positional or logical identifier (e.g., 'A', 'B', or numeric indices).
• <text>: Used for complex items where a single key maps to multiple text/choice options (common in multiple-choice questions).
• <checkbox class="..."/>: Used within <text> or <field_item> to denote selection UI.

<field_region>
    <field_heading>M31</field_heading>
    <field_item>
        <marker>1</marker>
        <key>Tipo</key>
        <value></value>
    </field_item>
    <field_item>
        <marker>2</marker>
        <key>Codice Stato estero</key>
        <value></value>
    </field_item>
</field_region>