ScienceBeam Parser

The LLM engine enforces strict data integrity to ensure no hallucinated text is introduced into the document.

• Text Integrity: No text reaches a document unless it existed in the source. Under values mode, every value is validated against the original token sequence. Under lines mode, only line numbers are returned.
• Parsing Failures: If the engine receives bad JSON or a malformed entry, it retries up to max_malformed_response_retries (default 1).
• Claim Validation: If a model claims a value that is not in the reference or is already claimed by a previous field, the value is dropped and logged to sciencebeam.dropped_fields. This prevents the model from 'adding' data that wasn't there.
• Strict Mode: Using *_raises settings (e.g., unanswered_reference_raises) turns these warnings into fatal errors.
• Zero Data Retention (ZDR): Every request enforces zero data retention via settings like data_collection: deny and allow_fallbacks: false to ensure privacy and security.

The reasoning_enabled setting controls whether the model uses its internal reasoning capabilities.

• Behavior:
  • false: Sends {'enabled': false}. This is safe for all models. If a model doesn't support reasoning, the engine retries once without the parameter.
  • true: Sends {'enabled': true}.
  • Omitted: The engine sends nothing, letting the model use its default behavior.
• Important Constraints:
  • Budget: Reasoning tokens count against max_output_tokens. If a model thinks by default, it may exhaust the budget before providing an answer.
  • Syntax: Must be an unquoted boolean in YAML (e.g., reasoning_enabled: false). Quoted values are rejected.
  • Provider-specific settings: Use extra_body for settings like effort. Do not use both reasoning_enabled and extra_body for reasoning settings, as reasoning_enabled will overwrite the extra_body value.

extra_body:
  reasoning:
    effort: 'low'

ScienceBeam Parser uses three distinct document abstractions to represent data at different stages of processing:

1. Layout Document

Represents the parsed input document without layout information.

• Immutability: Layout Document and its associated classes should be treated as immutable.
• Views: You can create a 'view' by constructing a layout document with a subset of tokens (e.g., passing only header tokens to a header model).

2. Semantic Document

Represents the intermediate semantically extracted document. This is the core of the semantic analysis.

• Mutability: This document is mutable and can be changed during processing.
• Semantic Labels: The type class serves as the main semantic label (e.g., SemanticTitle for titles, SemanticPageRange for page ranges).
• Hierarchy: Content can be hierarchical (e.g., a SemanticTitle inside a SemanticFront is a manuscript title, while a SemanticTitle inside a SemanticReference is a reference title).
• Preservation: All text should be preserved. Content without a known semantic meaning should use SemanticNote. The document aims to maintain the original order and formatting.
• Partial Parsing: Some types, like SemanticRawAuthors, represent partially parsed content intended for further processing by downstream models.

3. TEI Document

Used for the final TEI XML output.

• Mapping: It maps the Semantic Document to XML elements.
• Output: By default, it aims to output all content from the semantic document (often as a note) and includes page coordinates and preserved formatting.

Every run of generate_data writes a quality.jsonl file per model. This file contains one JSON line per source document, tracking the status and cardinality of labels at various stages. This allows for comparing a corpus against its aligned JATS source without post-processing counts.

File Location:

• If --use-directory-structure is used: <model>/quality.jsonl inside the output directory.
• Otherwise: <model>.quality.jsonl in the output path.

Key JSON Fields:

• document_id: The ID of the document.
• status: ok, missing (no JATS matched), unparsable, unreadable, timeout, or error.
• jats.status: The status of the JATS alignment.
• jats.reference_count: Total references declared in the JATS.
• aligned_reference_count: Number of references successfully placed by the aligner.
• entity_element_count: Number of entities written (e.g., bibl for citation models). Absent for region-marking models.
• written: Boolean indicating if the model wrote a file.
• label_counts: Per-label counts. jats counts references carrying a sub-field for that label; marked counts references actually marked in the training data.

{
  "document_id": "PPR459453",
  "source_filename": "PPR459453.pdf",
  "status": "ok",
  "model": "citation",
  "jats": {"status": "ok", "reference_count": 45, "aligned_reference_count": 2},
  "written": true,
  "entity_element_count": 2,
  "label_counts": {"<title>": {"jats": 44, "marked": 2}}
}

The citation model processes references in batches to improve throughput.

Key Parameters:

• max_references_per_request: The number of references sent in a single call. Larger batches are more efficient (roughly 2x improvement) but can trigger provider timeouts if the generation becomes too long. Smaller batches are generally preferred for stability.
• max_concurrent_requests: The number of calls in flight at once. Increasing this provides roughly 2x throughput improvement up to 4 calls, after which provider limits usually become the bottleneck.

Error Handling in Batches:

• If a reference is skipped or numbered incorrectly, it is counted in sciencebeam.unanswered_references. The engine will retry skipped references in a new batch (up to max_missing_reference_retries, default 1).
• If a response hits max_output_tokens, the batch is halved and retried until individual references are processed.
• If unanswered_reference_raises is set, any unrecovered reference makes the entire process fatal.

max_references_per_request: 10   # references per call
max_concurrent_requests: 4       # calls in flight at once