dedoc Documentation

The dedoc.readers module provides a suite of reader classes designed to ingest different document formats and convert them into a standardized representation. All readers inherit from BaseReader.

Available readers include:

• Office Documents: DocxReader, ExcelReader, PptxReader
• Web/Markup: HtmlReader, MhtmlReader, JsonReader
• Data/Text: CSVReader, RawTextReader, NoteReader
• PDF Specialized Readers:
  • PdfBaseReader: Base class for PDF processing.
  • PdfImageReader: For scanned PDFs (image-based).
  • PdfTabbyReader: Specialized for PDF table extraction.
  • PdfTxtlayerReader: For PDFs containing a text layer.
  • PdfAutoReader: Automatically selects the appropriate PDF strategy.
  • PdfBrokenEncodingReader: For PDFs with encoding issues.
• Other: ArchiveReader (for compressed files), EmailReader, ArticleReader.

When using DefaultStructureExtractor with document_type="other", you can use Structure Patterns to flexibly configure how line types and levels are identified during the extraction step. Patterns allow you to define rules for recognizing lists, headers, and other structural elements.

Available Pattern Types

• RegexpPattern: Matches lines based on regular expressions.
• StartWordPattern: Matches lines starting with a specific word.
• TagPattern / TagHeaderPattern / TagListPattern: Matches elements based on specific tags.
• BracketListPattern / BracketRomanListPattern: Matches lists enclosed in brackets (e.g., [1], [I]).
• BulletListPattern: Matches standard bulleted lists.
• DottedListPattern: Matches lists using dots.
• LetterListPattern: Matches lists using letters (e.g., a), b)).
• RomanListPattern: Matches lists using Roman numerals.
• PatternComposition: Allows you to combine multiple patterns into a single rule.

DOCX styles follow a specific hierarchy for property resolution: documentDefault $\rightarrow$ tables $\rightarrow$ paragraphs $\rightarrow$ numbering $\rightarrow$ characters $\rightarrow$ direct formatting (properties defined directly in document.xml).

Property Toggling: For properties like bold or italic, if values differ across the hierarchy, the first value encountered in the hierarchy is used.

Style Types (w:style):

• character: Applies to text elements (rPr).
• numbering: Used in paragraph properties (pPr) via numPr. It does not use list levels directly; instead, it references a numId in numberings.xml, which links to an abstractNum and a specific level.
• paragraph: Defines paragraph properties (pPr).
• table: Defines table properties.

Key Style Attributes:

• styleId: Unique identifier/name of the style.
• type: The type of style (character, numbering, paragraph, table).
• basedOn: Inheritance link (paragraphs inherit from paragraphs, characters from characters; numbering does not inherit).
• default: Indicates if the style is the default for its type.
• aliases: Alternative names for the style.
• next: The style to be applied to the next paragraph created.
• qFormat: Indicates if the style is the primary style for the document.
• uiPriority: The priority of the style in the UI.

In a DOCX file, numbering information is stored in word/numberings.xml. The structure relies on a hierarchy of definitions:

• abstractNum: Defines the properties of a specific list type. It is assigned an abstractNumId which is referenced in word/document.xml.
  • styleLink: Defines list properties that other lists can reference.
  • numStyleLink: A link to another abstractNum whose properties are used.
  • restartNumberingAfterBreak: Controls numbering restarts.
• num: An instance of a list type, identified by numId.
• lvlOverride: Used within a num to override properties inherited from the abstractNum for a specific level.
  • startOverride: Resets numbering for that specific level.
• ilvl: The nesting level of the list.
• lvl: Describes properties for a specific level within lvlOverride or abstractNum:
  • isLgl: If present, forces the list to use Arabic numerals regardless of other settings.
  • lvlText: The text representation for numbering (e.g., val="some text %num some text"). The %num token is replaced by the actual number.
  • numFmt: The numbering style (e.g., upperLetter, lowerRoman). If omitted, defaults to decimal numbers.
  • pPr: Paragraph properties (e.g., ind for indentation) that override properties in word/document.xml.
  • rPr: Element properties applied to the content of lvlText.
  • start: The starting value for numbering.
  • suff: The separator between the numbering text and the paragraph text (defaults to a tab).
  • lvlJc: Justification for the level (val="start" or val="end").
  • lvlRestart: Determines if the list restarts at this level. If val="0", the list does not restart.

Dedoc represents documents as a tree hierarchy where document lines are nodes. The structure is determined by styles found in the input file (e.g., docx styles) or via regular expression analysis for lists (dotted, bracket, and bullet lists).

Node importance is determined by depth: nodes closer to the root are considered more important.

Available Line Types:

• root: The single, obligatory auxiliary node of highest importance. It contains an empty string as text and serves as the parent for all other nodes.
• header: Optional nodes representing document headers. They can be nested based on their importance level.
• list: Optional auxiliary nodes that mark the beginning of a list (e.g., dotted, roman, bracket, or bullet). They are more important than list_item and raw_text. Lists can be nested (e.g., a bullet list nested inside a dotted list item).
• list_item: Represents a single item within a list. These are nested inside a list node and are more important than raw_text.
• raw_text: Represents simple document lines. These have the least importance and are typically located at the leaves of the tree.

When implementing a feature extractor, consider grouping features into three main categories to improve classification accuracy:

1. Visual features: Information about document formatting, such as spacing between lines, font size, and boldness. Use the helper methods provided in AbstractFeatureExtractor to extract these.
2. Textual features: Domain-specific information derived from regular expressions, keywords, or predefined lists.
3. Statistical features: General information about the line, such as its line number in the document or its character length.

Advanced Tips:

• Numerated items: Use the ListFeaturesExtractor class and the _list_features method to handle numbered lists.
• Normalization: Use the _normalize_features method to handle features that vary in scale (e.g., relative font size).
• Contextual features: Incorporate information from the previous or next lines into the current line's feature vector to capture document context.

An UnstructuredDocument object provides access to several key attributes:

• lines: A list of LineWithMeta objects. Each line has a text attribute and an annotations attribute containing formatting information.
• tables: A list of Table objects. Each table consists of rows and cells (CellWithMeta). For merged cells, the colspan attribute in the cell metadata indicates the span.
• attachments: A list of AttachedFile objects. These represent files (like images) embedded in the document. The tmp_file_path provides the location of the extracted file on disk.
• metadata: A dictionary containing document metadata (initially empty if not processed by a metadata extractor).
• warnings: A list of strings describing issues encountered during parsing.

# Accessing lines
for line in document.lines:
    print(line.text)
    print(line.annotations)

# Accessing tables
for table in document.tables:
    for row in table.rows:
        for cell in row:
            print(cell.text)
            print(cell.metadata)

# Accessing attachments
for attachment in document.attachments:
    print(attachment.tmp_file_path)

A .docx file is a ZIP archive containing XML files (.xml, .rels) and media files (images, etc.). To process or parse these files, you must understand three logical components:

1. Content Types: A list of media file types (e.g., png) and document part types (e.g., document, header).
2. Parts: Individual components of the document, such as document.xml, footer1.xml, header1.xml, comments.xml, and endnotes.xml.
3. Relationships: Identifiers that link parts together (e.g., linking a document section to a footer) and define external parts like hyperlinks.

Key metadata and configuration files often found in the archive include:

• docProps/core.xml: Core metadata (Dublin Core).
• docProps/app.xml: Document statistics (page count, word count, etc.).
• word/settings.xml: Document-specific settings.
• word/styles.xml: Document styles (separates data from presentation).
• word/fontTable.xml: List of fonts used.
• word/theme1.xml: Document theme (colors, fonts, formatting).