Inscriptis HTML to Text Converter

Inscriptis is a Python-based library, command line client, and Web service designed for converting HTML to plain text. It is specifically optimized for knowledge extraction tasks where preserving the spatial arrangement and layout of text is critical.

Use Inscriptis when:

• You need layout-aware conversion that preserves the spatial alignment of text elements (e.g., correctly rendering nested tables or text alignment determined by CSS like display, white-space, margin-top, or vertical-align).
• You want to avoid the artifacts (like concatenated words) often produced by scraping libraries like Beautiful Soup or lxml when encountering complex structures like lists and tables.
• You need to preserve semantic information from the original HTML (such as headings, emphasized text, or table structures) by using annotation rules to map HTML tags/attributes to the extracted text.
• You are preparing data for downstream NLP tasks like entity recognition, sentiment analysis, or text summarization that benefit from knowing the original document's structure.

Beyond simple text extraction, Inscriptis supports annotated text conversion. This allows you to map structural and semantic information from HTML tags and attributes onto the resulting plain text.

Key benefits of annotations:

• Enhanced NLP: Provides downstream components (like summarizers or sentiment analyzers) with cues such as bold/italic text or heading levels.
• Manual Annotation Support: Assists in creating gold standards for qualitative analysis by exporting to formats compatible with annotation tools.
• Content Extraction: Enables tasks that rely on the document's structural hierarchy.

Supported Export Formats:

• XML
• Annotated HTML (can be triggered via the --postprocessor html CLI option)
• JSONL (compatible with the doccano annotation tool)

Developers can extend these capabilities by implementing custom annotation post-processors.

The project includes a benchmarking script to compare different HTML-to-text conversion algorithms (e.g., Lynx, Justext, HTML2Text, BeautifulSoup, and Inscriptis) against a list of URLs.

Setup

1. Prepare URLs: Create a file named url_list.txt in the benchmarking directory. Add the target URLs, one per line, ensuring they include the protocol (e.g., http:// or https://).
2. Configure Algorithms: Open run_benchmarking.py and modify the pipeline() function to enable or disable specific algorithms by setting their boolean flags to True or False:
   • run_lynx
   • run_justext
   • run_html2text
   • run_beautifulsoup
   • run_inscriptis

Execution

Run the script from within the benchmarking folder:

python run_benchmarking.py

Results

Results are saved in a timestamped folder within benchmarking/benchmarking_results:

• Text Outputs: The converted text for manual comparison.
• speed_comparisons.txt: A file containing the measured processing speed of every algorithm per URL.

# Inside run_benchmarking.py
def pipeline():
    run_lynx = True
    run_justext = True
    run_html2text = True
    run_beautifulsoup = True
    run_inscriptis = True

Use the ParserConfig class to modify how HTML tags are interpreted and how elements like anchors, captions, and images are rendered.

Use strict CSS profiles

To achieve Firefox-like whitespace handling (where <div> and <span> do not automatically add extra whitespace), use the strict profile from inscriptis.css_profiles.CSS_PROFILES.

Configure link and anchor display

You can enable the display of inline links and anchor URLs by setting display_links=True and display_anchors=True in ParserConfig.

Hide elements (e.g., forms)

You can prevent specific elements from appearing in the output by overriding the CSS profile and setting the element's display property to Display.none using inscriptis.html_properties.Display.

from lxml.html import fromstring
from inscriptis import Inscriptis
from inscriptis.css_profiles import CSS_PROFILES
from inscriptis.model.config import ParserConfig

# Example: Using strict CSS and enabling link/anchor display
css = CSS_PROFILES['strict']
config = ParserConfig(css=css, display_links=True, display_anchors=True)

html_tree = fromstring(html_content)
parser = Inscriptis(html_tree, config)
text = parser.get_text()

A TableCell represents a single cell in a table and manages its own content, dimensions, and alignment. It operates in two distinct phases:

1. Construction Phase: The cell's content blocks are mutable. You can set the content using the blocks property (a list of strings).
2. Normalization Phase: Calling normalize_blocks() converts multi-line blocks into individual lines and freezes the content. Once normalized, the _content_blocks become an immutable tuple, and attempting to modify blocks will raise a FrozenError.

Dimensions vs. Content:

• Content Dimensions: The actual width (longest line) and height (number of lines) of the text inside.
• Requested Dimensions: You can set a minimum width and height. The cell's actual dimensions will be the maximum of its content dimensions and its requested dimensions. This allows for padding and alignment (horizontal and vertical) within the cell.

from inscriptis.model.table import TableCell
from inscriptis.html_properties import HorizontalAlignment, VerticalAlignment

# 1. Initialize with alignment
cell = TableCell(align=HorizontalAlignment.CENTER, valign=VerticalAlignment.MIDDLE)

# 2. Set content (mutable phase)
cell.blocks = ["Line 1", "Line 2\nLine 3"]

# 3. Normalize (freezes content)
cell.normalize_blocks()

# 4. Set requested dimensions (padding/alignment will apply)
cell.width = 20
cell.height = 5