extractous

To extract the text content and metadata from a local file as a string using Python, use the Extractor.extract_file_to_string method. You can also configure the maximum length of the extracted string using set_extract_string_max_length.

from extractous import Extractor

# Create a new extractor
extractor = Extractor()
extractor = extractor.set_extract_string_max_length(1000)
# if you need an xml
# extractor = extractor.set_xml_output(True)

# Extract text from a file
result, metadata = extractor.extract_file_to_string("README.md")
print(result)
print(metadata)

In Rust, the Extractor uses a consuming builder pattern. You can extract file content directly to a string using extract_file_to_string.

use extractous::Extractor;

fn main() {
    // Create a new extractor. Note it uses a consuming builder pattern
    let mut extractor = Extractor::new().set_extract_string_max_length(1000);
    // if you need an xml
    // extractor = extractor.set_xml_output(true);

    // Extract text from a file
    let (text, metadata) = extractor.extract_file_to_string("README.md").unwrap();
    println!("{}", text);
    println!("{:?}", metadata);
}

For large files or when you need to process content incrementally, use extract_file, extract_url, or extract_bytes to obtain a reader (stream) and metadata. This allows for buffered reading of the content.

from extractous import Extractor

extractor = Extractor()
# if you need an xml
# extractor = extractor.set_xml_output(True)

# for file
reader, metadata = extractor.extract_file("tests/quarkus.pdf")
# for url
# reader, metadata = extractor.extract_url("https://www.google.com")
# for bytearray
# with open("tests/quarkus.pdf", "rb") as file:
#     buffer = bytearray(file.read())
# reader, metadata = extractor.extract_bytes(buffer)

result = ""
buffer = reader.read(4096)
while len(buffer) > 0:
    result += buffer.decode("utf-8")
    buffer = reader.read(4096)

print(result)
print(metadata)

Extractous uses Apache Tika for non-native formats, compiled as native shared libraries via FFI for performance.

GraalVM

• GraalVM is required to build Tika as native libraries.
• You can set GRAALVM_HOME to use a specific local version.
• macOS Users: Official GraalVM JDKs may fail with code using java awt. It is recommended to use Bellsoft Liberica NIK (sdk install java 24.1.1.r23-nik).

Tesseract (for OCR)

• Debian/Ubuntu: sudo apt install tesseract-ocr and language packs like tesseract-ocr-deu.
• macOS: brew install tesseract tesseract-lang.

To build the project from source or run the test suite, use standard Cargo commands:

# Build the project
cargo build

# Run tests
cargo test

Install the extractous Python package using pip to use the Extractor API.

%pip install extractous

To extract text from scanned PDFs or images within PDFs, configure the extractor with TesseractOcrConfig and set a PdfOcrStrategy.

Requirement: You must have tesseract installed on your system along with the necessary language packs (e.g., tesseract-ocr-deu for German). If you encounter Parse error occurred : Unable to extract PDF content, verify that the OCR language pack is installed.

use extractous::Extractor;

fn main() {
  let file_path = "../test_files/documents/deu-ocr.pdf";

  let extractor = Extractor::new()
          .set_ocr_config(TesseractOcrConfig::new().set_language("deu"))
          .set_pdf_config(PdfParserConfig::new().set_ocr_strategy(PdfOcrStrategy::OCR_ONLY));
  
  let (content, metadata) = extractor.extract_file_to_string(file_path).unwrap();
  println!("{}", content);
  println!("{:?}", metadata);
}

Use extract_file_to_string to quickly get the text content and metadata of a file as a String and a metadata object. This is the simplest way to extract text when you don't need to stream the data.

use extractous::Extractor;

fn main() {
  let file_path = "path/to/file";

  let mut extractor = Extractor::new();
  // Extract text from a file
  let (content, metadata) = extractor.extract_file_to_string(file_path).unwrap();
  println!("{}", content);
  println!("{:?}", metadata);
}

If you need to handle large files or want to process data as a stream, use extract_file, extract_url, or extract_bytes. These methods return a StreamReader (which implements std::io::Read) and the metadata. This allows you to use standard Rust tools like BufReader for efficient processing.

use std::io::{BufReader, Read};
use extractous::Extractor;

fn main() {
  let file_path = "path/to/file";

  let extractor = Extractor::new();
  let (stream, metadata) = extractor.extract_file(file_path).unwrap();

  // Because stream implements std::io::Read trait we can perform buffered reading
  let mut reader = BufReader::new(stream);
  let mut buffer = Vec::new();
  reader.read_to_end(&mut buffer).unwrap();

  println!("{}", String::from_utf8(buffer).unwrap());
  println!("{:?}", metadata);
}