Apache Tika Documentation

The Tika Pipes Kafka integration tests verify the functionality of the Kafka-specific modules within the Tika Pipes framework. These tests utilize TestContainers to provide a real Kafka environment for validation. The tests specifically cover:

• Kafka Pipe Iterator: Validates the iteration logic for Kafka pipes.
• Kafka Emitter: Validates the emission logic for Kafka pipes.

Apache Tika is a toolkit designed to detect and extract metadata and text from over 1,000 different file types (including PPT, XLS, and PDF). It provides a unified interface for parsing diverse formats, making it suitable for search engine indexing, content analysis, and translation.

Key capabilities include:

• Unified Interface: Parse all supported file types through a single, consistent API.
• Broad Format Support: Support for over 1,000 different file types.
• Metadata Extraction: Automatic identification and extraction of document metadata.
• Text Extraction: Extraction of readable content from complex file formats.
• Content Detection: Identification of file types regardless of the file extension.

Tika Pipes is a framework designed for fault-tolerant and scalable document processing. It protects applications from malformed files by parsing each document in a forked JVM with configurable timeouts and memory limits. This prevents a single problematic file from crashing or hanging the entire application.

While a programmatic Java API is available, Tika Pipes is primarily intended to be used via:

• tika-app: For batch processing from the command line.
• tika-server: A REST API with built-in pipes-based robustness.
• tika-grpc: A gRPC API with pipes-based robustness (intended for use on trusted networks only).

The Tika Pipes GRPC Server manages a pool of Tika Pipes clients and provides the following capabilities:

• Tika Pipes Fetcher CRUD operations: Create, Read, Update, and Delete.
• Fetch + Parse: Perform fetch and parse operations on a given Fetch Item.

Security Note: By default, runtime fetcher/iterator management (mutations like Create/Update/Delete and reading stored configs) and per-request parse configuration are disabled.

To enable these features, you must explicitly set the following in the grpc section of your tika-config:

• allowComponentManagement
• allowPerRequestConfig

When management is disabled, Read RPCs will only return the component ID and class, omitting the configuration (which may contain secrets).

Tika provides two implementations of the org.apache.tika.language.detect.LanguageDetector SPI interface. Both are loaded automatically via Tika's service discovery:

• CharSoupLanguageDetector (tika-langdetect-charsoup): A built-in, hash-based detector with zero runtime dependencies beyond tika-core. This is the recommended detector for new deployments.
• OpenNLPDetector (tika-langdetect-opennlp): Based on Apache OpenNLP's language detection models.

The Atlassian JWT plugin (tika-pipes-atlassian-jwt) provides a fetcher designed for Atlassian Connect endpoints (such as Jira Cloud or Confluence Cloud) that require an Atlassian-style JWT bearer token.

This component is a fetcher-only plugin, meaning it must be paired with an emitter and an iterator to function within a Tika pipeline. For generic HTTP fetching that does not require Atlassian-specific JWT signing, use the http plugin instead.

Tika Pipes is extensible through four primary plugin interfaces. Developers can implement these interfaces to create custom plugins that integrate into a Tika Pipes pipeline:

• Fetcher: Retrieves document bytes from a specific source.
• Emitter: Writes parsed results to a destination.
• Iterator (PipesIterator): Enumerates documents to process as FetchEmitTuple records.
• Reporter (PipesReporter): Records the processing status for each document.

A PipesIterator is responsible for enumerating documents to be processed in a pipeline. It runs on its own thread and emits a stream of FetchEmitTuple records. Each tuple contains:

• fetch key: Passed to the fetcher to retrieve the document bytes.
• emit key: Passed to the emitter to determine the destination for results.
• id (optional): A unique identifier.
• metadata: Arbitrary metadata fields.

The pipeline workers consume these tuples by calling the bound fetcher, the parser, and the bound emitter.

Apache Tika provides two main types of Docker images for its server instance:

• Minimal version: Contains only Apache Tika and its core dependencies.
• Full version: Includes dependencies for GDAL and Tesseract OCR parsers.

By default, the full image includes language packs for English, French, German, Italian, Spanish, and Japanese. To include more languages during a build, set the LANGUAGES build argument or use an ADD command for custom packs.

The Microsoft Graph plugin (tika-pipes-microsoft-graph) provides a fetcher to retrieve files from OneDrive, SharePoint, and other Graph-accessible sources.

Note that this plugin is fetcher-only; you must pair it with an emitter and an iterator to create a complete pipeline. It uses the official microsoft-graph SDK.

Apache Tika offers four primary integration methods depending on your application requirements:

• Java API: Best for Java applications requiring tight integration and full control over parsing behavior.
• Command Line (tika-app): Best for shell scripts, batch processing, quick one-off extractions, and scripting.
• Server (REST API): Best for language-agnostic integration (e.g., Python, Node.js) and microservice architectures.
• gRPC: Best for high-performance, cross-language communication in high-throughput pipelines.

Note on gRPC Security: The tika-grpc implementation is more exposed by default than tika-server. It lacks transport security or per-caller authorization by default, and its core fetch-and-parse surface is always active. Only run gRPC on trusted, access-controlled networks.