tika-python

repository·master·Indexed 23 days ago

https://github.com/chrismattmann/tika-python

A Python wrapper for Apache Tika that provides a high-level interface for content extraction, metadata detection, and file type identification via the Tika REST service. It includes interfaces for parsing (text and XHTML), unpacking, MIME type detection, language identification, and translation, as well as a command-line tool.

Tokens
5.2K
Snippets
14
Records
37
Agent score
80%

What's inside tika-python

  1. Set up tika-python for airgap environments

    master

    To run tika-python in a disconnected environment without internet access, you must manually provide the Tika server JAR file.

    1. Download tika-server.jar and tika-server.jar.md5 from the Apache Tika Maven repository.
    2. Set the TIKA_SERVER_JAR environment variable to the local file path using the file:/// protocol.

    This prevents the library from attempting to check the version and download the latest JAR from Apache every time it runs.

  2. Understand the Unpack output format

    master

    The unpack interface returns a dictionary with the following keys:

    • content: A string containing the extracted text content from the __TEXT__ member of the archive.
    • metadata: A dictionary containing metadata extracted from the __METADATA__ member (parsed from CSV format).
    • attachments: A dictionary where keys are the filenames of the remaining members in the archive and values are the raw bytes of those files.
  3. Gzip compression for Tika streams

    master

    Since Tika 1.24.1, you can use gzip compression for both input and output streams to improve performance.

    Input Compression

    Compress your data using gzip or zlib before passing it to from_buffer:

    import zlib
    import gzip
    
    # Using zlib
    with open(file, 'rb') as file_obj:
        return tika.parser.from_buffer(zlib.compress(file_obj.read()))
    
    # Using gzip
    with open(file, 'rb') as file_obj:
        return tika.parser.from_buffer(gzip.compress(file_obj.read()))

    Output Compression

    Request compressed output by passing the Accept-Encoding header:

    with open(file, 'rb') as file_obj:
        return tika.parser.from_file(file_obj, headers={'Accept-Encoding': 'gzip, deflate'})
    import zlib
    
    with open(file, 'rb') as file_obj:
        return tika.parser.from_buffer(zlib.compress(file_obj.read()))
    
    ...
    
    import gzip
    
    with open(file, 'rb') as file_obj:
        return tika.parser.from_buffer(gzip.compress(file_obj.read()))
  4. Parse from a buffer using Parser or Detector

    master

    If you have already loaded content into memory, you can use the .from_buffer() method on the parser or detector interfaces to process a string or a BufferedIOBase object.

    import io
    from tika import parser
    
    # Parsing a string
    string_parsed = parser.from_buffer('Good evening, Dave')
    
    # Parsing bytes via BytesIO
    byte_data: bytes = b'B\xc3\xa4ume'
    parsed = parser.from_buffer(io.BytesIO(byte_data))
    import io
    from tika import parser
    
    string_parsed = parser.from_buffer('Good evening, Dave')
    byte_data: bytes = b'B\xc3\xa4ume'
    parsed = parser.from_buffer(io.BytesIO(byte_data))
  5. Extract XHTML content using the Parser Interface

    master

    The parser interface can output content as XHTML instead of plain text by setting the xmlContent parameter to True.

    Note: Set export PYTHONIOENCODING=utf8 in your console for correct printing.

    from tika import parser
    parsed = parser.from_file('/path/to/file', xmlContent=True)
    print(parsed['metadata'])
    print(parsed['content'])

    This option is also available when using parser.from_buffer().

  6. Enable Client Only mode

    master

    You can set Tika to ClientOnly mode to bypass the automatic check for a running local Tika service. This is useful when you know the server is already running or you are connecting to a remote endpoint and want to avoid the overhead of startup checks.

    import tika.tika
    tika.tika.TikaClientOnly = True
  7. Extract text and metadata with the Parser Interface

    master

    The parser interface is the primary way to extract text and metadata from files using the /rmeta interface. This is recommended for obtaining internal XHTML content.

    Note: To ensure extracted content prints correctly, set the following environment variable in your console: export PYTHONIOENCODING=utf8

    Basic Usage

    from tika import parser
    parsed = parser.from_file('/path/to/file')
    print(parsed['metadata'])
    print(parsed['content'])

    Using a specific Tika Server URL

    You can pass a custom Tika server URL (useful for Dockerized or multi-instance setups):

    parsed = parser.from_file('/path/to/file', 'http://tika:9998/tika')

    Parsing from a binary stream

    You can pass a file-like object (binary stream) directly:

    with open(file, 'rb') as file_obj:
        response = tika.parser.from_file(file_obj)
  8. Customize outgoing HTTP requests

    master

    You can customize the outgoing HTTP request to the Tika server by passing a requestOptions dictionary to the .from_file or .from_buffer methods (supported by Parser, Unpack, Detect, Config, Language, and Translate).

    This dictionary accepts any valid arguments for the Requests library. This will override defaults except for url and params/data.

    from tika import parser
    # Example: setting a custom timeout
    parsed = parser.from_file('/path/to/file', requestOptions={'timeout': 120})
    from tika import parser
    parsed = parser.from_file('/path/to/file', requestOptions={'timeout': 120})
  9. Translate text with the Translate Interface

    master

    The translate interface automatically translates extracted text from a source language to a destination language.

    from tika import translate
    # Example: Translate from Spanish (es) to English (en)
    print(translate.from_file('/path/to/spanish', 'es', 'en'))
    from tika import translate
    print(translate.from_file('/path/to/spanish', 'es', 'en'))