DOCSEARCHbyAlgolia
Submit Docs

Kaitai Struct Documentation

repository·master·Indexed 26 days ago

https://github.com/kaitai-io/kaitai_struct

A declarative language and compiler ecosystem for describing binary data structures. It allows developers to define formats in .ksy files and generate parsers across multiple programming languages using the kaitai-struct-compiler (ksc). The documentation covers the project workflow, the use of the Web IDE and ksv visualizer, and the required KaitaiStream methods for implementing new runtimes, including stream positioning, integer/floating-point reading, and bit/byte array operations.

Tokens
999
Snippets
0
Records
7
Agent score
39%

What's inside Kaitai Struct

  1. Overview of Kaitai Struct

    master
    Kaitai Struct (KS) is a declarative language used to describe binary data structures, such as binary file formats or network stream packet formats. By describing a format once in a .ksy file, you can compile it using the kaitai-struct-compiler (or ksc) into source code for various supported programming languages. This generated code provides a parser that reads the described data from a file or stream and provides an easy-to-use API to access the data components.

  2. Workflow for using Kaitai Struct in your project

    master

    To use Kaitai Struct formats in your application, follow these steps:

    1. Describe the format: Create a .ksy file defining the binary structure.
    2. Debug the format: Use a visualizer to ensure the format parses data correctly. Official tools include the Web IDE and the ksv console visualizer.
    3. Compile: Use the compiler to turn the .ksy file into source files for your target programming language.
    4. Integrate: Include the generated source files in your project.
    5. Add Runtime: Add the specific Kaitai Struct runtime library for your language to your project (required for code readability).
    6. Parse: Use the generated classes to parse your binary files or streams and access the data.

  3. Required Integer and Floating Point reading methods

    master

    Runtimes must support reading various integer and floating-point types with specific endianness:

    Signed Integers

    • read_s1: 1-byte signed integer.
    • read_s2be, read_s4be, read_s8be: 2, 4, or 8-byte big-endian signed integers.
    • read_s2le, read_s4le, read_s8le: 2, 4, or 8-byte little-endian signed integers.

    Unsigned Integers

    • read_u1: 1-byte unsigned integer.
    • read_u2be, read_u4be, read_u8be: 2, 4, or 8-byte big-endian unsigned integers.
    • read_u2le, read_u4le, read_u8le: 2, 4, or 8-byte little-endian unsigned integers.

    Floating Point

    • read_f4be, read_f8be: 4 or 8-byte big-endian floating point numbers.
    • read_f4le, read_f8le: 4 or 8-byte little-endian floating point numbers.

  4. Required Stream positioning methods

    master

    Runtimes must implement these methods for managing the current position within the data stream:

    • eof?: Check if the end of the stream has been reached.
    • seek(n): Move the stream position to a specific offset.
    • pos: Get the current stream position.
    • size: Get the total size of the stream.

  5. Required Byte Array processing and Misc operations

    master

    Runtimes must provide these specialized processing and utility methods:

    Byte array processing

    • process_xor(data, key): XOR data with a key.
      • process_xor_one(data, key)
      • process_xor_many(data, key)
    • process_rotate_left(data, amount, group_size): Perform a left rotation on data.
    • process_zlib(data): Decompress data using zlib.

    Misc runtime operations

    • static mod(a, b): Perform modulo operation.

  6. Required KaitaiStream methods for runtime implementation

    master
    When implementing a new Kaitai Struct runtime, the KaitaiStream class (or its equivalent) must provide a specific set of methods to ensure compatibility with compiled Kaitai Struct decoders. These methods cover stream positioning, integer reading (signed/unsigned, big/little-endian), floating point numbers, bit manipulation, byte array operations, and specific byte processing algorithms.

  7. Required Bit and Byte Array methods

    master

    Runtimes must implement the following for bit-level access and byte array handling:

    Unaligned bit values

    • align_to_byte(): Align the current bit position to the next byte boundary.
    • read_bits_int(n): Read n bits as an integer.
    • read_bits_array(n): Read n bits as an array.

    Byte arrays

    • read_bytes(n): Read n bytes.
    • read_bytes_full: Read all remaining bytes.
    • read_bytes_term(String encoding, int term, boolean include_term, boolean consumeTerm, boolean eosError): Read bytes until a terminator is found.
    • ensure_fixed_contents(expected): Validate that the content matches an expected size/value.
    • static bytes_strip_right(bytes, pad_byte): Strip padding bytes from the right.
    • static bytes_terminate(bytes, term, include_term): Terminate a byte array based on a term.
    • static bytes_to_str(bytes, encoding): Convert bytes to a string using a specific encoding.