Apache Avro Documentation

Avro Python is a library that implements the Avro Specification for data serialization. It provides the following core capabilities:

• Schema Assembly: Programmatically build Avro schemas.
• Schema Parsing: Parse Avro schemas (written in JSON) into Schema objects.
• Binary Encoding/Decoding: Use primitive functions to encode data into Avro format and decode it back.
• Data Streams: Provides streams for storing and reading data used by Encoders and Decoders.
• Avro DataFile Support: Support for the Avro DataFile format.

Apache Avro is a data serialization system designed for efficiency and flexibility. It provides rich data structures and a compact, fast, binary data format.

Key features include:

• Container Files: For storing persistent data.
• Remote Procedure Call (RPC): For communication between systems.
• Dynamic Language Integration: Code generation is not required to read or write data files or to use/implement RPC protocols. This makes it ideal for dynamic or scripting languages.
• Optional Code Generation: While not required, code generation can be used as an optimization for statically typed languages.

Before using the Avro PHP library, ensure your environment meets the following requirements:

• PHP Version: 8.1 or higher
• 32-bit Platforms: Requires the GMP PHP extension
• Zstandard Compression: Requires ext-zstd
• Snappy Compression: Requires ext-snappy
• Testing: Requires PHPUnit

On many systems, GMP and PHPUnit can be installed via package managers (e.g., php8.1-gmp and phpunit).

In avro-js, every Avro type maps to a corresponding JavaScript Type class (e.g., int maps to IntType, record to RecordType). An instance of a Type provides methods to encode and decode data, perform validity checks, and generate random data.

Key capabilities of a Type instance:

• Buffer Encoding/Decoding: Use toBuffer(val) to encode a value into an Avro-encoded buffer and fromBuffer(buf) to decode it back.
• JSON Encoding/Decoding: Use toString(val) for JSON-encoded strings and fromString(jsonString) to parse them.
• Validity Checks: Use isValid(val) to check if a JavaScript object conforms to the schema.
• Random Generation: Use random() to generate a random value matching the type.

Avro binary data does not include type information or field names to minimize size. Consequently, a schema must be used to read Avro data correctly.

To ensure successful deserialization:

• Always include the writer's schema with the stored data.
• In RPC systems, guarantee that recipients have the schema used to write the data.
• For schema evolution (reading data with a newer schema), use the Schema Resolution process.
• While it is possible to read data with a different schema if primitive values are compatible (e.g., int can be deserialized as long), it is highly recommended to use schemas with identical Parsing Canonical Forms to ensure reliability.

Avro uses Spotless to ensure Java code complies with project conventions (including a specific two-space indentation rule). If your code fails compilation due to formatting issues, you can automatically apply the correct style using Maven.

mvn spotless:apply

API documentations for C, C++, and C# modules are generated by their respective build.sh dist implementations. To include these in the website, copy the generated HTML to the external documentation folder:

cp ../avro/build/avro-doc-1.12.0-SNAPSHOT/api/c/* content/en/docs/external/c/

Avro IDL supports Java-style comments:

• Single-line comments: Use // to ignore text on a line.
• Multi-line comments: Use /* ... */ to ignore text spanning multiple lines.
• Documentation strings: Comments starting with /** are used as the documentation string for the type or field definition that immediately follows.

To automate the compilation of .avdl files in a Maven project, add the avro-maven-plugin to your pom.xml with the idl goal.

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro-maven-plugin</artifactId>
      <executions>
        <execution>
          <goals>
            <goal>idl</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>