Avro4k

The AvroObjectContainer class is used to encode and decode Avro 'Object Containers' (also known as Avro Data Files). This format prefixes the binary data with the full schema, making it self-describing.

This is ideal for long-term storage of many objects in a single file.

Usage Note: AvroObjectContainer.decodeFromStream<T>(...) returns a Sequence. Because this is a 'hot' source, you must consume the sequence before closing the underlying stream. Be cautious with sequence.toList() on very large files to avoid OutOfMemoryError.

package myapp

import com.github.avrokotlin.avro4k.*
import kotlinx.serialization.*

@Serializable
data class Project(val name: String, val language: String)

fun main() {
    // Serializing objects
    val valuesToEncode = sequenceOf(
        Project("kotlinx.serialization", "Kotlin"),
        Project("java.lang", "Java"),
        Project("avro4k", "Kotlin"),
    )
    Files.newOutputStream(Path("your-file.bin")).use { fileStream ->
        AvroObjectContainer.openWriter(fileStream).use { writer ->
            valuesToEncode.forEach { writer.write(it) }
        }
    }

    // Deserializing objects
    Files.newInputStream(Path("your-file.bin")).use { fileStream ->
        AvroObjectContainer.decodeFromStream<Project>(fileStream).forEach {
            println(it) // Project(name=kotlinx.serialization, language=Kotlin) ...
        }
    }
}

The AvroSingleObject class implements the Avro Single Object encoding. This format prefixes the binary data with a 64-bit schema fingerprint, allowing the reader to identify the writer schema.

This is highly efficient for message brokers like Kafka or RabbitMQ because it is compact and schema-aware. However, to decode the data, you must provide a mechanism (like a schema registry) to resolve the fingerprint back to the actual schema.

package myapp

import com.github.avrokotlin.avro4k.*
import kotlinx.serialization.*
import org.apache.avro.SchemaNormalization

@Serializable
data class Project(val name: String, val language: String)

fun main() {
    val schema = Avro.schema<Project>()
    val schemasByFingerprint = mapOf(SchemaNormalization.parsingFingerprint64(schema) to schema)
    val singleObjectInstance = AvroSingleObject(schemasByFingerprint::get)

    // Serializing objects
    val data = Project("kotlinx.serialization", "Kotlin")
    val bytes = singleObjectInstance.encodeToByteArray(data)

    // Deserializing objects
    val obj = singleObjectInstance.decodeFromByteArray<Project>(bytes)
    println(obj) // Project(name=kotlinx.serialization, language=Kotlin)
}

The plugin follows specific rules to ensure type safety and compatibility with Avro4k:

• Naming: Class names are derived from Avro schema names; package names are derived from the Avro namespace.
• Unions: Generates a sealed interface where each branch is a nested value class to enforce type safety.
• Nullability: Nullable record fields become nullable Kotlin properties. Nullable fields without Avro defaults are generated as optional with a null default. Maps/Arrays without defaults are generated with empty collections as defaults.
• Schema Preservation: Each generated class contains the original normalized JSON schema. This allows the Avro instance to bypass schema inference and ensures compatibility with Schema Registries.
• Root Types: To prevent losing schema precision for primitives, logical types, or collections, the plugin generates value classes wrapping these types. This ensures the original schema is always used during serialization without requiring the user to manually pass a schema object.

To use avro4k with Maven, you must first configure the kotlin-maven-plugin to include the kotlinx-serialization compiler plugin. Then, add the avro4k-core dependency to your project.

<!-- 1. Configure Kotlin Serialization Plugin -->
<build>
    <plugins>
        <plugin>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-maven-plugin</artifactId>
            <version>${kotlin.version}</version>
            <executions>
                <execution>
                    <id>compile</id>
                    <phase>compile</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <compilerPlugins>
                    <plugin>kotlinx-serialization</plugin>
                </compilerPlugins>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.jetbrains.kotlin</groupId>
                    <artifactId>kotlin-maven-serialization</artifactId>
                    <version>${kotlin.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

<!-- 2. Add avro4k dependency -->
<dependency>
    <groupId>com.github.avro-kotlin.avro4k</groupId>
    <artifactId>avro4k-core</artifactId>
    <version>${avro4k.version}</version>
</dependency>