AvroTurf

If you need to reduce output size (e.g., for Kafka or databases), use the AvroTurf::Messaging API. Instead of prepending the full schema to every message, it uses a schema ID. This requires a running Schema Registry service.

Note: You must explicitly require 'avro_turf/messaging' to use this API. The Messaging format is not compatible with the standard Avro data file API.

Key Methods:

• encode: Automatically registers the schema if it's new. You can also specify a subject and version, or a schema_id directly.
• decode: Fetches the schema from the registry using the ID embedded in the message and caches it.
• decode_message: Returns a result object containing both the message and the schema_id used.

require 'avro_turf/messaging'

# Initialize with your registry URL
avro = AvroTurf::Messaging.new(registry_url: "http://my-registry:8081/")

# 1. Automatic registration
data = avro.encode({ "title" => "hello" }, schema_name: "greeting")

# 2. Encode using specific subject and version
data = avro.encode({ "title" => "hello" }, subject: 'greeting', version: 1)

# 3. Encode using a specific schema_id
data = avro.encode({ "title" => "hello" }, schema_id: 2)

# 4. Decode a message
decoded_data = avro.decode(data)

# 5. Decode and inspect metadata
result = avro.decode_message(data)
puts result.message     # The decoded hash
puts result.schema_id   # The ID used for encoding

You can use AvroTurf.new to manage encoding and decoding using local .avsc files. Schemas are looked up from the directory specified by schemas_path. Each schema file must follow the naming convention <name>.avsc.

Key features:

• Validation: Pass validate: true to the encode method to catch errors early. This raises an Avro::SchemaValidator::ValidationError if the data does not match the schema.
• Schema Registration: By default, encoding may attempt to register schemas. Use register_schemas: false to prevent this behavior.

# Initialize with the directory containing your .avsc files
avro = AvroTurf.new(schemas_path: "app/schemas/")

# Decode data using a named schema
decoded = avro.decode(encoded_data, schema_name: "person")

# Encode data using a named schema
encoded = avro.encode({ "name" => "Jane", "age" => 28 }, schema_name: "person")

# Encode with validation enabled
avro.encode({ "titl" => "hello" }, schema_name: "person", validate: true)

# Encode without registering the schema
avro.encode({ "name" => "Jane" }, schema_name: "person", register_schemas: false)

The AvroTurf::Messaging class provides a way to encode and decode Avro messages without embedding the full schema in the encoded data. It uses Confluent's Schema Registry to manage schemas. When encoding, a schema ID is retrieved from the registry and prepended to the message. When decoding, the schema ID is read from the data to look up the writer's schema from the registry.

Key Methods

• encode(message, options): Encodes a message using a schema. Returns the encoded data as a String.
• decode(data, options): Decodes data into the original message. Returns the decoded message object.
• decode_message(data, options): Decodes data and returns a DecodedMessage object containing metadata about the schema used.

# Example usage of Messaging
# Assuming a registry_url is configured
messaging = AvroTurf::Messaging.new(registry_url: 'http://localhost:8081')

# Encoding
encoded_data = messaging.encode(my_hash, schema_name: 'User', subject: 'user-topic')

# Decoding
# Returns just the message
decoded_hash = messaging.decode(encoded_data, schema_name: 'User')

# Decoding with metadata
# Returns an AvroTurf::Messaging::DecodedMessage object
decoded_msg = messaging.decode_message(encoded_data, schema_name: 'User')
puts decoded_msg.schema_id
puts decoded_msg.message

Instantiate AvroTurf::Messaging with a configuration hash to connect to a Schema Registry and manage local schemas.

Configuration Options