Allegro JSON Avro Converter

repository·master·Indexed 43 days ago

https://github.com/allegro/json-avro-converter

A Java tool for converting between JSON and Apache Avro formats. It simplifies migration by allowing JSON data conversion to Avro without manual type wrapping for optional fields. Features include the JsonAvroConverter and AvroJsonConverter classes, custom conversion via the AvroTypeConverter interface, and a Validator CLI tool for schema validation and round-trip testing.

Tokens
1.5K
Snippets
7
Records
7
Agent score
5%

What's inside json-avro-converter

  1. Handle AvroConversionException

    master

    When performing conversions, if the JSON data does not conform to the provided Avro schema (e.g., type mismatch), the library throws an AvroConversionException.

    try {
        converter.convertToAvro(invalidJson.getBytes(), schema);
    } catch (AvroConversionException ex) {
        System.err.println("Caught exception: " + ex.getMessage());
    }
  2. Use AvroJsonConverter for Avro to JSON conversion

    master

    The AvroJsonConverter class is used to convert binary Avro data back into JSON format.

    import tech.allegro.schema.json2avro.converter.AvroJsonConverter;
    
    AvroJsonConverter converter = new AvroJsonConverter();
    // conversion from binary Avro to JSON
    byte[] binaryJson = converter.convertToJson(avro, schema);
  3. Use JsonAvroConverter for JSON to Avro conversion

    master

    The JsonAvroConverter class provides methods to convert JSON byte arrays into Avro formats. It supports converting to binary Avro and to Avro GenericData.Record objects. It also handles optional fields (unions) without requiring explicit type wrapping in the JSON and ignores unknown fields not declared in the schema.

    Key methods:

    • convertToAvro(byte[] jsonBytes, String schema): Returns a byte[] containing the binary Avro data.
    • convertToGenericDataRecord(byte[] jsonBytes, Schema schema): Returns an org.apache.avro.generic.GenericData.Record.
    import tech.allegro.schema.json2avro.converter.JsonAvroConverter;
    import org.apache.avro.generic.GenericData;
    import org.apache.avro.Schema;
    
    String schema = "{\"type\":\"record\",\"name\":\"Acme\",\"fields\": [{\"name\": \"username\", \"type\": \"string\"}]}";
    String json = "{ \"username\": \"mike\" }";
    
    JsonAvroConverter converter = new JsonAvroConverter();
    
    // conversion to binary Avro
    byte[] avro = converter.convertToAvro(json.getBytes(), schema);
    
    // conversion to GenericData.Record
    GenericData.Record record = converter.convertToGenericDataRecord(json.getBytes(), new Schema.Parser().parse(schema));
  4. Extend the converter with AvroTypeConverter

    master

    If the library does not support certain Avro types (like specific logical types), you can implement the AvroTypeConverter interface to define custom conversion logic. You can target specific fields by checking the path in the canManage method.

    To apply your custom converter, instantiate JsonAvroConverter using a CompositeJsonToAvroReader containing your custom converter.

    public class CustomFieldConverter implements AvroTypeConverter {
        @Override
        public Object convert(Schema.Field field, Schema schema, Object jsonValue, Deque<String> path, boolean silently) {
            return "custom-" + jsonValue;
        }
    
        @Override
        public boolean canManage(Schema schema, Deque<String> path) {
            return "customField".equals(path.getLast());
        }
    }
    
    // Usage:
    JsonAvroConverter converter = new JsonAvroConverter(new CompositeJsonToAvroReader(new CustomFieldConverter()));
  5. Use the Validator CLI tool

    master

    The Validator is a command-line tool for validating JSON/Avro documents against a schema.

    Modes (-m):

    • Default: JSON to Avro validation.
    • avro2json: Converts Avro binary data into JSON.
    • json2avro2json: Performs a round-trip (JSON $\rightarrow$ Avro $\rightarrow$ JSON) to show how messages look after encoding/decoding.

    Arguments:

    • -s <schema>: Path to the Avro schema file.
    • -i <input>: Path to the input file (JSON or Avro).
    • -m <mode>: The conversion mode.
    # JSON to Avro validation
    java -jar json2avro-validator.jar -s user.avcs -i user.json
    
    # Avro to JSON conversion
    java -jar json2avro-validator.jar -s user.avcs -i user.avro -m avro2json
    
    # JSON to Avro to JSON round-trip
    java -jar json2avro-validator.jar -s user.avcs -i user.json -m json2avro2json