DOCSEARCHbyAlgolia
Submit Docs

Apache Fory

repository·main·Indexed 26 days ago

https://github.com/apache/fory

A high-performance, multi-language serialization framework designed for idiomatic domain objects, schema IDL, and efficient cross-language data exchange. It includes the fory-compiler (FDL compiler) and supports shared/circular references and optimized native serialization modes. The project provides comprehensive benchmarks for C++, C#, Dart, Go, and Java to compare performance against Protocol Buffers and MessagePack.

Tokens
347.6K
Snippets
966
Records
1.5K
Agent score
88%

What's inside Apache Fory

  2. Overview of Fory integration tests

    main

    The integration_tests directory contains suites designed to verify Fory's compatibility and interoperability across different environments and languages. Key test areas include:

    • JDK Compatibility: Verifies Fory works across multiple JDK versions (jdk_compatibility_tests).
    • GraalVM Support: Tests support for GraalVM native images (graalvm_tests).
    • JPMS Support: Tests Java Platform Module System (JPMS) module names (jpms_tests).
    • IDL Cross-Language: Tests Fory IDL cross-language generation and round-trip capabilities (idl_tests).
    • gRPC Interoperability: Tests Fory gRPC companion interoperability across different programming languages (grpc_tests).
    • CPython Performance: Includes CPython microbenchmarks for Fory (cpython_benchmark).

  3. Overview of Xlang Cross-language Serialization

    main

    Apache Fory™ xlang serialization provides a dynamic binary format for automatic cross-language object serialization. It is designed to work across multiple languages (including Java, Python, C++, Go, Rust, JS/TS, C#, Swift, Dart, Scala, and Kotlin) without requiring intermediate IDL definitions or schema compilation steps.

    Key capabilities include:

    • Automatic Serialization: Objects are serialized directly without manual protocol conversion.
    • Reference Management: Built-in support for shared references and circular references to prevent duplication or infinite recursion.
    • Polymorphism: Supports object polymorphism with concrete type resolution.
    • Dynamic Format: Uses a dynamic wire format to enable flexibility and ease of use.

  4. Understand the Fory Deserialization Security Model

    main

    The Fory deserialization security model defines how the system protects against resource and policy failures when processing untrusted or partially trusted binary data.

    Key Security Boundaries:

    • Resource Ownership: Protection of memory, CPU progress, stream buffering, file handles, native allocations, and retained read-side tables.
    • Runtime Safety: Prevention of crashes, panics, undefined behavior, and out-of-bounds memory access.
    • Fory Policy Checks: Enforcement of type, function, method, class, or registration policies intended to restrict materialized objects.
    • Cleanup Boundaries: Ensuring state created during a failed read is released or reset before the next operation.

    Note on Application Responsibility: Fory does not protect the semantic meaning of a successfully deserialized value. Application-level authorization, business rules, and domain-specific validation are the responsibility of the developer, not the Fory runtime.

  5. Understand the Fory Swift Benchmark

    main
    The Fory Swift Benchmark is a performance comparison tool used to measure the serialization and deserialization throughput of Apache Fory against Protocol Buffers and JSON within the Swift environment. It evaluates different data types including NumericStruct, Sample, MediaContent, and their respective list variants.

  6. Understand JavaScript Benchmark Semantics

    main

    The JavaScript benchmarks use specific data handling semantics for each format:

    • Fory: Uses serializer-native typed values. Fory receives the pre-normalized Fory value used by its schema.
    • Protobuf: Receives the prebuilt protobuf-shaped value. Note that Protobuf timings do not include the overhead of toProto, fromProto, protobufjs.create, or toObject conversion work.
    • JSON: Receives the standard benchmark JavaScript object.

  7. Understand the differences between Xlang and Python Native mode

    main

    While the binary protocol and API are similar, Xlang mode has stricter requirements than Python Native mode:

    • Allowed Objects: Xlang mode only supports a specific subset of types designed for cross-language compatibility.
    • Prohibited Objects: You cannot serialize global functions, local functions, lambdas, local classes, or types that rely on custom serialization via __getstate__, __reduce__, or __reduce_ex__ in Xlang mode.

  8. Understand Fory Row Format features

    main

    Fory Row Format is a cache-friendly binary format designed for efficient random access and partial serialization. Unlike standard object graph serialization, it allows you to read individual fields without deserializing the entire object.

    Key features include:

    • Zero-Copy Random Access: Read specific fields directly from binary data.
    • Partial Serialization: Skip unnecessary fields during serialization.
    • Cross-Language Compatibility: Data can be shared between Java, Python, and C++.
    • Apache Arrow Integration: Convert row format to/from Arrow RecordBatch for analytics workloads.

  9. Use the Fory Annotation Processor for static serialization

    main

    The fory-annotation-processor generates static serializers for Java classes annotated with @ForyStruct. This is useful for optimizing serialization performance and compatibility in specific environments.

    Use cases

    • Android applications: Provides support for Android Record construction (via desugared Java Record accessors), GraalVM metadata, and automatic R8 rules.
    • Ordinary JVM applications: Use this when configuring ForyJsonBuilder#withCodegen(false).
    • Android model classes: Required for models using Fory type-use annotations like @Ref, @UInt8Type, or @Float16Type.
    • Fory JSON object models: Use when you need direct member access, Android Record support, GraalVM metadata, or automatic R8 rules.
    • GraalVM native images: Use the processor for every reachable @JsonType JSON model and @JsonMixin source. For ordinary Fory serialization in native images, use Fory's native-image build-time serializer generation.

  10. Understand Apache Fory security boundaries and responsibilities

    main

    Apache Fory is an in-process serialization library, not a standalone network service or daemon. Because it operates within your application's process, security is a shared responsibility between the library and your implementation.

    Fory's Security Scope

    Fory focuses on protecting the runtime during the deserialization of bytes or streams. Its security boundaries include:

    • Runtime safety: Preventing crashes, panics, undefined behavior, and out-of-bounds memory access.
    • Resource ownership: Managing memory, CPU progress, stream buffers, native allocations, and callbacks.
    • Policy enforcement: Respecting explicit Fory policies (e.g., TypeChecker or allow-lists) that restrict which types, functions, or methods can be materialized.
    • Cleanup: Ensuring state from failed operations does not leak into subsequent operations.

    Application Responsibilities

    Since Fory does not provide encryption, authentication, or transport security, applications must handle the following:

    • Data Integrity: Authenticate or integrity-check bytes (e.g., via MACs or signatures) before passing them to Fory if they come from untrusted sources.
    • Transport Security: Manage TLS, credentials, and transport lifecycles (Fory's gRPC service companions do not provide this).
    • Configuration:
      • Choose appropriate deserialization modes based on the trust level of the source.
      • Register only safe types and serializers.
      • Configure depth and resource limits appropriate for expected data shapes.
    • Validation: Perform application-level validation on the business logic/meaning of deserialized values.
    • Trust Management: Treat cross-language peers and schemas as part of your application's trust boundary.

  11. Choose a Fory serialization format for Java

    main

    Apache Fory provides three distinct serialization formats depending on your use case:

    1. Binary Object Serialization (org.apache.fory:fory-core): Use for compact object graphs in Java-native mode or for cross-language (xlang) payloads.
    2. Row Format (org.apache.fory:fory-format): Use for zero-copy random access, partial reads, or Apache Arrow integration.
    3. Fory JSON (org.apache.fory:fory-json): Use for high-throughput standard JSON serialization for Java applications.

  12. Choose the appropriate Apache Fory™ Java format

    main

    Apache Fory™ Java offers three primary formats depending on your use case. Ensure all Fory modules used in a single application are kept on the same version.

    • Binary Object Serialization (fory-core): Use for compact object graphs in Java-native mode or for cross-language communication.
    • Row Format (fory-format): Use for zero-copy random access, partial reads, or when integrating with Apache Arrow.
    • Fory JSON (fory-json): Use for high-throughput standard JSON serialization/deserialization in Java applications.