Palantir Conjure

repository·master·Indexed 19 days ago

https://github.com/palantir/conjure

An opinionated toolchain for defining APIs in YAML and generating type-safe client and server implementations across multiple languages, including Java, TypeScript, Python, Rust, and Go. Conjure uses a compiler-and-generator model to produce an Intermediate Representation (IR) from YAML definitions, which is then used by language-specific generators to eliminate serialization bugs and abstract low-level networking details.

Tokens
23.7K
Snippets
66
Records
108
Agent score
67%

What's inside conjure

  1. What is Conjure?

    master

    Conjure is an opinionated toolchain used to define APIs once in a concise, human-readable YAML format and then generate type-safe client and server interfaces in multiple programming languages. It is designed to eliminate serialization bugs and abstract away low-level networking details through ergonomic, idiomatic interfaces.

    Key benefits include:

    • Cross-language compatibility: Enables teams to work together across different language stacks.
    • Type safety: Generates clean abstractions that reduce errors in network requests.
    • Backwards compatibility: Helps preserve compatibility so old clients can communicate with new servers.
    • Domain modeling: Supports expressive types like enums, union types, maps, lists, and sets.
  2. Use Extensions for Metadata in the Conjure Intermediate Representation

    master

    The extensions section is a map of named extensions used to convey metadata about definitions without modifying their core semantics.

    Important Implementation Rules:

    • The structure of an extension is user-defined.
    • Consumers of an extension should treat its presence as optional.
    • Consumers must be tolerant of unknown fields within an extension.
    • Extensions must not change the semantics of the definitions they describe.
    {
      "extensions": {
        "productVersion": "1.0.0"
      }
    }
  3. JSON serialization of Conjure container types

    master

    How container types are represented in JSON:

    • optional<T>: Serialized as JSON(T). If the value is absent, the key should be omitted from JSON Objects, or set to null. Inside a JSON Array, an absent optional must be serialized as null.
    • list<T>: Serialized as a JSON Array. Order must be maintained.
    • set<T>: Serialized as a JSON Array. Order is insignificant.
    • map<K, V>: Serialized as a JSON Object. Keys are strings. For optional<?> values, the key should be omitted if the value is absent, though null is permitted.
  4. Contract for Conjure generators

    master

    Conjure generators are expected to follow a standardized interface to ensure compatibility with build tools like gradle-conjure.

    Distribution

    Generators are typically distributed as a .tgz archive containing a platform-independent executable. The standard structure includes:

    • bin/: Contains the executable script (e.g., conjure-<lang>) for Mac/Linux.
    • lib/: Contains necessary JAR dependencies.

    Execution Model

    Generators must implement a generate command. They consume an intermediate representation (IR) JSON file and write the resulting code to a specified directory.

    Key requirements:

    • Input: An IR JSON file provided as a positional argument.
    • Output: A target directory provided as a positional argument. Generators assume this directory exists and is empty.
    • Exit Codes: Must exit with 0 on success and a non-zero code on failure.
    • Error Reporting: Errors must be written to stderr.
    • File Hygiene: Generators should not produce repository-specific files like .gitignore.
    • Up-to-date checking: Generators are not responsible for diffing or checking if files are up-to-date; this logic belongs to the calling build tool.
    conjure-<lang> generate <input-json> <output-directory>
  5. Client and Server compatibility behaviors

    master

    To ensure robust communication, Conjure follows these compatibility rules:

    • Forward compatible clients: Clients must tolerate extra headers, unknown fields in JSON objects, and unknown variants of enums and unions.
    • Server field strictness: Servers must reject unexpected JSON fields to help developers catch bugs early.
    • Server header tolerance: Servers must tolerate extra headers not defined in the endpoint (e.g., X-Forwarded-For).
    • Enum/Union round-tripping: Clients should be able to round-trip unknown variants of enums and unions.
    • CORS: Servers must support the HTTP OPTIONS method for browser preflight requests.
    • Void endpoints: Clients must tolerate an endpoint expected to return no value actually returning an arbitrary JSON value.
  6. Naming conventions for Conjure build tools

    master

    Build tools (such as gradle-conjure) are designed to integrate Conjure CLIs into specific build ecosystems. They are intentionally named differently from code generators to avoid confusion. They do not use the conjure-<foo> pattern.

    For the Palantir-supported gradle-conjure tool:

    • Maven coordinates: com.palantir.gradle.conjure:gradle-conjure:<version>
    • Gradle plugin name: com.palantir.conjure
    • Git repository: gradle-conjure
  7. Define named types in Conjure

    master

    Conjure allows you to define several kinds of named types that can be referenced throughout your definitions:

    • Object: A collection of named fields, where each field has its own Conjure type.
    • Enum: A type consisting of named string variants (e.g., RED, GREEN, BLUE).
    • Alias: A named shorthand for an existing Conjure type, used to improve readability.
    • Union: Also known as 'algebraic data types' or 'tagged unions', these represent different named variants, each of which can contain different types.
  8. Understand types of wire-format compliance

    master

    Compliance for a Conjure-generated client is categorized into three requirements based on the wire specification:

    1. Understand spec-compliant server responses: The client must correctly deserialize valid responses.
    2. Always send spec-compliant requests: The client must generate requests that follow the wire spec.
    3. Reject non-compliant server responses: The client must fail when receiving responses that violate the spec.

    Note: Client-side and server-side serialization requirements differ. For example, clients must tolerate extra fields in a server response to allow for forward compatibility, whereas servers may reject unknown fields in a JSON request body.

  9. Manage LogSafety for types

    master

    Log safety is used to categorize types according to the SLS specification to prevent sensitive data from being logged.

    Allowed values for safety:

    • safe: The data is safe to log.
    • unsafe: The data is not safe to log.
    • do-not-log: The data must never be logged.

    Rules for declaring safety:

    • Safety can be declared on primitives, aliases, object fields, and union types.
    • Safety for complex types is computed based on the type graph.
    • bearertoken is always do-not-log and its safety cannot be overridden.
    • Safety cannot be declared on maps or wrappers around maps; instead, use alias types to declare safety for the map's components.
    • Safety cannot be declared on a type that is simply a reference to another type (the underlying type is responsible for declaring safety).
    types:
      definitions:
        default-package: com.palantir.product
        objects:
          MySimpleAlias:
            alias: string
            safety: safe
          MyWrapperAlias:
            alias: list<optional<string>>
            safety: unsafe
          MyObject:
            fields:
              wrappedPrimitive:
                type: set<rid>
                safety: do-not-log
              token:
                type: bearertoken
              reference:
                type: MySimpleAlias
              mapField:
                type: map<string,string>
    services:
      MyService:
        package: com.palantir.product
        endpoints:
          send:
            http: POST /send
            args:
              arg:
                type: string
                param-type: body
                safety: safe
  10. How query parameters are serialized in Conjure

    master

    Query parameters use the Conjure paramId as the query key.

    • Optional values: If a parameter of type optional<T> is not present, the key must be omitted from the query string.
    • Lists and Sets: For list<T> or set<T>, each value results in one key=value pair separated by &. For list<T>, the order of values must be preserved.
    • Serialization: Values must be serialized using the PLAIN format and any reserved characters must be URL encoded.
    demoEndpoint:
      http: GET /recipes
      args:
        filter:
          param-type: query
          type: optional<string>
        limit:
          param-type: query
          type: optional<integer>
        categories:
          param-id: category
          param-type: query
          type: list<string>
  11. HTTP endpoint abstractions in Conjure

    master

    When defining APIs in Conjure, you can utilize standard HTTP abstractions to structure your service. These include:

    • HTTP Methods: GET, PUT, POST, and DELETE.
    • Query Parameters: Key-value pairs appended to the URL (e.g., ?foo=bar&baz=2).
    • Path Parameters: Variable segments within a URL path (e.g., /repo/{owner}/{repo}/pulls/{id}).
    • Headers: Non-case sensitive string names associated with Conjure values.
    • Cookie auth: HTTP Cookies, typically used for authentication.
  12. Rules for Conjure JSON deserialization

    master

    When deserializing JSON into Conjure types, follow these rules:

    1. Handling null or absent keys:
      • If a key is absent or null, optional, list, set, and map types must be initialized to their empty variants.
      • Attempting to coerce null or an absent key to any other type (like string or integer) must result in an error.
    2. No automatic casting:
      • Do not coerce types automatically. For example, if a field is defined as boolean, the JSON strings "true" or "false" must be rejected; only actual JSON booleans are valid.

    Note: To avoid ambiguity, Conjure definitions do not allow the type optional<optional<T>>.