DOCSEARCHbyAlgolia
Submit Docs

Zed Language

website·Indexed 33 days ago

https://zed.brimdata.io

A query language for search, analytics, and transformation. The documentation covers the Zed data model, type conventions, and a comprehensive reference of operators, functions, and aggregate functions. It also includes guides for the zq command-line tool, which uses the Zed language for pipeline-style search and analytics.

Tokens
821.8K
Snippets
5.5K
Records
6.7K
Agent score
42%

What's inside Zed

  1. Overview of the Zed Super-structured Data Model

    v1.11.0
    Zed is a 'super-structured' data model that unifies the relational data model (SQL tables) and the document model (JSON). Unlike traditional tabular models that require a single composite schema for all values, or semi-structured models that require traversing data to discover structure, Zed uses a granular type system where each value expresses its own type. This allows Zed to represent both relational tables and hierarchical documents within a single model, making it a superset of formats such as JSON, BSON, Ion, Avro, ORC, Parquet, CSV, JSON Schema, and XML.

  2. Overview of the Zed Lake Storage Format

    v1.6.0
    The Zed lake storage format is an open specification for organizing eclectic data based on a super-structured type system rather than relational tables. It allows for a universal data representation where relations, schemas, Parquet, Avro, JSON, CSV, and XML are treated as subsets of the Zed type system. All data and metadata in a Zed lake conform to the Zed data model; for example, search indexes are implemented as ZNG files with embedded B-Tree structures.

  3. Overview of the Zed Project and Data Model

    Zed is a data system based on a "super-structured" data model that allows messy JSON data to be treated with the precision of relational tables while maintaining the flexibility of JSON. The system is built on composable building blocks and self-describing data structures, eliminating the need for external schema registries (like Protobuf) or auxiliary databases (like Hive metastore).

  4. Overview of zq command-line tool

    v1.7.0
    zq is a command-line tool that uses the Zed language for pipeline-style search and analytics. It can query data from local files, HTTP/HTTPS URLs, and S3 storage. It is designed to blend the functionality of tools like jq, awk, and grep with the database capabilities of sqlite and duckdb. The tool is most performant when using the Zed-native ZNG format.

  5. Overview of zq command-line tool

    v1.5.0
    zq is a command-line tool for processing data in diverse input formats using the Zed language. It allows for pipeline-style search and analytics, blending the approach of tools like jq, awk, and grep with embedded databases like sqlite and duckdb. It can query files, HTTP/HTTPS URLs, and S3 storage. The tool is most performant when using the Zed-native ZNG format.

  6. Overview of data shaping in Zed

    v1.10.0
    Shaping is the process of cleaning up disparate data sources to fit a well-defined set of schemas. In Zed, this involves converting leaf field types, adding or removing fields, and reordering fields to unify data into a consistent structure.

  7. Overview of zq command-line tool

    v1.16.0
    zq is a command-line tool for processing data in diverse input formats using the Zed language. It allows for search, analytics, and transformations via a pipeline-style approach. It can query data from local files, HTTP/HTTPS URLs, S3 storage, or standard input (-). It is optimized for the Zed-native ZNG format.

  8. Overview of Zed record operators

    v1.11.0

    Zed provides several operators specifically designed for sequences of records (structured logs or events):

    • cut: Extract subsets of record fields into new records.
    • drop: Remove specific fields from record values.
    • fuse: Coerce all input values into a single merged type (uniform schema).
    • put: Add or modify fields of records.
    • rename: Change the name of record fields.

  9. Overview of Zed client libraries

    Zed provides client libraries for a limited number of languages, allowing developers to manipulate Zed data and interact with a Zed service through its remote API. The project aims to expand support for a broader range of languages and welcomes open-source contributions for new library development.

  10. Overview of Zed built-in dataflow operators

    v1.10.0

    Dataflow operators in Zed process a sequence of input values to create an output sequence and serve as the primary components of a dataflow pipeline. In addition to built-in operators, Zed supports user-defined operators.

    Commonly used built-in operators include:

    • assert: Evaluate an assertion
    • combine: Combine parallel paths into a single output
    • cut: Extract subsets of record fields into new records
    • drop: Drop fields from record values
    • file: Source data from a file
    • fork: Copy values to parallel paths
    • from: Source data from pools, files, or URIs
    • fuse: Coerce all input values into a merged type
    • get: Source data from a URI
    • head: Copy leading values of input sequence
    • join: Combine data from two inputs using a join predicate
    • load: Add and commit data to a pool
    • merge: Combine parallel paths into a single, ordered output
    • over: Traverse nested values as a lateral query
    • pass: Copy input values to output
    • put: Add or modify fields of records
    • rename: Change the name of record fields
    • sample: Select one value of each shape
    • search: Select values based on a search expression
    • sort: Sort values
    • summarize: Perform aggregations
    • switch: Route values based on cases
    • tail: Copy trailing values of input sequence
    • uniq: Deduplicate adjacent values
    • where: Select values based on a Boolean expression
    • yield: Emit values from expressions

  11. Overview of zq command-line tool

    v1.6.0
    zq is a command-line tool that uses the Zed language for pipeline-style search and analytics. It can query data from local files, HTTP/HTTPS URLs, or S3 storage. It is designed to blend the functionality of tools like jq, awk, and grep with the embedded database capabilities of sqlite and duckdb. It is most performant when using the Zed-native ZNG format.

  12. Overview of the ZJSON format

    v1.10.0

    ZJSON is a JSON-based protocol used to serialize Zed data without losing type information or precision (e.g., preventing int64 or timestamp precision loss in JavaScript). A ZJSON stream consists of a sequence of JSON objects, where each object represents a Zed value with the following structure:

    {
  "type": <type>,
  "value": <value>
}

    ZJSON files are formatted as newline delimited JSON (NDJSON).