DOCSEARCHbyAlgolia
Submit Docs

SuperDB Documentation

website·Indexed 32 days ago

https://superdb.org/

An analytics database built on super-structured data and algebraic types. It features SuperSQL, a backward-compatible language for querying CSV, JSON, and Parquet files, and provides integrations for Zeek logs, Amazon S3, and Grafana, along with client libraries for Go and JavaScript.

Tokens
86.5K
Snippets
597
Records
711
Agent score
60%

What's inside SuperDB

  1. Overview of SuperSQL capabilities

    SuperSQL is a Pipe SQL adaptation designed for super-structured data, making it ideal for ETL, data exploration, and discovery. Key features include:

    • Pipe syntax for compositional queries.
    • Syntactic shortcuts and keyword search.
    • Recursive functions with re-entrant subqueries for traversing nested data.
    • Polymorphic operators that define behavior for every possible input type.

  2. Introduction to SuperSQL

    SuperDB uses SuperSQL, a query language adapted for super-structured data based on Pipe SQL. It is designed to be aspirationally backward-compatible with PostgreSQL.

  3. Overview of SuperDB and Super-structured Data

    SuperDB is an analytics database that uses a 'super-structured' data model to unify relational tables and eclectic JSON. Unlike traditional relational databases that use schema inference or 'variant' columns to handle JSON, SuperDB treats both as special cases of a more general model. Super-structured data is characterized as being dynamic (not handcuffed by schemas), strongly typed (allowing static type checking and optimization), and self-describing (obviating the need for upfront schema definitions).

  4. Introduction to SuperSQL

    SuperSQL is a query language adapted for super-structured data. It is based on Pipe SQL syntax and is designed to be aspirationally backward-compatible with PostgreSQL.

  5. Overview of SuperDB and Super-structured Data

    SuperDB is an analytics database utilizing a 'super-structured' data model that unifies relational tables and JSON. Unlike traditional relational databases that use schema inference or specialized JSON/variant columns, SuperDB treats both relational and dynamic data uniformly.

    Key characteristics of super-structured data:

    • Dynamic: Data collections can vary by type and are not restricted by fixed schemas.
    • Strongly Typed: Supports static type checking and type-based query optimizations even for dynamic data.
    • Self-Describing: Eliminates the need to define schemas upfront, facilitating easier data transmission and serialization without information loss.

  6. Overview of SuperDB integration with Zeek

    SuperDB provides specialized functionality and reference configurations for ingesting and searching logs from the Zeek open source network security monitoring tool. It supports Zeek data through three primary mechanisms:

    1. Compatibility with Zeek's type system.
    2. Native reading of Zeek's TSV format to maintain super-structured data during ingestion.
    3. Support for Zeek's JSON format, which can be converted back into richly typed data using SuperSQL data-shaping scripts.

  7. Overview of SuperSQL pipe operators

    SuperSQL pipelines consist of pipe operators that perform specific operations on a sequence of values. Most operators produce incremental output (streaming results as they are produced), while some—such as aggregate and sort—read all input before producing output.

    Key operator categories include:

    • Filtering: search and where find matching values and drop others.
    • Derivation: values emits output values based on expressions.
    • Routing: fork copies input to parallel branches; switch routes input to a single branch based on clauses.
    • Record Manipulation: put (adds/modifies fields), cut (extracts fields), and aggregate (forms new records via aggregate functions).

  8. Overview of the Super Column (CSUP) Format

    Super Columnar (CSUP) is a file format for storing semi-structured data in columnar form to enable efficient analytics and search. Unlike Parquet, CSUP is schema-less; it does not require a schema declaration during writing. Instead, data columns self-organize based on their type structure. A CSUP file represents a bounded, ordered sequence of values and is designed for random access (e.g., via range requests in cloud object stores or file system seeks).

  9. Overview of the Super (SUP) Format

    Super (SUP) is a human-readable, text-based serialization format for super-structured data. It is a superset of JSON, meaning all valid JSON documents are also valid SUP values. SUP uses UTF-8 encoding and supports single-line (//) and multi-line (/* ... */) comments, which are treated as whitespace and ignored.

  10. Introduction to SuperSQL

    SuperSQL is the query language for SuperDB. It is an adaptation of Pipe SQL designed for super-structured data and is intended to be backward-compatible with PostgreSQL.

  11. Overview of SuperDB and Super-structured Data

    SuperDB is an analytics database based on a 'super-structured' data model. Unlike traditional relational databases that struggle with mixed-type JSON or require rigid schemas, super-structured data is dynamic, strongly typed, and self-describing. This allows it to treat relational tables and eclectic JSON as special cases of the same model, enabling users to mix Parquet and JSON files in queries without requiring strict schema compatibility.

  12. Overview of Super-Structured Data Formats

    SuperDB utilizes a unified data model for super-structured data, allowing for lossless conversion between human-readable, row-based binary, and columnar formats. This ensures that data can be easily programmed in text form while remaining efficient for storage and analysis in binary formats.