DOCSEARCHbyAlgolia
Submit Docs

Tenzir Documentation

website·Indexed 34 days ago

https://tenzir.com/docs/

A data pipeline engine for security teams to collect, transform, and enrich security data. Documentation covers the Tenzir CLI, Tenzir Node for pipeline execution, and the Tenzir Platform for fleet management. Includes guides on TQL pipelines, v6 migration, node deployment (Docker, Linux, AWS), performance tuning, and platform configuration including OIDC identity providers and PostgreSQL databases.

Tokens
505.8K
Snippets
2.6K
Records
3.9K
Agent score
71%

What's inside Tenzir

  1. Overview of Tenzir Query Language (TQL)

    Tenzir Query Language (TQL) is a dataflow language designed for processing unstructured byte streams and structured events. It allows developers to build pipelines that collect, parse, transform, and route data. TQL operates as a layer between the streaming executor (which manages the DAG and columnar batch processing) and reusable integrations (e.g., Kafka, S3, Syslog) and packages.

  2. Overview of Tenzir Microsoft Integration capabilities

    Tenzir provides end-to-end connectivity for the Microsoft ecosystem, allowing developers to stream, store, and ingest telemetry using a single pipeline language. Supported integrations include:

    • Streaming & Storage: Azure Event Hubs and Blob Storage (for storage and replay).
    • SIEM & Analytics: Microsoft Sentinel and Azure Log Analytics (for querying and ingestion).
    • Telemetry Sources: Microsoft 365, Microsoft Defender, Microsoft Graph, and Windows Event Logs.

    The Tenzir Library provides a dedicated Microsoft package containing over 70 operators designed to parse and map telemetry from Windows, Active Directory, Azure, Microsoft 365, and Defender into the Open Cybersecurity Schema Framework (OCSF).

  3. Overview of ClickHouse integration in Tenzir

    Tenzir integrates with ClickHouse to use it as an analytical store for security telemetry. It provides two primary operators: to_clickhouse for writing normalized security telemetry to ClickHouse tables, and from_clickhouse for reading tables or SQL query results back into Tenzir pipelines. This allows Tenzir to act as a programmable pipeline that collects, parses, maps to OCSF, and enriches data before storing it in ClickHouse.

  4. Overview of SecDataOps and Tenzir

    SecDataOps applies DataOps principles to security data management to move away from rigid, vendor-controlled ingestion models. Tenzir implements this by providing security-native data pipelines that process, normalize, enrich, and correlate data in motion before it reaches a SIEM. This approach reduces ingestion costs by filtering noise and duplicates upstream and ensures interoperability by leveraging open standards like Apache Arrow and OCSF mappings.

  5. Overview of Tenzir Security Data Pipeline Platform

    Tenzir is a data pipeline platform designed for security use cases (SecDataOps). It enables the collection, shaping, enrichment, and routing of security data between various technologies. The platform is built on open standards, including Apache Arrow, Parquet, SIGMA, and STIX, to prevent vendor lock-in and allow data portability.

  6. Introduction to Tenzir AI Workbench

    The Tenzir AI Workbench allows you to integrate an AI agent and configure it with Tenzir's agent skills. This setup enables the AI agent to perform specialized tasks such as writing TQL pipelines, interpreting OCSF schemas, generating parsers, and creating data mappings.

  7. Overview of Amazon Security Lake integration

    Tenzir integrates with Amazon Security Lake to send security data in the Open Cybersecurity Schema Framework (OCSF) format. This is achieved using the to_amazon_security_lake operator. Due to Amazon Security Lake's requirement that each Parquet file contains records of only one OCSF event class, Tenzir follows a pattern of creating one custom source and one dedicated pipeline per OCSF event class.

  8. Overview of the Tenzir Data Lifecycle

    The Tenzir data lifecycle transforms raw security data (text/bytes) into structured information usable by detections, dashboards, and auditors. The lifecycle consists of 13 stages grouped into four phases. While most stages can run concurrently in separate pipelines connected by publish and subscribe operators, the first four stages must occur in a specific order: Collect $\rightarrow$ Parse $\rightarrow$ Shape $\rightarrow$ Normalize. All stages are implemented using TQL (Tenzir Query Language).

  9. Overview of Tenzir Query Language (TQL)

    TQL (Tenzir Query Language) is a native, domain-specific language used within Tenzir to manipulate security data in a pipeline. It is designed for intuitiveness and composability, allowing security analysts to automate workflows through complex operations such as parsing, shaping, and transforming data. Its functionality is conceptually similar to a modernized combination of command-line pipes and specialized query languages.

  10. Overview of Tenzir security data pipeline

    Tenzir is an AI-powered security data pipeline platform designed for builders and cyber defenders. It enables the collection, shaping, enrichment, routing, and storage of security telemetry through the use of programmable pipelines.

  11. Overview of SecDataOps and Security Data Pipeline Platforms (SDPP)

    SecDataOps applies DataOps principles (pipeline automation, as-code workflows, and dynamic transformations) to security telemetry. The goal is to minimize data costs (computational and storage) while maximizing utility. This is implemented via a Security Data Pipeline Platform (SDPP), a control layer situated between telemetry sources and downstream consumers like SIEMs, data lakes, and detection tools. Key techniques include:

    • Reshaping and Reduction: Filtering, deduplicating, and trimming data at the edge to reduce input volume by 30-80%.
    • Compaction: Using aggregation (e.g., turning raw flows into a communication matrix) to preserve context while reducing data volume.
    • Edge Enrichment and Detection: Adding context (user, asset, threat intel) and running detection rules (e.g., Sigma rules) early in the pipeline to avoid duplicated effort downstream.

  12. Overview of ZeroMQ integration in Tenzir

    Tenzir integrates with ZeroMQ (0mq) to send, receive, and route messages asynchronously. It supports PUB (publisher) and SUB (subscriber) socket types in both bind and connect modes. To listen on all available network interfaces, use the IP address 0.0.0.0.

    ZeroMQ does not have a native topic abstraction; Tenzir implements this using an optional prefix prepended to outgoing messages. Subscribers use ZeroMQ's native byte-prefix filtering to match these. By default, receivers strip the matched prefix before processing the event unless keep_prefix=true is specified.