DOCSEARCHbyAlgolia
Submit Docs

Tenzir Documentation

website·Indexed 34 days ago

https://docs.tenzir.com

Tenzir is a data pipeline engine for security teams used to collect, transform, and enrich security data. The documentation covers the Tenzir CLI, Tenzir Node for pipeline execution, and the Tenzir Platform for fleet management. It includes guides on deploying nodes via Docker, Linux, and AWS, configuring TLS, tuning performance, and migrating legacy TQL pipelines to Tenzir v6.

Tokens
502.6K
Snippets
2.5K
Records
3.9K
Agent score
60%

What's inside Tenzir

  1. Overview of Tenzir Security Data Pipeline Platform

    Tenzir is a data pipeline platform designed for security use cases (SecDataOps). It enables the collection, shaping, enrichment, and routing of security data between various technologies using security-native operators. The platform is built on open standards, including Apache Arrow, Parquet, SIGMA, and STIX, to prevent vendor lock-in and allow users to move data across different security solutions.

  2. Overview of Tenzir for AWS Security Data Pipelines

    Tenzir provides a high-performance data fabric for AWS security stacks, allowing users to ingest, process, and normalize telemetry from services like CloudTrail, VPC Flow Logs, and S3. It is designed to replace complex custom 'glue code' or Lambda scripts with a configuration-driven pipeline that reduces data volume (by 30-80%) and minimizes latency (< 0.1s) for high-volume event streams (>1M events per second).

  3. Overview of Tenzir Microsoft ecosystem integrations

    Tenzir provides end-to-end connectivity for the Microsoft estate using a single pipeline language. Supported integrations include:

    • Streaming: Azure Event Hubs
    • Storage & Replay: Azure Blob Storage
    • SIEM/Analytics: Microsoft Sentinel and Azure Log Analytics
    • Telemetry Collection: Microsoft 365, Microsoft Defender, Microsoft Graph, and Windows Event Logs.

    Users can utilize the Microsoft package from the Tenzir Library, which contains over 70 operators designed to parse and map telemetry from Windows, Active Directory, Azure, Microsoft 365, and Defender to the Open Cybersecurity Schema Framework (OCSF).

  4. Overview of onboarding a data source

    Onboarding a data source is the process of converting raw product output into structured events for the data lifecycle. This involves two primary stages:

    1. Parsing: Converting raw text into structured events while preserving the product's original field names and ensuring correct data types (e.g., IP addresses, timestamps).
    2. Mapping: Translating those structured events into a target schema, typically OCSF (Open Cybersecurity Schema Framework).

  5. Overview of ClickHouse integration in Tenzir

    Tenzir uses ClickHouse as an analytical store for security telemetry. It allows users to write normalized security telemetry to ClickHouse using the to_clickhouse operator and read tables or SQL query results back into Tenzir pipelines using the from_clickhouse operator. Tenzir acts as a programmable pipeline that collects, parses, maps to OCSF, and enriches data before writing it to ClickHouse.

  6. Overview of the window operator

    The window operator runs a subpipeline over time-based or count-based windows. It is used to provide bounded semantics similar to streamstats in Splunk SPL. It supports fixed windows (tumbling or sliding), trailing windows (anchored to events), and session windows (grouped by inactivity gaps).

  7. Overview of the Tenzir data lifecycle

    The Tenzir data lifecycle transforms raw security text into structured data for detections, dashboards, and auditors. The lifecycle consists of 13 stages grouped into four phases. While most stages can run concurrently in separate pipelines connected by publish and subscribe operators, the first four stages must occur in a specific order: Collect $\rightarrow$ Parse $\rightarrow$ Shape $\rightarrow$ Normalize. All stages are implemented using standard TQL.

  8. Overview of Tenzir Query Language (TQL)

    TQL (Tenzir Query Language) is a native, domain-specific language used within Tenzir to manipulate security data in a pipeline. It is designed for intuitiveness and composability, enabling security analysts to automate workflows through complex operations such as parsing, shaping, and transforming data. Its functionality is similar to a modernized combination of command-line pipes and specialized query languages.

  9. Introduction to Tenzir Query Language (TQL)

    Tenzir Query Language (TQL) is a low-code, pipe-based language used to build security data pipelines. It follows a Unix-shell philosophy where data is taken from a source and passed through a series of modular operators to filter, shape, enrich, and route it. TQL is designed to be readable and intuitive, allowing for complex logic without requiring full-scale programming or complex GUI configurations.
    // Perform a paginated API request.
from_http "api.example.org", max_retry_count=3, paginate=(x => x.next_url?)

// Unroll array of records in response.
unroll data

// Reshape data
set src_ip = data.src.ip()
set src_port = data.port.int()
set hostname = data.host

// Enrich with threat intelligence.
context_enrich "iocs", field=hostname

// Escalate high-severity events immediately to Splunk.
if iocs.severity > 0.8 {
  to_splunk "https://splunk.local:8088", hec_token=secret("splunk-token")
}

// Write full data feed as Hive-partitioned Parquet files into S3.
set year = data.year()
set month = data.month()
to_hive "s3://my-bucket/some/subdirectory",
  partition_by=[year, month],
  format="parquet",
  max_size=10G

  11. Overview of Tenzir Solution Initiatives

    Tenzir provides funded programs (Initiatives) designed to solve specific security data challenges. These include:

    • AI Observability: Collect prompts, tool calls, and model traffic as security telemetry to detect prompt injection, data leakage, and runaway spend.
    • SIEM Cost Optimization: Reduce SIEM license costs by 30-80% by filtering, aggregating, and routing logs before indexing.
    • Security Data Lake: Support for writing to AWS S3, Google Cloud Storage, Azure Blob, Snowflake, Databricks, or MinIO, with querying via Tenzir, Spark, Trino, DuckDB, or Parquet-compatible tools.
    • SIEM Migration: Facilitates side-by-side operation of old and new platforms to migrate sources incrementally without visibility gaps.
    • Compliance & Audit: Long-term retention of audit-ready security data in cost-effective storage for regulatory requirements.
    • Data Sovereignty: Deployment options for on-premises, EU regions, or fully air-gapped environments to meet sovereignty rules.

  12. Overview of Tenzir Platform Workspaces

    Workspaces in the Tenzir Platform are used to logically group nodes, secrets, and dashboards. They can be managed in two ways: dynamically via the CLI (best for ad-hoc management) or statically via configuration files (best for GitOps-based infrastructure management).