DOCSEARCHbyAlgolia
Submit Docs

Lakewatch Documentation

website·Indexed 30 days ago

https://docs.lakewatch.com

Documentation for Lakewatch, the Databricks Security Lakehouse. Lakewatch provides security information and event management (SIEM) capabilities directly within the Databricks lakehouse. The docs cover data ingestion via Lakeflow Connect, datasource management using the UI, CLI, and Python client, threat detection, natural-language investigation with Genie, and the content marketplace for prebuilt security content.

Tokens
59.9K
Snippets
166
Records
395
Agent score
54%

What's inside Lakewatch

  2. Overview of Lakewatch SIEM capabilities

    Lakewatch is a Databricks feature that provides Security Information and Event Management (SIEM) capabilities within the lakehouse. It allows security teams to perform the following tasks:

    • Data Ingestion: Import security data from sources such as CrowdStrike, Okta, and AWS CloudTrail via managed connectors, cloud storage, or existing workspace tables.
    • Data Normalization: Convert data into the Open Cybersecurity Schema Framework (OCSF) format to ensure detection rules and queries work across diverse data sources.
    • Threat Detection: Execute scheduled SQL-based detection rules to identify threats.
    • Incident Investigation: Use security cases to group related alerts, track entity risk scores, and link involved events.
    • Security Querying: Run SQL queries directly in the Lakewatch UI with time-range filtering and no-code refinements.
    • AI-Assisted Investigation: Use Genie, a context-aware AI assistant, to write queries, build detection rules, and investigate threats using natural language.

  3. Overview of Lakewatch System Health monitoring

    Lakewatch provides system health features to monitor and address operational issues within a workspace. Key monitoring capabilities include tracking detection rule failures, datasource pipeline errors, and data ingestion volume anomalies. This is achieved through three primary mechanisms: System Cases (grouping alerts into investigations), Operational Alerts (event records for configuration or health problems), and Datasource Health Monitoring (detecting latency, missing timestamps, and volume anomalies).

  4. Overview of the Lakewatch Content Marketplace

    The Lakewatch Content Marketplace serves as a central hub for discovering and importing prebuilt security content into a Lakewatch workspace. It provides users with ready-to-use content packs, datasource presets, and detection rules authored by Databricks and the Databricks community.

  5. Overview of the Lakewatch Query Page

    The query page in Lakewatch serves as a central hub for security data analysis. It allows users to search, filter, and analyze security data using SQL, write custom queries, apply real-time filters, and visualize results over time.

  6. Overview of CrowdStrike Falcon Event Stream ingestion via Lakeflow Connect

    Databricks provides a connector as part of Lakeflow Connect (currently in preview) that allows users to ingest data from the CrowdStrike Falcon Event Stream directly into Databricks. This connector handles the data pipeline from the Falcon event stream to destination tables in Databricks.

  7. Overview of Lakewatch datasource health checks

    Lakewatch monitors a datasource's bronze table using three types of scheduled health checks. When a check fails, it emits an operational alert and creates a system case. The affected datasource will display an 'Error' badge on the datasources page.

    • Volume check: Detects if the most recent hour's row count deviates significantly from a historical baseline (e.g., data stopped flowing or a sudden spike).
    • Latency check: Detects if a high percentage of rows arrived in the bronze table significantly after their event time (e.g., upstream pipeline delays).
    • Null timestamp check: Detects if a high percentage of rows are missing original event timestamps (e.g., upstream format changes where the timestamp field moved or changed type).

  8. Overview of Databricks Genie in Lakewatch

    Databricks Genie allows users to investigate security events, write detection logic, and explore data using natural language. It is implemented in two distinct ways depending on the task complexity:

    1. Genie Code: An AI assistant embedded in the Lakewatch UI available on every page. It is best for quick, context-aware tasks such as writing SQL queries, building detection rules, and exploring datasources.
    2. Genie spaces: Dedicated environments powered by the Genie Research Agent for multi-step investigative work. These spaces allow you to connect OCSF-normalized gold tables, system tables, and threat intel feeds to perform complex threat investigations, gap analyses, and MITRE ATT&CK mappings.

  10. Overview of 1Password ingestion via Lakeflow Connect

    Databricks provides a Lakeflow Connect connector (currently in preview) to ingest data from 1Password into Databricks. The connector supports the ingestion of three specific event types: audit events, item usages, and sign-in attempts. Lakewatch specifically provides presets for audit events and sign-in attempts.

  11. Overview of Lakeflow Connect for security data ingestion

    Lakeflow Connect provides managed connectors designed to ingest security data from popular enterprise applications directly into Lakewatch bronze tables. When a datasource is created using a Lakeflow Connect connector, Lakewatch automatically handles the creation and management of the underlying ingestion pipeline to replicate data from the external source.

  12. Overview of Databricks Genie for Lakewatch

    Databricks Genie allows users to investigate security events, write detection logic, and explore data using natural language. Within Lakewatch, Genie is available through two primary interfaces: Genie Code and Genie spaces.