DOCSEARCHbyAlgolia
Submit Docs

Matano Documentation

repository·main·Indexed 23 days ago

https://github.com/matanolabs/matano

Matano is an open-source, cloud-native security data lake platform for AWS. It enables security teams to collect, normalize, and query security logs using serverless infrastructure, Apache Iceberg, and the Elastic Common Schema (ECS). The platform includes a CLI for environment initialization, deployment, and management, supports log transformation via Vector Remap Language (VRL), and allows for threat detection using Python and YAML configurations.

Tokens
28.4K
Snippets
39
Records
183
Agent score
81%

What's inside Matano

  1. Understand the Matano directory structure

    main

    A Matano directory is used to manage all resources in your project, including log sources, detections, and configuration. A typical structure looks like this:

    ├── detections
│   └── <detection_name>
│       ├── detect.py          # Python detection logic
│       └── detection.yml      # Detection configuration
├── log_sources
│   ├── <source_name>
│   │   ├── log_source.yml    # Log source configuration
│   │   └── tables
│   │       └── <table_name>.yml
├── matano.config.yml         # Global Matano configuration
└── matano.context.json       # Matano context file
    tree
├── detections
│   └── aws_root_credentials
│       ├── detect.py
│       └── detection.yml
├── log_sources
│   ├── cloudtrail
│   │   ├── log_source.yml
│   │   └── tables
│   │       └── default.yml
│   └── zeek
│       ├── log_source.yml
│       └── tables
│           └── dns.yml
├── matano.config.yml
└── matano.context.json

  3. Configure shell autocomplete for Matano

    main

    To enable command completion in your terminal, use matano autocomplete [SHELL]. This will provide instructions for installing autocomplete for your specific shell (e.g., bash, zsh).

    $ matano autocomplete bash
$ matano autocomplete zsh

  5. Set up Matano for local development

    main

    To build all Matano packages and install the Matano CLI locally for testing, run the following commands from the repository root:

    make build-all
make local-install

    Warning: Be aware of potential path conflicts if you already have a version of the Matano CLI installed from a published binary.

    make build-all
make local-install

  6. Configure NVM for Node.js development

    main

    If you use NVM (Node Version Manager) to manage your Node.js installation, you must explicitly set the MATANO_REPO_DIR environment variable to the absolute path of your local Matano repository root. This ensures the build system can locate the source files correctly.

    export MATANO_REPO_DIR="/home/myname/workplace/matano"

  7. Install the Matano CLI

    main

    To deploy and manage Matano in your AWS account, you must install the Matano CLI. Choose the command corresponding to your operating system.

    Linux

    curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-linux-x64.sh
chmod +x matano-linux-x64.sh
sudo ./matano-linux-x64.sh

    macOS

    curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-macos-x64.sh
chmod +x matano-macos-x64.sh
sudo ./matano-macos-x64.sh
    curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-linux-x64.sh
chmod +x matano-linux-x64.sh
sudo ./matano-linux-x64.sh

  8. Create detections with Python

    main

    A detection is a Python program invoked with real-time log data to identify threats and create alerts. You can define logic using a detect(record) function. For more complex rules, you can also implement title(r) for custom alert naming and dedupe(r) to group related events (e.g., by IP address).

    def detect(record):
  return (
    record.deepget("event.action") == "CreateInstanceExportTask"
    and record.deepget("event.provider") == "ec2.amazonaws.com"
    and record.deepget("event.outcome") == "failure"
  )

  9. Initialize and deploy Matano

    main

    To start using Matano, run the matano init command.

    Prerequisites:

    • Ensure you have AWS credentials configured in your environment or in an AWS CLI profile.

    Process:

    1. The interactive CLI wizard will guide you through generating an initial Matano directory.
    2. It will initialize your AWS account.
    3. It will deploy the infrastructure into your AWS account (this typically takes a few minutes).

    Once your project is set up, you can use matano deploy from anywhere within your Matano directory to push changes (such as new log sources or detections) to your account.

    matano init

  10. Initialize Matano with `matano init`

    main

    Use the matano init command to start a wizard that sets up your Matano environment. This process creates the necessary AWS resources, initializes your account, and performs the initial deployment.

    $ matano init

# Using a specific AWS profile
$ matano init --profile prod

  11. Develop managed log sources

    main

    Managed log sources are defined in data/managed/log_sources/. To create a new one:

    1. Create a new subdirectory under data/managed/log_sources/ named after your log source.
    2. Add the schema and transform definition within that directory.
    3. Testing: Create a log source in your Matano directory and set managed.type to your new managed log source name.
    4. Send test data to the log source and verify the output.

    VRL Transform Testing: You can test VRL (Vector Remap Language) transforms locally without deploying using:

  12. Install prerequisites for Matano development

    main

    To develop on the Matano source code, you must install the following dependencies depending on which components you are working on:

    • Rust: version 1.63.0
    • Cargo Lambda: Install via pip install cargo-lambda
    • Java: version 11
    • Python: version 3.9
    • Node.js: version >= 14

    Mac OS specific requirement: Install patchelf using Homebrew:

    brew install patchelf
    # No single command installs all, but individual requirements are:
pip install cargo-lambda
brew install patchelf