DOCSEARCHbyAlgolia
Submit Docs

Malcolm Network Traffic Analysis Suite

repository·main·Indexed 25 days ago

https://github.com/cisagov/malcolm

A containerized network traffic analysis suite that automates the processing of PCAP, Zeek, and Suricata data. It provides security visibility via OpenSearch Dashboards and Arkime, supporting both long-term SOC deployments on Linux servers and local incident response. The suite includes capabilities for data ingestion, ICS protocol visibility, and specialized builds for Raspberry Pi Hedgehog sensors.

Tokens
131.7K
Snippets
231
Records
615
Agent score
82%

What's inside Malcolm

  1. Overview of the Malcolm REST API

    main
    Malcolm provides a REST API that allows you to programmatically query Malcolm's status and its underlying data. The API surface includes endpoints for dashboard exports, ingest statistics, document lookup, event logging, field aggregations, field definitions, index management, and system health (ping, ready, and version).

  2. Overview of Malcolm network traffic analysis

    main

    Malcolm is a network traffic analysis tool suite designed for network security monitoring. It automates the normalization, enrichment, and correlation of network data to provide visibility into communications.

    Key Capabilities:

    • Data Ingestion: Accepts full packet capture (PCAP) files, Zeek logs, and Suricata alerts. Data can be uploaded via a browser-based interface or forwarded live using lightweight forwarders.
    • Analysis Interfaces:
      • OpenSearch Dashboards: Provides data visualization through prebuilt dashboards for monitoring network protocols.
      • Arkime: Used for finding and identifying network sessions during security incident investigations.
    • Deployment Model: Operates as a cluster of isolated software containers, making it suitable for both long-term SOC deployment on Linux servers and incident response on local machines (e.g., a MacBook).
    • Security: All communications (UI and remote log forwarders) are secured using industry-standard encryption.
    • ICS Visibility: Ongoing development focuses on adding parsers for Industrial Control Systems (ICS) protocols.

  3. What is Hedgehog Linux?

    main

    Hedgehog Linux is Malcolm's dedicated Network Sensor operating system. It is a Debian-based OS designed to perform the following tasks:

    • Monitor network interfaces.
    • Capture packets to PCAP files.
    • Detect file transfers in network traffic and extract/scan those files for threats.
    • Generate and forward Zeek logs, Arkime sessions, and other telemetry to a Malcolm instance.

    Note on Installation: As of Malcolm v25.12.0, Hedgehog Linux and Malcolm share a single codebase. The Hedgehog Linux installer ISO is a specific 'flavor' of the Malcolm installer ISO, preconfigured to use the Hedgehog run profile. Because they share a codebase, installation and configuration procedures are identical for both platforms.

  4. Overview of Malcolm's network traffic analysis capabilities

    main

    Malcolm is a network traffic analysis tool suite designed for network security monitoring. It automates the normalization, enrichment, and correlation of network data to provide visibility into communications.

    Key capabilities include:

    • Data Ingestion: Accepts full packet capture (PCAP) files, Zeek logs, and Suricata alerts. Data can be uploaded via a browser-based interface or forwarded live using lightweight forwarders.
    • Analysis Interfaces:
      • OpenSearch Dashboards: Provides data visualization and prebuilt dashboards for protocol overviews.
      • Arkime: Used for identifying and finding specific network sessions during security incidents.
    • Deployment Model: Operates as a cluster of software containers, allowing for deployment on various platforms ranging from Linux servers in a SOC to individual MacBooks for incident response.
    • Security: Uses industry-standard encryption for all communications (UI and remote log forwarders).
    • ICS Visibility: Ongoing development focuses on adding parsers for Industrial Control Systems (ICS) protocols.

  5. Overview of Malcolm's core components

    main

    Malcolm is a network security monitoring platform that integrates several open-source tools to provide traffic analysis, threat detection, and visualization. Key functional areas include:

    • Traffic Processing & Analysis: Uses Arkime (formerly Moloch) for PCAP processing and browsing, and Zeek or Suricata for network analysis and IDS.
    • Search & Analytics: Relies on OpenSearch (or Elasticsearch) for indexing session metadata, with OpenSearch Dashboards (or Kibana) for visualization. It supports anomaly detection via Random Cut Forests and machine learning via the ML Commons plugin.
    • Log Ingestion: Uses Logstash and Filebeat to ingest and parse Zeek logs into OpenSearch.
    • File Scanning & Threat Hunting: Employs Strelka for real-time file scanning, utilizing YARA, Capa, and ClamAV to identify malware and capabilities.
    • Deployment & Orchestration: Supports local deployment via Docker or Podman, and distributed/cloud deployment via Kubernetes (including the Malcolm Helm chart).

  6. OpenSearch Dashboards query languages: DQL, Lucene, SQL, and PPL

    main

    OpenSearch Dashboards supports several query languages for searching Zeek logs and Suricata alerts:

    1. Dashboards Query Language (DQL): The default syntax for OpenSearch Dashboards. It is designed for ease of use and is the primary way to interact with the search bar.
    2. Apache Lucene: A legacy syntax used for query_string queries. It is powerful and supports wildcards and searching across multiple fields.
    3. SQL and PPL: For more complex, on-demand analysis, OpenSearch provides Structured Query Language (SQL) and Piped Processing Language (PPL). These can be run via the Query Workbench tool in OpenSearch Dashboards.

  7. Understand the Malcolm hardening implementation

    main

    The Malcolm aggregator base operating system incorporates hardening via the harbian-audit scripts, which are designed for Debian GNU/Linux 9 through 13. The implementation is built upon the OVH-debian-cis framework and includes security checks derived from:

    • CIS recommendations
    • DISA STIG for Red Hat Enterprise Linux 7 (V2R5) and Ubuntu (V1R2), adapted for Debian
    • Additional checks contributed by the HardenedLinux community

  8. Supported Network Protocols in Malcolm

    main

    Malcolm utilizes Zeek and Arkime to analyze network traffic. The visibility into specific protocols depends on whether Arkime, Zeek, or both tools support them. The supported protocols range from standard Internet layer protocols to specialized Industrial Control Systems (ICS) protocols.

    Protocol Analysis Capabilities

    • Standard Protocols: Includes HTTP, DNS, DHCP, FTP, SSH, TLS/SSL, SMB, SMTP, SNMP, and more.
    • ICS/SCADA Protocols: Includes BACnet, DNP3, EtherNet/IP (CIP), Modbus, OPC UA Binary, PROFINET, and S7comm.
    • Tunneling & VPNs: Supports various tunnel protocols (GTP, GRE, Teredo, etc.), IPsec, OpenVPN, and WireGuard.

    For a complete list of supported protocols and their specific analyzer support in Arkime and Zeek, refer to the protocol matrix in the documentation.

  9. Current limitations of Malcolm on Kubernetes

    main

    Deploying Malcolm with Kubernetes is currently an experimental feature and does not yet support the full range of Malcolm capabilities. Key limitations include:

    • Live Traffic Analysis: Network traffic artifacts cannot be captured directly in the cloud. Currently, traffic must be provided via forwarding from a remote Hedgehog Linux instance or via PCAP upload.
    • Horizontal Scaling: Malcolm services in Kubernetes are currently configured with replicas: 1. Horizontal scaling is not yet fully supported or tested for all containers.
    • Deployment Method: Deployment is managed via standard Kubernetes manifests located in the repository's kubernetes/ directory. A Helm chart is not currently available.

  10. Understand Malcolm installer ISO wrapper images

    main

    The Malcolm installer ISOs (for the Malcolm and Hedgehog profiles) are ISO 9660-formatted files. To facilitate distribution and testing, they are wrapped in container images (e.g., ghcr.io/username/malcolm/malcolm:main).

    These wrapper images serve two purposes:

    1. Providing an HTTP server to download the ISO.
    2. Booting the live ISO image in QEMU (using tianon/qemu:native).

    Important Note on Workflow Execution: The Malcolm installer ISO workflow includes a snapshot of all other Malcolm container images within its filesystem. Therefore, the installer ISO workflow should not be executed until all other container images have been built and pushed at least once, as the ISO will contain the images from the fork's most recent successful workflow runs.

  11. Use and extend Logstash enrichments

    main

    Malcolm automatically performs several enrichments in the logstash/pipelines/enrichment pipeline, such as GeoIP, ASN, and MAC address to vendor lookups via OUI.

    To leverage these existing enrichments for your own data, normalize your new fields to use the same standardized field names Malcolm uses for IP addresses, MAC addresses, etc.

    To add new enrichment logic, create new .conf files containing Logstash filters within the logstash/pipelines/enrichment directory.

  12. Understand Malcolm Analytics and filtering

    main

    Malcolm Analytics provides visualizations and dashboards designed to highlight specific aspects of a network. All analytics components are subject to global OpenSearch filters. When interacting with any visualization or dashboard, your view will be constrained by:

    • The Time Range filter
    • The search query bar