DOCSEARCHbyAlgolia
Submit Docs

CloudMapper Documentation

repository·main·Indexed 27 days ago

https://github.com/duo-labs/cloudmapper

A tool for analyzing Amazon Web Services (AWS) environments to audit security misconfigurations and generate resource reports. It includes a CLI for metadata collection, security auditing, and network visualization, as well as a CloudMapper Auditor for automated nightly scans across multiple accounts with Slack notifications and S3 metadata storage.

Tokens
8.1K
Snippets
14
Records
50
Agent score
91%

What's inside CloudMapper

  1. Use the Network Visualization UI

    main

    The network visualization UI allows you to interact with your VPC resources (EC2, RDS, ELB, Redshift, etc.). Note that edges are based on Security Group ingress rules, not Resource Policies, NACLs, or Route Tables.

    Navigation & Selection

    • Pan/Zoom: Use UI controls, arrow keys, or +/- keys.
    • Select Node: Click a node (turns yellow). Double-click a node to make its deleted neighbors visible again.
    • Unselect: Click a new node or hold Shift and click the selected node.
    • Multi-select: Hold Shift and click, or hold Shift and drag over an area.
    • Move Node: Click and drag a node.

    UI Commands

    • Delete (d): Select a node and click the 'eye with a slash' icon to hide it. Click the 'eye' icon to unhide all deleted nodes. Nodes connected to a deleted node will have a black border.
    • Highlight (h): Select a node and click the connected nodes symbol to highlight neighbors. Use the inverse symbol to unhighlight.
    • Collapse All: Use the 'arrows pointed toward each other' icon to collapse all nodes, or the 'arrows pointed away' icon to uncollapse.
    • Collapse/Expand (c/e): Use the 'minus' symbol to collapse a node and the 'plus' symbol to expand it.
    • Randomize Layout (r): Click the 'hammer' symbol to re-run the layout algorithm.
    • Save Image: Click the 'camera' symbol to save a high-resolution image of the current diagram.
    • Import/Export: Save the current layout as a JSON file to preserve node positions for later use.

  2. Prepare data for network visualizations

    main

    Convert collected AWS metadata into a web/data.json file suitable for browser visualization using the prepare command. You can apply various filters to reduce complexity and improve rendering performance for large environments.

    Filtering Options

    Resource & Location Filtering:

    • --regions: Restrict the diagram to specific regions (e.g., --regions us-east-1,us-east-2).
    • --vpc-ids: Restrict the diagram to specific VPC IDs.
    • --vpc-names: Restrict the diagram to specific VPC names.
    • --tags: Filter by tags. Multiple tags in a single set are AND'd; multiple sets are OR'd (e.g., --tags Env=Prod --tags Env=Test,Name=Bastion).
    • --collapse-by-tag: Reduces all nodes sharing a specific tag into a single displayed node.

    Edge & Node Visibility:

    • --internal-edges (default) / --no-internal-edges: Use --no-internal-edges to show only publicly accessible paths.
    • --inter-rds-edges (default) / --no-inter-rds-edges: Use --inter-rds-edges to show communication paths between RDS nodes.
    • --read-replicas (default) / --no-read-replicas: Use --no-read-replicas to hide RDS read replica nodes.
    • --azs (default) / --no-azs: Use --no-azs to hide Availability Zones.
    • --no-collapse-asgs: Prevents Auto Scaling Groups from being collapsed, showing all individual instances instead.
    python cloudmapper.py prepare --account my_account

  3. Install CloudMapper on macOS

    main

    To install CloudMapper on macOS, you need Python 3, pip, virtualenv, jq, and the pyjq library. Follow these steps to clone the repository and set up a virtual environment with the required dependencies.

    # clone the repo
git clone https://github.com/duo-labs/cloudmapper.git
# Install pre-reqs for pyjq
brew install autoconf automake awscli freetype jq libtool python3
cd cloudmapper/
python3 -m venv ./venv && source venv/bin/activate
pip install --prefer-binary -r requirements.txt

  5. Set up the CloudMapper Auditor

    main

    The CloudMapper Auditor runs collection and audit capabilities nightly across multiple accounts, sends findings to Slack, and stores metadata in S3.

    To set it up:

    1. Clone the repository and the required CloudMapper resources:
      git clone https://github.com/duo-labs/cloudmapper.git
cd cloudmapper/auditor
git clone https://github.com/duo-labs/cloudmapper.git resources/cloudmapper
npm install
    2. Create an S3 bucket (e.g., MYCOMPANY-cloudmapper) to store configuration and metadata.
    3. Create an SNS topic for error alarms.
    4. Configure target accounts with SecurityAudit and ViewOnlyAccess privileges and IAM trust policies allowing the auditor account to assume them.
    5. Prepare configuration files in the s3_bucket_files directory and upload them to your S3 bucket:
      • config: The ~/.aws/config used for assuming roles (must be named CloudMapper and use credential_source = EcsContainer).
      • config.json: Specifies the accounts to scan.
      • audit_config_override.yaml: Used for muting audit findings.
      • run_cloudmapper.sh: Execution script (do not modify).
      • cdk_app.yaml: CDK configuration.
    6. Deploy the application using CDK:
      cdk deploy
      (Enter 'y' when prompted).
    git clone https://github.com/duo-labs/cloudmapper.git
cd cloudmapper/auditor
# Clone CloudMapper again into the auditor (weird, I know, but the only way to keep this all one repo)
git clone https://github.com/duo-labs/cloudmapper.git resources/cloudmapper
npm install

  6. Collect AWS account metadata

    main

    Use the collect command to gather metadata about an AWS account. This process uses describe and list calls and saves the resulting JSON in the account-data folder under the specified account name.

    Required AWS Privileges

    You must use AWS credentials with read permissions. Because CloudMapper collects IAM information, you MUST use MFA. Recommended policies include:

    • arn:aws:iam::aws:policy/SecurityAudit
    • arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
    python cloudmapper.py collect --account my_account

  7. Run CloudMapper via Docker

    main

    You can run CloudMapper in a Docker container. Note that for data collection, you cannot use aws-vault session credentials; you must pass role credentials directly or configure them manually inside the container.

    Warning: The following method exposes raw credentials inside the container.

    (                                                              
    export $(aws-vault exec YOUR_PROFILE --no-session -- env | grep ^AWS | xargs) && \ 
    docker run -ti \ 
        -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ 
        -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \ 
        -p 8000:8000 \ 
        cloudmapper /bin/bash
)

  8. Configure Slack notifications via Secrets Manager

    main

    To send audit findings to a Slack channel, create a secret in AWS Secrets Manager named cloudmapper-slack-webhook containing your Slack webhook URL in a JSON object with the key webhook.

    aws secretsmanager create-secret --name cloudmapper-slack-webhook --secret-string '{"webhook":"https://hooks.slack.com/services/XXX/YYY/ZZZ"}'

  9. Install CloudMapper on Linux

    main

    To install CloudMapper on Linux, ensure you have the necessary build tools and Python dependencies. For Debian/Ubuntu systems, you may need build-essential in addition to the listed packages.

    # clone the repo
git clone https://github.com/duo-labs/cloudmapper.git
# (Debian, Ubuntu etc.):
sudo apt-get install autoconf automake libtool python3.7-dev python3-tk jq awscli
cd cloudmapper/
python3 -m venv ./venv && source venv/bin/activate
pip install -r requirements.txt

  10. Configure AWS account for CloudMapper

    main
    Before collecting data, you must configure your config.json. Copy config.json.demo to config.json and edit it to include your AWS account ID, name (e.g., "prod"), and any external CIDR names (e.g., 1.2.3.4/32).

  11. Mute audit findings and manage accounts

    main

    To prevent Slack from being flooded with findings, you must manage issues before enabling automated nightly runs.

    • Muting findings: Modify audit_config_override.yaml in your S3 bucket. You can test these changes by downloading account-data from S3 and running the CloudMapper audit command locally.
    • Adding new accounts:
      1. Manually run CloudMapper's audit on the new account first.
      2. Fix or mute findings in that account.
      3. Add the account to config.json and config files in the S3 bucket.
      4. Set up the necessary IAM trust relationships in the target account.

  12. Override audit configurations

    main
    To ignore specific audit items or specific resources, create an override file. Copy config/audit_config_override.yaml.example to config/audit_config_override.yaml and edit it according to the comments provided in the file.