AWS ClickOps Notifier Terraform Module

When setting up the aws_kinesis_firehose_delivery_stream to work with the ClickOps Notifier, use dynamic_partitioning_configuration and processing_configuration to organize logs in S3 using Hive-style paths (e.g., recipientAccountId=!{partitionKey...}).

To achieve this, you must:

1. Enable dynamic_partitioning_configuration.
2. Use a MetadataExtraction processor with a MetadataExtractionQuery (using JQ) to extract fields like recipientAccountId and awsRegion from the JSON log.
3. Use a RecordDeAggregation processor if the input records are aggregated.
4. Use an AppendDelimiterToRecord processor to ensure valid record separation in S3.
5. Define a prefix using Firehose interpolation variables to build the folder structure.

resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
  name        = local.naming_prefix
  destination = "extended_s3"

  extended_s3_configuration {
    # ... role and bucket config ...

    dynamic_partitioning_configuration {
      enabled = "true"
    }

    prefix = join("/", [
      "data",
      "recipientAccountId=!{partitionKeyFromQuery:recipientAccountId}",
      "awsRegion=!{partitionKeyFromQuery:awsRegion}",
      "year=!{timestamp:yyyy}", "month=!{timestamp:MM}", "day=!{timestamp:dd}", "hour=!{timestamp:HH}",
      ""
    ])

    processing_configuration {
      enabled = "true"

      processors {
        type = "RecordDeAggregation"
        parameters {
          parameter_name  = "SubRecordType"
          parameter_value = "JSON"
        }
      }

      processors {
        type = "MetadataExtraction"
        parameters {
          parameter_name  = "JsonParsingEngine"
          parameter_value = "JQ-1.6"
        }
        parameters {
          parameter_name  = "MetadataExtractionQuery"
          parameter_value = "{recipientAccountId:.recipientAccountId, awsRegion:.awsRegion}"
        }
      }

      processors {
        type = "AppendDelimiterToRecord"
      }
    }
  }
}

To use the ClickOps Notifier as a standalone module (rather than in organizational mode), you must set the standalone input to true. In this mode, you are responsible for providing the CloudTrail log group name and configuring your webhook endpoints for Slack and MS Teams.

Key requirements for standalone mode:

• Set standalone = true.
• Provide cloudtrail_log_group (the name of the CloudWatch Log Group used by your CloudTrail).
• Define webhooks_for_slack_notifications and webhooks_for_msteams_notifications as maps where the key is a notification identifier and the value is the webhook URL.
• Ensure the module depends_on the CloudTrail infrastructure to ensure logs are available.

module "clickops_notifications" {
  source = "../.."

  standalone = true

  naming_prefix = "my-prefix"

  webhooks_for_slack_notifications = {
    my-first-notification : "https://fake.com"
  }
  webhooks_for_msteams_notifications = {
    my-second-notification : "https://fake.com"
  }

  tags = { usage = "testing" }

  cloudtrail_log_group = "my-cloudtrail-log-group"

  depends_on = [
    module.aws_cloudtrail
  ]
}

To encrypt the SNS topic created by the module, you must provide a kms_key_id_for_sns_topic. This requires creating an aws_kms_key and an aws_iam_policy_document that grants the necessary permissions to the S3 bucket (to generate/decrypt data keys) and the root account (to manage the key).

# To encrypt the SNS topic
data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "clickops_sns_topic" {
  statement {
    sid = "Enable IAM User Permissions"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
    actions   = ["kms:*"]
    resources = ["*"]
  }
  statement {
    principals {
      type        = "Service"
      identifiers = ["s3.amazonaws.com"]
    }
    actions = [
      "kms:GenerateDataKey*",
      "kms:Decrypt"
    ]
    resources = ["*"]
    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_s3_bucket.test_bucket.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
  }
}

resource "aws_kms_key" "clickops_sns_topic" {
  description             = "KMS key for SNS topic ${local.naming_prefix}"
  deletion_window_in_days = 7
  policy                  = data.aws_iam_policy_document.clickops_sns_topic.json

  tags = local.tags
}

This example demonstrates how to implement a basic setup of the clickops_notifications module. It includes configuring Slack and MS Teams webhooks, providing a CloudTrail S3 bucket name, and optionally encrypting the SNS topic using a custom KMS key.

Key configuration steps:

• Provide a run_id to ensure resource uniqueness.
• Set cloudtrail_bucket_name to the S3 bucket where CloudTrail logs are stored.
• Define webhooks_for_slack_notifications and webhooks_for_msteams_notifications as maps of notification names to webhook URLs.
• (Optional) Provide kms_key_id_for_sns_topic to encrypt the SNS topic.

module "clickops_notifications" {
  source = "../../"

  naming_prefix          = local.naming_prefix
  cloudtrail_bucket_name = aws_s3_bucket.test_bucket.id

  webhooks_for_slack_notifications = {
    my-first-notification : "https://fake.com"
  }
  webhooks_for_msteams_notifications = {
    my-second-notification : "https://fake.com"
  }

  tags = local.tags

  # Optional
  kms_key_id_for_sns_topic = aws_kms_key.clickops_sns_topic.arn
}

To use the clickops-notifier module in a standalone configuration (rather than within an organizational structure), set the standalone = true argument. You must provide a naming_prefix and link the module to your existing CloudTrail and Kinesis Firehose infrastructure.

Key configuration requirements:

• standalone = true: Enables standalone mode.
• naming_prefix: A string used to prefix resources.
• webhooks_for_slack_notifications: A map of notification names to Slack webhook URLs.
• webhooks_for_msteams_notifications: A map of notification names to MS Teams webhook URLs.
• cloudtrail_log_group: The name of the CloudWatch Log Group associated with your CloudTrail.
• firehose_delivery_stream_name: The name of the Kinesis Firehose delivery stream used for processing logs.

module "clickops_notifications" {
  source = "../.."

  standalone = true

  naming_prefix = local.naming_prefix

  webhooks_for_slack_notifications = {
    my-first-notification : "https://fake.com"
  }
  webhooks_for_msteams_notifications = {
    my-second-notification : "https://fake.com"
  }

  tags = local.tags

  cloudtrail_log_group = local.naming_prefix_cloudtrail

  firehose_delivery_stream_name = aws_kinesis_firehose_delivery_stream.extended_s3_stream.name

  depends_on = [
    module.aws_cloudtrail
  ]
}