tail -f /dev/null

If you haven't had any obstacles lately, you're not challenging. be the worst.

AWS Configは通知ノイズが大きい

AWS Configを通して何が実現出来るのか

  • AWS Resourceの変更通知。
  • AWS Resourceの変更履歴の検索。
  • AWS Resourceの構成情報や設定変更ログのSnapshotをS3に保存。
  • Multi accounts, Multi regionsでAWS resourceの変更を追跡、通知可能。
  • Managed Rule, Custom RuleによるAWS Resourceの評価。

    • Inboundが全開放になっているSGはないか
    • 特定のタグがついているEC2 Instanceがどれくらいあるのか
  • SupportされているResource

Multi Accounts有効化の流れ

Aggregator account = Managed account

  • Aggregator accountを作成する。
    • 手動でMulti Accountsにするか、Organizationの場合は自動登録も可能。
  • Source accountのregionか、All regionsかを選択する。
  • Source accountがAggregatorのアクセスを承認する。
    • Organizationでは、このflowは無い。
  • 承認されていない場合は、Aggregator accountは以下のエラーを経験する。
AWS Config does not have permission from the source account to replicate data into an aggregator account. Authorize aggregator account to replicate data from source accounts and region.

Lambda実行の権限周りはAWS公式Blogを参照。

AWS resource変更通知

変更通知を受け取る方法は色々と考えられる。既存の環境や運用に合ったものを選択する。

  • AWS Config > S3(Snapshot) > Lambda > Slack (Mail)
  • AWS Config > CloudTrail > Lambda > Slack (Mail)
  • AWS Config > SNS > Lambda > Slack (Mail)
  • AWS Config > CloudWatch Events > Lambda > Slack (Mail)

Resource変更Logの検索方法

AWS Resourceの変更を追う方法も色々と考えられる、目的や手段に合ったものを選択する。

production以外でノイズとなる通知を抑止する

例えばインスタンスの起動、停止を行うと以下のように複数のResourceに変更が走る為、通知が複数届きノイズとなってしまう。 CloudWatch EventsのEvent patternでフィルタをかけるか、Lambda等で不必要な resourceType は除外する等が好ましい。

{
  "changedProperties": {
    "Configuration.State.Name": {
      "previousValue": "running",
      "updatedValue": "stopping",
      "changeType": "UPDATE"
    },
    "Configuration.StateTransitionReason": {
      "previousValue": "",
      "updatedValue": "User initiated ()",
      "changeType": "UPDATE"
    },
    "Configuration.StateReason": {
      "previousValue": null,
      "updatedValue": {
        "code": "Client.UserInitiatedShutdown",
        "message": "Client.UserInitiatedShutdown: User initiated shutdown"
      },
      "changeType": "CREATE"
    }
  },
  "changeType": "UPDATE"
}

インスタンスの起動、停止に伴うRoute Tableのstate変更 (AWS::EC2::RouteTable rtb-xxx)

   "Configuration.Routes.7": {
     "previousValue": null,
     "updatedValue": {
       "destinationCidrBlock": "",
       "destinationIpv6CidrBlock": null,
       "destinationPrefixListId": null,
       "egressOnlyInternetGatewayId": null,
       "gatewayId": null,
       "instanceId": "",
       "instanceOwnerId": "",
       "natGatewayId": null,
       "networkInterfaceId": "",
       "origin": "CreateRoute",
       "state": "blackhole",
       "vpcPeeringConnectionId": null
     },
     "changeType": "CREATE"
   },

   "Configuration.Routes.3": {
     "previousValue": {
       "destinationCidrBlock": "",
       "destinationIpv6CidrBlock": null,
       "destinationPrefixListId": null,
       "egressOnlyInternetGatewayId": null,
       "gatewayId": null,
       "instanceId": "",
       "instanceOwnerId": "",
       "natGatewayId": null,
       "networkInterfaceId": "",
       "origin": "CreateRoute",
       "state": "active",
       "vpcPeeringConnectionId": null
     },
     "updatedValue": null,
     "changeType": "DELETE"
   },

インスタンスの起動、停止に伴うNetworkInterfaceのrelation変更 (AWS::EC2::VPC vpc-xxx)

{
  "changedProperties": {
    "Relationships.0": {
      "previousValue": {
        "resourceId": "",
        "resourceName": null,
        "resourceType": "AWS::EC2::NetworkInterface",
        "name": "Contains NetworkInterface"
      },
      "updatedValue": null,
      "changeType": "DELETE"
    }
  },
  "changeType": "UPDATE"
}

CloudWatch Eventsでの通知除外方法

Event Pattern

設定変更に関して、特定の通知を受け取る場合以下を選択する。
参考: Monitoring AWS Config with Amazon CloudWatch Events - AWS Config

  • Events Type
    • Config Configuration Item Change
  • Message type
    • ConfigurationItemChangeNotification
      • 特定のResourceの設定が変更された場合
    • ComplianceChangeNotification
      • Rule評価するResource typeが変更された場合
    • ConfigRulesEvaluationStarted
      • 指定のResourceに対しAWS ConfigがRuleの評価を開始した場合
    • ConfigurationSnapshotDeliveryCompleted
      • Config設定のSnapshotをS3 bucketに正常に送信した場合
    • ConfigurationSnapshotDeliveryFailed
      • Config設定のSnapshotをS3 bucketに正常に送信出来なかった場合
    • ConfigurationSnapshotDeliveryStarted
      • Config設定のSnapshotをS3 bucketに配信開始した場合
    • ConfigurationHistoryDeliveryCompleted
      • AWS Configの設定履歴をS3 bucketに送信した場合

Targets Configure Input

Targets Configure inputでLambdaへ渡す際にカスタマイズしてもよいかもしれない。

Terraform

cloud_watch.tf

resource "aws_cloudwatch_event_rule" "aws_config" {
  name        = "aws-config"
  description = "AWS Config notification"
  is_enabled  = true

  event_pattern = <<PATTERN
{
  "source": [
    "aws.config"
  ],
  "detail-type": [
    "Config Configuration Item Change"
  ],
  "detail": {
    "messageType": [
      "ConfigurationItemChangeNotification"
    ],
    "configurationItem": {
      "resourceType": [
        "AWS::EC2::CustomerGateway",
        "AWS::ElasticBeanstalk::Application",
        "AWS::ElasticBeanstalk::ApplicationVersion",
        "AWS::ElasticBeanstalk::Environment",
        "AWS::ElasticLoadBalancing::LoadBalancer",
        "AWS::XRay::EncryptionConfig"
      ]
    }
  }
}
PATTERN
}

# cloudwatch
resource "aws_cloudwatch_log_group" "aws_config" {
  name = "/aws/lambda/aws-config"
}

resource "aws_cloudwatch_event_target" "lambda" {
  rule      = "${aws_cloudwatch_event_rule.aws_config.name}"
  target_id = "aws-config"
  arn       = "${var["lambda_endpoint"]}"
}

aws_config.tf

resource "aws_config_delivery_channel" "aws_config" {
  name           = "aws-config"
  s3_bucket_name = "${var["s3_logs_bucket"]}"
  s3_key_prefix  = "${var["s3_logs_bucket_prefix"]}"
  sns_topic_arn  = "${aws_sns_topic.aws_config.arn}"

sns_topic.tf

resource "aws_sns_topic" "aws_config" {
  name         = "aws-config"
  display_name = "aws-config"
}

resource "aws_sns_topic_policy" "aws_config" {
  arn    = "${aws_sns_topic.aws_config.arn}"
  policy = "${data.aws_iam_policy_document.sns_topic_policy.json}"
}

resource "aws_sns_topic_subscription" "aws_config_lambda" {
  protocol                        = "lambda"
  topic_arn                       = "${aws_sns_topic.aws_config.arn}"
  endpoint                        = "${var["lambda_endpoint"]}"
  confirmation_timeout_in_minutes = "1"
  endpoint_auto_confirms          = "false"
  raw_message_delivery            = "false"
}

lambda.tf

data "archive_file" "sample_function" {
  type              = "zip"
  source_dir    = "lambda/sample_function"
  output_path = "lambda/upload/sample_function.zip"
}

resource "aws_lambda_function" "sample_function" {
  filename                 = "${data.archive_file.sample_function.output_path}"
  function_name       = "sample_function"
  role                         = "${aws_iam_role.lambda_sample_function.arn}"
  handler                   = "lambda_function.lambda_handler"
  source_code_hash = "${data.archive_file.sample_function.output_base64sha256}"
  runtime                   = "python3.6"

  memory_size = 128
  timeout     = 30
}

S3 Bucket policy

data 読み込み用。

参考: Permissions for the Amazon S3 Bucket - AWS Config

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSConfigBucketPermissionsCheck",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "config.amazonaws.com"
                ]
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::{bucket_name}"
        },
        {
            "Sid": " AWSConfigBucketDelivery",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "config.amazonaws.com"
                ]
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::{bucket_name}/aws-config/AWSLogs/{account_id}/Config/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        },

SNS topic policy

data 読み込み用。

{
    "Id": "",
    "Statement": [
        {
            "Sid": "AWSConfigSNSPolicy20150201",
            "Action": [
                "SNS:Publish"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sns:region:account-id:myTopic",
            "Principal": {
                "AWS": [
                    "account-id1",
                    "account-id2",
                    "account-id3",
                ]
            }
        }
    ]
}

Lambda

CloudWatch Eventから渡ってくるJSONは例えば以下のような形式。

{
    "version": "0",
    "detail-type": "Config Configuration Item Change",
    "source": "aws.config",
    "account": "",
    "time": "",
    "region": "ap-northeast-1",
    "resources": [
        "arn:aws:lambda:ap-northeast-1::"
    ],
    "detail": {
        "recordVersion": "1.3",
        "messageType": "ConfigurationItemChangeNotification",
        "configurationItemDiff": {
            "changedProperties": {
                "Configuration.codeSha256": {
                    "previousValue": "",
                    "updatedValue": "",
                    "changeType": "UPDATE"
                },
                "Configuration.lastModified": {
                    "previousValue": "",
                    "updatedValue": "",
                    "changeType": "UPDATE"
                },
                "Configuration.revisionId": {
                    "previousValue": "",
                    "updatedValue": "",
                    "changeType": "UPDATE"
                },
                "Configuration.codeSize": {
                    "previousValue":,
                    "updatedValue":,
                    "changeType": "UPDATE"
                }
            },
            "changeType": "UPDATE"
        },
        "notificationCreationTime": "2019-03-05T12: 58: 49.180Z",
        "configurationItem": {
            "relatedEvents": [],
            "relationships": [
                {
                    "resourceName": "",
                    "resourceType": "AWS: :IAM: :Role",
                    "name": "Is associated with "
                },
                {
                    "resourceId": "",
                    "resourceType": "AWS: :EC2: :SecurityGroup",
                    "name": "Is associated with "
                },
                {
                    "resourceId": "",
                    "resourceType": "AWS: :EC2: :Subnet",
                    "name": "Is contained in "
                },
                {
                    "resourceId": "",
                    "resourceType": "AWS: :EC2: :Subnet",
                    "name": "Is contained in "
                }
            ],
            "configuration": {
                "functionName": "",
                "functionArn": "arn:aws:lambda:ap-northeast-1:",
                "runtime": "python3.6",
                "role": "arn:aws:iam:",
                "handler": "lambda_function.lambda_handler",
                "codeSize":,
                "description": "uploaded by lambda-uploader",
                "timeout": 300.0,
                "memorySize": 128.0,
                "lastModified": "",
                "codeSha256": "",
                "version": "LATEST",
                "vpcConfig": {
                    "subnetIds": [
                        "",
                        ""
                    ],
                    "securityGroupIds": [
                        ""
                    ]
                },
                "tracingConfig": {
                    "mode": "PassThrough"
                },
                "revisionId": "",
                "layers": []
            },
            "supplementaryConfiguration": {
                "Tags": {}
            },
            "tags": {},
            "configurationItemVersion": "1.3",
            "configurationItemCaptureTime": "",
            "configurationStateId":,
            "awsAccountId": "",
            "configurationItemStatus": "OK",
            "resourceType": "AWS: :Lambda: :Function",
            "resourceId": "",
            "resourceName": "",
            "ARN": "arn:aws:lambda:ap-northeast-1:",
            "awsRegion": "ap-northeast-1",
            "availabilityZone": "Not Applicable",
            "configurationStateMd5Hash": ""
        }
    }
}

resourceTypeconfigurationItemDiff 等、必要なもののみ抽出する。

#!/usr/bin/env python

...{略}...

    resource_type = message['configurationItem']['resourceType']
    notification_created_at = message['notificationCreationTime']
    resource_id = message['configurationItem']['resourceId']
    changed_trigger = message['configurationItemDiff']['changeType']
    configuration_item_diff = message['configurationItemDiff']

    slack_message = {
        'text': "resouces changed at %s.\n change type: %s \ndiff: %s %s\ndetails: \n```\n%s\n```\n" %
        (notification_created_at, changed_trigger, resource_type, resource_id,
         json.dumps(configuration_item_diff,  indent=2))
    }

余談

Slackで通知する際、通知のJSONサイズが大きい場合がありSnippetでbot投稿したくなるが、ファイルと同様ストレージ容量を圧迫していく為おすすめしない。