How to Auto-Scale EC2 Instances with Encrypted Root Volume

amazon ec2amazon-kmsamazon-web-servicesautoscaling

I am trying to configure an autoscaling setup in AWS where the Node Launch Template includes encrypting the root volume (EBS). I have configured a service linked role, and a CMK in Amazon KMS with an IAM policy as per the documentation.

However, I am getting the following error when the ASG looks to create the instances:

Launching a new EC2 instance: i-0123456789xxx. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch

The troubleshooting documentation simply points at the original documentation and suggests the IAM policy isn't configured properly – I'm struggling to work out what's not right though.

The service linked role is configured on the ASG: SLR on ASG, and the SLR has the correct permissions in the IAM policy for the key that was used to encrypt the volume:

{
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms: Encrypt",
            "kms: Decrypt",
            "kms: ReEncrypt*",
            "kms: GenerateDataKey*",
            "kms: DescribeKey"
        ],
        "Resource": "*"
    },
    {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms: CreateGrant",
            "kms: ListGrants",
            "kms: RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    }

Note that launching the same AMI manually, specifying the root volume is encrypted with the same key, works. This suggests a problem with the SLR perhaps?

Or, do I need to create an AMI where the root volume is already encrypted?

UPDATE 11/05/2020:

Turns out there was a formatting error – there is a space after each colon in the Actions section. Removing that has fixed it and it works as expected now.

{
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
        ],
        "Resource": "*"
    },
    {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    }

Best Answer

I had the same problem and resolved it by adding the Service-Linked Role for Auto Scaling to the Key policy of the pertinent key (AWS Console -> KMS -> Customer managed keys -> YOUR_KEY -> 'edit' under the Key policy tab) as follows:

{
    "Version": "2012-10-17",
    "Id": "key-default-1",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::READCTED:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
                    "arn:aws:iam::REDACTED:root"
                ]
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}