Amazon CloudFormation – AWS::CloudFormation::Init Not Executing Commands

amazon-cloudformationamazon-web-services

I'm trying to get ansible installed on an instance. I figured I could use AWS::CloudFormation::Init to execute sudo pip install ansible. That doesn't seem to be working, though. This is my instance resource:

ansibleInstance:
  Type: 'AWS::EC2::Instance'
  Metadata:
    'AWS::CloudFormation::Init':
      commands:
        ansible:
          command: "sudo pip install ansible"
          test: "pip --version"
          ignoreErrors: 'false'
  Properties:
    ImageId: ami-467ca739
    KeyName: Candidate-EyMm7zuOcn
    InstanceType: t2.micro
    SubnetId: !Ref subnetTest
    SecurityGroupIds:
    - !Ref allowSSH
    Tags:
      - Key: Name
        Value: Test

Is there a way to see what is happening when this is attempted so I can figure out where/why it is failing? Is there anything others might suggest looking at to figure this out?

EDIT1: I removed the sudo just in case that might have been getting in the way for whatever reason (I didn't think it would, but I wanted to eliminate it all the same). That had no impact. I also verified that the aws-cli tools are installed which was expected since it is the AWS Linux AMI

EDIT2: This is a version of the ansibleInstance resource in which I tried to execute commands via UserData:

ansibleInstance:
  Type: 'AWS::EC2::Instance'
  Properties:
    ImageId: ami-467ca739
    KeyName: Candidate-EyMm7zuOcn
    InstanceType: t2.micro
    SubnetId: !Ref subnetTest
    SecurityGroupIds:
    - !Ref allowSSH
    Tags:
      - Key: Name
        Value: Test
  UserData:
    Fn::Base64:
      !Sub |
        #!/bin/bash -xe
        pip install ansible

I suspect there is something wrong with the formatting, but I can't make out what it is.

EDIT3: I ran cfn-init per the suggestion jordanm. This does appear to provide the metadata to the instance and, if I then log in and manually execute cfn-init the commands are processed:

[ec2-user@ip-192-168-1-121 ~]$ sudo /opt/aws/bin/cfn-init -v -s cfTest --resource ansibleInstance
[ec2-user@ip-192-168-1-121 ~]$ ansible --version
ansible 2.5.4
  config file = None
  configured module search path = [u'/home/ec2-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.13 (default, Jan 31 2018, 00:17:36) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

So now my question is: why won't cfn-init execute per the UserData?

SOLUTION: Thanks to @jordanm I was able to get though this. My working snippet:

  ansibleInstance:
    Type: 'AWS::EC2::Instance'
    Metadata:
      'AWS::CloudFormation::Init':
        config:
          commands:
            ansible:
              command: "sudo pip install ansible"
              test: "pip --version"
              ignoreErrors: 'false'
    Properties:
      ImageId: ami-467ca739
      KeyName: Candidate-EyMm7zuOcn
      InstanceType: t2.micro
      SubnetId: !Ref subnetTest
      SecurityGroupIds:
      - !Ref allowSSH
      Tags:
        - Key: Name
          Value: Test
      UserData: 
        Fn::Base64: 
          !Sub |
            #!/bin/bash -xe
            # Install Ansible from the metadata
            /opt/aws/bin/cfn-init -v -s ${AWS::StackName} --resource ansibleInstance

Best Answer

You have to execute cfn-init in order for AWS::CloudFormation::Init to do anything. This is most commonly done via cloud-init user data. Here is the example from the cfn-init documentation:

UserData: !Base64 
  'Fn::Join':
    - ''
    - - |
        #!/bin/bash -xe
      - |
        # Install the files and packages from the metadata
      - '/opt/aws/bin/cfn-init -v '
      - '         --stack '
      - !Ref 'AWS::StackName'
      - '         --resource WebServerInstance '
      - '         --configsets InstallAndRun '
      - '         --region '
      - !Ref 'AWS::Region'
      - |+

Related Solutions

AWS CloudFormation: VPC default security group

Referencing the default security group is possible using:

{ "Fn::GetAtt" : ["VPC", "DefaultSecurityGroup"] }

Where "VPC" is your VPC resource name.

With AWS::EC2::SecurityGroupIngress and AWS::EC2::SecurityGroupEgress, you can augment the permissions of this default security group.

I think this is what you want:

"VPCDefaultSecurityGroupIngress": {
  "Type" : "AWS::EC2::SecurityGroupIngress",
  "Properties" : {
    "GroupId": { "Fn::GetAtt" : ["VPC", "DefaultSecurityGroup"] },
    "IpProtocol":"tcp",
    "FromPort":"22",
    "ToPort":"22",
    "CidrIp":"0.0.0.0/0"
  }
},

As mentioned by @artbristol and @gabriel, this allows Ingress/Egress rules to be added to the default security group for the VPC in a single stack deployment.

I'm pretty sure that the self-referential problem still impacts any attempts at changing any of the other properties on the default security group of the VPC. A good example of this would be adding Tags, or a Description. If you wish to change these things, you'll have to deal with extraneous security groups laying around.

AWS CloudFormation – Custom variables in templates

I was looking of the same functionality. Using a nested stack as SpoonMeiser suggested came to mind, but then I realised that what I actually needed was custom functions. Luckily CloudFormation allows the use of AWS::CloudFormation::CustomResource that, with a bit of work, allows one to do just that. This feels like overkill for just variables (something I would argue that should have been in CloudFormation in the first place), but it gets the job done, and, in addition, allows for all the flexibility of (take your pick of python/node/java). It should be noted that lambda functions cost money, but we're talking pennies here unless you create/delete your stacks multiple times per hour.

First step is to make a lambda function on this page that does nothing but take the input value and copy it to the output. We could have the lambda function do all sorts of crazy stuff, but once we have the identity function, anything else is easy. Alternatively we could have the lambda function being created in the stack itself. Since I use many stacks in 1 account, I would have a whole bunch of leftover lambda functions and roles (and all stacks need to be created with --capabilities=CAPABILITY_IAM, since it also needs a role.

Create lambda function

Go to lambda home page, and select your favourite region
Select "Blank Function" as template
Click "Next" (don't configure any triggers)
Fill in:
- Name: CloudFormationIdentity
- Description: Returns what it gets, variable support in Cloud Formation
- Runtime: python2.7
- Code Entry Type: Edit Code Inline
- Code: see below
- Handler: index.handler
- Role: Create a Custom Role. At this point a popup opens that allows you to create a new role. Accept everything on this page and click "Allow". It will create a role with permissions to post to cloudwatch logs.
- Memory: 128 (this is the minimum)
- Timeout: 3 seconds (should be plenty)
- VPC: No VPC

Then copy-paste the code below in the code field. The top of the function is the code from the cfn-response python module, that does only get auto-installed if the lambda-function is created through CloudFormation, for some strange reason. The handler function is pretty self-explanatory.

from __future__ import print_function
import json

try:
    from urllib2 import HTTPError, build_opener, HTTPHandler, Request
except ImportError:
    from urllib.error import HTTPError
    from urllib.request import build_opener, HTTPHandler, Request


SUCCESS = "SUCCESS"
FAILED = "FAILED"


def send(event, context, response_status, reason=None, response_data=None, physical_resource_id=None):
    response_data = response_data or {}
    response_body = json.dumps(
        {
            'Status': response_status,
            'Reason': reason or "See the details in CloudWatch Log Stream: " + context.log_stream_name,
            'PhysicalResourceId': physical_resource_id or context.log_stream_name,
            'StackId': event['StackId'],
            'RequestId': event['RequestId'],
            'LogicalResourceId': event['LogicalResourceId'],
            'Data': response_data
        }
    )
    if event["ResponseURL"] == "http://pre-signed-S3-url-for-response":
        print("Would send back the following values to Cloud Formation:")
        print(response_data)
        return

    opener = build_opener(HTTPHandler)
    request = Request(event['ResponseURL'], data=response_body)
    request.add_header('Content-Type', '')
    request.add_header('Content-Length', len(response_body))
    request.get_method = lambda: 'PUT'
    try:
        response = opener.open(request)
        print("Status code: {}".format(response.getcode()))
        print("Status message: {}".format(response.msg))
        return True
    except HTTPError as exc:
        print("Failed executing HTTP request: {}".format(exc.code))
        return False

def handler(event, context):
    responseData = event['ResourceProperties']
    send(event, context, SUCCESS, None, responseData, "CustomResourcePhysicalID")

Click "Next"
Click "Create Function"

You can now test the lambda function by selecting the "Test" button, and select "CloudFormation Create Request" as sample template. You should see in your log that the variables fed to it, are returned.

Use variable in your CloudFormation template

Now that we have this lambda function, we can use it in CloudFormation templates. First make note of the lambda function Arn (go to the lambda home page, click the just created function, the Arn should be in the top right, something like arn:aws:lambda:region:12345:function:CloudFormationIdentity).

Now in your template, in the resource section, specify your variables like:

Identity:
  Type: "Custom::Variable"
  Properties:
    ServiceToken: "arn:aws:lambda:region:12345:function:CloudFormationIdentity"
    Arn: "arn:aws:lambda:region:12345:function:CloudFormationIdentity"

ClientBucketVar:
  Type: "Custom::Variable"
  Properties:
    ServiceToken: !GetAtt [Identity, Arn]
    Name: !Join ["-", [my-client-bucket, !Ref ClientName]]
    Arn: !Join [":", [arn, aws, s3, "", "", !Join ["-", [my-client-bucket, !Ref ClientName]]]]

ClientBackupBucketVar:
  Type: "Custom::Variable"
  Properties:
    ServiceToken: !GetAtt [Identity, Arn]
    Name: !Join ["-", [my-client-bucket, !Ref ClientName, backup]]
    Arn: !Join [":", [arn, aws, s3, "", "", !Join ["-", [my-client-bucket, !Ref ClientName, backup]]]]

First I specify an Identity variable that contains the Arn for the lambda function. Putting this in a variable here, means I only have to specify it once. I make all my variables of type Custom::Variable. CloudFormation allows you to use any type-name starting with Custom:: for custom resources.

Note that the Identity variable contains the Arn for the lambda function twice. Once to specify the lambda function to use. The second time as the value of the variable.

Now that I have the Identity variable, I can define new variables using ServiceToken: !GetAtt [Identity, Arn] (I think JSON code should be something like "ServiceToken": {"Fn::GetAtt": ["Identity", "Arn"]}). I create 2 new variables, each with 2 fields: Name and Arn. In the rest of my template I can use !GetAtt [ClientBucketVar, Name] or !GetAtt [ClientBucketVar, Arn] whenever I need it.

Word of caution

When working with custom resources, if the lambda function crashes, you're stuck for between 1 and 2 hours, because CloudFormation waits for a reply from the (crashed) function for an hour before giving up. Therefore it might be good to specify a short timeout for the stack while developing your lambda function.