I was looking of the same functionality. Using a nested stack as SpoonMeiser suggested came to mind, but then I realised that what I actually needed was custom functions. Luckily CloudFormation allows the use of AWS::CloudFormation::CustomResource that, with a bit of work, allows one to do just that. This feels like overkill for just variables (something I would argue that should have been in CloudFormation in the first place), but it gets the job done, and, in addition, allows for all the flexibility of (take your pick of python/node/java). It should be noted that lambda functions cost money, but we're talking pennies here unless you create/delete your stacks multiple times per hour.
First step is to make a lambda function on this page that does nothing but take the input value and copy it to the output. We could have the lambda function do all sorts of crazy stuff, but once we have the identity function, anything else is easy. Alternatively we could have the lambda function being created in the stack itself. Since I use many stacks in 1 account, I would have a whole bunch of leftover lambda functions and roles (and all stacks need to be created with --capabilities=CAPABILITY_IAM
, since it also needs a role.
Create lambda function
- Go to lambda home page, and select your favourite region
- Select "Blank Function" as template
- Click "Next" (don't configure any triggers)
- Fill in:
- Name: CloudFormationIdentity
- Description: Returns what it gets, variable support in Cloud Formation
- Runtime: python2.7
- Code Entry Type: Edit Code Inline
- Code: see below
- Handler:
index.handler
- Role: Create a Custom Role. At this point a popup opens that allows you to create a new role. Accept everything on this page and click "Allow". It will create a role with permissions to post to cloudwatch logs.
- Memory: 128 (this is the minimum)
- Timeout: 3 seconds (should be plenty)
- VPC: No VPC
Then copy-paste the code below in the code field. The top of the function is the code from the cfn-response python module, that does only get auto-installed if the lambda-function is created through CloudFormation, for some strange reason. The handler
function is pretty self-explanatory.
from __future__ import print_function
import json
try:
from urllib2 import HTTPError, build_opener, HTTPHandler, Request
except ImportError:
from urllib.error import HTTPError
from urllib.request import build_opener, HTTPHandler, Request
SUCCESS = "SUCCESS"
FAILED = "FAILED"
def send(event, context, response_status, reason=None, response_data=None, physical_resource_id=None):
response_data = response_data or {}
response_body = json.dumps(
{
'Status': response_status,
'Reason': reason or "See the details in CloudWatch Log Stream: " + context.log_stream_name,
'PhysicalResourceId': physical_resource_id or context.log_stream_name,
'StackId': event['StackId'],
'RequestId': event['RequestId'],
'LogicalResourceId': event['LogicalResourceId'],
'Data': response_data
}
)
if event["ResponseURL"] == "http://pre-signed-S3-url-for-response":
print("Would send back the following values to Cloud Formation:")
print(response_data)
return
opener = build_opener(HTTPHandler)
request = Request(event['ResponseURL'], data=response_body)
request.add_header('Content-Type', '')
request.add_header('Content-Length', len(response_body))
request.get_method = lambda: 'PUT'
try:
response = opener.open(request)
print("Status code: {}".format(response.getcode()))
print("Status message: {}".format(response.msg))
return True
except HTTPError as exc:
print("Failed executing HTTP request: {}".format(exc.code))
return False
def handler(event, context):
responseData = event['ResourceProperties']
send(event, context, SUCCESS, None, responseData, "CustomResourcePhysicalID")
- Click "Next"
- Click "Create Function"
You can now test the lambda function by selecting the "Test" button, and select "CloudFormation Create Request" as sample template. You should see in your log that the variables fed to it, are returned.
Use variable in your CloudFormation template
Now that we have this lambda function, we can use it in CloudFormation templates. First make note of the lambda function Arn (go to the lambda home page, click the just created function, the Arn should be in the top right, something like arn:aws:lambda:region:12345:function:CloudFormationIdentity
).
Now in your template, in the resource section, specify your variables like:
Identity:
Type: "Custom::Variable"
Properties:
ServiceToken: "arn:aws:lambda:region:12345:function:CloudFormationIdentity"
Arn: "arn:aws:lambda:region:12345:function:CloudFormationIdentity"
ClientBucketVar:
Type: "Custom::Variable"
Properties:
ServiceToken: !GetAtt [Identity, Arn]
Name: !Join ["-", [my-client-bucket, !Ref ClientName]]
Arn: !Join [":", [arn, aws, s3, "", "", !Join ["-", [my-client-bucket, !Ref ClientName]]]]
ClientBackupBucketVar:
Type: "Custom::Variable"
Properties:
ServiceToken: !GetAtt [Identity, Arn]
Name: !Join ["-", [my-client-bucket, !Ref ClientName, backup]]
Arn: !Join [":", [arn, aws, s3, "", "", !Join ["-", [my-client-bucket, !Ref ClientName, backup]]]]
First I specify an Identity
variable that contains the Arn for the lambda function. Putting this in a variable here, means I only have to specify it once. I make all my variables of type Custom::Variable
. CloudFormation allows you to use any type-name starting with Custom::
for custom resources.
Note that the Identity
variable contains the Arn for the lambda function twice. Once to specify the lambda function to use. The second time as the value of the variable.
Now that I have the Identity
variable, I can define new variables using ServiceToken: !GetAtt [Identity, Arn]
(I think JSON code should be something like "ServiceToken": {"Fn::GetAtt": ["Identity", "Arn"]}
). I create 2 new variables, each with 2 fields: Name and Arn. In the rest of my template I can use !GetAtt [ClientBucketVar, Name]
or !GetAtt [ClientBucketVar, Arn]
whenever I need it.
Word of caution
When working with custom resources, if the lambda function crashes, you're stuck for between 1 and 2 hours, because CloudFormation waits for a reply from the (crashed) function for an hour before giving up. Therefore it might be good to specify a short timeout for the stack while developing your lambda function.
Best Answer
I recently had to do this for some layered deployments which referenced shared services. In addition to parameters, here are some other options:
Named exports: this is a good option if you have some resources which were created by a separate CloudFormation stack and you just want to reference them (e.g. an infrastructure admin sets up the lower-level portion for the application team to deploy on top of).
Substitution: in many cases you might simply need to reference a well-known name which is constant but the ARN varies depending on the AWS account ID. You can use
Fn::Sub
to expand a few “pseudo-parameters” such as the account ID or region:"TaskRoleArn": { "Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/YourSharedServiceTaskRole" }
SSM parameters: you can have a dynamic reference which retrieves an SSM property. This is handy for being completely abstracted from the source of that value – it could be created by CloudFormation but could literally also be someone running a one-off command line script and it supports secure storage of passwords and other secrets which can be configured to prevent retrieval by anyone other than the target service (e.g. an EC2 / ECS IAM instance role) — for example, I used this to store SES credentials:
aws ssm put-parameter --type String --name "/project/mail/EmailHost" --value email-smtp.us-east-1.amazonaws.com aws ssm put-parameter --type String --name "/project/mail/EmailUser" --value <SES_ACCESS_KEY> aws ssm put-parameter --type SecureString --key-id alias/your-well-known-iam-kms-alias --name "/project/mail/EmailPassword" --value <SES PASSWORD>`
Macros: this was recently announced and is a very powerful mechanism where you can have a Lambda function which returns arbitrary JSON for inclusion in the template. That could do almost anything from provisioning extra resources which the CloudFormation stack creator doesn't have direct permission to create to looking up values in a database and returning a template configured with, for example, VPC CIDR allocations out of a larger reservation pool which is managed by the parent organization.