AWS – How to Use aws-iam-authenticator with Remote Terraform Cloud Runs

amazon-iamamazon-web-serviceskubernetesterraform

I am already successfully using Terraform with the Kubernetes provider to manage various part of and services on an EKS cluster in AWS. I would like to use Terraform Cloud to manage it (and take advantage of the nice Github/VCS integrations). However, the kubeconfig that I have for that cluster uses the aws-iam-authenticator binary to leverage AWS credentials (and hence IAM permissions) to authenticate and grant permissions, both in AWS and within the cluster via IRSA.

The Terraform Cloud instances do not have the aws-iam-authenticator binary installed and hence, although they have the correct AWS credentials, cannot leverage these to authenticate and connect with the EKS cluster.

Using regular (non-cloud) Terraform does not have this issue, you just make sure the machine you are running your plan/apply from has the binary and credentials and it "just works", how can I make Terraform Cloud work the same way?

Best Answer

Terraform Cloud docs cover this, and it is not forbidden, but it is not exactly encouraged either. There are 2 methods mentioned in those docs, using a local-exec provisioner (the use of which is described as a "last resort" and potentially problematic), or using a submodule. The docs also mention that only "Standalone Binaries" should be used. Thankfully aws-iam-authenticator is indeed available as a static standalone binary for many systems. Unfortunately the documentation, beyond a mention of submodules, does not cover the steps needed to make this work.

Hopefully Hashicorp will provide a less hacky and more supported way to customize these environments in future, but until then...

First, we need a compatible binary, and although we do not know exactly what OS/Image is being run by the Terraform Cloud instances, we know from the documentation that it is Linux x86_64 compatible. Therefore, since the aws-iam-authenticator is available as a standalone static binary for Linux x86_64, what we need to do is introduce that binary into the environment, and then make it available to the instance generally (since we do not control how it is called specifically).

Terraform Cloud allows you to manipulate the environment variables, we can achieve this by manipulating the $PATH variable once we have the binary on the host. The simplest way to do this would be to just add the binary to the config inside a folder, however if you would like to use this for multiple configs without having to manage the binary separately in each (imagine having to update them all separately), I'd recommend doing this as a sub-module in your git repository.

Here's an example of a public repo containing the binary, you would add this to your repo as a submodule as follows:

git submodule add https://github.com/comerford/tc-tools.git

You can then commit/push as usual, making sure that your version control settings for your workspace are set to clone submodules as below:

submodule cloning enabled

Finally, to make sure that no matter how the binary is called that it works, you need to make the changes to the PATH environment variable so that it will see your new binary, but also not break anything. Here's the ENV variable that worked for me given the above submodule:

Working PATH Variable on Terraform Cloud

To give that string in a more usable format:

"~/tc-tools:/home/terraform/tc-tools:./tc-tools:/usr/bin:/usr/local/bin:/bin:$PATH"

It's taking a guess at the likely paths, and I am not 100% sure which one works because I didn't want to cause broken runs for the sake of a few characters.

Although this is specifically for the aws-iam-authenticator binary, it should work as a generic method for any such Linux x86_64 static binary that you happen to need.

Related Topic