Terraform – Resolving Cycle Error with AWS EC2 Instance, Volume, and Template File

Terraform's aws_instance resource requires an AMI, and the root device is always a new volume produced from that AMI's underlying EBS snapshot. This is (as far as I know) a limitation of the underlying EC2 API.

As you've seen, it's possible to attach an existing EBS volume to a running instance using aws_ebs_volume_attachment but there's no way to use an existing EBS volume as the root filesystem of a new instance.

The official documentation on root volumes states that the closest we can come to what you want here is:

• Shut down the old instance while retaining its root EBS volume.
• Create a snapshot of the old root EBS volume.
• Create a new AMI using the created snapshot.
• Launch a new image using that new AMI.

This sort of multi-step process is difficult to orchestrate using Terraform alone, since Terraform isn't able to keep track of it's position in such a process. However, a config like the following would allow you to achieve it with a bit of manual workflow when you need to replace the instance:

variable "source_volume_id" {
}

resource "aws_ebs_snapshot" "new" {
  volume_id = "${var.source_volume_id}"
}

resource "aws_ami" "new" {
  name = "from-${aws_ebs_snapshot.new.id}"
  virtualization_type = "hvm"
  root_device_name = "/dev/xvda"

  ebs_block_device {
    device_name = "/dev/xvda"
    snapshot_id = "${aws_ebs_snapshot.new.id}"
    volume_size = "${aws_ebs_snapshot.new_volume_size}"
  }
}

resource "aws_instance" "new" {
  ami = "${aws_ami.new.id}"
  # ...etc...
}

This config assumes that you've already manually terminated the old EC2 instance and noted its root volume EBS volume id. You then pass that volume id in via the source_volume_id variable and it will do the remaining steps.

Each time source_volume_id changes Terraform should repeat this process, creating a new snapshot, AMI and instance. If you run Terraform again with the same source_volume_id as most recently used, it should leave everything unchanged.

A weird gotcha with this semi-manual process is that it has a chicken-and-egg problem where on the first run you probably won't already have a volume to base your AMI on. In that case you'd have to initially comment out everything except the aws_instance resource, hard-code an AMI id to bootstrap with, and let Terraform create that initial instance. On subsequent runs you can then follow the process as described above.

As noted above, Terraform is not currently well-suited for this sort of problem. Given that there's some manual steps anyway, it may work out simpler to just script this entire process using a more imperative approach e.g. using one of the AWS SDKs with your favorite programming language, though using Terraform for part of the process does at least free you from the task of keeping track of previously-used snapshot and AMI ids to ensure that they get cleaned up.

Thanks to jbardin https://github.com/hashicorp/terraform/issues/14056

In order to fix this you need to use an output from your VPC module. You can use the aws_nat_gateway attribute public_ip but since I had a route created after the aws_nat_gateway I used that instead. I then made a dummy variable and dummy resource in my Jenkins module.

resource "null_resource" "dummy" {
  provisioner "local-exec" {
    command = "echo ${var.dummy}"
  }
}

Make sure you assign that dummy variable to the output you have chosen. Also it needs to be a string. dummy = "${join(",", module.vpc.private_nat_gw_routes)}"

After that I used depends_on = ["null_resource.dummy"] on my ASG resource. This made that resource wait until after the NAT Gateway + routes were created but doesn't have the nasty side effect of recreating the resource every time.

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed