Terraform Error – What Does ‘Error: Cycle’ Mean in Terraform?

terraform

The Terraform docs for some weird reason do not explain what "Error: Cycle" means. I've looked everywhere but there is no mention of it on the official docs.

Best Answer

As part of Terraform's work, it analyzes the dependencies between your resource blocks, data blocks, and other configuration constructs in order to determine a suitable order to process them so that the necessary input data will be available.

For example, consider the following contrived simple configuration:

resource "null_resource" "foo" {
}

resource "null_resource" "bar" {
  triggers = {
    foo = null_resource.foo.id
  }
}

Terraform will analyze the above and notice that the configuration of null_resource.bar contains a reference to null_resource.foo, and therefore the operations related to null_resource.foo must happen before null_resource.bar. We can visualize that as a graph where an arrow represents "must happen after", or "depends on":

Operations for null_resource.bar must happen after operations for null_resource.foo.

Consider now what happens if we modify that configuration like this:

resource "null_resource" "foo" {
  triggers = {
    bar = null_resource.bar.id
  }
}

resource "null_resource" "bar" {
  triggers = {
    foo = null_resource.foo.id
  }
}

Now null_resource.foo also refers to null_resource.bar. There are now two "must happen after" relationships implied by this configuration:

Operations for null_resource.bar must happen after operations for null_resource.foo.
Operations for null_resource.foo must happen after operations for null_resource.bar.

The two statements above contradict one another: null_resource.bar cannot be processed both before and after null_resource.foo. Terraform will respond to this situation by reporting a dependency cycle, using the error message you've seen:

Cycle: null_resource.foo, null_resource.bar

When Terraform returns this error, the solution is to remove at least one of the "must happen after" arrows (dependencies) from the configuration so that it's no longer contradictory. Without seeing your configuration I can't suggest what specific change might achieve that in your case, but it's likely that somewhere in your configuration you have two mutually-dependent resources like this, or perhaps a resource referring to itself.

If you are sighted, depending on how complicated your configuration is, it might help to ask Terraform to produce a graph similar to the ones I've included above in this answer, but highlighting the cycles. To do this, you can use the terraform graph command, like this:

terraform graph -draw-cycles

The output of this command is a description of the graph in the format accepted by Graphviz. If you don't have Graphviz installed on your local computer, you could instead copy-paste the output into Graphviz Online to produce a graph image. The -draw-cycles command causes Terraform to mark the arrows that are related to the cycle being reported using the color red. If you cannot visually distinguish red from black, you may wish to first edit the generated Graphviz code to replace red with some other color you can distinguish.

The graph visualization of the configuration tends to become unusable for non-trivial configurations because there are so many graphs and edges, so if your configuration has many objects it may be better to follow the dependencies through the configuration itself.

Related Solutions

Create multiple resources terraform

Your issue is

default = ["${aws_instance.my_instance1.id}", "${aws_instance.my_instance2.id}", "${aws_instance.my_instance3.id}", "${aws_instance.my_instance4.id}", "${aws_instance.my_instance5.id}"]

That either has to be passed in or set statically.

If my_instance1-5 are the same and can be set with a count in them like above. Then its better to do

InstanceId = "${element(aws_instance.my_instance.*.id, count.index)}"

Terraform Multi-Account Deployment – Deploying to Multiple Accounts with Terraform?

There are two problems that need to be overcome to use the same terraform configurations in multiple AWS accounts. The first, is that you will need to use a different state file for each account. The second, will be making sure the AWS provider uses the correct profile/credentials for the desired account.

Using a different state file for each account

At a high level, terraform works simply by creating a dependency resource graph from the terraform configuration files (*.tf) given; and compares them with a state file. The state file can be either local (terraform.tfstate) or in a remote backend; a common AWS remote backend is s3.

Even if you specify the AWS provider to use separate accounts through variables (more on this below), if both accounts use the same backend, terraform will continuously crash. This will occur, because as it switches from one account to another, the resource AWS IDs won't match, and/or resources will be stated in the state file that don't exist in the account terraform is currently looking at.

A simple way around this is to use workspaces. Workspaces allow you to use different state files without specifying different backend keys. Using workspaces is very straightforward. If you have a simple terraform block such as:

terraform {
    backend "s3" {
       bucket = "aws_bucket"
       key    = "terraform.tfstate"
       region = "us-east-1"
    }
}

You can create new workspaces using terraform workspace new <workspace-name>. So, you can create a workspace for both of your accounts using a sequence such as:

terraform workspace new account-1
terraform workspace new account-2

And you can switch you're current workspace using terraform workspace select:

terraform workspace select account-1

Configuring the AWS provider to be workspace aware

Besides using separate state files. You will also need to specify different AWS credentials (i.e. access keys) for each account needed. AWS credentials are stated using providers. In this case, we are obviously using the AWS provider.. You essentially want to use interpolation within the AWS provider resource block, so when you switch workspaces, terraform will use the correct credentials. Assuming you have your aws credentials file setup with a separate profile per account:

[account-1]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

[account-2]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

Then you can simply write your aws provider block as:

provider "aws" {
    region  = "us-east-1"
    profile = "${terraform.workspace}"
}

So as long as you name your workspaces as your aws profile names, then when you switch workspaces, terraform will use a different aws credential profile when refreshing state and building a dependency graph.

Alternative methods

This is by far not the only way to accomplish this. You could build out modules; and then have different directories that call the module with different AWS providers and state files specified. There are pros and cons to each method.

The workspace method is how you would do it with a single directory, and you would be using the exact same terraform configuration files as written.

Modules would require separate configuration files that at least call the modules and specify the providers and backends. Modules would make a lot of sense if you where going to do multi accounts, multi region, and multi environments. For instance, if you were going to run in two regions for each account, and two different environments (staging and production). Modules make sense here because it provides flexibility using arguments such as count (perhaps staging will run less number of instances, etc.).

Best Answer

Related Solutions

Create multiple resources terraform

Terraform Multi-Account Deployment – Deploying to Multiple Accounts with Terraform?

Related Topic