Terraform AWS: Files and Variables

In the previous article of our series about Terraform AWS, we looked at an initial project that allowed us to create an SQS-type resource, understanding how to configure Terraform so that it can communicate with AWS thanks to programmatic access users.

We also explored how to use the basic terraform commands to generate our resources and terminate them when we’re done, such as:

• init
• plan
• apply
• destroy

In this article, we will see in more detail how to structure a Terraform project in more depth, dividing the logic into multiple files, in order to make everything more well-ordered and maintainable.

By the end of the article, you will also understand how to use variables, and what tools are available to you to make your project more dynamic. In this way, you can create a single template and use it in multiple environments, for example.

You will also understand the interdependence between resources and how to retrieve information that is present in your cloud account in order to build increasingly complex resources.

A GitHub project, with a ready-to-use example that I will refer to, accompanies this article.

Prerequisites

In order to start, you will need to know these prerequisites:

1. UNIX-based environment (I used Ubuntu or WLS in Windows 10)
2. Terraform CLI (if you haven’t installed it yet, see the previous article)
3. AWS: basic knowledge of terminology, and security and resource cost issues
4. At least one AWS account (I used the free tier for this lab) which will allow us to create our resources without initial investments
5. Basic knowledge of any programming language

Before starting, I want to return to the importance of knowledge of the cloud, and the great responsibilities this brings, especially from an economic point of view.

In fact, we are going to create real resources and it is necessary to be very careful with the tests that are carried out to avoid finding yourself paying for resources that have been created only once as a test, but which, from the point of view of AWS, are in effect resources with associated costs due from the account

I speak from personal experience, so I strongly advise you to insert alerts (more than one) that will notify you immediately if you are starting to pay unexpected amounts of money.

A dynamic Terraform AWS project in continuous growth

While in the first article we treated an example of a static, simple and immediate project to get to the Terraform juice, now we want to get into the specifics, simulating a possible work project.

We will focus more on the organisation of the project, dividing it into several files that will simplify management.

In the simulation, we are managing a request from the customer who wants an EC2 VM that can be created in different environments (DEV, STAGE, PROD), in order to support various development phases.

To be able to do this, therefore, we must begin to deal in depth with Terraform’s management of variables. In this way, we can create a single template valid for all environments that will behave differently depending on the value of our variables.

Input Variables in Terraform

If we want to have an orderly and dynamic project, the tool that immediately comes to our aid is the use of variables, which allows the creation of a unique template that will behave differently depending on the values ​​we are going to use.

Not only that – another advantage of variables is the ability to give a business meaning to each value, in order to simplify the problem determination in case of errors, or the expansion of the project itself.

For more in-depth information, check the documentation pages dedicated to input variables and expressions.

Variables: structure and types

Variables in Terraform come in three macro types:

• primitive:
  • string
  • number
  • boolean
• collections
  • maps
  • list
  • set
  • tuple
  • objects
• generic
  • any (when you don’t know or don’t want to force a type). If no type is entered, terraform automatically considers a variable as type ANY

In this article we will deal only with the primitive types for simplicity. In the next article of this series, we will deal with the other types in more detail.

Variables: concrete example

This is an example of the variables that we are going to use.

The variables can be defined anywhere in Terraform, as long as it occurs inside a file with the TF extension. In the previous example we used the main.tf.

In this case, to simulate a more complex project the use of a dedicated file called vars.tf is highly recommended. This must be in the same folder that contains the main.tf.

The structure of a variable is:

• variable "<variable name>" =
  Here, the name of the variable is defined (in fact, if we wanted to, we could stop here too – terraform assigns ANY as the default type, e.g. plain).  Alternatively, we can choose to be more detailed and invest our time in being precise in the definition and construction. From personal experience, when the project begins to grow without any such precautions, when a problem arises it becomes difficult to get to the point in a short time.
  
  The remaining information is:
  • description (optional): the description of our variable;
  • type (optional): defines the type of the variable (see the previous list for the various types offered, or check the documentation for more details);
  • default (optional): defines the default value that we want the variable to have if no value is entered. Obviously the default value follows the type of the variable, so a variable of type list has as default [], while a variable of type bool will have a default value of true. It is very important to keep this in mind, as sometimes there can be correlation errors between variables.

There are at least two ways to use the variables we have just created:

• through interpolation:

  

• plain-direct:

  

  Note that in previous versions of Terraform 0.11, the use of ${var.env} was always mandatory.

Locals

Within a real project, we cannot limit ourselves to having only AS-IS variables; we also need to be able, for example, to concatenate information, or make it dynamic, in the case of a machine name that depends on the environment. For instance, we can have names such as test_micro_vm or dev_micro_vm.

In order to achieve this goal, Terraform provides us with the locals tool:

This tool allows the creation of a new special variable, starting from the manipulation of previous variables.

To be able to use the new local variable, it will be enough to recall them in the following way:

Be careful! If I use a variable I will call it using var.xyz, while if it is local it is called with local.abc. 

How to give value to our variable’s Variable

Precedence

Before talking about the various methods we have to value our variables and therefore eventually give them a fundamental business value for our project to take shape, it is important to discuss the precedence of variables.

Terraform uses hierarchical logic to decide which value to consider in the event of a conflict.

The order is as follows:

1. Environment variables
2. The terraform.tfvars file, if present
3. The terraform.tfvars.json file, if present
4. Any *.auto.tfvars or *.auto.tfvars.json files, processed in lexical order of their filenames
5. Any -var and -var-file options on the command line, in the order they are provided

In this guide we will not talk about auto.tfvars, as in my opinion these are difficult to manage, but it is useful to know that this method also exists.

Environment variables

This is the first entry point when it comes to giving a value to variables. By variables, I mean here not only the variables we have defined within our var file, but also the configuration variables that terraform makes available to configure, for example, management of the logs or the location of the project files.

In this article we will only talk about how to manage the value of our variables; for everything else, refer to this link which gives a better explanation of all the various options.

To be able to value our newly created personal variables, Terraform uses a very simple convention:

TF_VAR_<name variable>Code language: HTML, XML (xml)

So for example, if we had created an “env” variable, in order to value it we must use:

export TF_VAR_env = PRODCode language: JavaScript (javascript)

This method of valuing variables is very useful when you want to:

• avoid inserting a sensitive value in the code, such as the aws secret key. In this case terraform knows the name of the variable and when it starts the project, the first thing it does is query the environment variables to load the sensitive value:
  export AWS_ACCESS_KEY_ID = DASDSAXCKSCNK
• For automation: in this case, taking advantage of the fact that terraform gives absolute priority to environment variables, I can overwrite the value of the environment where I want to deploy terraform, for example, and force a certain behaviour,

terraform.tfvars

This file is used to define the default values ​​of a variable.

In fact, even if it is very convenient to be able to use various files to contain the variants of our variables, it is true that there will always be variables that are constant even if we change the environment. Having to map these every time to every single file leads us not only to a waste of time, but also to possible errors.

For this there is terraform.tfvars This is the first file loaded by Terraform to enhance our variables and can consequently be used as a repository for all the global or default variables that we want to share within the project.

To be able to use it, just create a file called terraform.tfvars in the same folder as the main.tf. The syntax inside is very simple, as shown in the example:

my_default_variable = "default"Code language: JavaScript (javascript)

Var files

With all that achieved, we are finally going to see how to organise the values ​​according to the environments and start to approach our imaginary customer’s request.

This is where the concept of vars files comes in handy.  These are files with the TFVARS extension, and unlike variable files, can be anywhere in our project.

Here is an example snippet of a vars file base_2\vars_files\variables_dev.tfvars:

As you can see, these recall the name of our variables, and we feed them with the values ​​we want.

To be able to recall these variables, just use the command:

terraform <command> -var-file=vars_files\variables_dev.tfvarsCode language: HTML, XML (xml)

Thanks to this, we can use any vars file that respects our variables. In this case, you contain the info for other environments such as STAGE, for example.

Variables from command line

Another way in which we can value our variables is via the command line. In this case, we simply use the command:

terraform <command> -var="<variable name>=<value>"Code language: HTML, XML (xml)

Example:

terraform <command: apply/plan> -var="env=dev"Code language: JavaScript (javascript)

This type of use is definitely not recommended as it is difficult to maintain. I strongly suggest using  the other methods previously discussed.

Variables with user input

If you want to have user input, it can be done simply – just don’t value the variable.

In this case, Terraform automatically realises that the variable has not yet been evaluated, and will ask you to enter its value on the screen, as each variable that is defined within the Terraform must have a value.

In this example, I purposely created a variable that is not valued anywhere and terraform immediately asked me for its value.

Data resources: how to retrieve information outside Terraform

While developing your project, at some point (very soon) you will find yourself in a situation where you need information that is located outside your terraform project.

For example in our sample project, in order to build my EC2, I need to know the AMI ID to use for my vm. This ID could change over time or depending on the region where I am, therefore it is not possible to have it static in the code. I need to ask AWS to form it for me.

This is where the data source component comes into play – it allows us to upload information outside of our project and then be able to use it at will.

Terraform provides an infinite range of datasources for nearly every resource it creates. As a result, it is possible not only to create the resources, but also to query them and loading the necessary data and be able to reuse them at a later time.

Here is an example (from here):

As can be seen within the Terraform documentation for each resource (in this case VPC) there is both a subsection Resources (where their use to create the resource is explained), but also a subsection Data Sources.

Here you will find:

• a brief explanation of the component
• examples 
• arguments (necessary and optional) for their use
• attributes returned: i.e., the values ​​that you can obtain to be able to use them later

In our example project we have:

Where in order to define a data source, it is necessary to use the following syntax:

data "<datasource id>" "<local name / id>" {
    <attribute>=<value>
    ...        
}Code language: HTML, XML (xml)

In order to use the attributes that are made available, just use the following notation:

given.<datasource id>.<local name>.<attribute>Code language: HTML, XML (xml)

For instance:

data.aws_ami.amazon_linux_ami.idCode language: CSS (css)

When you execute a plan / apply command you will see that terraform will load the requested information with the data source, so that it is available immediately afterwards:

Outputs

This is the last feature we will talk about. For now we will focus on the simplest use of outputs, which is to print the result of a process on the screen, or the content of the variables we used. Further on, we can see how output is used within modules to make a value available to whoever invokes it.

In our example project we can see how we defined the outputs:

with the following result:

The outputs are very useful in both the development phase (as debugging a run with Terraform is not always easy), and in terms of having useful information on screen in order to understand the progress of resource creation, such as when you have automatic pipelines and want to see in the logs, a report of what they have created.