Terraform AWS: Up and Running in 25 Minutes

Working with the Cloud is a great opportunity, as it offers access to a series of services and infrastructures that until recently were beyond the reach of all except corporations.

However, “with great powers come great responsibilities“. Managing the amount of information related to this new infrastructure is not an easy task and can lead,  if not handled properly, to cost or security issues.

Fortunately, Terraform can help us to manage our entire infrastructure. First, by providing a series of tools and abstractions that allow growth within a short time. Second, by keeping infrastructure consistent and under control.

Moreover, Terraform also allows full adoption of the IaC (Infrastructure as Code) philosophy, by exploiting programming advantages within the realm of infrastructures. In other words, it greatly simplifies cloud management.

What is Terraform?

Terraform is a tool developed in 2012 on the basis of the Infrastructure as Code (IaC) “philosophy”, exploiting years of infrastructure management experience. If you are reading this with thoughts of utilising Terraform, you probably wish to boost the overall effectiveness and efficiency of your work. This tool offers valuable support in managing your cloud provider without incurring excessive complexity.

By fully managing the IaC, Terraform offers the following possibilities:

• Code versioning: Terraform uses a pseudo language called HCL that allows us to write our structure as a code;
• Management infrastructure: a simplified platform and abstract structures enable the creation of business value, while taking care of everything else at the same time (API calls, driver updates, functionality additions etc …);
• Idempotency: prevents duplication of efforts and operations. In other words, if a resource is already present no further operation will be performed, and the resource’s state will be protected from any unexpected behavior;
• Infrastructure consistency: any change is tracked and reported. In this way, our code straightforwardly portrays what is available in our infrastructure, and any changes are promptly reported.

In this article, we will learn how to create an SQS queue – a very simple object that carries no cost if we forget to destroy it once the exercise is over.I will refer to the GitHub repository with a ready-to-use example that accompanies this article.

Prerequisites

To begin, we need the following prerequisites:

• UNIX based environment (I use Ubuntu or WSL)
• Terraform CLI (a simple guide to installation is available)
• AWS: Basic knowledge of the terminology, security issues, and the cost of resources 
• At least one AWS account (I used the free tier for this lab) – this allows us to create our resources for free
• Basic knowledge of a programming language

Before proceeding, I would like to remind readers of the importance of knowing the cloud you are working with. This is of crucial importance to avoid incurring excessive costs. We are creating real resources, and in carrying out our tests must keep in mind the existing AWS thresholds in terms of both timing and quantity.

I strongly recommend you use alerts (ideally, more than one) to provide timely warnings if you are approaching the threshold, to avoid paying unexpected amounts of money.

What you will learn

• Get familiar with the basic concepts of Terraform
• Create your first automated infrastructure
• Understand how Terraform manages infrastructure changes

Terraform: Installation of the CLI

You can download Terraform at this link. For this example, I chose to use version 0.12.28.

One of the advantages of Terraform is its ability to work on almost all platforms. For this tutorial, an Ubuntu environment inside Windows 10 has been chosen, thanks to the opportunity to use the Windows Ubuntu app. The required steps to install Terraform on Ubuntu/WSL are very simple:

• Download the zip file containing the terraform executable (inside a temporary folder, for example)
• Extract the zip file (which will have a name similar to terraform_0.12.28_linux_amd64.zip)
• After extraction there will be a file named terraform; copy it into the folder /usr/local/bin, so that you can access it from any location in the shell

Once the Terraform steps have been followed, it can be called from the terminal. To verify that the program works correctly just run the command:

terraform version

Creation and configuration of a user dedicated to Terraform

Once the Terraform cli has been installed, it is necessary to create an IAM User with Programmatic access-type access, in order to use it with AWS. We don’t want to use our user here, instead creating a dedicated user with specific and controlled permits in order to limit the area of use and any security issues. During user creation you will need to assign the AmazonSQSFullAccess predefined policy, which allows full use of SQS resources.

NOTE: For reasons of simplicity, it is also possible to directly assign “AdministratorAccess”. However, it is always preferable to insert only the most relevant policies, to limit the damage done should someone manage to steal our credentials.

At the end of the creation process we will have two keys:

• ACCESS KEY ID
• SECRET ACCESS KEY ID 

Both are necessary for Terraform to be able to connect to our account, using the newly created user and its specific policies. A .csv file containing the same data is also provided.

Such information must be saved immediately as there is no way for security to resume these, unless you destroy the user and create a new one (even if it has the same name).

NOTE: These data are very sensitive and it is necessary to be extremely careful i.e., never save them on a public repo or similar.

(Optional) Billing alert creation

Lastly,  I strongly recommend inserting billing alarms into the account to keep control of costs and avoid nasty surprises. Although the AWS tier is free, it has very precise conditions of use, anything not included will be charged.

Terraform: start!

To start using Terraform, we first need a folder that will contain our project. If you are using my example project, I am referring here to the base_1 folder. Inside the folder we will create a file called main.tf, and inside this we can create our first Terraform object, allowing us to connect to the AWS provider. This is the code example:

Before you can really start using Terraform, however, you will still need to complete the last setup actions in order to log in to the AWS account.

Connect Terraform to AWS

In order to use AWS, you first need to enter the information of the user created previously, i.e., the two keys:

• SECRET ACCESS KEY ID
• ACCESS KEY ID

To do this, simply load our keys as environment variables, using the example code below. To load the keys, just run the following commands:

# <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions>
export AWS_DEFAULT_REGION=us-east-2 #<region id>
export AWS_ACCESS_KEY_ID=DASDSAXCKSCNK #<acces key>
export AWS_SECRET_ACCESS_KEY=DSFK432_dsaedi #<secret access key>Code language: HTML, XML (xml)

Again, it is a great idea to automate this operation and simplify the insertion of information. Nevertheless, bear in mind that it is best to keep such information private. 

Once these operations are complete we can finally use Terraform with AWS. 

NB: If you start a new session, these environment variables will not be available.  It will be necessary to repeat the procedure, to avoid receiving an error message from Terraform.

Terraform init

terraform init

This command must be executed inside the working folder in which the tf files are saved. After execution, all the init operations will be taken care of, namely:

• download the various providers;
• plugins for the platform used;
• connect or download the modules used;

This command is essential to use our new Terraform project. Now we can begin to analyze the various components of the code.

Provider: AWS

The provider is responsible for understanding how the APIs (of the Cloud provider) interact and expose their resources.

We can imagine it as the engine that allows us to communicate with AWS resources

As you can see, each provider has at least:

• name: in this case, aws → provider “<provider name>”

Other optional attributes:

• Version: the provider version. If not entered automatically, Terraform will take care of downloading the latest version available. This is recommended, as a new version of the provider could break compatibility or bring bugs into the system to interact with what we currently have. For this reason, even if optional, insertion is always recommended.

Other configurable attributes (but not recommended for security reasons, as you directly expose the keys of your Terraform user) will be taken automatically by Terraform as they are present as environment variables

• region: the region on which we must operate for example “us-west-2”
• access_key: the access key of our user
• secret_key: the user’s secret key id

For more information on providers you can consult the following pages:

• https://www.terraform.io/docs/providers/index.html
• https://registry.terraform.io/providers/hashicorp/aws/latest/docs

Resources

The resource components are the main Terraform object. In fact, each block describes one or more infrastructure objects, such as virtual machines, VPCs, etc …

In the example used, we are creating an SQS queue with a tag.

The syntax of the code block is explained below:

resource "<type of resource you want to create / manage>" "<local id / name of that resource>" {

  "Attributes ...."

}Code language: JavaScript (javascript)

• aws_sqs_queue → <type of resource to be created / managed> defines the type of resource to be created. If you want to see the entire catalog offered by Terraform, visit this page.

  

• terraform_queue → <local id / name of that resource>: defines a unique id / name for the resource (we can think of it as the name of the class instance in the programming world). This allows us to have N resources of the same type, for example, and to be able to recall them later. If you try to create two resources with the same name, Terraform will identify an error, as the second resource conflicts with the name.
  We will talk about how to recall the information about a specific resource in the next lesson. For now, all you need to know is that you need to give each resource a unique name.
• <attributes ...>: Inside the resource block, on the Terraform page relating to the resource, the attributes that can be configured are available to view. You can see all the available attributes, and if they are mandatory, if there are limitations and many other pieces of information. Further on, we will talk specifically about how to work with attributes.

Infrastructure creation – the journey begins

Now that complete base is ready, we can finally start using Terraform and see what happens to our account.

Terraform plan

terraform plan

This command executes what in Terraform is called an execution plan; that is, it allows calculation (without any concrete performance on our account) of the actions that must be performed to achieve the desired state of the configuration files. Done for the first time, the result should look something like this:

As you can see, Terraform will logically generate all the resources (and their respective configurations) necessary to achieve our goal. In the end it will carry out a summary of the type of operations:

• add: resources to add
• change: resources that need to be changed
• destroy: resources that must be destroyed. As shown in the last figure, we must be careful, because depending on the type of resource, what for us logically looks like a change might actually be an operation of destruction and recreation

We are finally ready to really create our resources on our account. To do this we would use the ‘apply’ command.

Terraform apply

With the Terraform apply command it is finally possible to concretely create the resources that we have previously seen with the plan command. To launch the creation, just run the command:

terraform apply

Once the Terraform command is launched, it will do the following:

• re-execute a resource plan, showing us the same information, in order to know which operations will be carried out

For safety, it will make a confirmation request to be sure that the operations to be carried out are those we expect

• When you confirm the operation, Terraform will begin to call the bees to proceed with creation, displaying a log of the operations carried out on the screen.
• At the end of the creation process you will be presented with a summary of the operations carried out.

Here is an example of an apply command in use:

Terraform state: the historical memory of Terraform

Once our resources have been created, we can see that a file has been created inside the project folder called terraform.tfstate – Terraform needs this file to keep track of the real infrastructure. In fact, this file contains the current status of the resources created so that it can act in the event of subsequent changes, by deciding what actions it must take.

Do not manually delete or modify this file. In subsequent articles, I will write in a more in-depth way  about the function of the tfstate and its uses.

Terraform and idempotency

We have now seen how Terraform takes care of creating resources, but what if we rerun the Terraform apply command?If you try again, you will activate a log in which Terraform will show you that no operations must be performed and that everything is aligned.

This magic is made possible by using terraform.tfstate which then displays the state of the last operation performed by Terraform, in comparison with the current state. If both coincide, no operation will be carried out.

Terraform and change management

Contrary to what we have seen before, if there are changes (performed within the code or directly from the interface) and the ‘terraform apply’ command is run, there will also be a comparison here between the local state and what is really present.

Case 1: We made a change within our code

increase the delay_seconds = 120

If we then rerun the ‘terraform apply’ command, then we would see that even this simple modification is noticed by Terraform, showing us the changes it will make and asking if we are sure we want to make them.

Case 2: We made the same change as before but directly from the web console (and not on the code)

Here, Terraform notes a change that is inconsistent with what is present in the code, and will propose a modification action that will return the state to what is described in the code.

This is one of Terraform’s strengths, as it allows us to maintain our infrastructure’s alignment with what is written in the code. At the same time, especially during development, it requires us to be more careful, as any changes made to test a new functionality, if not reported within Terraform, will be lost. From now on, this should always be taken into consideration.

One very important thing for the future is to not be fooled by the number of changes. Depending on how the Cloud provider treats its resources, even a field that may seem trivial could lead to the destruction and then recreation of resources. This happens in Azure’s Virtual Machines, for example.

Terraform destroy

it is clear that we have concluded our project, and it is now time to destroy (delete) all the resources created to avoid any additional costs on our account. In order to destroy the resources created by Terraform just run the command:

terraform destroy

The steps performed will be very similar to those performed by the apply command, but in this case with a view to removal:

• re-execute a resource plan, but unlike ‘plan’ and ‘apply’, this time Terraform will show us which resources will be deleted
• For security reasons, a confirmation request will be made
• When the operation is confirmed, Terraform will begin to call the “bees” to proceed with the destruction, displaying a log of the operations carried out on the video.
• At the end of the destruction operation, you will be presented with a summary of the operations you have carried out.

Here is an example of the destruction log:

To check the result, simply log into the console.

What we have learned

• Terraform interfaces with AWS
• Basic Terraform concepts: plan, apply and destroy commands
• What the terraform.tfstate file is for
• How Terraform uses idempotency and manages changes
• The usefulness of having code that describes the state of our infrastructure.

Launching your event

Want to launch a developer event despite the challenges of COVID-19?

If you want to know more about how modern technologies and tools can support you for – and during – the organisation of a virtual event, don’t miss this article showcasing the best tools we used to host our online conferences since the COVID-19 outbreak.