Terraform Data Sources: Examples

Have you ever felt like you’re pulling data out of thin air when crafting your infrastructure with Terraform? It’s a common issue. You define your resources, but what about the info you need to make those resources work together? That’s where Terraform data sources come into play. They’re the unsung heroes of declarative infrastructure, pulling in the necessary details to connect all the pieces. This article will dive deep into the world of Terraform data sources, giving you real-world examples to help you understand how to use them to build more robust and dynamic infrastructure.

What Are Terraform Data Sources?

Terraform data sources let you use information from outside your current configuration. Think of it like this: instead of hardcoding details like specific IDs or network configurations, you can use data sources to grab that data from live systems or even other Terraform states. This makes your infrastructure code far more flexible and less likely to break when things change in the real world. They read data, and they don’t create or change anything, which means they are safe to use in any scenario.

You might be wondering: how are data sources different from resources? Resources are the things you create, update, or delete, like virtual machines or databases. Data sources, on the other hand, are all about reading existing info. They’re the sidekicks that gather intel, and resources do the heavy lifting.

Why Use Data Sources?

Data sources are an amazing tool that can save you time and headaches. They bring key benefits to your workflows:

Dynamic Configurations: Data sources allow your configuration to adapt to changes in your environment. If an ID or a name changes on your cloud account, your Terraform code doesn’t need manual updates. The data source will fetch the new data.
Reduced Hardcoding: Avoid hardcoding values. Using data sources makes your infrastructure far more maintainable and less prone to errors. Instead of writing specific values directly into your code, you can pull them from an authoritative source.
Reusability: Data sources make your modules more reusable. You can use the same module across different environments without having to make changes to the config files.
Simplified Complex Setups: Data sources help simplify the process of setting up complex infrastructure by providing info for your resources in a clean and organized manner. You avoid dealing with scattered configuration.
Integration: They allow your Terraform code to interact with existing systems. Data sources also let your infrastructure interact with resources that were created outside of Terraform.

In short, data sources make your infrastructure code smarter, more adaptable, and easier to manage.

Basic Syntax of a Data Source

Before diving into examples, it is best to understand the structure of a data source block. Here’s a typical data source syntax in Terraform:

data "provider_name_resource_type" "data_source_name" {
  # Configuration options specific to the data source
  attribute1 = "value1"
  attribute2 = "value2"
  ...
}

Let’s break down each part:

data: This keyword tells Terraform that you are defining a data source.
"provider_name_resource_type": This is the type of data source you are using. It specifies which provider and resource type this data source is for. For example, "aws_ami" for an Amazon Machine Image.
"data_source_name": This is a local name that you’ll use to refer to the data source in your code. Choose a descriptive name that makes sense within your infrastructure.
Configuration Options: Inside the curly braces ({}), you add options specific to the data source. These are usually filters or identifiers to select the exact data you want. They can be attributes like names, IDs, or even more complex filters.

Once the data source is defined, you can access the attributes from the data source object in your resources. You refer to them as data.data_source_name.attribute. For example, you can reference the id attribute of an aws_ami data source as data.aws_ami.example.id.

Common Data Sources and Examples

Terraform has a very wide variety of data sources. They can be used to gather all sorts of info, and they can cover almost anything you can imagine. Let’s explore some common examples to see how you can use them in practice.

AWS Data Sources

AWS is one of the most common platforms to use data sources. It has a huge ecosystem with lots of different options to use. Here are a few you may want to explore:

`aws_ami`

The aws_ami data source fetches details about an Amazon Machine Image (AMI). It’s very common when you need to use an image that you don’t have the exact ID of, but rather, you have a specific name or other criteria.

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["099720109477"] # Canonical
  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
  }
}

resource "aws_instance" "example" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"
}

In this example:

The data "aws_ami" "ubuntu" block looks for the most recent Ubuntu 22.04 image.
It uses a filter to specify the name and the Canonical owner ID.
The resource "aws_instance" "example" then uses the ID of that AMI to launch an EC2 instance.

This way, you don’t need to manually find the ID of the latest Ubuntu AMI every time you want to launch an instance. The data source takes care of fetching it for you.

`aws_vpc`

The aws_vpc data source retrieves details about an existing Virtual Private Cloud (VPC). It is useful when you have a VPC created outside of Terraform, or in another state.

data "aws_vpc" "main" {
  default = true
}

resource "aws_subnet" "example" {
  vpc_id            = data.aws_vpc.main.id
  cidr_block        = "10.0.1.0/24"
}

In this snippet:

The data "aws_vpc" "main" block gets the details of your default VPC.
The resource "aws_subnet" "example" then creates a subnet within that VPC, using the VPC’s ID.

This allows you to create subnets and other resources within an existing VPC, without hardcoding its ID.

`aws_security_group`

The aws_security_group data source fetches information about an existing security group. This helps you reference security groups created outside of your current Terraform config.

data "aws_security_group" "web_sg" {
  name = "web-security-group"
}

resource "aws_instance" "example" {
  ami                    = data.aws_ami.ubuntu.id
  instance_type          = "t2.micro"
  vpc_security_group_ids = [data.aws_security_group.web_sg.id]
}

In this configuration:

The data "aws_security_group" "web_sg" block retrieves the security group by its name.
The resource "aws_instance" "example" then assigns this security group to your new instance.

This helps to apply an existing security group to new resources.

Azure Data Sources

Azure has a wide array of services. So you can benefit a lot from data sources. Here are a few useful examples:

`azurerm_resource_group`

The azurerm_resource_group data source gets details about an existing Azure Resource Group. It’s great when you have created the group outside of Terraform.

data "azurerm_resource_group" "example" {
  name = "my-resource-group"
}

resource "azurerm_virtual_network" "example" {
  resource_group_name = data.azurerm_resource_group.example.name
  location            = data.azurerm_resource_group.example.location
  address_space       = ["10.0.0.0/16"]
}

In this setup:

The data "azurerm_resource_group" "example" block pulls the details of the resource group.
The resource "azurerm_virtual_network" "example" block uses the resource group’s name and location to deploy a Virtual Network.

This way you can keep reusing a resource group defined outside of your current Terraform config.

`azurerm_virtual_network`

The azurerm_virtual_network data source obtains the configuration of an existing Virtual Network. This is helpful when creating subnets or other resources within that network.

data "azurerm_virtual_network" "example" {
  name                = "my-virtual-network"
  resource_group_name = "my-resource-group"
}

resource "azurerm_subnet" "example" {
  name                 = "my-subnet"
  virtual_network_name = data.azurerm_virtual_network.example.name
  resource_group_name  = data.azurerm_virtual_network.example.resource_group_name
  address_prefixes     = ["10.0.1.0/24"]
}

In this example:

The data "azurerm_virtual_network" "example" block retrieves an existing Virtual Network by name and resource group.
The resource "azurerm_subnet" "example" block uses the data to create a subnet within that Virtual Network.

This way, you don’t have to hardcode the Virtual Network details.

`azurerm_key_vault`

The azurerm_key_vault data source retrieves the properties of an existing Azure Key Vault.

data "azurerm_key_vault" "example" {
  name                = "my-key-vault"
  resource_group_name = "my-resource-group"
}

resource "azurerm_key_vault_secret" "example" {
  name         = "my-secret"
  value        = "some-secret-value"
  key_vault_id = data.azurerm_key_vault.example.id
}

In this code:

The data "azurerm_key_vault" "example" block fetches the Key Vault details by its name and resource group.
The resource "azurerm_key_vault_secret" "example" block then uses the Key Vault’s ID to create a new secret in that Key Vault.

This lets you manage secrets and other resources within a specific key vault.

Google Cloud Platform Data Sources

GCP is a big player in cloud computing. So it has many data sources that can help you manage it:

`google_compute_image`

The google_compute_image data source is very similar to AWS’ aws_ami data source, it allows you to pull details about a Google Compute Engine image. It’s helpful when creating instances from a specific image.

data "google_compute_image" "ubuntu" {
  family  = "ubuntu-2204-lts"
  project = "ubuntu-os-cloud"
}

resource "google_compute_instance" "example" {
  name         = "example-instance"
  machine_type = "e2-medium"
  zone         = "us-central1-a"
  boot_disk {
    initialize_params {
      image = data.google_compute_image.ubuntu.self_link
    }
  }
  network_interface {
    network = "default"
  }
}

Here:

The data "google_compute_image" "ubuntu" block gets the latest Ubuntu 22.04 image from its project.
The resource "google_compute_instance" "example" then launches an instance using that image.

This lets you use a specific image from a common project in your instances.

`google_compute_network`

The google_compute_network data source obtains details about a Compute Engine network.

data "google_compute_network" "default" {
  name = "default"
}

resource "google_compute_subnetwork" "example" {
  name                     = "example-subnet"
  ip_cidr_range            = "10.0.0.0/24"
  region                   = "us-central1"
  network                  = data.google_compute_network.default.id
}

Here:

The data "google_compute_network" "default" block retrieves the details of your default network.
The resource "google_compute_subnetwork" "example" then creates a subnet in that network.

This lets you manage your subnet using info from the network.

`google_project`

The google_project data source fetches details about the current Google Cloud Project being used.

data "google_project" "current" {}

resource "google_project_service_identity" "gcs_service_identity" {
  provider = google-beta
  project  = data.google_project.current.project_id
  service  = "storage.googleapis.com"
}

In this example:

The data "google_project" "current" block fetches the ID of your Google Cloud Project.
The resource "google_project_service_identity" "gcs_service_identity" block then uses the project ID to create a service identity for GCS.

This is useful when you need to reference your project ID, especially when setting IAM policies.

More Complex Use Cases

Data sources are not limited to fetching basic resource details. They can handle more complex scenarios, adding more flexibility to your Terraform configurations.

Using Filters

Many data sources allow you to use filters to refine the data you retrieve. This is very useful when you have multiple resources and need to pick the right one. For example, in AWS:

data "aws_instances" "web_servers" {
  filter {
    name   = "tag:Environment"
    values = ["production"]
  }
  filter {
    name   = "instance-state-name"
    values = ["running"]
  }
}

resource "null_resource" "example" {
  provisioner "local-exec" {
    command = "echo Instance IDs: ${join(",", data.aws_instances.web_servers.ids)}"
  }
}

In this case:

The aws_instances data source filters instances based on the Environment tag, only fetching running production instances.
The output lists all the IDs for running production instances.

Filters give you the power to select the exact data you need.

Chaining Data Sources

You can chain multiple data sources together to build more complex logic. This allows you to take info from one source and feed it into another. Here’s an example of how to pull info from Azure and use it in another data source.

data "azurerm_resource_group" "example" {
  name = "my-resource-group"
}

data "azurerm_virtual_network" "example" {
  name                = "my-virtual-network"
  resource_group_name = data.azurerm_resource_group.example.name
}

data "azurerm_subnet" "example" {
    name                 = "my-subnet"
    virtual_network_name = data.azurerm_virtual_network.example.name
    resource_group_name  = data.azurerm_resource_group.example.name
}

resource "azurerm_network_interface" "example" {
  name                = "my-nic"
  location            = data.azurerm_resource_group.example.location
  resource_group_name = data.azurerm_resource_group.example.name
  ip_configuration {
    name                          = "internal"
    subnet_id                     = data.azurerm_subnet.example.id
    private_ip_address_allocation = "Dynamic"
  }
}

This code:

First, it pulls a resource group.
Then uses the resource group name to pull a virtual network.
Then, it uses both the virtual network and resource group to pull a subnet
Finally, all this information is used to configure a network interface.

This way, you can make very specific references to existing resources.

Data Sources in Loops

Data sources can be used in loops (for_each or count), which makes your code more flexible when dealing with multiple resources.

locals {
  azs = ["us-west1-a", "us-west1-b", "us-west1-c"]
}

data "google_compute_zones" "available" {
    provider = google-beta
    region = "us-west1"
}

resource "google_compute_instance" "example" {
  for_each = toset(local.azs)
  name         = "instance-${each.value}"
  machine_type = "e2-medium"
  zone         = each.value
  boot_disk {
    initialize_params {
        image = "debian-cloud/debian-11"
    }
  }
  network_interface {
    network = "default"
  }
}

This configuration:

It gets the availability zones on the us-west1 region.
Then it creates one VM per zone using a for_each loop, dynamically setting the zone and the instance name.

This makes it easy to manage multiple resources without repeating yourself.

Best Practices When Using Data Sources

Data sources can be very helpful, but you must follow some best practices to make the most of them:

Use Specific Filters: Don’t be too broad in your filtering. Instead, use the most specific filters to get the exact data you want. This avoids unexpected results.
Avoid Unnecessary Data Sources: Only use data sources when you truly need to read data from the outside. Don’t overuse data sources for values you can easily define.
Understand Caching: Some data sources may cache data. Understand if the data source caches, and how long the data is stored. It may affect how fresh the data you retrieve is.
Be Aware of API Limits: Data sources can make API calls. Be mindful of the API limits for the provider you are using to avoid any rate-limiting issues.
Error Handling: Plan for errors when using data sources. Use the error handling methods available in the terraform language to manage situations when a data source can’t retrieve the data.
Keep it Readable: Choose descriptive names for your data sources and format your code well. It will help anyone reading your code to understand it better.
Security: Use data sources in a way that protects sensitive data. Avoid exposing secrets or sensitive information through data source outputs.
Documentation: Always check the official documentation for each provider to know how to use them in the most secure and reliable way possible.
Testing: Test data sources thoroughly in a non-prod environment. This helps to catch and fix any issue before it affects your production environment.

By following these practices, you can make your Terraform configurations more efficient, reliable, and secure.

How Data Sources Enhance Your Infrastructure Management

Data sources are a vital tool in the Terraform ecosystem. They can dynamically fetch external data and adapt to changes.

Maintainability: They reduce hardcoding. Your code is easier to maintain as it won’t break with external changes.
Scalability: They make your modules more reusable. You can deploy them to multiple environments without hardcoding specific details.
Flexibility: They allow your infrastructure to interact with resources that were created outside of Terraform. They also let your code adapt to changes in existing resources.
Efficiency: Data sources provide a clean way to read info about external resources. They prevent you from having to use manual steps to find IDs and other details.

By using data sources effectively, you make your infrastructure management more efficient and flexible.

Embracing Data Sources for Dynamic Infrastructure

Data sources are the unsung heroes of Terraform. They let you build infrastructure that adapts to real-world changes. They also reduce the need to manually hardcode information into your configurations. As you have learned, data sources can read from cloud provider APIs, external systems, and even other Terraform states. This makes your infrastructure more dynamic, reusable, and easier to manage.

From fetching the latest AMI IDs to reading VPC configurations, data sources are an essential tool for any Terraform user. As you become more skilled with them, you’ll find yourself building more sophisticated and maintainable infrastructure. Start by experimenting with the simple examples in this guide, and you’ll be on your way to mastering this powerful Terraform feature. Don’t be afraid to dive into the docs, and play around with different options. Your future self will thank you for it, as you make your infrastructure more dynamic and easy to manage.

Terraform Data Sources: Examples

What Are Terraform Data Sources?

Why Use Data Sources?

Basic Syntax of a Data Source

Common Data Sources and Examples

AWS Data Sources

aws_ami

aws_vpc

aws_security_group

Azure Data Sources

azurerm_resource_group

azurerm_virtual_network

azurerm_key_vault

Google Cloud Platform Data Sources

google_compute_image

google_compute_network

google_project

More Complex Use Cases

Using Filters

Chaining Data Sources

Data Sources in Loops

Best Practices When Using Data Sources

How Data Sources Enhance Your Infrastructure Management

Embracing Data Sources for Dynamic Infrastructure

Related Posts

Terraform Import: Existing Resources

Ultimate Guide to Azure Pipelines

`aws_ami`

`aws_vpc`

`aws_security_group`

`azurerm_resource_group`

`azurerm_virtual_network`

`azurerm_key_vault`

`google_compute_image`

`google_compute_network`

`google_project`