Proxmox and Terraform¶

These are my notes on using Terraform to provision VMs in Proxmox.

Hashicorp Language (HCL)¶

Terraform uses Hashicorp Language (HCL) to define Infrastructure as Code (IaC).

There are a decent number of other tools that also use HCL:

Tool	Main Use of HCL
Terraform	Infrastructure as code (mainstream)
Packer	Machine image build templates
Vault	Secrets access policy management
Nomad	Job and workload scheduling
Consul	Service network configuration
Boundary	Secure remote access configuration
OpenTofu	Terraform-compatible IaC

So HCL is definitely a good thing to learn, since it's portable across all these different tools.

Terraform Setup¶

Quick install of Terraform on my main Proxmox node (home-pve).

Recommended installation method from the Terraform installation guide:

wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

I wrote my own custom installer script for portability, to work on both Debian-based and RedHat-based systems -- mainly distros that are binary-compatible with RHEL (Rocky, Alma).

Terraform Authetication on Proxmox¶

Terraform will need root access to the Proxmox server.

Assuming the Proxmox server is already set up, We can set up authentication for the Terraform user in two ways:

Username/password auth:
- Use the root user's username/password.
API key auth:
- Use a separate username with an API key.

If we want to use API keys, we can set up a separate Terraform user and give it permissions to create VMs using the Proxmox API.

API Key Auth¶

To use an API key for authentication with a dedicated user account, we'll need to create a new user, then generate an API key associated with the new user account.

Create a New User¶

We're opting for the API authentication method here.

There are two main ways to create a new user and generate an API key.

Using the Proxmox Web UI
Using the command line with pveum (Proxmox VE user management tool)

Using the Web UI¶

In order to set up a new user account and gnerate an API token, we can go through the Proxmox Web UI.
That will be the IP of your Proxmox server at port 8006 (e.g., http://192.168.1.49:8006).

Once there, navigate to the "Datacenter" tab on the left, then find the "Permissions" dropdown on the left.

From there, create a user:

"Users" tab
- Add (top left of center panel)
  - Create the user account (e.g., terraform)

Once the user is created, go to the "API Tokens" tab and generate a new API key:

"API Tokens" tab
- Add (top left of the center panel)
  - Generate an API key linked to the new user terraform

Save that API key in a secure place, as it will be used in the Terraform files.

Using the CLI¶

I prefer to do things from the CLI, so this is the approach I took.

We can utilize the pveum (Proxmox VE User Management) tool to create a new user and assign it a new API key.

Create the new user (in the PVE realm, for API key compatibility).
```
pveum user add terraform@pve
```
- We can also verify the realm name before adding if we want:
```
pveum realm list
```
  We don't use the pam realm because we can't create API tokens for the user. PAM is password-based auth only, so it's not ideal for automation and app integrations.
Verify that the user account has been created.
```
pveum user list
```
Set a password for the user.
```
pveuser passwd terraform@pve
```
- This can be done with the --password option in pveum user add, however I don't encourage people to pass passwords as command line arguments. Those arguments live in shell history.
Generate an API token (decide on privilege separation or not).
```
pveum user token add terraform@pve TOKEN_ID
# or
pveum user token add terraform@pve TOKEN_ID --privsep 0
```
The <TOKEN_ID> is going to be a user-specified token identifier.
This ID is basically just a name for your API key.
I'll name mine tf-token.
```
pveum user token add terraform@pve tf-token
```
- This will output a table containing the full-tokenid, info, and the value (the token itself). The full-tokenid follows the pattern user@realm!token-id, so in my case terraform@pve!tf-token.
- Privilege separation will require us to configure the ACL by adding a role to the token itself. If it's on (default), the token will inherit the privileges of the user it belongs to.
Verify the token has been created.
```
pveum user token list terraform@pve
```

Once the user account is created and has an API key, we can move forward to configuring the Terraform Proxmox provider.

Configure ACL¶

In order for the token and user to have permissions, we need to add some roles via the ACL.

The easiest way to do this is to grant the Administrator role.

We can add a role with pveum acl modify.

sudo pveum acl modify / -user terraform@pve -role Administrator

If privilege separation is enabled on the API token generated earlier, we must also add the role to the token.

sudo pveum acl modify / -token terraform@pve!tf-token -role Administrator

Configure Terraform Provider¶

Terraform has this concept of "providers."

Providers are essentially plugins that allow Terraform talk to an external system, usually via that system's API.

Each provider is programmed to interact with specific APIs (e.g., AWS, Proxmox), and exposes Terraform resources that correspond to that system.

Providers are downloaded from the Terraform registry

How Terraform providers work

When we run terraform init, it reads the .tf files and detects which providers are being used, then downloads and installs them into the .terraform/providers directory.

Then, when we run terraform plan or terraform apply, it will load each provider's plugin binary, passes in the config (via RPC), and the provider makes the appropriate calls to the API. The provider will report back what changed (or what will change).

All that to say, Terraform has a Proxmox provider.

So, we'd create a directory for our Terraform project.

mkdir pve-tf && cd pve-tf

Then we'd make a main.tf file to define the provider we want to use.

touch main.tf
vi main.tf

Inside the main.tf file, we'd add:

terraform {
  required_providers {
    proxmox = {
      source  = "Telmate/proxmox"
      version = "3.0.2-rc04"
    }
  }
}

This specifies the provider we want to use. We also specify a version alongside the source to use a specific version of the provider.

Next, we configure the provider.

provider "proxmox" {
  pm_api_url  = "https://192.168.1.49:8006/api2/json"
  pm_user     = "terraform"
  pm_password = "api-key-goes-here"
  pm_tls_insecure = true
}

This configures the Proxmox provider (Telmate/proxmox), and specifies the necessary information for Terraform to interact with the Proxmox API.

These three things are essential:

pm_api_url: The PVE API endpoint
pm_user: The Proxmox user
pm_password: The Proxmox user's password

Setting user and password as environment variables

The user and password can be left out of the provider definition as long as these are set as environment variables:

export PM_USER="terraform@pve"
export PM_PASS="api-key-goes-here"

Once the provider is configured, we'd add the new resource that we want to create, also in main.tf.

resource "proxmox_vm_qemu" "new_vm" {
  name        = "new_vm"
  target_node = "home-pve"
  clone       = "ubuntu-template"
  storage     = "vmdata"
  cores       = 2
  memory      = 2048
}

This specifies a single VM to create. In this case, a clone of a pre-existing VM or template named ubuntu-template.
Any additional VMs would be another resource entry, all with their own configurations.

Here, we're using a proxmox_vm_qemu resource, which will create a new VM by either cloning an existing VM/template, or by creating a new one from an ISO file.

Using a clone or ISO is less granular than using CloudInit.
CloudInit will allow us to configure the hostname, set up SSH keys, and perform other system configurations that need to be done before the VM is ready for general use.

Once we have the provider and our resource set up in main.tf, we can go ahead and do terraform init to intialize the provider.

terraform init

Then, we can do a plan to see the changes that will be made.

terraform plan

Finally, we apply to actually do the magic.

terraform apply

Hiding the API Key¶

So, storing API keys in plain text is usually not great practice.

Using Environment Variables¶

One very common method of avoiding plaintext secrets is using environment variables.

Using Boilerplate Environment Variables¶

Terraform will look for these variables when utilizing the proxmox provider without a pm_user and pm_password set in the provider definition:

PM_USER
PM_PASS
PM_API_TOKEN_ID
PM_API_TOKEN_SECRET

Set these as environment variables in either .bashrc, or in a .env file and then run source .env to make sure they're in your current shell environment.

This way, the API key can be set in a separate file without specifying the username and password in the main.tf.

export PM_USER='terraform@pve'
export PM_PASS='password-here'
export PM_API_TOKEN_ID='terraform@pve!tf-token'
export PM_API_TOKEN_SECRET="afcd8f45-acc1-4d0f-bb12-a70b0777ec11"

Using Custom Variables¶

We can define variables in a variables.tf file (or just in main.tf).
We'll make a variable for the API endpoint, the token ID (name), as well as the API key itself.

variables.tf

variable "proxmox_api_url" {
    description = "Proxmox API endpoint"
    type        = string
}

variable "proxmox_api_token_id" {
    description = "Proxmox API token ID"
    type        = string
}

variable "proxmox_api_token_secret" {
    description = "Proxmox API token secret (API key itself)"
    type        = string
    sensitive   = true
}

Then, we'd reference these variables inside the provider block.

We access variables by simply using var.<VAR_NAME>.

provider "proxmox" {
    pm_api_url          = var.proxmox_api_url
    pm_api_token_id     = var.proxmox_api_token_id
    pm_api_token_secret = var.proxmox_api_token_secret
    pm_tls_insecure     = true
}

Finally, we'd need to set the environment variables.
Any environment variable that is prepended with TF_VAR_ is automatically picked up by Terraform.

export TF_VAR_promox_api_url="https://192.168.1.49:8006/api2/json"
export TF_VAR_promox_api_token_id="terraform@pve!tf-token"
export TF_VAR_promox_api_token_secret="SECRET_TOKEN"

We can set this in our .bashrc, or source it from some other environment file (e.g., in a .env file).

Once those are set and sourced, we're all set with setting up the environment variables for our secrets.

Using Vault¶

Hashicorp Vault is the gold standard for hiding secrets used with Terraform.

Info

The following assumes a Vault dev server is already set up in the environment.
Quick overview:

Install via the Hashicorp Vault Installation Guide

Start a vault server and set the necessary variables

vault server -dev & # Start dev server and run in background
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<ROOT_TOKEN_HERE>' # root token from

With the Vault server running, you'd set the secrets:

vault kv put secret/proxmox/terraform \
    url="https://192.168.1.49:8006/api2/json" \
    token_id="terraform@pve!tf-token" \
    token_secret="TOKEN_SECRET"

Then, we'd need to specify the vault provider in our main.tf.

provider "vault" {
    address = "https://vault.example.com"
    token   = "your-root-or-app-token"
}

If we already set the VAULT_ADDR and VAULT_TOKEN environment variables, we can omit them in the provider.

provider "vault" {}

After specifying the vault provider, we can then access the data stored inside.

data "vault_kv_secret_v2" "pve_data" {
  mount = "secret"
  name  = "proxmox/terraform"
}

provider "proxmox" {
  pm_api_url          = data.vault_kv_secret_v2.pve_data.data["url"]
  pm_api_token_id     = data.vault_kv_secret_v2.pve_data.data["token_id"]
  pm_api_token_secret = data.vault_kv_secret_v2.pve_data.data["token_secret"]
  pm_tls_insecure     = true
}

We specify a data entry, then we can scope into each of the keys that we set earlier with vault kv put.

So, once it's all set up, the main.tf should look something like this:

provider "vault" {}

data "vault_kv_secret_v2" "pve_data" {
  mount = "secret"
  name  = "proxmox/terraform"
}

provider "proxmox" {
  pm_api_url          = data.vault_kv_secret_v2.pve_data.data["url"]
  pm_api_token_id     = data.vault_kv_secret_v2.pve_data.data["token_id"]
  pm_api_token_secret = data.vault_kv_secret_v2.pve_data.data["token_secret"]
  pm_tls_insecure     = true
}

Using CloudInit¶

Using CloudInit images with Proxmox offers extended levels of configuration and customization when provisioning VMs with Terraform.

In order to use Cloud-Init for Terraform, we typically need a VM built with a cloud image image that comes with Cloud-Init preinstalled. These are available for download for most major distributions (e.g., for Ubuntu Server.

We would:

Make a VM with a CloudInit drive (e.g., using qm).
Turn it into a Proxmox template (this will be our golden image).
Use Terraform to clone it and do the initial configuration.

In order to use CloudInit, we need to create a CloudInit drive.

The CloudInit drive is a special virtual disk, which is created automatically when we run:

qm set VMID --ide2 vmdata:cloudinit

This can be done using whatever storage you want, as long as you append :cloudinit to the end.

Creating a CloudInit Drive¶

We can create a CloudInit Drive by using the qm command line tool.

We'd start the same way as making a normal VM.

qm create 1400 --name 'rocky9-gencloud-template' \

    --memory 2048 \
    --net0 virtio,bridge=vmbr0 \
    --cores 1 \
    --sockets 1 \
    --cpu x86-64-v2-AES \
    --storage vmdata

qm importdisk 1400 /var/lib/vz/template/qcow/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 vmdata:vm-1400-disk-0
qm set 1400 --virtio0 vmdata:vm-1400-disk-0
qm set 1400 --boot c --bootdisk virtio0
qm set 1400 --ide2 vmdata:cloudinit
qm set 1400 --ciuser luser --cipassword luser
qm set 1400 --agent 1
qm template 1400

These are the command broken down:

qm create: This creates a new VM called rocky9-gencloud-template, and configures the network, memory, and CPU.
qm importdisk: Imports the downloaded Rocky Linux cloud image into the storage pool vmdata.
- Pulls from /var/lib/vz/template/qcow/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 and puts it into vmdata:vm-1400-disk-0.
qm set 1400 ...: Setting configuration values for the new VM with ID 1400.
- --virtio0 vmdata:vm-1400-disk-0: Attaches the imported disk as the VM's main boot disk using the VirtIO network interface.
- --boot c --bootdisk virtio0: Sets the boot order from the first hard disk (virtio0) .
- --ide2 vmdata:cloudinit: This adds a special CloudInit drive to IDE slot 2.
  - This disk acts as a metadata provider. CloudInit inside the guest reads from it during boot.
  - It's essentially a virtual drive that contains instructions for the VM's first boot configuration.
- --ciuser luser --cipassword luser: This defines the default CloudInit user credentials for the template (user: luser, pass: luser).
  - These values will be used if we don't override them later in Terraform.
- --agent 1: Enable the QEMU guest agent.
qm template 1400: Convert the new VM into a template.

Cloud-Init Snippets¶

Once the template is created, we can create snippets, which are used to pass additional configuration to the Cloud-Init package.

Note

Before wae can create a snippet, we need a place to store it. Best practice is to store snippets in the same location as the template.
Cloned VMs can't start if the snippet is not accessible.

Create a snippet directory:

mkdir /var/lib/vz/snippets

Create a snippet (this one will be used to ensure qemu-guest-agent is installed):

touch /var/lib/vz/snippets/qemu-guest-agent.yml
vi /var/lib/vz/snippets/qemu-guest-agent.yml

The file should be formatted as follows:

#cloud-config
runcmd:

  - apt-get update
  - apt-get install -y qemu-guest-agent
  - systemctl enable --now qemu-guest-agent

The qemu-guest-agent package will be installed and the daemon started.

Troubleshooting¶

When running terraform plan, if we get some sort of error, we will need to troubleshoot.

Token Privilege Error¶

For instance, I received this error:

│ Error: user terraform@pve has valid credentials but cannot retrieve user list, check privilege separation of api token
│
│   with provider["registry.terraform.io/telmate/proxmox"],
│   on main.tf line 9, in provider "proxmox":
│    9: provider "proxmox" {

This suggests that the API key that I'm using is improperly configured.

The privilege separation setting was enabled from that token, shown by the command:

sudo pveum user token list terraform@pve

There seems to be two options for fixing this problem:

Recreate the token with privilege separation disabled and give the user itself the permissions.
```
sudo pveum user token add terraform@pve TOKEN_ID --privsep 0
```

Configure ACLs for API tokens with privilege separation enabled.

sudo pveum acl modify / -token 'terraform@pve!tf-token' -role Administrator
# Verify:
sudo pveum acl list

I went for option 2.

The Administrator role seems to be required for provisioning VMs. I tried using the PVEAdmin role but that lacked the necessary Sys.Modify permission.

I had privilege separation enabled for my token, added the Administrator role,

When using privilege separation, the Administrator role needs to be added to both the user and the token.

If the user lacks the necessary privileges, the token privileges will not be enough.

Proxmox VM Qemu Resource¶

This is the resource that will be used to provision VMs on Proxmox.

There are several blocks that need to be specified on the later versions on the Telmate/proxmox provider.

Top Level Block
CPU Block
Network Block
Disk Block
- This block in particular has many sub-blocks that can be specified to further configure disk devices.

Without Using Cloud-Init¶

An example entry for this type of resource, a basic clone of a Proxmox VM template (no cloud-init):

resource "proxmox_vm_qemu" "test-tf-vm" {
  name        = "test-tf-vm"
  agent       = 1
  boot        = "order=scsi0"
  target_node = "home-pve"
  clone       = "ubuntu-22.04-template"
  disk {
    storage = "vmdata"
    size    = "16G"
    type    = "disk"
    slot    = "scsi0"
  }
  network {
    id     = 0
    model  = "virtio"
    bridge = "vmbr0"
  }
  cpu {
    cores = 1
  }
  memory = 4096
}

Using Cloud-Init¶

Some official examples of using the Telmate/proxmox provider with Cloud-Init can be found on the Telmate/proxmox GitHub and on the Terraform registry docs.

Steps taken:

# Pull down the qcow2 cloud init image into the appropriate directory
# Create directory and switch
mkdir /var/lib/vz/template/qcow && cd /var/lib/vz/template/qcow
# Download the image (can use wget too if you want)
curl -LO https://dl.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2

# Create the VM
qm create 9020 --name "rocky-10-cloudinit-template" --storage vmdata

# Create the config file if it doesn't exist
touch /etc/pve/qemu-server/9020.conf 

# Import the Cloud-Init image to the VM's disk (on scsi0 using vmdata)
# Can probably use `qm importdisk`
qm set 9020 --scsi0 vmdata:0,import-from=/var/lib/vz/template/qcow/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2

Example Terraform config for a VM using Cloud-Init:

resource "proxmox_vm_qemu" "cloudinit-test" {
  vmid        = 1400
  name        = "rocky-cloudinit-test-vm"
  target_node = "home-pve"
  agent       = 1
  cpu {
    cores     = 1
    sockets   = 1
  }

  memory      = 2048
  boot        = "order=scsi0" # has to be the same as the OS disk of the template
  clone       = "rocky-10-cloudinit-template" # The name of the template
  scsihw      = "virtio-scsi-single"
  vm_state    = "running"
  automatic_reboot = true

  # Cloud-Init configuration
  cicustom   = "vendor=local:snippets/qemu-guest-agent.yml" # /var/lib/vz/snippets/qemu-guest-agent.yml
  ciupgrade  = true
  nameserver = "1.1.1.1 8.8.8.8"
  ipconfig0  = "ip=192.168.1.10/24,gw=192.168.1.1,ip6=dhcp"
  skip_ipv6  = true
  ciuser     = "luser"
  cipassword = "luser"
  sshkeys    = <<EOF
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICGjGGUL4ld+JmvbDmQFu2XZrxEQio3IN7Yhgcir377t Optiplex Homelab key
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAQdazsCyvNGrXGT+zEc6l5X/JILWouFlnPchYsCeFZk kolkhis@home-pve
EOF

  # Most cloud-init images require a serial device for their display
  serial {
    id = 0
  }

  disks {
    scsi {
      scsi0 {
        # We have to specify the disk from our template, else Terraform will think it's not supposed to be there
        disk {
          storage = "local-lvm"
          # The size of the disk should be at least as big as the disk in the template. If it's smaller, the disk will be recreated
          size    = "10G" 
        }
      }
    }
    ide {
      # Some images require a cloud-init disk on the IDE controller, others on the SCSI or SATA controller
      ide1 {
        cloudinit {
          storage = "vmdata"
        }
      }
    }
  }

  network {
    id = 0
    bridge = "vmbr0"
    model  = "virtio"
  }
}

The custom Cloud-Init configurations are as follows:

 # Cloud-Init configuration
  cicustom   = "vendor=local:snippets/qemu-guest-agent.yml" # /var/lib/vz/snippets/qemu-guest-agent.yml
  ciupgrade  = true
  nameserver = "1.1.1.1 8.8.8.8"
  ipconfig0  = "ip=192.168.1.10/24,gw=192.168.1.1,ip6=dhcp"
  skip_ipv6  = true
  ciuser     = "root"
  cipassword = "Enter123!"
  sshkeys    = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE/Pjg7YXZ8Yau9heCc4YWxFlzhThnI+IhUx2hLJRxYE Cloud-Init@Terraform"

We can specify multiple SSH keys to add by using a heredoc:

  sshkeys    = <<EOF
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE/Pjg7YXZ8Yau9heCc4YWxFlzhThnI+IhUx2hLJRxYE example
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE/Pjg7YXZ8Yau9heCc4YWxFlzhThnI+IhUx2hLJRxYE example
EOF

The ciuser and cipass can be pre-configured when creating the template:

qm set 1400 --ciuser "luser" --cipassword "luser"

Resources¶

Cloud-Init resources:

Cloud-Init images: