Proxmox+Packer+Terraform+Ansible

Automating the continuous deployment of a virtual pentest machine using Proxmox, Packer, Terraform, Ansible and GitLab.

What on earth?

Setting up a new pentest VM for every project is tedious, time-consuming and error-prone. Thus, I've set out to build from scratch an automation that will:

  • Provision a release of Kali as VM template (Packer - IaC)

  • Provision a staging and production version (Terraform - IaC)

  • Modify the VM via simple changes in GitLab (GitLab + Ansible - CI/CD Pipeline)

  • Do all that on a Proxmox server

To demonstrate the minimum setup and combination of Packer, Terraform, Gitlab and Ansible, I've documented the initial setup. Please keep in mind, that some best-practices, such as using a Vault and secure credentials have been skipped for brevity. Should you choose to follow this guide, make sure to read official documentation for best practice recommendations!

In this guide, we won't look at how to install Proxmox or GitLab.

Overview

Use this chapter as a reference to keep in mind what components interact with eachother.

Set Up a Base VM

The base VM we will call gitlab-runner as it will perform all actions based on the instructions of a Pipeline that we are going to setup later. It is the one system that we will have to maintain manually. We could automate that too but that's out of scope.

  1. In Proxmox, create a new VM

    I will be using a Debian here but you are free to use whatever you want for your base VM. I will name it gitlab-runner and give it an ID of 420.

  2. Once the VM is ready, we will need to install a few things

That's it for the installation part. We will continue to make some changes and add things but I recommend you create a snapshot of that VM now, so you can rollback in case anything goes wrong.

Provision Kali with Packer

Packer is a tool for creating virtual images. Here, we will use it to automate downloading a Kali release from the official website, installing it in a VM and then creating a Proxmox template from it.

At the moment of writing, neither Packer nor Proxmox seem incredibly stable. Here, I am working with Proxmox 8.4.1, Packer 1.12.0 and the Packer Proxmox plugin 1.2.2. While many things work just perfectly, keep in mind that small or minor changes in either of these tools can break everything and result in headache-inducing rabbit holes of troubleshooting. On that note, I reported a bug in the Packer Proxmox plugin that you may encounter too.

  1. Before we dive into Packer, we want to create an API user in Proxmox that Packer (and later also Terraform) can use to orchestrate any changes, like creating and deleting VMs

    1. Navigate to Datacenter > Permissions > Roles

    2. Create a new role and call it APIProvision for example (the name cannot start with PVE)

    3. We want to assign permissions to this role, that the API only really requires:

    4. Next, navigate to Datacenter > Permissions > API Tokens and select Add

    5. Select a user (optimally it is a single-purpose API user but a normal account works too) and make sure that Privilege Separation is checked. This is especially important when using a normal account (or even root). When the checkbox is ticked, the API token will not inherit permissions from the user

    6. Set any token ID and click Add

    7. You will now see the complete token ID (i.e. <username>!<tokenname>) and the API secret - store both of them securely in your KeePass or elsewhere

    8. Lastly, go to Datacenter > Permissions and select Add

    9. Here, we merge the API token, the role, and a resource pool

      Select the created role, the created API token and the resource pool where you want to provision your systems - I will use a resource pool called Infrastructure that I had created under Permissions > Pools

If you loose your API key, you will have to create a new API token. Delete the previous one and reuse the role.

  1. Now we create our Kali Linix packer script kali.pkr.hcl

    The first item in the file defines the plugin we require to talk to the Proxmox API (documentation).

    Now we want to define some variables, so that we keep valuable secrets out of the code:

    Then follows the description of exactly what we want to build with that plugin. Consult the linked documentation for an explanation of each field.

Theory for the upcoming section: Preseeding is one way (of many) for automating OS installations. It consists of a file that contains all the answers to the questions we would otherwise see in a live install. Debian has some good documentation here.

Another method is cloud-init which provides a way of configuring basic system configuration such as credentials and IP configuration via a standard interface. Proxmox supports cloud-init and allows you to set these configurations easily via the GUI.

The default Kali ISO is perfect for customization with preseeding but does not come with cloud-init installed. For cloning and easy configuration later on, we do want cloud-init though. Thus, we install and activate it manually (alternatively, you may also experiment with the Kali Generic Cloud image).

  1. Now, as mentioned previously, we create a folder called boot-cfg in the current directory

  2. In that directory, we create the kali-preseed.cfg file, this one is based on the Kali example

  3. With this config we can change the hostname, username, password, install packages such as cloud-init, disable the swap partition (Don't get confused by the no_swap false option, it actually allows us to use no swap partition, see here.), etc.

    The d-i preseed/late_command is documented here and allows us to enable SSH and prepare sudo so that Packer can login via SSH once the setup is finished to perform final clean up tasks.

  4. With everything prepared, we must run the following command once

    This downloads the Proxmox plugin we specified in the beginning.

  5. From here on, we can use

That sums it up for Packer and Proxmox. Feel free to adapt any step to your needs - in the end you should see a success message like the following one and a new template in your Proxmox host.

Successful creation of a base template with Packer on Proxmox

Finally, you can verify that everything is working by cloning the template in the Proxmox Web interface. Before you start the new machine, switch to the Cloud-Init tab and set a username and password. Subsequently, press on Regenerate Image. Now you are ready to start the VM and login with the credentials you just have set. Note that cloud-init does alot of things for us as boot time, including changing the hostname to the name of the new virtual machine.

For testing purposes, clone the template manually and configure a user via cloud-init, then start the VM

You could configure the behaviour of cloud-init by modifying the file /etc/cloud/cloud.dfg but that is out of scope for this series. By default, it will add the specified user as the only user and grant it root permissions.

Deploy Staging and Production VM with Terraform

Having a template available, we now want to clone it to create a staging and production machine for our CI/CD pipeline. The staging VM will be used for feature and integration tests while the production VM is going to be the one we can use as base image to deploy during engagements.

For this we are going to use Terraform. Terraform is tool that we can use to declare the systems that we want to have in our infrastructure. Based on that, it will use the Proxmox API to build the environment.

  1. We start by creating a terraform directory on the gitlab-runner machine.

  2. Switch to the directory and create a kali-provision.tf

    The first part may look familiar. It works almost similar to Packer.

    The next part describes the infrastructure we want to build, using the cloud-init we configured for the template. Refer to the plugins documentation here for details.

  3. Save the file and then execute terraform init once - similar to what we did for Packer, this will prepare the environment and download the specified plugin

  4. Run terraform validate to check for any errors

  5. Then run terraform plan to check what Terraform plans to do next

  6. If all looks good (in our example, 2 resources should be created, none modified, none deleted), then we can run terraform apply (if you want to revert the VMs, use terraform destroy)

  7. Confirm you actions with Yes and wait a few minutes while Terraform instantiates the VMs

After a short while we should see two new VMs

That's already it for the Terraform part. Go ahead, login and test whatever requirements you have for your staging and production VM. Here, I am fine with the base installation and cloud-init being active. Next, I want to make sure that I can run Ansible playbooks against the VMs.

Configure the VMs via Ansible

Now would be a great time to take a snapshot of the gitlab-runner VM.

So far we've automated the provision of the infrastructure. Next, we want to automate the customization of our Kali. This may include custom programs, files, UI changes, you name it.

For this, we are going to use Ansible as it lets us define the exact state that we want our Kali to have without having to worry about scripting all of it with Bash and Python.

  1. Start by creating an ansible directory and navigate to it

  2. Create two directories: group_vars and inventory

  3. We want to create the following structure:

  4. First, the ansible.cfg

  5. Then the main playbook: kali.yml

  6. Next we want to add the SSH username and password as variables to the group_vars/all.yml but before we do that, we encrypt the password with ansible-vault encrypt_string 'kali' --name 'ansible_password'

    Copy the output and open group_vars/all.yml

  7. Finally, we define the inventory, where we specify the targets to run our playbook against

    and

  8. With this, we have configured a very basic Ansible setup. I highly encourage you to take a look at the documentation of ansible. While our playbok really only connects to the target and prints a message, we now have a base to include arbitrary Ansible roles easily. We also added some basic SSH optimizations and showcased the Ansible vault.

  9. Run echo kali | ansible-playbook --vault-password-file /bin/cat -i inventory/01-staging.yml kali.yml

    Using /bin/cat is a trick to accept the password from the command like. Alternatively, use a password file. Later, we will replace the password on the command line with a more secure option.

You should now have a functioning Ansible setup. We can customize both the staging and prod machine via the respective inventory file.

Finally, what's left is the automation and deployment via GitLab and CI/CD pipelines.

Create a GitLab Pipeline

GitLab pipelines can be configured to run any job you want on specific conditions (like a merge request, a commit, or even manually). Here, I simply demonstrate how to utilize the pipeline feature to automate all our previous steps. Checkout the documentation for more details to customize the pipeline for your needs.

Pipelines in GitLab are created by creating/editing the .gitlab-ci.yml file. You can do so by navigating to Build > Pipeline editor - on the left side of the GitLab menu.

A very straight forward pipeline may look like this:

With this basic pipeline configuration in place, we can attempt to run our pipeline automatically via a GitLab runner.

Note that, since credentials are still hardcoded, we have not yet uploaded all these files to GitLab. That would be our next step after removing all sensitive information. We could then edit all the scripts in Git and trigger a pipeline with our changes. Here, we skip that and use the static code on gitlab-runner.

Create a GitLab Runner

  1. In GitLab, navigate to Settings > CI/CD > Runners and select New project runner

  2. Select a tag if you like (here I've used debian to indicate a runner that runs on debian)

  3. Set a description and "Lock to current projects"

  4. Lastly, click on Create runner

  5. On the next page, select the operating system of the runner (here Linux) and follow the steps 1 to 3 as they are described on the page

  6. Once the gitlab-runner is running, it is ready to accept jobs from the pipeline

    You may want to experiment with the option to start the runner as a service instead of enabling it manually every time.

And that concludes this minimalistic guide to Proxmox + Packer + Terraform + Ansible + GitLab. As I have announced at the top, do pay attention to all the secrets floating around as plaintext and move them to appropriate keystores (using GitLab secrets for example)

Enjoy building your own automation pipeline!

To test everything, you can go to Build > Pipelines in GitLab and press New pipeline at the top right corner.

Successful pipeline
PreviousEmbedded Firmware ExtractionNextCTF - BugBase - RaaS

Last updated

Was this helpful?