search

CI/CD for a Pentest VM

Automating the continuous deployment of a virtual pentest machine using Proxmox, Packer, Terraform, Ansible and GitLab.

hashtag
What on earth?

Setting up a new pentest VM for every project is tedious, time-consuming and error-prone. Thus, I've set out to build from scratch an automation that will:

  • Provision a release of Kali as VM template (Packer - IaC)

  • Provision a staging and production version (Terraform - IaC)

  • Modify the VM via simple changes in GitLab (GitLab + Ansible - CI/CD Pipeline)

  • Do all that on a Proxmox server

I started out documenting the initial minimum setup and combination of Packer, Terraform, Gitlab and Ansible. Over time I've repeatedly improved upon some components and updated this blog accordingly. For example, the first version used hardcoded credentials and had not much functionality in the pipeline.

By now, the examples include a complete end-to-end pipeline that fetches the latest Kali release, configures it, runs Ansible on it and releases it as a VMware-ready virtual machine via SFTP.

As Proxmox (and other software) will continue to evolve, examples will break over time but the architecture and overall setup should remain valuable for you as an orientation. Though I do not cover every consideration, this guide will provide you with all the basics required to build your own automation pipeline.

Should you choose to follow this guide, make sure to always read official documentation for up-to-date and best practice recommendations!

circle-info

In this guide, we won't look at how to install Proxmox or GitLab.

hashtag
Overview

circle-info

Use this chapter as a reference to keep in mind what components interact with eachother.

hashtag
Set Up a Base VM (Gitlab Runner)

circle-info

The base VM we will call gitlab-runner as it will perform all actions based on the instructions of a Pipeline that we are going to setup later. It is the one system that we will have to maintain manually. We could automate that too but that's out of scope for now.

  1. In Proxmox, create a new VM (top right corner "Create VM") inside your favourite resource pool

    I will be using a Debianarrow-up-right here but you are free to use whatever you want for your base VM. I will name it gitlab-runner and give it an ID of 420.

    1. Supply enough disk space to allow conversion of VM backups later (~200 GB will suffice)

    2. 8192 MiB RAM suffice (though more is better)

    3. 4 sockets, 4 cores

    4. Choose a network config that will provide your VM with internet access and access to GitLab

      Beware of proxy setups - this can cause you a serious amount of trouble later on. It's possible, but a pain to maintain, trust me. Also, packet inspection may flag and block Kali/Tool downloads, so take that into account early during setup.

  2. Next, complete the ISO installation setup

    1. The hostname will be gitlab-runner

    2. Do not add a root user (leave the password blank to disable it)

    3. Add a normal user (I will call it user) and set a strong password

      Consider setting up a KeePass database for this project now because you will create more credentials and secrets.

  3. Once the VM is ready, we will need to install a few things

circle-check

That's it for the installation part. We will continue to make some changes and add things but I recommend you create a snapshot of that VM now, so you can rollback in case anything goes wrong.

hashtag
Additional Base VM Preparations

Right now we've laid the groundwork for a VM that will run our GitLab pipeline jobs. However, we need a few more things for later.

  1. Certificates

    In case you host your own GitLab instance, consider setting up a proper certificate and install the CA on the gitlab-runner:

  2. Configure Proxmox with a proper backup storage I will not cover the setup of a new storage. You require a storage that's large enough to hold at least one or two VM backups (200 GB will suffice). In Proxmox -> Server View -> Datacenter -> Storage, look for:

    1. Content must contain Backup

    2. Enabled must say Yes

    3. Note down the Path/Target value

  3. Then we configure a "Directory Mapping" that will mirror the backup directory into the runner VM

    In Proxmox -> Server View -> Datacenter -> Directory Mappings -> Add:

    1. Name: chose any

    2. Path: this must match the name of the backup storage from the step before

  4. Next we add a Virtiofs to our gitlab-runner to include that directory mapping

    1. Make sure to set Cache to never or you may experience disk exhaustion over time

  5. Restart the runner VM to activate the new Virtiofs

  6. Now we configure this backup directory and also add a second directory that will serve us for hosting purposes (it will contain the latest pentest vm release)

  7. To avoid the sftp user from being abused, we make sure that the SSH config in /etc/ssh/sshd_config contains the following lines. Only use the first line if you deleted the sftp user password intentionally (like in the example above).

  8. Run sudo systemctl restart ssh to enable the changes

    Now, only sftp will be able to connect with sftp sftp@<gitlab-runner-ip> and nobody else

  9. Another thing that you may or may not benefit from: Setting a static IP address for the gitlab-runner VM. This way you make sure that downloading the latest release is always possible with the same command later. If you got DNS setup correctly, you would not need this and work with hostnames. To configure a static IP, edit /etc/network/interfaces:

  10. Activate changes with sudo reboot now and confirm with ip a

hashtag
Provision Kali with Packer

Packer is a tool for creating virtual images. Here, we will use it to automate downloading a Kali release from the official website, installing it in a VM and then creating a Proxmox template from it.

circle-exclamation

At the moment of writing, neither Packer nor Proxmox seem incredibly stable. Here, I am working with Proxmox 8.4.1 (edit: guide updated for 9.1.4), Packer 1.12.0 (edit: guide updated for 1.14.3) and the Packer Proxmox plugin 1.2.2 (edit, guide updated for 1.2.3). While many things work just perfectly, keep in mind that small or minor changes in either of these tools can break everything and result in headache-inducing rabbit holes of troubleshooting. On that note, I reported a bug in the Packer Proxmox pluginarrow-up-right that you may encounter too.

  1. Before we dive into Packer, we want to create an API user in Proxmox that Packer (and later also Terraform) can use to orchestrate any changes, like creating and deleting VMs

    1. Navigate to Datacenter > Permissions > Roles

    2. Create a new role and call it APIProvision for example (the name cannot start with PVE)

    3. We want to assign permissions to this role, that the API only really requires:

    4. Next, create a user under Datacenter > Permissions > Users > Add (with no groups) and store the password in your KeePass - you should not see/use this password ever again.

    5. Next, navigate to Datacenter > Permissions > API Tokens and select Add

    6. Select the user we created and make sure that Privilege Separation is checked. When the checkbox is ticked, the API token will not inherit permissions from the user.

    7. Set any token ID and click Add

    8. You will now see the complete token ID (i.e. <username>!<tokenname>) and the API secret - store both of them securely in your KeePass

    9. Lastly, go to Datacenter > Permissions and select Add

    10. Here, we merge the API token, the role, and a resource pool

      Select the created role, the created API token and the resource pool where you want to provision your systems - I will use a resource pool called Infrastructure that I had created under Permissions > Pools

circle-info

If you loose your API key, you will have to create a new API token. Delete the previous one and reuse the role.

  1. Now we create our Kali Linix packer script kali.pkr.hcl

    The first item in the file defines the plugin we require to talk to the Proxmox API (documentationarrow-up-right).

    Now we want to define some variables, so that we keep valuable secrets out of the code:

    Then follows the description of exactly what we want to build with that plugin. Consult the linked documentation for an explanation of each field (these fields may change over time).

circle-info

Theory for the upcoming section: Preseeding is one way (of many) for automating OS installations. It consists of a file that contains all the answers to the questions we would otherwise see in a live install. Debian has some good documentation herearrow-up-right.

Another method is cloud-init which provides a way of configuring basic system configuration such as credentials and IP configuration via a standard interface. Proxmox supports cloud-init and allows you to set these configurations easily via the GUI.

The default Kali ISO is perfect for customization with preseeding but does not come with cloud-init installed. For cloning and easy configuration later on, we do want cloud-init though. Thus, we install and activate it manually (alternatively, you may also experiment with the Kali Generic Cloud image).

  1. Now, as mentioned previously, we create a folder called boot-cfg in the current directory

  2. In that directory, we create the kali-preseed.cfg file, this one is based on the Kali examplearrow-up-right

  3. With this config we can change the hostname, username, password, install packages such as cloud-init, activate passwordless SSH for the Packer script etc.

    The d-i preseed/late_command is documented herearrow-up-right and allows us to enable SSH and prepare sudo so that Packer can login via SSH once the setup is finished to perform final clean up tasks.

  4. With everything prepared, we must run the following command once

    This downloads the Proxmox plugin we specified in the beginning.

  5. From here on, we can use

circle-check

That sums it up for Packer and Proxmox. Feel free to adapt any step to your needs - in the end you should see a success message like the following one and a new template in your Proxmox host.

Successful creation of a base template with Packer on Proxmox

Finally, you can verify that everything is working by cloning the succesfully crafted template in the Proxmox Web interface. Subsequently, before you start the newly created machine, switch to the cloud-init tab of that VM and set a username and password. Then press on Regenerate Image. Now you are ready to start the VM and login with the credentials you just have set. Note that cloud-init does alot of things for us as boot time, including changing the hostname to the name of the new virtual machine. Just keep this in mind if you later try to change things via Ansible that cloud-init tries to manage (you can use Ansible to disable parts of cloud-init later on).

For testing purposes, clone the template manually and configure a user via cloud-init, then start the VM

You could configure the behaviour of cloud-init by modifying the file /etc/cloud/cloud.dfg but that is out of scope for this blog. By default, it will add the specified user as the only user and grant it root permissions.

hashtag
Deploy Staging and Production VMs with Terraform

Having a template available, we now want to clone it to create a staging and production machine for our CI/CD pipeline. The staging VM will be used for integration tests while the production VM is going to be the one we can use as base image to deploy during engagements.

For this we are going to use Terraform. Terraform is tool that we can use to declare the systems that we want to have in our infrastructure. Based on that, it will use the Proxmox API to build the environment.

  1. We start by creating a terraform directory on the gitlab-runner machine.

  2. Switch to the directory and create a kali-provision.tf

    The first part may look familiar. It works almost similar to Packer.

    The next part describes the infrastructure we want to build, using the cloud-init we configured for the template. Refer to the plugins documentation herearrow-up-right for details.

  3. Save the file and then execute terraform init once - similar to what we did for Packer, this will prepare the environment and download the specified plugin

  4. Run terraform validate to check for any errors

circle-info

Keep in mind that we used variables in our Terrafrom script (similar to the Packer script). So for the following terraform <action> commands, you should add these command line parameters:

  1. Then run terraform plan to check what Terraform plans to do next

  2. If all looks good (in our example, 2 resources should be created, none modified, none deleted), then we can run terraform apply (if you want to revert the VMs, use terraform destroy)

  3. Confirm you actions with Yes and wait a few minutes while Terraform instantiates the VMs

After a short while we should see two new VMs
circle-check

That's already it for the Terraform part. Go ahead, login and test whatever requirements you have for your staging and production VM. Here, I am fine with the base installation and cloud-init being active. Next, I want to make sure that I can run Ansible playbooks against the VMs.

hashtag
Configure the VMs via Ansible

circle-info

Now would be a great time to take a snapshot of the gitlab-runner VM.

So far we've automated the provision of the infrastructure. Next, we want to automate the customization of our Kali. This may include custom programs, files, UI changes, you name it.

For this, we are going to use Ansible as it lets us define the exact state that we want our Kali to have without having to worry about scripting all of it with Bash and Python.

You can follow the next steps on the gitlab-runner VM for testing purposes. Later of course, this will have to go into our GitLab repository.

  1. Start by creating an ansible directory and navigate to it

  2. Create three directories: group_vars, roles and inventory

  3. We want to create the following structure:

  4. First, the ansible.cfg

  5. Then the .ansible-lint (configures linter rules)

  6. Then the main playbook: kali.yml

  7. You can see that in the main playbook we gather all the roles that should be executed. Roles are automatically taken from the roles directory. Inside the roles directory, we use a requirements.yml that will let Ansible know, what collections should be used. It is not strictly required, but a good practice. It will be referenced later in the pipeline. For example:

  8. A single role inside the roles directory will look like this:

  9. For test purposes you can create a role directory with this tasks/main.yml (ignore the other directories and files as they are all optional):

  10. Next we want to add the SSH username and password as variables to the group_vars/all.yml but before we do that, we encrypt the password with ansible-vault encrypt_string 'kali' --name 'ansible_password' This way we leverage the Ansible vault. It allows us to include encrypted secrets in our code, so we do not have to place cleartext credentials in the files. Encrypted variables will be decrypted at runtime by Ansible.

    Copy the output and open group_vars/all.yml:

  11. Finally, we define the inventory, where we specify the targets to run our playbook against

  12. With this, we have configured a very basic Ansible setup. I highly encourage you to take a look at the documentationarrow-up-right of Ansible. While our playbok only executes a single role that installs an application, we now have a base to include arbitrary Ansible roles easily. We also added some basic SSH optimizations and showcased the Ansible vault.

  13. Run echo kali | ansible-playbook --vault-password-file /bin/cat --limit kali-prod kali.yml

    Using /bin/cat is a trick to accept the password from the command like. Alternatively, use a password file. Later, we will replace the password on the command line with a more secure option.

  14. If the playbook succeeds, it should run the my-first-role role against the previously generated kali-prod and when you login to the VM via Proxmox, htop should now be installed.

circle-check

You should now have a functioning Ansible setup. When everything works locally on the gitlab-runner VM, get ready to move all of the prior steps to CI/CD.

Finally, what's left is the automation and deployment via GitLab and CI/CD pipelines.

hashtag
Create a GitLab Pipeline

GitLab pipelines can be configured to run any job you want on specific conditions (like a merge request, a commit, or even manually). Here, we create two pipelines that will automate all of the above.

Checkout the documentationarrow-up-right for more details to customize the pipeline for your needs.

Pipelines in GitLab are created by creating/editing the .gitlab-ci.yml file. You can do so by navigating to Build > Pipeline editor - on the left side of the GitLab menu.

What follows is a complete pipeline example, that reacts to two triggers:

  • Scheduled Pipeline Execution:

    Whenever a scheduled pipeline (explained later) is run, the following stages and steps will be performed:

    • Query the Kali release page for the latest ISO file (and checksum)

    • Download the ISO and run Packer to create a Proxmox template from it

    • Build multiple VMs from that template using Terraform (at least a production and staging VM)

    • Create VM snapshots for all (non-production) VMs using the Proxmox API (this will allow you to reset the VMs again and again when you want to test new roles/features)

    • Execute the Ansible playbook against the production VM

    • Create a snapshot of the production VM (this snapshot can be used to rollback the production VM to the current clean release state - but usually, you do not use the production VM directly, see next step)

    • Backup the production VM and convert the Proxmox backup-format to another virtual machine format (using qemu-img convert) before moving it to the release directory

    • In between all steps, the Proxmox API is used via curl to query and manipulate the VM status

  • Merge Requests: This pipeline will always execute when a merge request is created (for example from a feature branch against the dev branch, or from dev against main):

    • Run ansible-lint on all Ansible code

    • Run the complete Ansible playbook against the staging VM

    • Should either of these steps fail, the merge requests gets blocked (this way we can ensure, that all changes introduced to our release do not break the existing Ansible code)

This pipeline is of course way bigger than a minimum working example. If you are just getting into CI/CD, I suggest you start with only the run_ansible_on_prod stage. It is the most straight forward one and simply automates what we've done manually before. If you got time to work through the entire setup, here is the directory setup that you should have in your GitLab Repository by now:

Below you will find the contents of all the scripts referenced in the .gitlab-ci.yml. We place the logic in separate files inside the ci directory to keep the actual pipeline configuration as clean as possible. You will notice, that the scripts contain much more than just packer init and terraform apply. However, don't let additional logic distract you. Much of what you will find in these scripts is curl commands that talk to the Proxmox API, starting/stopping VMS, creating snapshots and waiting for Proxmox jobs to finish. Most of it is "nice-to-have". When you boil it down, these scripts execute exactly what we have run before on the command line to execute packer, terraform and ansible.

With this pipeline configuration in place, we can attempt to run our pipeline automatically via a GitLab runner.

Note that we referenced several GitLab variables. To let the pipeline run successfully, you must go to Settings > CI/CD and add project variables to your repository. Namely:

  • GL_PROXMOX_API_USER (can be visible)

  • GL_PROXMOX_API_TOKEN (masked and hidden)

  • GL_ANSIBLE_VAULT (masked and hidden)

Next, we configure the runner that will run this pipeline for us.

hashtag
Create a GitLab Runner

  1. In GitLab, navigate to Settings > CI/CD > Runners and select New project runner

  2. Select a tag if you like (here I've skipped it because I only use one runner)

  3. Set a description and "Lock to current projects"

  4. Configure a global timeout (8 hours are more than enough for all our pipeline needs here)

  5. Lastly, click on Create runner

  6. On the next page, select the operating system of the runner (here Linux) and follow the steps 1 to 3 as they are described on the page

    1. Note that on step 1 you should use sudo for the command!

    2. Use shell in step 2

    3. Ignore step 3

  7. Once the gitlab-runner is running, it is ready to accept jobs from the pipeline

hashtag
The End

And that concludes this guide to Proxmox + Packer + Terraform + Ansible + GitLab.

circle-check

To test everything, you can go to Build > Pipeline schedules in GitLab and press New pipeline at the top right corner.

The image is an example of an earlier version of this blog post, where I only used three stages.

Successful pipeline

I hope this blog served you as a well rounded introduction to all these different technologies and gives you a good overview of how they can all play together to form a fully automated pipeline.

Enjoy building your own automation pipeline!

PreviousEmbedded Firmware ExtractionNextCTF - BugBase - RaaS

Last updated