Build Pipeline Security

This occurred on an AWS website (not a site hosted on AWS, but a site run by AWS). It shows that security is hard, even for a $51 billion business. This issue can occur not just on websites but even SDKs and libraries

📸 Erik Mclean via unsplash

While developers have a keen nose for code smells us operations types have a keen nose for infrastructure smells. When I opened this git repository for first time it hit me. A buildspec.yml file.

The humble buildspec.yml

For those unfamiliar, buildspec.yml is used by a service called CodeBuild and basically defines the steps used to build a project, including running shell commands. It’s basically remote code execution as a service.

The presence of this file in a repository isn’t call for alarm, but when it’s in a public repository it certainly raises red flags. The usual concern is someones committed some secret credentials into this file. In this case the file was clean of credentials.

All good right? Not so fast.

📸 Lachlan Gowen via unsplash

notices your deploy.sh

The buildspec.yml referenced a deploy.sh. This is when I verbally said “oh no”. Like before no secrets committed. A good start. deploy.sh contains instructions to deploy out the project - like aws s3 sync and the like, so we can determine that when this gets run it has access to upload to the production site.

📸 Nathan Anderson via unsplash

The issue here is that the buildspec.yml and deploy.sh could be modified by a malicious user.

The pull request

However malicious user doesn’t have access to commit to the repository and an admin isn’t going to merge malicious code, so this is no big deal right? Let’s see what happens when we lodge a pull request.

Upon creation of the pull request GitHub triggers a CodeBuild job. This is a fairly common practice to make sure nothing in the pull request breaks the build. What prevents the pull request build from deploying to production? Lets check deploy.sh

if [[ "$CODEBUILD_WEBHOOK_HEAD_REF" == "refs/heads/main" && ${CODEBUILD_SOURCE_VERSION:0:3} != "pr/" ]]; then

oh no.

So deployment is purely controlled by a script that can be changed in the pull request.

📸 Scott Walsh via unsplash

One last chance

At this stage we’ve got remote code execution into the pipeline. Apart from mining some Bitcoin this is pretty uneventful. What about the S3 sync we mentioned earlier? It’s possible that the role granted for pull requests is the same role used for deploying to production, so lets check it out.

I edited the shell script to have my code right at the start …

echo "testing a security issue" > test.html
aws s3 cp test.html s3://target_bucket/test.html
aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DIST_ID --paths "/*"
exit 1

target_bucket value was recovered from original deploy.sh

… and lodged a pull request. I checked the website and sure enough my file was there. 😮

📸 Nathan Anderson via unsplash

It doesn’t end there

It’s quite possible that the role used for deployment might have access to lots of interesting things, a private subnet, IAM admin, CloudFormation. I didn’t check further than this and submitted a disclosure reported to the security team immediately.

Prevention

If you still want pull requests to trigger builds on a public repository there a couple of things you can do to limit risk.

Place build scripts in a separate repo. Some build tools let you specify a separate repo to use for the build pipeline. Be careful though as this doesn’t guarantee that the project build can’t execute commands, depending on the programming language and build tools.

For services like CodeBuild you can utilize a separate IAM role for pull requests which is limited to just build requirements. Make sure the build agents for PRs aren’t within a a trusted network.