Vulnerability scanning Docker images with CoreOS Clair and Klar

What and motivation

This post is going to talk about how to get the CoreOS Clair container security tool running from the command line, with a view to integrating it into a CI/CD workflow using the Klar CLI tool.

The way that most people are using Docker is as a “better VM” in that they’re taking Ubuntu LTS releases and deploying on top of them. This is as opposed to using a “stripped down”/minimal image like Alpine Linux. The advantage of the minimal images is that they often remove a whole bunch of software that’s not needed in a container, reducing the potential attack surface and also make the images smaller and easier to work with.

One of the thorny issues everyone doesn’t think about in this scenario is how to upgrade/patch the “OS” (libraries etc…) running in Docker as it’s still an issue. Also, how to check whether a Docker image is missing patches.

To this end, CoreOS released a tool called Clair to allow someone to “scan” Docker images in order to ensure that all patches/upgrades have been applied.

tldr;

Install CoreOS Clair onto Minikube using the Helm chart. Ignore the PostgeSQL part of the chart as the PV doesn’t work with Minikube. Ensure it can talk to DB.

Expose Clair to the cluster. Download and install Klar, run it, pointing at the Clair service and pass it an image to scan (needs docker locally?).

Ignore v1 API error message from Clair logs

Install CoreOS Clair onto Minikube

Installing Clair onto Minikube is fairly straight forward thanks to Clair providing a Helm chart in GitHub, which you can find here. Personally, I found an issue with the “PostgreSQL” container within this helm chart, which caused it to fail to start, namely something to do with PhysicalVolumes:

Warning Failed 12h (x675 over 2d) kubelet, minikube Error: lstat /tmp/hostpath-provisioner/pvc-16c78bd3-12c1-11e8-8c48-080027fa3e9c: no such file or directory
Normal SuccessfulMountVolume 9m kubelet, minikube MountVolume.SetUp succeeded for volume "pvc-16c78bd3-12c1-11e8-8c48-080027fa3e9c"
Normal SuccessfulMountVolume 9m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-5v2tq"
Warning Failed 7m (x12 over 9m) kubelet, minikube Error: lstat /tmp/hostpath-provisioner/pvc-16c78bd3-12c1-11e8-8c48-080027fa3e9c: no such file or directory

Therefore, I just ran the “vanilla” PostgreSQL image and exposed it as a service with:

kubectl run postgres --image postgres
kubectl expose deployment postgres --port 5432 --target-port 5432 --type NodePort

and pointed Clair at the PostgreSQL instance using this bit of configuration in the custom values Helm YAML:

postgresURI: "postgres://postgres@postgres:5432/postgres?sslmode=disable"

(the full file can be found here)

Next up, we can install Clair itself from the Helm chart with our custom values:

helm dependency update clair
helm install clair -f mycustomvalues.yaml

NOTE: This assumes you’ve got Helm and Tiller installed and setup (have run helm init etc…)

Assuming it’s all gone well, you should see the following output when running “kubectl get pods”:

and the Clair pod logs should look something like the following:

Now we need to grab the port that the Clair service is running on in our Minikube cluster with the “kubectl get services” command:

The value we’re after is the port mapped onto 6060 (in the above screenshot 32527).

Assuming we’ve installed Klar (instructions here) we can now run Klar against a test image with the following command (update port and Minikube ip for you config):

CLAIR_ADDR=http://192.168.99.100:32527 CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 klar postgres:9.5.1

The output currently will likely be “Found 0 vulnerabilities” which is a bit of an issue (it should be > 0). The reason for this false report is that the Clair “updaters” take a while to download the full list of vulnerabilities  and lists from the Ubuntu/Debian/Alpine etc… databases.

Integrating into Continuous Integration workflow

So, at this point, we can automate the above to fail our build if the newly built/pushed image has an unacceptable level of vulnerabilities. The Klar command exit value is determined by whether the vulnerabilities exceeded the configured thresholds (0 = all good, 1 = fail). Therefore, it’s trivial to integrate into the build workflow.

We do need to sit down and work out where to set the thresholds at (i.e. what constitutes an unacceptable level of risk).

Conclusion and Further work

It would have been great if Klar supported scanning images locally (before they were pushed to a registry) as it would somewhat further reduce the risk that a “insecure” image might be deployed.

Also, we didn’t cover any alerting or notifications when Clair does find security issues in an image (who should be alerted? what should they do?).

Finally, there was no discussion of scanning already deployed images or any kind of periodic “background” scanning of a deployed environment.

For other options for Clair integration instead of Klar, have a look at the Clair integrations page.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.