Often when I read something on continuous delivery with docker at some point I stumble over a statement like this: “Testing the image is left as an exercise to the reader” or “Of course in the real work we’d add some testing here”. Some time ago I automated some build-pipelines for a couple of docker images, and I had a hard time in finding good resources on how to do it for real. I’ll summarize how I implemented it in the end in this article. If we consider that a CI/CD pipeline might eventually deploy new software versions to a production environment without human interaction the need for really good tests along that pipeline is obvious. But, how can you actually set up some basic tests for your freshly built image?

I will outline one approach using jenkins pipelines, docker, and goss, an opensource-tool for “Quick and Easy server testing/validation” created by Ahmed Elsabbahy. First, let me give some examples of the tests I want to run:

  • Starting a container from the freshly built image works
  • Certain files are present in the container
  • The container logs contain certain messages (“… started successfully”)
  • Running a shell command in the container gives an expected output (curl http://localhost:8080)

As a second requirement I didn’t want to reinvent the wheel for each image I’m building. Put differently: The Jenkinsfile should be identical for all kinds of different images I would test in this way. Additionally, I wanted the approach to be lightweight and not take too much time during the CI/CD pipeline.

Now, let’s review how the goss-test-tool works. It consists of a small binary which requires a test-specification in yaml-format. When running goss validate the tool verifies that the conditions in the test-specification are fulfilled. An example for such a specification could look like this:

package:
  java-1.8.0-openjdk:
    installed: true
user:
  java:
    exists: true
    uid: 1000
    gid: 1000
    groups:
    - java
command:
  cd / && java -version:
    exit-status: 0
  java -version:
    exit-status: 0
  pwd:
    exit-status: 0
    stdout:
    - \/home/java/app

It tests among other things, that java has been installed successfully in the docker container, that it contains a java user and that running java -version works fine from different working directories.

Originally goss was created to validate such conditions directly on the host on which it is running (docker out of scope). For using it with docker, the creator wrote a wrapper called “dgoss”. However, according to my experience this tool is great when interactively using goss with docker containers, but was difficult to use in conjunction with jenkins. Therefore, I went for the simpler approach of just adding the goss binary to the image under test.

After some experimenting, I came up with the following procedure, automated on jenkins:

  • Build the docker image we want to test. Referred to as “image under test”
  • Build a temporary second image inheriting FROM the image under test. Add the goss-binary and a goss-test-specification to this temporary image.
  • Launch a container from the temporary image using the docker-functions available in jenkins.
  • Launch goss validate inside that container as a second process. If it fails, fail the build.

For reusing this procedure in all images we can use a jenkins shared library function boiling down to the following code:

def dockerTest(String imageUnderTest) {
  if (!steps.fileExists('test/goss.yaml')) {
    throw new MissingResourceException("dockerTest cannot be run. ./test/goss.yaml could not be found in ${steps.pwd()}")
  }
  String dockerfileTest = ""
  // check if Dockerfile.test present, otherwise use default one
  if (!steps.fileExists("test/Dockerfile.test")) {
    dockerfileTest = steps.libraryResource('Dockerfile.test')
    log.info("docker test: Using generic Dockerfile.test")
  } else {
    dockerfileTest = steps.readFile("test/Dockerfile.test")
    log.info('docker test: Using provided Dockerfile.test. Replacing placeholder ${IMAGE_UNDER_TEST}')
  }
  // for newer docker versions the image-under-test can be passed as build-arg
  dockerfileTest = dockerfileTest.replace('${IMAGE_UNDER_TEST}', imageUnderTest)
  steps.writeFile(file: "test/Dockerfile.test", text: dockerfileTest)

  def randomImageId = UUID.randomUUID().toString()
  def testImage = docker.build(randomImageId, "--file test/Dockerfile.test test/}")
  testImage.withRun() { c ->
    steps.sh("docker exec ${c.id} goss validate --retry-timeout 35s --sleep 10s")
  }
}

The second temporary image may always be constructed from a generic Dockerfile.test, only the first FROM line changes depending on the image under test. This generic Dockerfile.test looks like this:

# in newer docker versions IMAGE_UNDER_TEST may be passed as a build argument. For now replace it through groovy-code on jenkins
# ARG IMAGE_UNDER_TEST
FROM ${IMAGE_UNDER_TEST}

USER root

# install goss
RUN set -x \
 && curl -fsSL https://github.com/aelsabbahy/goss/releases/download/v0.3.6/goss-linux-amd64 -o /usr/local/bin/goss \
 && chmod +rx /usr/local/bin/goss

# copy test specification into the image
COPY goss.yaml goss.yaml

# never ending command
CMD exec /bin/bash -c "trap : TERM INT; sleep infinity & wait"

# ensure no entrypoint inherited from image under test
ENTRYPOINT []

As this Dockerfile.test is identical for whatever image under test, we may just include it as a resource into the shared library (and not into each and every git-repository of the docker-images).

To wrap up until here, we are now able to run some goss tests on any image during the CI/CD pipeline on jenkins. The git-repository contains the following files:

git repo structure

  • src = application source code.
  • test/goss.yaml = our test specification.
  • Dockerfile = the Dockerfile of our application.
  • Jenkinsfile = our pipeline calling the shared library function dockerTest() at some point.

There is one of the initial requirements which we haven’t adressed yet: Being able to analyze the STDOUT-logs of our images under test. This part is a bit tricky: goss validate is running as a second process inside the container, so we cannot easily analyze the STDOUT of that container. However, with a simple workaround we can still achieve it: We provide a customized Dockerfile.test for the respective image and redirect the logs we are interested in to a file inside the container instead of STDOUT. As an example let’s assume we are dealing with a tomcat-based web-application and we want to assert that the server prints a successful startup message after some time.

In the directory structure we add an additional Dockerfile.test:

git repo with customized Dockerfile.test

This Dockerfile.test is identical to the file included in the resources of the shared library, except for one difference: The CMD instruction. We add an output redirection part to the usual catalina.sh run.

# ...

# modify CMD to redirect output to a file inside the container.
# this allows goss to validate that the server starts up fine by looking into the log file
CMD ["bin/catalina.sh", "run", "&> stdout.log"]

# ...

Now we can add the following check to our goss test specification:

file:
  /opt/tomcat/stdout.log:
    exists: true
    contains:
    - "Server startup in"

Note that the shared-library function first checks if a customized Dockerfile.test exists and only uses the generic one as a fallback. With these final changes we have a generic approach of testing any docker image in our pipeline with the help of goss. According to my experience the test-phase of the pipeline takes around 1 minute, where most of the time is spent on building the temporary docker image.

A final sidenote: During my experiments I really enjoyed the discovery of goss. However, I recently noticed that the project is looking for maintainers.