Docker

RWX supports running and building Docker containers inside of tasks using the docker CLI. To enable support for Docker in a task, specify docker: true:

tasks:
  - key: container
    docker: true
    run: docker run hello-world

RWX's Docker functionality is most commonly used to run background services as part of a task:

tasks:
  - key: packages
    run: |
      sudo apt-get update
      sudo apt-get install netcat redis-tools
      sudo apt-get clean

  - key: ping-redis
    use: [packages]
    docker: true
    background-processes:
      - key: redis
        run: docker run -p 6379:6379 redis
        ready-check: redis-cli ping
    run: |
      redis-cli SET mykey "Hello, Redis!"
      redis-cli GET mykey

Preserving Docker data

By default, RWX deletes all data related to Docker at the end of a task. If you want to preserve the images you have pulled or the volumes used by your containers, you can specify docker: preserve-data. This is very useful for pulling and caching images ahead of the task that actually uses them:

tasks:
  - key: docker-images
    docker: preserve-data
    run: docker pull hello-world

  - key: some-task
    use: docker-images
    docker: true
    run: docker run hello-world

RWX's internal networking and cache layer management is faster than Docker's, so typically pre-pulling large images in a cached task will be faster than pulling them on-demand in the task that needs them. You can also pull images with docker compose pull in an RWX task if you're using Docker compose.

Ready checks

As described in the documentation for background processes, it's recommended to use service-specific tools for ready checks. However, you may not want to have to install those tools outside of a docker container. In this case, you can use docker health commands.

tasks:
  - key: docker-images
    docker: preserve-data
    run: docker pull redis

  - key: demo-redis
    use: docker-images
    docker: true
    background-processes:
      - key: redis
        run: |
          docker run -p 6379:6379 \
            --name redis \
            --health-cmd 'redis-cli ping' \
            --health-interval 5s \
            redis
        ready-check: |
          docker ps -a # for observability / debugging
          docker inspect redis | jq -e '.[0].State.Health.Status == "healthy"'
    run: |
      echo -e '*1\r\n$4\r\nPING\r\n' | nc localhost 6379

RWX also provides an rwx-docker-ready-check command to make sure all docker services are healthy.

tasks:
  - key: docker-images
    docker: preserve-data
    run: docker pull redis

  - key: demo-redis
    use: docker-images
    docker: true
    background-processes:
      - key: redis
        run: |
          docker run -p 6379:6379 \
            --name redis \
            --health-cmd 'redis-cli ping' \
            --health-interval 5s \
            redis
        ready-check: rwx-docker-ready-check
    run: |
      echo -e '*1\r\n$4\r\nPING\r\n' | nc localhost 6379

Caching Docker volumes

You can also take advantage of preserved Docker data to cache Docker volumes needed by your task. For example, if your test suite relies on a Postgres database being set up a certain way, you can perform your database setup in one task and then have your testing task depend on that setup task:

tasks:
  - key: packages
    run: sudo apt-get update && sudo apt-get install netcat postgresql-client && sudo apt-get clean

  - key: postgres-image
    docker: preserve-data
    run: docker pull postgres

  - key: postgres-data
    use: [packages, postgres-image]
    docker: preserve-data
    background-processes:
      - key: postgres
        run: |
          docker volume create pgdata
          docker run -p '5432:5432' -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=password -v pgdata:/var/lib/postgresql/data postgres
        ready-check: pg_isready -U postgres -h 127.0.0.1 -p 5432
    run: |
      psql postgres://postgres:password@127.0.0.1:5432 -c "CREATE DATABASE db;"
      psql postgres://postgres:password@127.0.0.1:5432/db -c "CREATE TABLE users (id SERIAL PRIMARY KEY, email VARCHAR(255));"
      psql postgres://postgres:password@127.0.0.1:5432/db -c "INSERT INTO users (email) VALUES ('test@test.com');"

  - key: test
    use: postgres-data
    docker: true
    background-processes:
      - key: postgres
        run: docker run -p '5432:5432' -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=password -v pgdata:/var/lib/postgresql/data postgres
        ready-check: pg_isready -U postgres -h 127.0.0.1 -p 5432
    run: psql postgres://postgres:password@127.0.0.1:5432/db -c "SELECT * FROM users;"

Enabling incremental Docker builds

Preserving Docker data also allows you to enable incremental Docker builds inside your task. Using a tool cache, you can preserve the Docker build cache from a Docker build and restore it the next time the task attempts the same Docker build:

- key: dockerfile
  run: |
    cat << EOF > Dockerfile
    FROM ubuntu:22.04
    RUN apt-get update &&  apt-get install -y curl && apt-get clean && rm -rf /var/lib/apt/lists/*
    EOF
- key: docker
  use: dockerfile
  docker: preserve-data
  tool-cache: true
  cache: false
  run: docker build .

Running this sample run twice will show that the apt-get build step is a cache hit in the second run.