Gitlab Continuous Integration (CI) allows for automation of jobs for building, testing and deploying software. Such jobs are configured using the file .gitlab-ci.yml. Sometimes, you need to perform similar steps in different jobs. Rather than copy-pasting sections across multiple jobs, it is better to define common parts in one place and then reusing them. In software engineering this practice is called "Don't Repeat Yourself" (DRY). The benefit of not repeating yourself is that your code becomes easier to maintain: if you need to change the parts that are reused, you only need to do it in a single place. The .gitlab-ci.yml file is a (YAML) configuration file, but there are still ways to not have to repeat yourself. Two ways of doing this are:

1. Job templates and extensions
2. Referencing job sections

We will use an example to show how to apply these principles.

Example

Description

In our example, we are creating a software application for our client. To easily deploy our software on a cloud platform, we encapsulate our software inside a Docker image. We always do a test-deployment of the software on our own cloud platform account. If no problems arise in our test deployment, we deliver the docker image to the client.

Technicalities

Our Gitlab repository contains a Dockerfile with which the docker image can be created. Our git repository has two relevant branches: test and production. Whenever a new commit is added to test, we want to automatically build the docker image, push the image to our own image registry and deploy a docker container based on this image. Whenever a new commit is added to production, we want to automatically build the docker image and push the image to our client's registry.

CI configuration

Relevant sections of a .gitlab-ci.yml communicating with a non-specific cloud-provider is given below.

# Job to build docker image
build-docker-image:
  [irrelevant parts not shown]
  rules:
    - if: $CI_COMMIT_BRANCH == "test"
    - if: $CI_COMMIT_BRANCH == "production"

# Job to deploy image to our cloud account
deploy-to-our-account:
  stage: deploy
  image: cloud-provider-cli
  before_script:
    # Login to cloud provider registry to enable pushing an image
    - cloud-provider registry login --key=$IMAGE_REGISTRY_KEY_OURS
  script:
    # Push the image to the cloud provider registry
    - docker push $DOCKER_IMAGE $IMAGE_REGISTRY_OURS/$DOCKER_IMAGE
  after_script:
    # Run the container based on the image in the cloud provider's docker container engine
    - cloud-provider run container $IMAGE_REGISTRY_OURS/$DOCKER_IMAGE
  rules:
    - if: $CI_COMMIT_BRANCH == "test"

# Job to deploy image to client's cloud account
deploy-to-client-account:
  stage: deploy
  image: cloud-provider-cli
  before_script:
    # Login to cloud provider registry to enable pushing an image
    - cloud-provider registry login --key=$IMAGE_REGISTRY_KEY_CLIENT
  script:
    # Push the image to the cloud provider registry
    - docker push $DOCKER_IMAGE $IMAGE_REGISTRY_CLIENT/$DOCKER_IMAGE
  rules:
    - if: $CI_COMMIT_BRANCH == "production"

Analysis

There is some duplication of configuration in the deploy jobs defined by the .gitlab-ci.yml. In particular, both use the same stage, image, script and rules keyword sections.

Job templates and extensions

Gitlab CI specification allows disabling jobs by adding a dot (.) in front of the job name:

.hidden-and-disabled-job:
  [...]

These jobs are not executed and don't show up in the CI pipeline, and are therefore also called "hidden jobs". You can use the hidden/disabled jobs by using them as templates using the extends keyword.

Simple example

.template:
   image: docker:latest
   before_script:
     - echo This is the before script section.

job-using-template:
   extends: .template
   script:
     - echo This is the script section.

This YAML will be parsed and the jobs "merged" such that the following YAML is equivalent to the previous one.

job-using-template:
  image: docker:latest
  before_script:
    - echo This is the before script section.
  script:
    - echo This is the script section.

Overwriting template keyword values

If the job which uses the template (job-using-template) also sets some keyword section present in the template, the merged job will take the value set in the job-using-template job. For example, if job-using-template also sets the image keyword (e.g. to python), the following will happen:

.template:
  image: docker:latest
  before_script:
    - echo This is the before script section.

job-using-template:
  extends: .template
  image: python # <- `image` keyword is also set in .template
  script:
    - echo This is the script section.

Equates to:

job-using-template:
  image: python # <- no longer the value from .template
  before_script:
    - echo This is the before script section.
  script:
    - echo This is the script section.

Merging variables

Keyword section merging also works for the variables keyword, but on a per-variable basis. In the example below, the template defines the variables letter and number. Two jobs use the template: job-setting-its-own-letter and job-adding-another-variable:

.template:
   variables:
     number: 42
     letter: A

.job-setting-its-own-letter:
   extends: .letter
   variables:
     letter: Z

.job-adding-another-variable:
   extends: .template
   variables:
     symbol: !

This YAML configuration will be parsed and merged to the following equivalent:

.job-setting-its-own-letter:
  variables:
    number: 42
    letter: Z

.job-adding-another-variable:
  variables:
    letter: A
    number: 42
    symbol: !

Usage in the main example

We can apply hidden job templates and job extensions to simplify our example. In particular, we can create a .deploy hidden job that sets the image, before_script and script keywords which are quite similar in the jobs used to deploy to our own cloud account and to the client's cloud account: By adding a variables section we can set some variables that are used to distinguish between our and our client's deployment.

build-docker-image:
  [irrelevant parts not shown]
  only:
    - test
    - production

.deploy:
  stage: deploy
  image: cloud-provider-cli
  before_script:
    # Login to cloud provider registry to enable pushing an image
    - cloud-provider registry login --key=$IMAGE_REGISTRY_KEY
  script:
    # Push the image to the cloud provider registry
    - docker push $DOCKER_IMAGE $IMAGE_REGISTRY/$DOCKER_IMAGE

# Job to deploy image to our cloud account
deploy-to-our-account:
  extends: .deploy
  variables:
    IMAGE_REGISTRY: $IMAGE_REGISTRY_OURS
    IMAGE_REGISTRY_KEY: $IMAGE_REGISTRY_KEY_OURS
  after_script:
    # Run the container based on the image in the cloud provider's docker container engine
    - cloud-provider run container $IMAGE_REGISTRY_OURS/$DOCKER_IMAGE
  rules:
    - if: $CI_COMMIT_BRANCH == "test"

# Job to deploy image to client's cloud account
deploy-to-client-account:
  extends: .deploy
  variables:
    IMAGE_REGISTRY: $IMAGE_REGISTRY_CLIENT
    IMAGE_REGISTRY_KEY: $IMAGE_REGISTRY_KEY_CLIENT
  only:
    - if: $CI_COMMIT_BRANCH == "production"

Note that the variables $IMAGE_REGISTRY and $IMAGE_REGISTRY_KEY are used in the hidden (template) job .deploy, but they are not (and need not) be defined the hidden (template) job's variables keyword section. This is not necessary because these variables are defined in the jobs extending upon the template (deploy-to-ours and deploy-to-client). The merged jobs will be equivalent to the way they were defined before.

Referencing job sections

Gitlab CI specification allows referencing of keyword sections to reuse them and extend them as well. This allows picking only specific parts from other jobs to merge into the referencing job, which is more fine-grained than merging jobs based on a template. The syntax for referencing job parts !reference [referenced_job, referenced_keyword_section].

Simple example

Here is an example with two hidden jobs and a normal job.

.template-script:
  variables:
    number: 42
  script:
    - echo This is the script

.template-after-script:
  variables:
    letter: A
  after_script:
    - echo This is the after script

job-script-and-after-script:
  script:
    - !reference [.template-script, script]
  after_script:
    - !reference [.template-after-script, after_script]

This is equivalent to:

job-script-and-after-script:
  script:
    - echo This is the script
  after_script:
    - echo This is the after script

Note that that job-script-and-after-script does not have the variables defined in .template-script and .template-after-script because it is defined by referencing only the script and after_script keyword sections of .template-script and .template-after-script respectively. Also note that referencing allows reusing keyword sections from multiple jobs (both .template-script and .template-after-script). Although not shown in the example, it is allowed to reference sections from normal (non-hidden) jobs.

Extending keyword sections

You can use referencing to extend keyword sections.

.template-script:
  variables:
    number: 42
  script:
    - echo This is the script
    - echo With multiple commands

extended-job-script:
  script:
    - !reference [.template-script, script]
    - echo This is an extra part!

This is equivalent to:

extended-job-script:
  script:
    - echo This is the script
    - echo With multiple commands
    - echo This is an extra part!

Application to the main example

In our example we are pushing a docker image named $DOCKER_IAMGE to the cloud provider when commits are made to the test branch (our cloud account) or the production branch (client's cloud account). This means we only need to build the docker image when we are also going to try pushing it. We can reference the rules sections of the deploy jobs to decide when to run the build job.

build-docker-image:
  [irrelevant parts not shown]
  rules:
   - !reference [deploy-to-our-account, rules]
   - !reference [deploy-to-client-account, rules]

Read more:

Read more on Gitlab's documentation: - Gitlab documentation: extends keyword - Gitlab documentation: reference tags