Skip to content

GitLab CI/CD: A Complete Tutorial

  • 19 min read

Is your team struggling to keep up with the pace of software delivery? Do you find that deployments are often a source of stress and uncertainty? Well, you’re not alone. Many teams face these challenges. But there’s a way to make the process smoother, more reliable, and much faster. That’s where GitLab CI/CD comes in. It is a powerful tool that can transform how your team builds, tests, and releases software. In this article, we will explore the world of GitLab CI/CD, and show you how it can help you streamline your workflows and bring speed and consistency to your software delivery.

What is GitLab CI/CD?

GitLab CI/CD is a tool built into GitLab that allows you to automate your software development process. CI/CD stands for Continuous Integration and Continuous Delivery (or Continuous Deployment). It’s a set of practices that enable you to build, test, and deploy code changes more often and with less risk. Think of it as a well-oiled machine that takes your code from your developers’ workstations and puts it into the hands of your users, with many automated steps in between.

Continuous Integration (CI)

Continuous Integration is the practice of merging code changes from multiple developers into a shared repository, multiple times a day. Each change is then built and tested to ensure that it works well with the code that’s already there. This process catches integration issues early on. It reduces the time, effort, and risks that often comes with merging big changes at once, or later in the software development lifecycle.

Continuous Delivery (CD)

Continuous Delivery is the practice of automating the release of code changes into various environments, like testing, staging, and production. The process also includes running automated tests, which ensure the code is ready for release. This means that every change, when passed all the tests, is always ready to be deployed. It gives you the ability to release new features fast, and with confidence, once you are happy with how everything works.

Continuous Deployment, on the other hand, takes Continuous Delivery a step further. It means that every code change that passes the automated tests is released automatically to production. With this, you skip the manual approval step needed in Continuous Delivery. This speeds up the whole process even more. This practice can be great for small, nimble teams. But it might not be the best option if you need more control over your releases.

Why Use GitLab CI/CD?

There are several reasons why you may want to use GitLab CI/CD:

  • Speed: It automates many tasks, freeing up your team to focus on writing good code. This will make the whole software delivery process faster.
  • Consistency: Automated builds and tests guarantee that your code is always tested the same way. This reduces the risk of unexpected problems in production.
  • Reliability: Fewer errors are made when many steps are automated. This ensures software is released in a more reliable way.
  • Collaboration: Teams can work together on code and see the outcome of their work sooner. This reduces frustration that can come with complicated development and release processes.
  • Flexibility: GitLab CI/CD is customizable, so you can adapt it to your needs. It works well with different languages and deployment options.
  • Visibility: The whole CI/CD pipeline is visible in GitLab. This gives everyone a clear view of the progress of every code change.
  • Cost Savings: Automated processes save time, and reduce human error. In the long run, this will lower development costs.

Key Concepts in GitLab CI/CD

To use GitLab CI/CD, you must learn a few key concepts:

.gitlab-ci.yml File

The .gitlab-ci.yml file is a YAML file at the root of your project, that defines your CI/CD pipeline. It’s the file that tells GitLab CI/CD what to do, when, and how. This file defines the jobs to run, the stages they belong to, and other important configurations for your workflow.

Stages

Stages are used to group jobs into logical steps in your pipeline. The order of execution of these stages is fixed, meaning, each stage will start when the prior stage has finished successfully. A typical pipeline might have stages like build, test, and deploy. By using stages, you can make sure that jobs are run in the correct order. They make it easy to understand the progress of your pipeline.

Jobs

Jobs are the individual tasks that are run in a pipeline. A job can be anything from compiling your code, running tests, or deploying your app to a server. Jobs are defined under their respective stages in the .gitlab-ci.yml file. The definition will specify the script that needs to be executed, the image the job needs to use, and other configurations.

Runners

GitLab Runners are agents that run the jobs you have defined in your .gitlab-ci.yml file. These runners can be installed on your infrastructure, or can use shared runners provided by GitLab. Each runner will pick up the jobs assigned to them, execute the script for the job, and return the results to GitLab.

Pipelines

A Pipeline is the top-level view of your CI/CD workflow. Every change you make to your code triggers a new pipeline. A pipeline includes all the stages, the jobs, the runners that handle the execution, and the status of each. By seeing the whole pipeline, your team can monitor the state of every code change.

Artifacts

Artifacts are files or directories created by jobs in your pipeline. These can be binaries, images, documentation, and more. Artifacts can be used as inputs for later jobs in the pipeline. Or, they can be stored for later use. By using artifacts, you can pass the results of one job to another and make sure that all the jobs have what they need.

Environment Variables

Environment Variables are key-value pairs that store configuration data for your pipeline. They can hold database credentials, API keys, or any other piece of information that changes based on the environment where a job runs. This helps make your pipeline more secure, and allows your code to work in different places, without needing to hard-code specific values into your configuration files.

Setting Up GitLab CI/CD

To start using GitLab CI/CD, you need to set up your project and configure the pipeline. Here are some steps to do so:

Step 1: Create a GitLab Project

If you haven’t done so yet, create a new project in GitLab. This will be the repository for your code and the hub for your CI/CD pipeline. You can create a new project from scratch or import an existing one from a repository provider, such as GitHub, Bitbucket, or others.

Step 2: Add a .gitlab-ci.yml File

At the root of your project, create a new file named .gitlab-ci.yml. This file is where you define your CI/CD pipeline. Here’s a basic example of what a .gitlab-ci.yml file might look like:

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  image: node:16
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - dist/

test_job:
  stage: test
  image: node:16
  script:
    - npm install
    - npm run test

deploy_job:
  stage: deploy
  image: alpine:latest
  script:
    - echo "Deploying application..."

In this example:

  • The pipeline has three stages: build, test, and deploy.
  • The build_job uses a Node.js image and runs npm install and npm run build. It also creates an artifact named dist/ which contains your built project.
  • The test_job uses a Node.js image and runs npm install and npm run test.
  • The deploy_job uses an Alpine image, and has a placeholder command that prints a message.

Step 3: Commit and Push

After creating the .gitlab-ci.yml file, commit it to your project, and push the changes to your repository. This action will automatically trigger the first pipeline.

Step 4: Check the Pipeline Status

Navigate to the CI/CD section of your GitLab project. You should see the pipeline running, and you can follow the progress of every stage and job. If everything goes well, you will see that all stages and jobs pass successfully. If there is an error, you can inspect the logs of each job and take corrective actions.

Step 5: Configure Runners

If you do not see your pipeline running, you will need to configure GitLab Runners. In a small project, you can use GitLab’s shared runners. In larger or more specific situations, you should set up your runners. To configure a runner:
* Go to your Project Settings, and select “CI/CD”.
* Expand the “Runners” section.
* You will see instructions on how to register a runner on your machine.
* After the runner has been registered, make sure the pipeline is set to use your runner.

Creating a More Complex Pipeline

The above example shows a very simple pipeline. In real world scenarios, you’ll have to make more complex pipelines that fit your needs. Here are ways to create more complex workflows:

Using Multiple Jobs in a Stage

You can have multiple jobs running in the same stage. This can help you parallelize your workflow and finish a stage faster. For example, you might want to run your tests in many parallel jobs, which saves you time. In the .gitlab-ci.yml, you can define each job with the same stage, and they will run at the same time if runners are available.

stages:
  - test

test_job_1:
  stage: test
  image: node:16
  script:
    - npm run test-unit

test_job_2:
  stage: test
  image: node:16
  script:
    - npm run test-integration

test_job_3:
  stage: test
  image: node:16
  script:
    - npm run test-e2e

In this example, all three test_job will run concurrently. This speeds up the test stage, and gives you a faster feedback cycle.

Controlling Job Execution

You can use rules to control when a job should be run. You can specify that a job should only run on specific branches, tags, or when a specific file has been modified. This can help make your pipeline more efficient by not running jobs that do not make sense in certain circumstances. Here’s an example of how you might do this:

deploy_job:
  stage: deploy
  image: alpine:latest
  script:
    - echo "Deploying to production..."
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

In this example, deploy_job will only run when the changes are pushed to the main branch. This makes sure that deployment to production is done only when code is merged to your main branch.

Using Templates

If you have more than one project, you can create reusable pipeline templates. This will make your pipeline configuration more modular, and easier to maintain. You can define templates in a separate file and include them in your .gitlab-ci.yml files using the include keyword. For example, you could have a template for deployment, which includes logic for different environments, and then reuse this across your projects. This makes sure the pipelines across your organization are consistent.

include:
  - template: .gitlab-ci-deploy.yml

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  image: node:16
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - dist/

test_job:
  stage: test
  image: node:16
  script:
    - npm install
    - npm run test
deploy_job:
  stage: deploy

In this example, deployment configuration is kept in another file gitlab-ci-deploy.yml for reuse.

Using Services

When you need services, such as databases, to be available when running your jobs, you can use services. You can define your services directly in your .gitlab-ci.yml file, and GitLab CI/CD will create those containers before the job starts and destroy them when the job finishes. For example, if you need a MySQL database for integration tests, you can add it as a service to your pipeline.

stages:
  - test

test_job:
  stage: test
  image: node:16
  script:
    - npm install
    - npm run test-integration
  services:
    - mysql:latest
      variables:
        MYSQL_ROOT_PASSWORD: test

In this example, the test_job uses a MySQL service, available during the job. This lets your tests interact with a real database.

Advanced GitLab CI/CD Features

GitLab CI/CD has many other features that you may find useful. Here are some of them:

Auto DevOps

Auto DevOps is a set of predefined pipeline configurations that automates many common CI/CD tasks. With Auto DevOps, you can quickly set up a working pipeline for building, testing, and deploying a wide range of applications, with very little configuration needed. Auto DevOps is a good place to start if you want to get something running quickly.

Environments

GitLab Environments allow you to group your deployments into named logical environments, like staging, production, development, or others. Environments provide a way to organize your deployments, and track the status of your application in each environment. You can also use environment variables to provide unique settings for every environment.

Secret Variables

GitLab allows you to define secret variables that are hidden in your .gitlab-ci.yml file. This makes it possible to store sensitive information such as API keys and credentials without hardcoding them directly in your code. You can define secret variables in the CI/CD settings of your project, and use them in your pipeline configuration.

Security Scanning

GitLab CI/CD comes with a range of security scanning tools that let you check your code for vulnerabilities. These scanners can be run as part of your CI/CD pipeline and will help you find vulnerabilities before your code goes to production. Security scans can check for issues in dependencies, your code, and containers.

Merge Request Pipelines

Merge Request Pipelines are a specialized type of pipeline that runs when a merge request is created. These pipelines are run to ensure that code changes that you intend to merge are safe and don’t introduce new issues into your project. They will help make sure that only code that has passed all the tests is merged into your main branch.

Review Apps

Review Apps are temporary environments that are created automatically for each merge request. This makes it possible for reviewers to test code changes live. They can see how these changes work before they are merged into the main branch. This reduces the risk of issues making their way into production.

Custom Docker Images

While you can use pre-built Docker images for your jobs, you might need to create your images to meet the needs of your specific project. GitLab CI/CD works great with custom Docker images and helps you create powerful, specialized build environments. To create custom docker images you can either define your own Dockerfile or extend from existing images.

GitLab CI/CD Best Practices

To make the most of GitLab CI/CD, follow these best practices:

  • Keep your .gitlab-ci.yml file clean: Do not include a lot of logic in the .gitlab-ci.yml file. Instead, use separate scripts for complex tasks. This makes your configuration more simple to understand and more maintainable.
  • Use environment variables for secrets: Never store credentials in your .gitlab-ci.yml file. Instead, store them in environment variables, or secret variables, that you have configured in the CI/CD settings of your project.
  • Test your code thoroughly: Include a comprehensive set of tests in your pipeline. These tests should include unit tests, integration tests, and end-to-end tests to make sure your application works correctly.
  • Optimize your pipeline for speed: Look for ways to optimize your pipeline execution. Use parallel jobs when you can, or use caching to avoid repetitive operations. This will make your feedback loop faster, and your team more efficient.
  • Monitor your pipelines: Regularly check the status of your pipelines. Look for any issues, and take steps to fix them. By monitoring, you can catch errors early and prevent them from making their way into production.
  • Version control your pipeline configuration: Commit your .gitlab-ci.yml file to your repository. This will allow you to revert changes and make sure your pipeline configuration is tracked and version controlled like the rest of your code.
  • Use GitLab-managed runners: GitLab-managed runners provide better availability. It will help you avoid the management of runner infrastructure, and give you more time to focus on development.
  • Start simple and iterate: Start with a simple pipeline, and add more features to it over time. This will make it easier to understand, and will allow you to make changes slowly.
  • Make use of caching: You can use caching to speed up your pipeline by saving dependencies, built artifacts, and anything else that you may need later in the pipeline. This saves time, and resources, by avoiding repetitive steps.
  • Use review apps: Use review apps to test code changes before they make their way to production. This can reduce problems that could happen if code changes are merged without being properly tested.
  • Secure your CI/CD pipelines: Implement good security practices such as using secret variables, and security scans, to keep your pipeline safe.
  • Use merge request pipelines: Merge request pipelines help you keep code quality high, by making sure code changes that are merged into your main branch, have passed all tests and checks.
  • Always have proper logging: Proper logging can make it easier to debug issues when your pipelines break. A good practice is to log all important steps in your pipeline.

Troubleshooting Common Issues

Even with a good setup, you might still face problems when using GitLab CI/CD. Here are some tips to debug and resolve common issues:

  • Job failing: If a job fails, check the logs of that job for any errors. The logs will often provide information about the cause of the problem, which you can use to fix it.
  • Pipeline stuck: Sometimes a pipeline will get stuck. You can restart the pipeline or the job that got stuck. If restarting does not resolve the issue, check for configuration issues, and runner availability.
  • Runner not picking up jobs: If a runner is not picking up jobs, make sure the runner is properly registered, and configured in the project. Also, check that the runner has enough resources to execute the jobs.
  • Artifacts not being passed: If artifacts from a previous job are not being passed to the current job, make sure that the path to the artifacts is correctly defined in the .gitlab-ci.yml file.
  • Issues with docker images: If there are issues with the docker images, ensure that the images are accessible. You can test them by running the images locally to see what is going on.
  • Incorrect Environment variables: Make sure that the environment variables are configured correctly, and that you are not using any hardcoded secrets.
  • Incorrect syntax in .gitlab-ci.yml: Incorrect syntax can lead to pipeline errors. Make sure the .gitlab-ci.yml file is valid, and free of errors. You can use linters, or yaml validators, to help with this.

GitLab CI/CD in Practice

Now, let’s take a look at a practical example of how you can set up a CI/CD pipeline for a Node.js application.

Step 1: Initial Setup

Start by creating a simple Node.js application. This example application is a web server using the Express.js framework. Create a file named index.js with the following content:

const express = require('express');
const app = express();
const port = 3000;

app.get('/', (req, res) => {
  res.send('Hello World!');
});

app.listen(port, () => {
  console.log(`Server running on port ${port}`);
});

module.exports = app;

Then, create a package.json file:

{
  "name": "node-app",
  "version": "1.0.0",
  "description": "A simple Node.js app",
  "main": "index.js",
  "scripts": {
    "start": "node index.js",
    "test": "jest",
    "build": "echo 'Build completed'"
  },
  "dependencies": {
    "express": "^4.17.1"
  },
  "devDependencies": {
    "jest": "^29.3.1",
    "supertest": "^6.3.3"
  }
}

Install Jest and Supertest for testing.

npm install --save-dev jest supertest

Add a basic test file named index.test.js:

const request = require('supertest');
const app = require('./index');

describe('GET /', () => {
  it('responds with Hello World!', async () => {
    const response = await request(app).get('/');
    expect(response.statusCode).toBe(200);
    expect(response.text).toBe('Hello World!');
  });
});

Step 2: .gitlab-ci.yml Configuration

Now, set up the .gitlab-ci.yml file at the root of your project:

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  image: node:16
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - node_modules/
      - package*.json

test_job:
  stage: test
  image: node:16
  script:
    - npm install
    - npm run test
  dependencies:
    - build_job

deploy_job:
  stage: deploy
  image: docker:latest
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  rules:
     - if: '$CI_COMMIT_BRANCH == "main"'

This pipeline:
* Defines three stages: build, test, and deploy.
* The build_job installs dependencies, and creates artifacts.
* The test_job installs the dependencies, runs the tests, and uses the artifacts from the build job.
* The deploy_job builds a docker image, and pushes it to GitLab’s container registry. This job runs only on the main branch.

Step 3: Adding Dockerfile

To deploy the app, add a Dockerfile:

FROM node:16
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["npm", "start"]

Step 4: Push and Run the Pipeline

Commit and push your changes to GitLab. Then, check the pipeline status in the CI/CD section of your project.

Step 5: Set Secret Variables

Set secret variables in your project settings under “CI/CD > Variables”. You need to define:

  • CI_REGISTRY_USER as your GitLab username
  • CI_REGISTRY_PASSWORD as your GitLab password or a deploy token
  • CI_REGISTRY as your GitLab registry

After all these steps, your pipeline should be up and running, and you can follow the process of your software release.

The Final Verdict

GitLab CI/CD is a powerful tool that can transform how your team builds, tests, and releases software. By setting up a well-defined pipeline, you can automate many tasks, improve collaboration, and release new features faster and more reliably. Whether you are a beginner or a seasoned DevOps engineer, GitLab CI/CD can help your team deliver high-quality software with ease. With features like automated builds, security scans, and review apps, GitLab CI/CD can help you take your software development workflow to a new level. By using the concepts and best practices we’ve discussed in this article, you can set up a pipeline that fits your needs. This helps you to make the whole software delivery process more efficient, reliable, and faster. So, why wait? Start exploring GitLab CI/CD and change the way your team delivers software.

Related Posts

Jenkins Configuration as Code

  • by mark
  • CI/CD

The Secret to Parallel Stages

  • by mark
  • CI/CD