In recent years, software development industries have introduced and shifted to methodologies with radical new approaches. These methodologies aim to deliver impressive development velocity without compromising software quality in the process. CI/CD is one such methodology that has helped teams develop and release software much faster than before, enabling them to improve and grow their product rapidly.
Since it offered many enticing productivity improvements, our tech team ventured out to build a CI/CD pipeline for its development process. This post will explain all the preliminary information about CI/CD and then dive into the general details of how we implemented it through Gitlab CI to speed up our product development.
What is CI/CD pipeline?
The CI/CD pipeline consists of practices showcased within its very name: continuous integration and continuous delivery/deployment. It is essentially a majorly-automated workflow that consists of everything you need to test, stage, and deploy a developed software product. Automation is one of the crucial factors in all CI/CD pipelines since it helps accelerate the whole process while ensuring reliability.
The general pattern of a CI/CD pipeline involves an initial commit to a CI enabled approach that triggers a set of unit tests or a build, after which more automated rounds of tests happen that provide feedback after each round. Next, the software gets deployed to different staging servers and environments, which may involve manual testing. Finally, the approved changes go live through continuous delivery (manually) or continuous deployment (automatically).
What is Gitlab CI?
Gitlab CI is an industry-standard tool for implementing CI/CD pipelines. It helps you create customized pipelines for your product development approaches without involving additional tools. Therefore, it simplifies the whole CI/CD implementation by offering all tools you need to iteratively build, test, and deploy your software in one place. Additionally, Gitlab CI’s close integration with Git enables robust version control as you release improved code changes.
How did we build a CI/CD pipeline using Gitlab CI?
Sample Architecture
Before starting, we needed to look into CI/CD pipeline architectures for inspiration. We settled on using Gitlab CI/CD for orchestrating all the build, test, stage, and deploy stages of the pipeline that were to happen on the Amazon EKS cluster. With every new commit, the pipeline will fetch the code saved in an Amazon DynamoDB versioning table, build, and run the necessary integration tests.
Creating an EKS cluster and installing the controllers
Our CI/CD pipeline needed to accommodate the end-to-end testing of our product’s complex ecosystem of microservices. Therefore, we created an AWS EKS cluster and set up temporary Kubernetes namespaces to deploy the microservices.
Our CI/CD pipeline needed to accommodate the end-to-end testing of our product’s complex ecosystem of microservices. Therefore, we created an AWS EKS cluster and set up temporary Kubernetes namespaces to deploy the microservices.
https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-eks.gitYou can set up the EKS cluster through Gitlab by importing their example project repository. First, go to Import Project and choose Repo by URL. Next, insert the following URL and select Create Project.
Now your project has an Amazon Virtual Private Cloud, an AWS EKS cluster, and the GitLab agent for Kubernetes.
We manually provisioned the cluster once our project was up and configured for CI/CD. After deployment, the cluster becomes visible within GitLab and AWS. Next, it was time to provide the cluster with Kubernetes controllers that watch over its state and make or request any needed changes. You can use the Kubernetes control manager daemon to set up and install controllers within GitLab Runner.
For example, this is how a sample yml file looks for a Replica Controller.
apiVersion: v1
kind: ReplicaController
metadata:
name: exampleapp-replicaset
labels:
app: exampleapp
type: front-end
spec:
template:
metadata:
name: exampleapp-pod
labels:
app: exampleapp
type: front-end
spec:
containers:
- name: nginx-container
image: nginx
replicas: 3You can use the following command to install this controller.
$ kubectl create -f replicacontroller.ymlCreating a DynamoDB table for CI/CD Versioning
Next, we need to keep track of all the versions of our software. We utilize Amazon DynamoDB for this purpose since version tracking is crucial for continuous application deployments. The versions are present in a DynamoDB versioning table, and Gitlab CI retrieves and updates it. Executing the following bash file will create the required DynamoDB.
export TABLE_NAME=example-versioning
export REPO_NAME=example-app
aws dynamodb create-table \
--table-name $TABLE_NAME \
--attribute-definitions \
AttributeName=example_app,AttributeType=S \
--key-schema \
AttributeName=example_app,KeyType=HASH \
--provisioned-throughput \
ReadCapacityUnits=1,WriteCapacityUnits=1
aws ecr create-repository \
--repository-name $REPO_NAME Gitlab CI/CD pipeline
Once our infrastructure is up, we can finish building our CI/CD pipeline. Implementing it requires a .gitlab-ci.yml that specifies all the settings and executes every stage in the pipeline. Additionally, you need to set up a private Gitlab repo and put in the relevant environment variables for your CI/CD pipelines.
Our yml file had to incorporate that the source code gets Dockerized at each commit and pushed to the EKS cluster. The entire microservices ecosystem gets deployed on an AWS EKS cluster in shortlived sandbox namespaces for end-to-end testing on a merge request. Once merged, each microservice’s docker image gets automatically deployed on a staging environment on the EKS cluster with the latest code. Finally, the staging environment has nightly tests for the frontend and backend services.
Afterward, you have to add the Gitlab repo as the origin and push it through the following commands, which will initiate the CI/CD pipeline within Gitlab.
git remote set-url origin <YOUR_GITLAB_REPOSITORY_URL>
git remote -v
git push origin masterWrapping Up
This article showed each part of our process as we set up a CI/CD pipeline on Gitlab CI. We employed Amazon EKS clusters, Kubernetes controllers, and Amazon DynamoDB to create our pipeline architecture. With such robust technologies, our CI/CD pipeline will significantly improve software quality and release velocity.
