How to create an automatic ECS deployment with Jenkins

One of the biggest challenges in programming is smoothly integrating any changes when we have several people working on the same project at once. For these cases Victor Rentea says:

If it hurts, do it more often, and bring the pain forward.

Here is where Continuous Integration and Continuous Deployment comes into action. We know that integrating and deploying small changes are far easier to implement than larger, more significant changes and the difficulty increases exponentially as these changes grow. So what is the solution to this problem? The solution is to do it more often, integrating and deploying small pieces of code constantly and as soon as possible.

But we know that in order to reach this goal, it is not as simple as merging and deploying our code constantly. We need to bring the following concepts into our project:

• Test: This is really important to verify that the changes we are integrating are working correctly and that they don’t break anything else.
• Automate: As stated previously, our goal is to make these changes more often, so for this reason the best way is to have an automatic process to save time and reduce human errors.

In this article, we are going to focus on the second point and more specifically explain how to automate the integration and deployment of our code in ECS(Elastic Container Service) using Jenkins. Before we get started, I would recommend taking a look at my previous article on how to deploy a microservice using Elastic Container Service in AWS as we are going to deploy the code from there into this cluster.

Overview

In this example we are going to create a simple pipeline to deploy our code. You can see the code for this example in Github (user-api code). It is a really simple microservice with two endpoints:

• Get /users: Return a list with the users name
• Post /users: Create a user receiving a json with the name in the body

The pipeline:

• Build/Test: The first step is to build and test our code using maven.
• Build Image: The second step is to create a Docker image.
• Push To ECR: With the Docker image created in the previous step it is time to push it to our ECR(Elastic Container Registry).
• Deploy In ECS: Finally, we have to update our task definition with the new image version and deploy it.

Setting Up Jenkins

The first step here is to install Jenkins and set up it. To install Jenkins I recommend following the official guide. In addition to this we need the following tools:

• Java 11
• Maven 3.6.3
• Docker 20.10.2
• AWS CLI 1.18.178

And the following plugins:

• Docker Pipeline 1.25
• HTML Publisher 1.25
• Pipeline 2.6
• Pipeline AWS Steps 1.43
• Pipeline Maven integration 3.9.3
• Pipeline Utility steps 2.6.1
• Pipeline: GitHub Groovy Libraries 1.0
• Workspace Cleanup Plugin 0.38

Multibranch Pipeline

Now that we have installed Jenkins and set up it, we can begin. The first step is to create a Multibranch pipeline, we can find this option by clicking New Item (left menu). Below you can see the configuration:

JenkinsFile

In my opinion the best way to set up our pipeline is to use jenkinsfile, this is a text file that contains the definition of a Jenkins Pipeline. I’ve outlined my reasons below:

Why should we use jenkinsfile?

• The pipeline will be versioning in our repository (we can see the changes and rollback them if necessary). There is no magic behind the scene.
• We move the responsibility from DevOS to the developers. No one knows how to deploy a project better than its creators.
• Easier and faster, we can have a starter template (using the company standards) that we can copy in our project, modify with the specific configuration and add extra stages if necessary.

The main structure of our jenkinsfile looks like this:

Options

This directive allows configuring Pipeline-specific options from within the Pipeline itself. In our case we are using the following options:

• buildDiscarder: In this option we are specifying how many artifacts and logs we want to persist. In this case we are going to persist the last 10 builds.
• disableConcurrentBuilds: Disallows concurrent executions of the Pipeline.
• timeout: Sets a timeout period for the Pipeline run, after which Jenkins should abort the Pipeline. Our timeout is 1 hour. This is a good practice to prevent infinite blocks in our Jenkins.
• timestamps: Prepends all console output generated by the Pipeline run with the time at which the line was emitted.

Tools

This section defines the tools to auto-install and put on the path:

Environment

Here we can specify a sequence of key-value pairs which will be defined as environment variables for all steps:

Stages

This section is where we are going to define the different steps that we need to perform in our pipeline in order to deploy the code.

Build and Test

In this stage we are going to build our project and test it using maven. We use the maven plugin to publish the last successful artifact:

Build Docker Image

At this stage we are going to build the docker image using a Dockerfile. You can find this file in the project’s root path:

The Dockerfile is really simple, we are using openjdk:13-alpine to execute our java application and we are installing curl in line 5. We are using curl in ECS to check if the container is up or down.

In this stage we are using the docker plugin to build the image and tag it with the ECR url that we have been saving previously in the Jenkins’s credential. We’ll look at how to set up the credentials in Jenkins later in this article.

Push Image to ECR

After building the docker image it is time to push it to our ECR. In this section we are using the AWS plugin to use the AWS credential necessary to push the image. In line 7 we have a different sh command, the reason for this is so we hide the login command to prevent printing it in our log file.

Deploy in ECS

In the last stage we are updating our task definition in AWS with the new image version and deploying this service:

Post

This section defines one or more additional steps that are run upon the completion of a Pipeline’s run.

Once the pipeline is completed, we want to show the code’s coverage information using jacoco and clean our directory and remove the docker image from our local repository.

Jenkins Credentials

The Credentials allow us to save sensible data such as usernames, passwords and so on in Jenkins in a secure way. In this article we are using the credentials to save our AWS user. Important! You need to create an AWS user only for Jenkins with the minimum permissions necessary. In our case Jenkins’s user only has permissions for push image to ECR and perform the deployment.

To create the credentials we firstly have to go to our pipeline, in the left menu we are going see the option credentials. From this menu we can select our pipeline scope and finally we are going to have the option to add credentials.

After adding the credentials this will be available from the pipeline using the ID to identify it.

Summary

After completing this guide we can execute our pipeline and deploy our code (user-api Github code):

Now we can take this basic pipeline and add as many steps as we need to have the perfect pipeline for our project. For example, we can add one more stage after the deployment stage in ECS that monitors and waits until the new version of the application has started successfully, or shows an error.