Creating Pipeline for EC2 on AWS with GitLab End to End Guide

Hello everyone, in this post, we will discuss how to deploy your web application (nodejs in our case) to the EC2 machine on AWS using GitLab ci/cd.

Here’s a quick diagram you can follow along.

So what we are trying to achieve here? we wanted to push our code to the GitLab (main branch) and that code will be squeezed into the zip file, and then pushed to the s3 bucket. After that, the code pipeline will be triggered when there would be a new version on s3 for our code and after that, it will be used by code deploy to deploy our code into the target group(which is EC2 in our case).

I know it might sound complicated on the first look but I will try to divide it and explain part by part.


Creating an IAM User

Before diving into the creating of our pipeline, We will need an IAM user for that and you might want to add the following permissions:

  • CodePipelineFullAccess
  • AmazonEC2FullAccess
  • AmazonS3FullAccess
  • AWSCodeDeployFullAccess

Please follow the best practices provided by AWS by scoping your IAM policies and tightening your security.


Creating EC2 Instance & Role

Create a Role on IAM and attach the following policies:

  • AmazonEC2FullAccess
  • AmazonS3FullAccess
  • AWSCodeDeployRole

Attach that role to your EC2 Instance.


Install CodeDeploy Agent on EC2

Since we will use CodeDeploy we have to install its agent in our EC2 Instance.

Follow this guide to install CodeDeploy:

https://docs.aws.amazon.com/codedeploy/latest/userguide/codedeploy-agent-operations-install.html


Create an S3 repository

Please follow this article to create your S3 repository. Don’t forget to enable versioning on them.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html


Creating an application on AWS CodeDeploy

Since the basic elements are set we will start creating our application on CodeDeploy.

  1. Go to the CodeDeploy > Applications then click Create Application
  2. Choose the application name and Compute platform (EC2 in our case).
  3. After that click on Create deployment group.
  4. Choose a deployment group name, service role, deployment type, Environment configuration, Deployment settings, Load balancer, and then click on Create deployment group.

Creating a pipeline on AWS CodePipeline

We will create a CodePipeline so whenever there is a new version of our code on the S3 bucket it will be notified and it will push the necessary changes to the CodeDeploy.

  1. Click on Create Pipeline
  2. Choose a pipeline name and service role.
  3. Pick your source provider (S3 in our case). Pick your S3 bucket and enter the format for your zip file (yourcode.zip). Don’t forget to pick AWS CodePipeline.
  4. Skip build stage.
  5. For the Deploy stage pick our CodeDeploy and fill the necessary fields.
  6. Click on Create Pipeline.

Setting up the environment on GitLab

Since sharing your credentials on GitLab-ci.yml is a security flaw, we will follow the best practices provided by GitLab and we will add our variables at GitLab > CI/CD > Variables.

  • AWS_DEFAULT_REGION
  • AWS_SECRET_ACCESS_KEY
  • AWS_ACCESS_KEY_ID

Our GitLab runners will automatically sign you in for your AWS account, so you don’t have to do that manually.


CloudPipelines role in that architecture

CodePipeline detects the change in the revision of the S3 zip file

CodePipeline validates the file

CodePipeline sends a signal that the bundle for CodeDeploy is ready

CodeDeploy role on that architecture

CodeDeploy executes the deployment steps

Start – initialization of the deployment

Application Stop – Executes defined script for hook

DownloadBundle – Gets the bundle file from the S3 repository through the CodePipeline

BeforeInstall – Executes defined script for hook Install – Copies the contents to the deployment location as defined by the files section ofappspec.yml

AfterInstall – Executes defined script for hook

ApplicationStart – Executes defined script for hook

ValidateService – Executes defined script for hook End – Signals the CodePipeline that the deployment has completed successfully

Since our Infrastructure looks okay to me, time to start with gitlab-ci.yml file

image: node:latest

# This folder is cached between builds
# https://docs.gitlab.com/ee/ci/yaml/index.html#cache
cache:
  paths:
    - node_modules/

before_script:  
  - apt-get update -qq && apt-get install
  - apt-get install zip -y

# You specify the stages. Those are the steps that GitLab will go through 
# Order matters. 
stages:
  - build

Build:
  # Script to run for deploying an application to AWS
  
  when: manual # Manual deployments switch
  stage: build
  script:
    - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" # Downloading and installing awscli
    - unzip awscliv2.zip
    - ./aws/install
    - npm install
    - zip -r zipfile.zip .
    # - aws s3 cp zipfile.zip s3://s3-name # Uploads the zip file to S3 and expects the AWS Code Pipeline/Code Deploy to pick up
    - aws deploy push --application-name CodeDeployAppName --s3-location s3://s3-name/zipfile.zip # Adding revision to s3 bucket
    - aws deploy create-deployment --application-name CodeDeployAppName --s3-location bucket=s3-lcoation,key=zipfile.zip,bundleType=zip --deployment-group-name CloudDeploymentGroupName --deployment-config-name CodeDeployDefault.OneAtATime --file-exists-behavior OVERWRITE # Ordering the deployment of the new revision
    
  environment: production
  cache:
    key: $CI_BUILD_NAME:$CI_BUILD_REF_NAME
    untracked: true
    paths:
        - build/
    when: always

So I wanted to show you the two ways of doing it. You can uncomment the - aws s3 cp zipfile.zip s3://mlm-deployments # Uploads the zip file to S3 and expects the AWS Code Pipeline/Code Deploy to pick up line and upload your zip file to the S3 and wait for CodePipeline to be triggered and send it to the CodeDeploy. But in the second approach (uncommented section) it will deploy directly to the CodeDeploy. I found both of them useful but it is up to you to choose your environment by your needs.

when: manual # Manual deployments switch this code snippet lets you run your pipeline manually. I highly recommend using that switch if you are just adopting the DevOps culture. You can remove that snippet if you wanted to automatically run your pipelines whenever a code change.

After that CodeDeploy has to know what kind of scripts does it needs to run. We will create an appspec.yml file for that.

version: 0.0
os: linux
files:
  - source: /
    destination: /home/ubuntu/app
hooks:
  BeforeInstall:
    - location: scripts/before_install.sh
      timeout: 300
      runas: ubuntu
  AfterInstall:
    - location: scripts/after_install.sh
      timeout: 300
      runas: ubuntu
AppStart:
   - location: scripts/app_start.sh
      timeout: 300
      runas: ubuntu

After that, we will create the scripts we defined at the appspec.yml file.

#before_install.sh
#!/bin/bash
HOME=/home/ubuntu
APP=$HOME/app
LOG=$HOME/deploy.log

sudo apt-get update -qq && sudo apt-get install
#add any necessary bash needs here.

/bin/echo "## Before Install Completed ##" >> $LOG
#after_install.sh
#!/bin/bash
HOME=/home/ubuntu/
LOG=$HOME/deploy.log

# verify that the application directory has the correct owner/group
/usr/bin/sudo /bin/chown -R ubuntu:ubuntu /home/ubuntu/app
#add any necessary bash needs here.

/bin/echo " ## After Install Completed ##" >> $LOG
#app_start.sh
#!/bin/bash
HOME=/home/ubuntu
APP=$HOME/app
# CMD=$APP/app.sh
LOG=$HOME/deploy.log

cd $APP
sleep 5
sudo chown ubuntu:ubuntu /home/ubuntu/.pm2/rpc.sock /home/ubuntu/.pm2/pub.sock
sudo pm2 reload all --force --update-env

/bin/echo " ## App Start Completed ##" >> $LOG

Don’t forget to add additional scripts for your needs and modify the scripts for your architecture.

Successful deployment screenshots:

Reference:

https://docs.aws.amazon.com/codedeploy/latest/userguide/tutorials.html

https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials.html

https://about.gitlab.com/blog/2020/12/15/deploy-aws/

https://docs.gitlab.com/ee/ci/cloud_deployment/

https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-example.html

Related Posts