How to set up Jenkins and a Pipeline on AWS

Introduction
Create an IAM User
Create a Key pair
Create a Security Group
Create an EC2 instance
Install and Configure Jenkins
Create a Pipeline
Clean Up
Summary
Referrals

Introduction

In this blog we are going to set up Jenkins on an EC2 instance and then set up a pipeline to copy a file from S3.
We are going to start something very basic and enhance it as we move along.

What is Jenkins?

Jenkins offers a simple way to set up a continuous integration or continuous delivery (CI/CD) environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks. While Jenkins doesn’t eliminate the need to create scripts for individual steps, it does give you a faster and more robust way to integrate your entire chain of build, test, and deployment tools than you can easily build yourself.

What is Jenkins Pipeline?

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines as code via the Pipeline domain-specific language (DSL) syntax.

Demo

Let's get started with the demo.

Step 1. Create an IAM User

Navigate to IAM > Users > Click on Add users
Enter User Name as JenkinsUser
Select Access key - Programmatic access and Click Next: Permissions
Click Attach existing policies directly and then Create policy
In the Create policy window, Click on JSON and add the following JenkinsEC2Policy and save the policy as JenkinsEC2Policy
Select AmazonS3ReadOnlyAccess and JenkinsEC2Policy under Attach existing policies directly
Click Next:Tags, Next:Review and then Create user

Download the credentials .csv file as we will need this during Jenkins configuration at Step 6.

JenkinsEC2Policy

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "Stmt1312295543082",
        "Action": [
            "ec2:DescribeSpotInstanceRequests",
            "ec2:CancelSpotInstanceRequests",
            "ec2:GetConsoleOutput",
            "ec2:RequestSpotInstances",
            "ec2:RunInstances",
            "ec2:StartInstances",
            "ec2:StopInstances",
            "ec2:TerminateInstances",
            "ec2:CreateTags",
            "ec2:DeleteTags",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceTypes",
            "ec2:DescribeKeyPairs",
            "ec2:DescribeRegions",
            "ec2:DescribeImages",
            "ec2:DescribeAvailabilityZones",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSubnets",
            "iam:ListInstanceProfilesForRole",
            "iam:PassRole",
            "ec2:GetPasswordData"
        ],
        "Effect": "Allow",
        "Resource": "*"
    }
]
}

IAM_JenkinsUser1

IAM_JenkinsUser2

IAM_JenkinsUser3

Step 2. Create a key pair

Navigate to EC2 > Key Pairs (under Network & Security).
Click Create key pair.
Navigate to the folder where the key pair is downloaded and run.

   chmod 400 <key_pair_name>.pem

Jenkins_Keypair

Step 3. Create a Security Group

We are going to create a Security Group for SSH and Jenkins web access.
Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
- Name: JenkinsSG.
- Add an Inbound rule to allow SSH (TCP 22) traffic from My IP.
- Add another Inbound rule to allow Custom (TCP 8080) traffic from My IP.

Jenkins_SG

Step 4. Create an EC2 instance

Navigate to EC2 > EC2 Dashboard > Click on Launch instance.
Launch an instance with the following values as shown in the screenshots.
When the instance state is Running, select the instance and click on Connect and then copy the connections details.

Connect to the instance.

ssh -i "Jenkins-Keypair.pem" ec2-user@ec2-54-211-70-130.compute-1.amazonaws.com

Jenkins_EC1

Jenkins_EC2

Step 5. Install and Configure Jenkins

Install Jenkins

Ensure that the software packages are up to date on the instance by executing the following command:
```
[ec2-user ~]$ sudo yum update –y
```

Add the Jenkins repo using the following command:

[ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo

Import a key file from Jenkins-CI to enable installation from the package:

[ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

[ec2-user ~]$ sudo yum upgrade

Install Java:

[ec2-user ~]$ sudo amazon-linux-extras install java-openjdk11 -y

Install Jenkins:

[ec2-user ~]$ sudo yum install jenkins -y

Enable Jenkins service to auto start at boot:

[ec2-user ~]$ sudo systemctl enable jenkins

Start Jenkins as a service:

[ec2-user ~]$ sudo systemctl start jenkins

You can check the status of the Jenkins service using the command:
```
[ec2-user ~]$ sudo systemctl status jenkins
```
Just in case we ever need to restart Jenkins during this setup/configuration.
```
[ec2-user ~]$ sudo systemctl restart jenkins
```

Configure Jenkins

Copy the Public IPv4 DNS of EC2 instance and paste the URL as following in the browser For Example: http://ec2-54-211-70-130.compute-1.amazonaws.com:8080

Enter the initialAdminPassword from /var/lib/jenkins/secrets/initialAdminPassword and Click Continue

[ec2-user@ip-172-31-89-157 ~]$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword
    671x0x5x3xxx46xxxxxx099x1xf0149x

Select Install suggested plugins.
Once the installation is complete, Create First Admin User will open. Enter your information, and then select Save and Continue.
Click Dashboard, select Manage Jenkins, and then select Manage Plugins.
Click Available plugins, Search and Select Amazon EC2 and then **Install without restart*
Once the installation is complete, Navigate back to Dashboard, select Manage Jenkins, select Manage nodes and clouds and then click on Configure Clouds.
Select Add a new cloud, and select Amazon EC2. A new pop up window opens with more fields.
Click Add under Amazon EC2 Credentials and Select Jenkins
- From the Jenkins Credentials Provider: Jenkins, select AWS Credentials as the Kind.
- Enter Access Key ID, Secret Access Key from the key pair and Click Add.
Scroll down to Region and select your Region.
Click Add under EC2 Key Pair's Private Key and Select Jenkins.
- From the Jenkins Credentials Provider: Jenkins, select SSH Username with private key as the Kind and set the Username to ec2-user.
- Select Enter Directly under Private Key, then select Add.
- Open the private key pair you created in the creating a key pair step and paste in the contents from -----BEGIN RSA PRIVATE KEY----- to -----END RSA PRIVATE KEY-----. Select Add when completed.
Scroll down to Test Connection and ensure it states Success and then Click Save.

Jenkins_Configure1

Jenkins_Configure2

Jenkins_Configure3

Jenkins_Configure4

Jenkins_Configure5

Jenkins_Configure6

Jenkins_Configure7

Jenkins_Configure8

Jenkins_Configure9

Jenkins_Configure10

Jenkins_Configure10.1

Jenkins_Configure10.2

Jenkins_Configure11

Jenkins_Configure11.1

Jenkins_Configure11.2

Jenkins_Configure12

Step 6. Create a Pipeline

Navigate to Dashboard > Manage Jenkins and copy the ID of IAM User created in Step 1, which we need to replace xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the ID.
Navigate to Dashboard > Manage Jenkins > Plugin Manager > Click on Available plugins
Search for AWS Steps and Install without restart.
Navigate to Dashboard > New Job > Enter download-a-file-from-s3 > Select Pipeline and Click OK.
Scroll down to Pipeline, add the following Pipeline script and Click Save
Create an S3 bucket and copy some files to the bucket.
Replace the s3bucket with your bucket name and filename with one of the files in your S3 bucket.
Navigate to Dashboard > download-a-file-from-s3 and Click on Build Now.

Navigate to the latest Build History link and check out the Console Output.

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    s3Download(file: "filename", bucket: 's3bucket', path: '')
                }
            }
        }
    }
}

Jenkins_Pipeline1

Jenkins_Pipeline2

Jenkins_Pipeline3

Jenkins_Pipeline4

The following is the console log, which shows the pipeline has been successful. ✅

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh 
Finished: Downloading from s3bucket/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

The file that was copied from S3 to the EC2 instance. ✅

[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/
total 0
drwxr-xr-x 2 jenkins jenkins 55 Dec 31 03:45 receiveMessages.sh

Let's run the build again, this time the build has failed with the following error ❓
The error message suggests to use set force=true.

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/ 
Download failed due to existing target file; set force=true to overwrite target file
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Target exists: file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:146)
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:113)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

So we need to add force:true as per the Pipeline: AWS Steps documentation

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    s3Download(file: "filename", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}

The build succeeded after adding force:true and now we can run the build multiple times.

Now let's parameterise the destination foldername.

Navigate to Dashboard > download-a-file-from-s3 and click on Configure
Select This project is parameterised and then Add String parameter
Enter Name as foldername, Default Valu as foldername and Save
Click Build with Parameters
Navigate to the latest Build History link and check out the Console Output.

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    echo "${foldername}" 
                    s3Download(file: "${foldername}", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}

Jenkins_Pipeline_Parameter

Jenkins_Pipeline_Build-with-Parameters

The following is the console log, which shows the pipeline has been successful. ✅

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] echo
s3files
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
Finished: Downloading from sqssri/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

The files were copied to s3files directory on the EC2 instance. ✅

[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
total 8
-rw-r--r-- 1 jenkins jenkins 952 Dec 31 03:40 receiveMessages.sh
-rw-r--r-- 1 jenkins jenkins 646 Dec 31 03:40 sendMessages.sh

Blog