Home / Integrating Jenkins CI with Azure Pipelines

Overview

Jenkins is a very popular Java-based open source continuous integration (CI) server that allows teams to continuously build applications across platforms. Azure Pipeline includes the ability to build any application on any platform including Windows, Linux and Mac. However, it also integrates well with Jenkins for teams who already use or prefer to use Jenkins for CI.

There are two ways to integrate Jenkins with Azure Pipelines:

  • One way is to run CI jobs in Jenkins separately. This involves configuration of a CI pipeline in Jenkins and a web hook in Azure DevOps that invokes the CI process when source code is pushed to a repository or a branch.

  • The alternate way is to wrap a Jenkins CI job inside an Azure pipeline. In this approach, a build definition will be configured in Azure Pipelines to use the Jenkins tasks to invoke a CI job in Jenkins, download and publish the artifacts produced by Jenkins.

An Azure CD pipeline can be configured to pick these build artifacts, irrespective of the approach, for deployment. While there are pros and cons with both the approaches, the latter approach has multiple benefits:

  1. End-to-end traceability from work item to source code to build and release
  2. Triggering of a Continuous Deployment (CD) when the build is completed successfully
  3. Execution of the build as part of the branching strategy

What’s covered in this lab

This lab covers both the approaches and the following tasks will be performed

  • Provision Jenkins on Azure VM using the Jenkins template available on the Azure Marketplace
  • Configure Jenkins to work with Maven and Azure DevOps
  • Create a build job in Jenkins
  • Configure Azure Pipeline to integrate with Jenkins
  • Configure a CD pipeline in Azure Pipelines to deploy the build artifacts

Before you begin

  1. Microsoft Azure Account: You will need a valid and active Azure account for the Azure labs. If you do not have one, you can sign up for a free trial

  2. Putty a free SSH and Telnet client

  3. Set up your Azure DevOps project using the MyShuttle template in the Azure DevOps Demo Generator. We will use a Java web app that connects to a MySQL backend.

Setting up the Jenkins VM

  1. To configure Jenkins, the Jenkins VM image available on the Azure MarketPlace will be used. This will install the latest stable Jenkins version on an Ubuntu Linux VM along with the tools and plugins configured to work with Azure. Click the Deploy to Azure button below to get started.

    Jenkins Configuration

  2. Once the Jenkins VM is provisioned, select the Connect button and make a note of the username and the ip address. This information will be required to connect to the Jenkins VM from Putty

    Connecting to the virtual machine

  3. To initiate an SSH tunnel, the following command needs to be run from a Command Prompt. An SSH tunnel creates a secure connection between your host and remote computer through which services can be relayed. If this command is successful, you should be able to access the remote Jenkins on port 8080 on your local machine.

     putty.exe -ssh -L 8080:localhost:8080 <username>@<ip address>
    

    Initiating a SSH tunnel

  4. Log in with the user name and password that you provided while provisioning the Jenkins VM.

  5. Once the connection is successful, open a browser on the host machine and navigate to the URL http://localhost:8080. The Getting Started page for Jenkins will be displayed.

  6. For security reasons, Jenkins generates an initial password and save it in a file on the server. This password will need to be retrieved and provided to unlock Jenkins. Return to the Putty terminal and type the following command to open the password file and copy the password. You can double click on the password text and use CTRL+C to copy the text and place it in the clipboard. Press the Esc button and then type :q! at the prompt to exit the vi editor without saving the file.

    sudo vi /var/lib/jenkins/secrets/initialAdminPassword

  7. Return to the browser, paste the copied text in the Administrator password text box and click on the Continue button.

    Unlock Jenkins - First Time

  8. Jenkins has a vast ecosystem with a strong and active open source community users contributing hundreds of useful plugins. While configuring Jenkins, you can choose between installing the most commonly used plugins or go for specific selected plugins. Select Install suggested plugins to initiate the configuration with default plugins. We will install other plugins such as Maven, Azure DevOps manually later.

    Customize Jenkins Plugins

  9. The final step is to create a new Admin user. Provide User name, Password, Full name and Email address for the user. Select Save and Finish when you are done.

    Create Admin User for Jenkins

  10. Jenkins is now ready for use. Select Start using Jenkins to start using it.

    Jenkins Ready

Installing and Configuring Plugins

  1. We will now install the Maven and VSTS (yet to be renamed Azure DevOps!) plugins that we require for this lab. Click Manage Jenkins on the Jenkins home page and select Manage Plugins. Select the Available tab and search for team services

    Manage Jenkins

  2. Select VS Team Services Continuous Deployment plugin and select Install without restart

  3. Select Manage Plugins, select the Available tab and search for maven-plugin

  4. Select the Maven Integration Plugin and select the Install without restart button to install the plugin.

    Install Maven

  5. Once the plugin is installed, go back to Manage Jenkins and select the Global Tool Configuration option.

    Global Tool Configuration

  6. To install Maven, select the Install automatically option and select the Apply button. The latest version of Maven at the time of writing this lab is 3.5.4

    Maven Installer

  7. Select the Back to Dashboard button to return to the home page. We are done with the setup. Let’s go and create a new CI job.

Creating a new build job in Jenkins

  1. From the Jenkins home page, click theNew Item link. Provide a name for the build definition, select Maven project and click OK.

    New Build Definition

  2. Now scroll down to the Source code Management section. Select Git and provide the clone URL of the Azure DevOps Git repo in the format http://dev.azure.com/{your org name}/{team project name}/_git/MyShuttle. If you do not see Git under Source code management (not a usual thing), you will need to install/enable the Git plugin.

    Configuring Azure DevOps Git URL

  3. Your Azure repo is very likely to be private. Unless you have a public repo, you should provide the credentials to access the repository. If you do not have one or don’t remember the credentials, go to your Azure Repos and select the Clone option. Select Generate Credentials and enter a User name and Password. Click Save Git Credentials to save.

    Generating Git Credentials

  4. Select the Add | Jenkins option to add a new credential. Provide the User name and Password created earlier and click the Add button to close the wizard

    Adding Credentials to Jenkins

  5. Select the credential created in the previous step from the drop-down. The error message should disappear.

    VSTS Git config in Jenkins

  6. The source code for this application has both unit tests and UI tests. We are not ready to run the UI test at this point. So, we will specify to run only the unit tests. Scroll down to the Build section and provide the text package -Dtest=FaresTest,SimpleTest in the Goals and options field.

    Build Settings in Jenkins

  7. Once the build is complete, you can specify what action you want to take. For instance, you can archive the build artifacts, trigger an Azure CD pipeline, deploy directly to Azure App Service, etc., We will choose the Archive the artifacts option in the Post-build Actions.

    Post Build Action

  8. Enter target/*.war in the Files to archive text box. Select the Save button to save the settings and return to the project page.

    Archive War

  9. The configuration is now completed, Select the Build Now option to initiate an Ad-hoc build. The build progress will be displayed on the left pane in the Build History section

    Running Ad-hoc Build

  10. To view the build details and the list of build artifacts, select the build number displayed in the Module Builds section.

    Build Details

    Build Details

  11. Select the Test Result option to view the results of the unit tests that were included in the build definition.

Next, we will explore the two different options available to trigger the Jenkins CI job when a code is pushed to Azure Repos.

Approach 1: Triggering the CI via a service hook in Azure DevOps

In this approach, a service hook will be configured in Azure DevOps to trigger a Jenkins build upon a code commit. Service hooks enable you to perform tasks on other services when events happen in your Azure DevOps Services projects.

  1. To configure the service hook, navigate to the Azure DevOps project settings page and select Service hooks under General. Select + Create subscription.

  2. In the New Service Hook Subscriptions screen, select the Jenkins option and then click the Next button. Jenkins service supports three events - Build completed, Code Pushed and Pull request merged. We are only interested in the code push event - so, select Code pushed for the Trigger on this type of event field. Select the MyShuttle repository and then click Next

    VSTS - Trigger Code Pushed

  3. Provide the following details in the Select and configure the action to perform screen
    1. Select the Trigger Git build option

    2. Provide the Jenkins base URL in http://{ip address or the host name} format

    3. Provide the User name and Password to trigger the build. Note the username and password is the credentials of the Jenkins administrator user that you configured earlier

  4. Click the Test button to validate the configuration and then click Finish. This will set the trigger to initiate the Jenkins CI build whenever a source code change is committed on the repository.

  5. Try making a commit to the code - src/main/webapp/index.jsp would be a good candidate. This should trigger the MyShuttle build on Jenkins. You can confirm it by checking the history tab of the Jenkins services hook.

    Jenkins Trigger History

Approach 2: Wrapping Jenkins Job within Azure Pipelines

In this approach, Jenkins CI job will be nested within an Azure CI pipeline. The key benefit of this approach is you can have end-to-end traceability from work items to source code to build and release pipelines.

To begin, an endpoint to the Jenkins Server for communication with Azure DevOps will be configured.

  1. Go to your project settings. Select Pipelines and Service connections, click New service connection and choose Jenkins from the dropdown.

  2. Provide a connection name, Jenkins server URL in the format http://[server IP address or DNS name] and Jenkins user name with password. Select Verify Connection and validate the configuration. If it successful, then select Ok.

    Jenkins Endpoint

    The next step would be to configure the build pipeline.

  3. Go to Azure Pipelines and Builds, Click +New and select New build pipeline to create a new build definition.

  4. At the time of writing this lab, Azure Pipelines did not support Jenkins in YAML. Select Use the Visual Designer to create a pipeline without a YAML.

  5. Select Myshuttle project, repository and click Continue.

  6. Scroll down and select the standard Jenkins template Click Apply.

    [Jenkins Template

  7. Select Hosted VS2017 for the Agent Queue, provide MyShuttle as the Job name ( name of the build definition that was created in Jenkins) and then select the Jenkins service endpoint created earlier.

    Jenkins Settings in Team Build

  8. Next, select the Get Sources step. Since Jenkins is being used for the build, there is no need to download the source code to the build agent. To skip syncing with the agent, select Don’t sync sources option.

    Get Sources Settings in Team Build

  9. Next, select the Queue Jenkins Job step. This task queues the job on the Jenkins server. Make sure that the services endpoint and the job name are correct. The Capture console output and the Capture pipeline output options available at this step will be selected.

    The Capture console output and wait for completion option, when selected, will capture the output of the Jenkins build console when the Azure build pipeline runs. The build will wait until the Jenkins Job is completed. The Capture pipeline output and wait for pipeline completion option is very similar but applies to Jenkins pipelines (a build that has more than one job nested together).

    Jenkins Settings in Team Build

  10. The Jenkins Download Artifacts task will download the build artifacts from the Jenkins job to the staging directory.

    Download Jenkins Artifact

  11. The Publish Artifact drop will publish to Azure Pipelines.

  12. Click Save & queue button to save and initiate a new build.

Linking the build artifact for deployment in a CD pipeline

Next, you will configure an Azure CD pipeline to fetch and deploy the artifacts produced by the build. Since the deployment is being done to Azure, an endpoint to Azure will be configured. An endpoint to Jenkins server will also be configured, if not configured earlier.

  1. After the endpoint creation, go to the Releases tab in Azure Pipelines. Open the + drop-down in the list of release pipelines, and choose Create release pipeline.

  2. Select the Azure App Service Deployment template.

    New Release Definition

  3. The default environment for deployment will be named as Dev

    New Environment

  4. In the Artifacts section in the Pipeline tab, choose the + Add link to select your build artifact.

    1. If you have used the first approach, select Jenkins as the Source type, select the Jenkins endpoint configured earlier and provide MyShuttle for the Source(Job), choose the Default version as Latest. The Source(Job) should map to the project name configured in Jenkins.

      If the Jenkins server and the source location is configured correctly, once the publishing of the artifacts is completed, a message with the output file name myshuttledev.war will be displayed.

      Add Jenkins artifact

    2. Otherwise, point this to the Azure CI build pipeline from which the Jenkins CI is executed.

  5. Now, the artifact is linked for deployment. Please refer the Deploying a MySQL Backed Tomcat app on Azure Web App lab for deploying the WAR file to Azure App Service.