Jenkins and SVN – Organizing Around Branches and Deployment Environments

automationdeploymentjavaJenkins

Current setup

We currently have one Jenkins server that automates deployments for about 15 different Java web applications. Each application has three deployment environments on separate Linux boxes.

  1. Development
  2. Test
  3. Production

In Jenkins every application has it's own "view". Inside every view there is a job for every branch. Those jobs simply build a war file and place it on the Jenkins server. Then to deploy a DevOps person will SSH into the deployment environment (whether its dev, test, prod) for a particular application and run a SCP script that moves the war file from the Jenkins server to the deployment environment (usually into a Tomcat webapps folder for deployment).

Issues

Obviously this is a very silly process. I want to have a Jenkins deployment process that actually deploys the application (so no one needs to SSH into a box and run an SCP script shell script). I have got this process working for some of our smaller applications, but I am running into two issues.

  1. I don't know the best way to organize my jobs
  2. Some coworkers are worried about how easy this would be to accidentally push a production build.

Explained further

On issue #1 – An obvious brute force way would be to create a job for each deployment environment. However, in that case I don't know how to make it so that the "Test deployment environment" job can take in as a parameter the branch I wish to deploy. I don't want to have to make three jobs for every new branch (one for each deployment environment). Additionally, I could create a job for every branch, but then I don't know how send the deployment environment in as a parameter? I don't want to have to reconfigure these jobs each build which is the only way I've found thus far.

On issue #2 – Is there any type of plugin or generally strategy used to protect against nefarious production deployments? I realize this question might be heavily based off my answer to question #1. Is having a separate account specifically for production builds an efficient way to protect against this?

Best Answer

I'm going to suggest you move to a slightly more complex but far more flexible system. Basically, I recommend you shift away from SVN branches that are linked to environments. We used this for several years and had no serious problems.

  • If you don't already have one, install a repository product like Nexus.
  • Write deployment jobs for Jenkins that deploy a numbered version to Nexus, but do not deploy to the DEV, UAT or PROD hosts. This job will tag all files with the version number, and put the versioned .war file into Nexus. Assuming you use Maven or Gradle as your build tool, this is out of the box functionality for Jenkins.
  • Write deployment jobs for Jenkins that ask the user for a version number and target host. These are simply jobs that use SVN checkouts augmented by bash scripts that Jenkins will execute. You will need to add a few Jenkins plugins to make life easier (e.g Setenv to set variables visible to the whole job)
  • Install a second Jenkins instance for use of DEVOPS. This will host jobs that deploy a particular version to the the higher environments. This is mainly for security if developers are not allowed to do deployments to some environments.
  • Your use of SVN will change. We simply used the trunk for development, and branches for each deployed version. You only need to create version branches on demand (branch from tag). You typically will also create an occasional branch to try out a new feature.
  • Put up a white board in a prominent position that lists what versions are currently in each environment.

So what will this give you?

  • You don't need to do a code merge when moving between environments, as branches are no longer linked to environments.
  • You can quickly regress any environment to a different version. This is great for chasing obscure bugs.
  • Insist that defect reports include the version number. This gives good traceability of when the defect was fist detected.

There are a few fish hooks.

  • Your code must be written so that the same .war file will run in any environment. This means all configuration data must be stored as environment variables and/or JNDI variables. You can also use configuration files that are separately controlled and versioned (typically by DEVOPS). The actual location of these files will be specified by an environment or JNDI variable.
  • You will need to setup certificate-based trust between Jenkins and the target deployment machines. This is necessary so that you can use SSH to move files to the target hosts and execute commands on the target host - SSH does not support username/password authentication from a script.

I realise this may be quite a change for your team, so good luck!

Related Topic