Gluu CI CD pipeline blueprint

Problem statement

Gluu, like any other top company, has the customer's satisfaction as its primary goal. Given the number of Gluu products, it is difficult to meet that goal without a complete, clear and comprehensive workflow.

Below is the list of problems we have to fix:

1. How to provide patches as fast as possible
2. How to catch bugs as soon as possible before reaching production
3. How to react quickly to production outages
4. How to monitor the production server
5. Etc

Solution

DevOps is a software development approach that involves continuous development, continuous testing, continuous integration, continuous deployment, and continuous monitoring of the software throughout its development lifecycle.

Gluu is 100% a software company, and we must take advantage of a philosophy like DevOps, as well as tools and technologies like Jenkins, GitLab, Jira, SCM, etc.

The solution for all of the above problems is a CI/CD pipeline. CI stands for Continuous Integration and CD stands for Continuous Delivery/Continuous Deployment. You can think of it as a process similar to a software development lifecycle.

CI/CD pipeline

Version control

Developer teams use version control to track changes on files, and the code is stored in GitHub. When a developer is done with a feature or a fix, the code is pushed to GitHub.

Build

When code is pushed to GitHub, the Build phase can start. Most of our products are Java-based and use Maven as a build tool. Most of the time, this phase checks if there is a compilation error.

Unit testing

When the build phase is successful, the Unit Test phase runs all unit tests defined in the application. The unit test is intended to validate the domain object class. JUnit or TestNG can be used to write such a test.

Deploy to Development Environment

When the Unit Test is successful, the artifact generated by the build phase can be used to deploy the application in a development environment.

There are two possible scenarios:

1. The dev servers are up and ready: In this case, a script can be used to move artifacts to dev servers.
2. The dev servers aren't ready: In this case, we may have to use configuration management tools like Chef or Ansible to provision the servers and then move the artifacts.

QA

In the last verification phase, QA test cases are performed. Cucumber is the de facto tool to write and automate QA validation.

Deploy to Production Environment

When the QA validation is successful, the final product can safely be deployed to a production server for clients/users. At this stage, we are confident about the product quality because it has passed all verification steps.

Monitoring/Validation (performance monitoring for example)

This phase can be ignored depending on the project, but for transaction applications, it is very important to automate monitoring of the application state/metrics.

Tools

Our task is to automate the entire process, from the time a developer commits a change to the time we get it into production. We will automate the pipeline in order to make the entire software development lifecycle agile, maintainable and spotless. For this, we will need automation tools.

Jenkins:

The leading open source automation server, Jenkins provides hundreds of plugins to support building, deploying and automating any project.

Docker:

Building and deploying new applications is faster with containers. Docker containers wrap up software and its dependencies into a standardized unit for software development that includes everything it needs to run: code, runtime, system tools and libraries. This guarantees that your application will always run the same and makes collaboration as simple as sharing a container image.

Maven:

Apache Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of information.

JUnit:

JUnit 5 is the next generation of JUnit. The goal is to create an up-to-date foundation for developer-side testing on the JVM. This includes focusing on Java 8 and above, as well as enabling many different styles of testing.

Selenium:

Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) be automated as well.

Cucumber:

Cucumber is a tool that supports Behavior-Driven Development(BDD). Cucumber reads executable specifications written in plain text and validates that the software does what those specifications say. The specifications consist of multiple examples or scenarios.

Git:

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

Provisioning tool

SaltStack:

SaltStack uses event-driven automation to transform complex systems into an organized and intelligent digital infrastructure controlled by a central nervous system that senses, learns, and reacts to important IT events across your environment.

Ansible:

Ansible is a universal language, unraveling the mystery of how work gets done. Turn tough tasks into repeatable playbooks. Roll out enterprise-wide protocols with the push of a button. Give your team the tools to automate, solve, and share.

Terraform:

HashiCorp Terraform enables you to safely and predictably create, change, and improve infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared among team members, treated as code, edited, reviewed, and versioned.

Questions

1. What is the best provisioning tool to use?
2. How can we modify the setup.py command to run automatically without user intervention?