Building a CI/CD Pipeline: the Complete Guide
DevOps has taken over the software development world — and continuous integration/continuous delivery (CI/CD) pipelines are a key reason why. CI/CD pipelines automate the software production process, helping you reliably deliver applications in less time while keeping quality high.
Keep reading to learn more about how CI/CD pipelines can keep you ahead of the competition. And if you’re already familiar with the concept of a CI/CD pipeline, use the table below to skip ahead to the sections on creating a CI/CD pipeline and CI/CD pipeline tools.
A CI/CD pipeline is a set of connected, heavily automated processes that encompass everything needed to deliver software. They’re designed to deploy software efficiently and keep errors to a minimum. With a CI/CD pipeline, software developers can quickly make commits that are automatically built and tested, fostering an agile, iterative approach to development.
While there’s no single way to develop a CI/CD pipeline — and your organization’s CI/CD pipeline should be uniquely suited to its needs — they typically involve the following steps:
- Building the software through frequent commits.
- Testing the software each time a commit is made, which continuously provides valuable feedback to developers.
- Deploying the software to staging and production environments.
Continuous integration and continuous delivery are important, but distinct, parts of a CI/CD pipeline. CI involves the continuous merging — or integration — of code into a repository, with automated building and testing of those builds. CD encompasses the rest of the process, taking the new build through release to the end user.
If you’re not already using a CI/CD pipeline, you’re missing out on several key benefits:
- Efficiency - Automating key parts of the software development process lets you deliver applications faster and frees your developers to focus on more important tasks like creating code.
- Reliability - Human error is taken out of the equation for the automated portions of the CI/CD pipeline, and developers will have more time to monitor and troubleshoot those problems that do arise.
- Flexibility - A CI/CD pipeline keeps your team agile and responsive. They can make necessary changes and iterate on previous work without delay, and the ability to adapt to the ever-changing needs of end users leads to higher customer satisfaction.
- Transparency - A CI/CD pipeline records data on everything from code commits to test results, letting your developers peer under the hood of the process at any time.
- Profitability - Faster, more reliable development at a lower cost shows up in the bottom line.
Say, for example, you’re looking to launch and support a new application on your Drupal website. Without a CI/CD pipeline, your developers will be stuck manually committing and testing their code, all leading up to a scheduled release where users will finally see the results of their hard work. But if your organization has instead adopted a CI/CD approach, you’ll be able to automatically integrate and deploy new code that successfully passes the requisite tests. This provides your users with a regular stream of new features and quality of life improvements as soon as they’re ready, while letting your developers focus on what they do best — software creation.
Now that you know what a CI/CD pipeline looks like and the advantages it provides, let’s review each stage of the pipeline in detail. We’ll stick with the example of a CI/CD pipeline used for the Drupal web application described above and go through it step by step. The goal of this process is to add a new feature to this web application — for instance, a way for existing members to set up automatic renewals — as quickly as possible without disrupting the user experience.
It all starts with a member of the development team pushing a commit to the source code repository. The CI/CD pipeline then begins its work, building a functional version of the application that includes the new feature. Depending on the programming language you’re using, this may require compiling the software as well. Given that we’re working with Drupal here, PHP is our language of choice, so no compiling is needed.
Before building, ensure your local environment — as well as any container you’re using — is identical to (or at least very closely resembles) your production environment. For example, when building this Drupal pipeline, you’d want to make sure your version of PHP is current and you’re using the same database as your production environment. If you’re using a container like Docker, that should be updated as well. If the application is successfully built, the CI/CD pipeline will move on to testing it.
If building instead fails, don’t panic. With a properly designed pipeline, the issues are limited to your separate branch and won’t affect the master or interrupt the work of other developers. You’ll also have the information at your fingertips needed to identify what went wrong — whether the renewal feature had unintended consequences on the rest of the membership app, for example. Regardless of why the build failed, you’ll have to resolve the issue before the pipeline can proceed to the next stage.
After the build stage completes successfully, the pipeline automatically tests the code to discover whether it performs as intended. This stage ensures that the application remains relatively bug-free and that major errors are caught before users — here, the members accessing your Drupal website — are exposed to them. Excitement about these new renewal capabilities among your customers is likely to be muted if the update prevents them from accessing existing features they rely on.
While the tests are carried out without any manual input, that doesn’t mean the pipeline is devising these tests on its own. The development team will need to determine what tests should be included in this stage for the relevant software and, if necessary, create those tests. While this may take some time, it’s a key part of enabling efficient coding processes, and developers can dedicate some of the additional bandwidth they have thanks to the CI/CD pipeline to this task.
Finally, if you’re running many different types of tests, schedule those that take longer last. You don’t want to wait for the lengthiest tests to complete successfully only to find that one of the quickest tests to carry out has failed. Remember that no one has unlimited testing capability, so prioritize your tests based on the key features of your application that you can’t afford to break.
It’s now time to move from CI to CD. In most cases, you’ll be deploying your software first in a staging environment where your team can conduct further testing and validation. If these tests are again successful, then you can release the new build automatically to your production environment.
Even if your deployment process requires some manual input along the way, automatically pushing every approved change to production is critical. This allows your users to experience the benefits of your development team’s work — here, the automatic renewal feature — in real time, rather than waiting for a set of features to be packaged together in a single release.
There are a variety of tools designed to help you create the perfect CI/CD pipeline for your development team. Let’s dig into the pros and cons of a few of them.
Jenkins is a popular online, open source tool you can use to create CI/CD pipelines. Regardless of what language or repository your team uses, chances are that Jenkins supports it. Jenkins also features many plugins so you can customize it to meet your organization’s needs.
While Jenkins is a powerful tool that costs nothing to use, it’s not perfect. The vast number of plugins can be daunting — and beyond the sheer amount of extensions available, many perform the same function, so it requires some care to sort through them. In addition, as an open source product, there’s no support team that has your back when problems inevitably occur. Finally, again due to its open source nature, you’ll have to handle deploying and configuring Jenkins on your own.
Thankfully, there are plenty of online resources available if you’re interested in trying Jenkins out. For instance, all the Drupal developers out there can take a look at how to integrate Drush, a scripting interface for Drupal, with Phing, a project build system, and then use both as part of a CI/CD pipeline powered by Jenkins.
GitLab — through its built-in tool, GitLab CI/CD — is another popular platform that comes in an open source community edition and two commercial editions. It’s relatively simple to install and configure, and ongoing management is also enhanced thanks to issue tracking capabilities. Like Jenkins, there’s also an active community that can provide support when necessary, and its commercial products are supported by an official team as well.
However, GitLab’s UI may take some time to get used to. And if you're using GitLab for its non-CI/CD features, you'll be hooked into using it for that purpose as well, rather than another tool you may prefer.
There are many online guides for getting started with using Gitlab to create a CI/CD pipeline. If you’re looking to use GitLab CI/CD with your Drupal site, take a look at this freely available integration for Drupal 8.
Enterprises looking for a powerful, easy-to-use CI/CD tool for web development that they can run on their own infrastructure or in the cloud will find what they seek in Cascade. It’s quick to set up and use, and it offers additional configuration options for those users who have custom requirements or infrastructure. Organizations with large IT teams will especially benefit from Cascade, which was designed to be easily used by developers, admins, and all other team members.
Whether your website relies on Drupal, WordPress, Node.js, or another LAMP-based stack, Cascade can accelerate site delivery significantly — in fact, one Cascade user realized a 48-fold reduction in deployment time. It’s also supported by a team of experts that have the knowledge and skill necessary to help your organization craft the perfect CI/CD pipeline.
Establishing and fine-tuning a CI/CD pipeline that truly meets all your organization’s requirements isn’t easy — especially when it comes to web development. That’s why BlackMesh by Contegix stands ready to assist, with its CI/CD platform Cascade and its flexible and compliant BlackMesh hosting service. Whether your development processes are complex or simple, BlackMesh can provide the services needed to automate them and speed up deployment.
Get BlackMesh's take on how your organization can best use a CI/CD pipeline by contacting its experts today.