From Integration to Deployment: A CI/CD Primer

11.09.2024 Jan Marker No comments

Adopting proven practices like continuous integration (CI) and continuous deployment (CD) is a key part of modern software design. These methodologies enhance software quality and team productivity while shortening development cycles – what’s not to love? Arguably, it’s the short-term pain of getting these systems installed and configured for your environment and the long-term pain of maintaining them. But you’ve come to the right place; this blog gives you some practical tips in getting your CI/CD system spun up that avoid common pitfalls and maximize long-term benefits.

Not sure you want to take the plunge? Here’s why we recommend CI/CD practices.

Advantages of CI/CD

CI/CD practices have several advantages that that can transform your development process. Here are some key reasons why implementing them is a smart move:

• Comprehensive and efficient testing: CI runs all relevant tests, not just those immediately impacted by recent changes, so it uncovers potential hidden issues. This allows developers to focus on local testing for quick checks but rely on CI for thorough, asynchronous testing that can run on separate systems in the background.
• Automated code reviews: A CI system acts as an additional, automated code reviewer. By building and testing code across all platforms, it can catch issues that might be missed in peer reviews, all without consuming valuable engineering resources. Plus, it allows you to easily incorporate tools for static analysis, linting, and sanitizing to enhance code quality and stability.
• Consistent code style: CI enforces consistent code formatting across all source files, eliminating style discrepancies that can affect readability. This standardization is crucial for large, distributed teams and helps avoid time-consuming (and unproductive) debates during peer reviews.
• No testing needed: It might seem like you need a fully developed set of unit tests to take advantage of CI. You don’t. You can still get a lot of value from cross-compiling on multiple platforms, doing static analysis, and running format checkers. And once your CI system is in place, a small number of effective tests can validate critical items. Don’t be discouraged if you’ve just started creating your test suites – building up a few tests at a time as you find and fix bugs still offers a lot of benefit.

Essential components for effective CI/CD setup

A successful CI/CD pipeline requires several key components to function smoothly. These elements work together to ensure that your development process is efficient, scalable, and reliable. Here are a few basic requirements to get started:

• Repository: Hopefully this is already a given, but if it’s not, you’ll need a shared repository for all developers.
• CI servers: You need hardware to do the CI builds but it can be on-prem or in-cloud and we’ll discuss the pros and cons of both later. But if you’re considering an on-prem solution, you’ll want at least one dedicated CI server, and possibly multiple CI worker machines based on project size and complexity.
• Team structure: You need people that know in detail how the CI system works. But whether it’s better to have a dedicated specialist or to distribute the necessary CI knowledge and responsibilities throughout the team probably depends on which CI tools you choose. In either case, you probably want to have 2 or 3 people who are part-time focused on CI tasks who can help other team members troubleshoot and act as backup if your dedicated person is unavailable.
• Toolchain: Cloud-based tools like GitHub Actions provide easy, pre-configured setups. But a cloud system isn’t always the best idea. For an on-prem system like Jenkins or Buildbot, expect to invest time in server setup, software installation, worker machine configuration, and ongoing OS maintenance. You also want automation tools like Chef or Ansible to allow systems to run unattended so that the CI/CD system can reset and reboot them automatically.

Cloud versus on-premises

Deciding whether to host your CI/CD system in the cloud or on-premises is an important decision that impacts cost, scalability, and security. Each option has its advantages and disadvantages, depending on your specific needs and constraints. Here are some key considerations to help guide your choice:

Cloud pros

• Simplicity and quick setup: Cloud systems are ready to use without hardware acquisition, installation, or configuration.
• Scalability: Easy scalability is an advantage of cloud-based systems, allowing for quick adjustments to resources as development needs change.
• Integration: Cloud hosted CI/CD systems integrate easily with cloud hosting platforms like GitHub and GitLab, especially when you leverage built-in automation tools like GitHub Actions.

Cloud cons

• Cost: Continuous operation can lead to significant cloud processing and storage fees, especially with frequent multiplatform builds.
• Data privacy and security: Regulations such as GDPR or specific industry requirements may restrict cloud usage, necessitating on-prem solutions for enhanced control over data security and compliance.

On-prem pros

• Cost control: On-prem can be more cost-effective in the long run, especially if large-scale CI operations are needed frequently.
• Data security: On-prem provides control over data privacy and security, which can be essential for regulation

On-prem cons

• Maintenance: On-prem systems need to be maintained, repaired, and upgraded, requiring ongoing engineering or IT resources.
• Spin-up: While a cloud system is available instantly, an on-prem equivalent needs to be built and configured, which can take a significant amount of time, especially if it’s the company’s first.

Optimizing CI builds

CI is only effective when developers use it. And if your CI system is too slow to deliver results, developers will tend to ignore it. How do you achieve a balance of speed and thoroughness?

Essentially, you want to implement two main CI builds – one that allows a quick per-commit build and one that builds nightly. The quick build is run throughout the day as developers make commits and it should use pre-built dependencies and caches for optimal speed. The nightly build executes a full build from scratch every night with all static checks and installers.

If developers are in the same time zone, the two builds can be run on the same machine – otherwise, you’ll want a dedicated machine for nightly builds. You may also consider a third CI build for packaging. This can be used for teams that share external updates, such as customer testing or third-party validation services. This build is somewhere between the quick and nightly builds – it can rely on bits of the quick build (including packaging if it’s fast enough), while benefitting from the deeper coverage of the nightly build.

Ensuring reliability in your CI workflow

In a CI workflow, using developer tools like GitHub, Gerrit, and Bitbucket, CI-gated check-ins are essential. These tools ensure that all changes are validated by the CI system, confirming that commits pass necessary compiler and unit tests before merging into the main branch. Making CI gating optional can lead to the dangerous practice of skipping or bypassing CI builds, which significantly reducing a system’s value.

That’s why your CI system needs to be rock solid. If it frequently experiences issues – like excessive build times due to network problems – developers might be tempted to bypass CI checks altogether. Continuous maintenance is essential to ensure CI systems function effectively as a non-negotiable part of the development workflow.

Where to go next?

This short blog just scratches the surface on CI/CD. If you’re looking for more information and details on some of the most commonly used CI/CD tools – and when you might be better off choosing one over the other – check out our CI/CD best practice guide.