Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

by David Farley

This book describes how to revolutionize software delivery by making the path from idea to realized business value—the cycle time—shorter and safer.

Software delivers no revenue until it is in the hands of its users.

the first principle of the Agile Manifesto: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”

The pattern that is central to this book is the deployment pipeline.

The way the deployment pipeline works, in a paragraph, is as follows. Every change that is made to an application’s configuration, source code, environment, or data, triggers the creation of a new instance of the pipeline.

The aim of the deployment pipeline is threefold. First, it makes every part of the process of building, deploying, testing, and releasing software visible to everybody involved, aiding collaboration. Second, it improves feedback so that problems are identified, and so resolved, as early in the process as possible. Finally, it enables teams to deploy and release any version of their software to any environment at will through a fully automated process.

let’s be clear about the kinds of process failures that we are trying to avoid. Here are a few common antipatterns that prevent a reliable release process, but nevertheless are so common as to be the norm in our industry.

Antipattern: Deploying Software Manually

There should be two tasks for a human being to perform to deploy software into a development, test, or production environment: to pick the version and environment and to press the “deploy” button.

Releasing packaged software should involve a single automated process that creates the installer.

If you use the same script to deploy to every environment, then the deployment-to-production path will have been tested hundreds or even thousands of times before it is needed on release day. If any problems occur upon release,

All aspects of each of your testing, staging, and production environments, specifically the configuration of any third-party elements of your system, should be applied from version control through an automated process.

Indeed it should not be possible to make manual changes to testing, staging, and production environments. The only way to make changes to these environments should be through an automated process.

our goal as software professionals is to deliver useful, working software to users as quickly as possible.

Speed is essential because there is an opportunity cost associated with not delivering software. You can only start to get a return on your investment once your software is released.

Delivering fast is also important because it allows you to verify whether your features and bugfixes really are useful.

Visibility is one of the most important benefits of using a CI server.

Most CI server software ships with a widget that you can install on your development machine to show you the status of the build in the corner of your desktop.

Simian is a tool that identifies duplication in most popular languages (including plain text).

identify code duplicates

ratcheting. This means comparing the number of things like warnings or TODOs with the number in the previous check-in. If the number increases, we fail the build.

In large projects with distributed teams, tools like VoIP (e.g., Skype) and instant messaging are of enormous importance to enable the fine-grained communications necessary to keep things running smoothly. Everyone associated with development—project managers, analysts, developers, testers—should have access to, and be accessible to, everyone else on IM and VoIP. It is essential for the smooth running of the delivery process to fly people back and forth periodically, so that each local group has personal contact with members from other groups. This is important to build up trust between team members—often the first thing to suffer in a distributed team.

Virtualization can also work well in conjunction with centralized CI services, providing the ability to spin up new virtual machines from stored baseline images at the press of a button. You can use virtualization to make provisioning new environments a completely automated process, which can be self-serviced by delivery teams. It also ensures that builds and deployments always run on a consistent, baseline version of these environments. This has the happy effect of removing continuous integration environments that are “works of art,” having accumulated software, libraries, and configuration settings over many months that bear no relation to what is present in testing and production environments.

If you have to build binaries or installers locally, it becomes even more essential to ensure that you manage the configuration of your toolchain rigorously to ensure exactly the same binaries are created everywhere. One approach to enforce this is to automatically generate hashes of your binaries, using md5 or a similar algorithm, and have your CI server automatically check them against the hashes of the “master” binaries to ensure there are no differences.

It should be clear that CI requires good team discipline—but then, any process requires this. What is different about continuous integration is that you have a simple indicator of whether or not discipline is being followed:

An established CI system is a foundation on which you can build more infrastructure: • Big visible displays which aggregate information from your build system to provide high-quality feedback • A system of reference for reports and installers for your testing team • A provider of data on the quality of the application for project managers • A system that can be extended out to production, using the deployment pipeline, which provides testers and operations staff with push-button deployments

Too many projects rely solely on manual acceptance testing to verify that a piece of software conforms to its functional and nonfunctional requirements. Even where automated tests exist, they are often poorly maintained and out-of-date and require supplementing with extensive manual testing. This and the related chapters in Part II of this book aim to help you to plan and implement effective automated testing systems. We provide strategies for automating tests in commonly occurring situations and describe practices that support and enable automated testing. One of W. Edwards Deming’s fourteen points is, “Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place” [9YhQXz].

Testing establishes confidence that the software is working as it should, which means fewer bugs, reduced support costs, and improved reputation. Testing also provides a constraint on the development process which encourages good development practices. A comprehensive automated test suite even provides the most complete and up-to-date form of application documentation, in the form of an executable specification not just of how the system should work, but also of how it actually does work.

Brian Marick came up with Figure 4.1, which is widely used to model the various types of tests that you should have in place to ensure the delivery of a high-quality application.

Acceptance tests are critical in an agile environment because they answer the questions, “How do I know when I am done?” for developers and “Did I get what I wanted?” for users.

Modern automated functional testing tools, such as Cucumber, JBehave, Concordion, and Twist, aim to realize this ideal by separating the test scripts from the implementation, while providing a mechanism that makes it simple to keep them synchronized. In this way, it is possible for users to write the test scripts, while developers and testers work together on the code that implements them.

In general, for each story or requirement there is a single canonical path through the application in terms of the actions that the user will perform. This is known as the happy path. This is often expressed using the form “Given [a few important characteristics of the state of the system when testing begins], when [the user performs some set of actions], then [a few important characteristics of the new state of the system] will result.” This is sometimes referred to as the “given-when-then” model for tests.

alternate paths.

sad paths.

In general, we tend to limit our automated acceptance testing to complete coverage of happy path behaviors and only limited coverage of the most important other parts. This is a safe and efficient strategy, assuming that you already have a comprehensive set of automated regression tests of other kinds.

Automated test coverage in this context includes unit, component, and acceptance tests, each of which should cover 80% of the application (we don’t subscribe to the naive idea that you can gain 80% coverage with 60% unit test coverage and 20% acceptance test coverage).

The most important automated test to write is the main happy path test.

Every story or requirement should have at least one automated happy path acceptance test.

Deployment tests are performed whenever you deploy your application. They check that the deployment worked—in other words, that your application is correctly installed, correctly configured, able to contact any services it requires, and that it is responding.

Probably the most comprehensive paper laying out how to use mocks correctly is “Mock Roles, Not Objects” [duZRWb]. Martin Fowler also gives some pointers in his article “Mocks Aren’t Stubs” [dmXRSC].

It is important to remember that you should only write automated tests where they will deliver value. You can essentially divide your application into two parts. There is the code that implements the features of your application, and there is the support or framework code underneath it.

Thus the process modeled by the deployment pipeline, the process of getting software from check-in to release, forms a part of the process of getting a feature from the mind of a customer or user into their hands. The entire process—from concept to cash—can be modeled as a value stream map. A high-level value stream map for the creation of a new product is shown in Figure 5.1. This value stream map tells a story.

The ability to deploy the system at all stages of its development by pressing a button encourages its frequent use by testers, analysts, developers, and (most importantly) users.

Finally, it’s important to remember that the purpose of all this is to get feedback as fast as possible. To make the feedback cycle fast, you need to be able to see which build is deployed into which environment, and which stages in your pipeline each build has passed. Figure 5.5 is a screenshot from Go showing what this looks like in practice.

The second principle is to always build upon foundations known to be sound. The binaries that get deployed into production should be exactly the same as those that went through the acceptance test process—and indeed in many pipeline implementations, this is checked by storing hashes of the binaries at the time they are created and verifying that the binary is identical at every subsequent stage in the process.

The environment you deploy to least frequently (production) is the most important. Only after you have tested the deployment process hundreds of times on many environments can you eliminate the deployment script as a source of error.

It should be possible to consult one single source (a version control repository, a directory service, or a database) to find configuration settings for all your applications in all of your environments.

Using the same script to deploy to production that you use to deploy to development environments is a fantastic way to prevent the “it works on my machine” syndrome [c29ETR].

the most important step in achieving the goals of this book—rapid, repeatable, reliable releases—is for your team to accept that every time they check code into version control, it will successfully build and pass every test. This applies to the entire deployment pipeline. If a deployment to an environment fails, the whole team owns that failure. They should stop and fix it before doing anything else.

A new instance of your deployment pipeline is created upon every check-in and, if the first stage passes, results in the creation of a release candidate. The aim of the first stage in the pipeline is to eliminate builds that are unfit for production and signal the team that the application is broken as quickly as possible. We want to expend a minimum of time and effort on a version of the application that is obviously broken.