Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

by David Farley

The FitNesse open source testing tool takes a very similar approach, allowing Fit fixtures to be created as the “drivers” to whatever it is that you need to test. This is an excellent tool that comes into its own in this context.

Gui testing tool FITNesse

When implementing the window driver pattern, you should write the equivalent of a device driver for each part of your GUI. Acceptance test code only interacts with the GUI through an appropriate window driver. The window driver provides a layer of abstraction, which forms part of your application driver layer, to insulate your test code from changes in the specifics of the UI. When the UI changes, you change the code in the window driver, and all of the tests that depend upon it are fixed. The window driver pattern is shown in Figure 8.3

Using the Window Driver Pattern to Create Maintainable Tests In one very large project we had chosen to use an open source GUI test scripting tool. During the development of the first release, we almost managed to keep pace with development: Our automated acceptance test suite was running, although lagging on the versions of the software by a week or two. In the second release, our acceptance test suite fell behind more and more rapidly. By the end of that release it was so far behind that none of the tests from the first release were runnable at all—not one! We implemented the window driver pattern in the third release, and changed some aspects of the process of test creation and maintenance, most notably making developers responsible for test maintenance. By the end of that release we had a working deployment pipeline, which included our automated tests being run immediately after each successful commit.

automated testing does apply a pressure on you to make your code more modular and better encapsulated, but if you are breaking encapsulation to make it testable you are usually missing a better way to achieve the same goal.

In most cases, you should treat a desire to create a piece of code that only exists to allow you to verify the behavior of the application with a great deal of suspicion. Work hard to avoid such privileged access, think hard before you relent, take a hard line with yourself—and don’t give in to the easy option until you are absolutely certain that you can’t find a better way. However, sometimes inspiration fails and

Automated acceptance testing is not the same as user acceptance testing. One of the differences is that automated acceptance tests should not run in an environment that includes integration to all external systems. Instead, your acceptance testing should be focused on providing a controllable environment in which the system under test can be run.

Recording Acceptance Tests for Debugging A common problem with automated UI tests is understanding why exactly a test failed. Since these tests are necessarily of very high level, there are many potential points of failure. Sometimes these may not be related to the project at all. At other times there may be a failure early in a test suite, in a different window or dialog, that leads to the problem later on. Often the only way to find out what went wrong is to rerun the test and watch it as it proceeds. On one project we found a way to make this much easier. Before the tests started, we would start a screen capture recording on the test machine with an open source tool called Vnc2swf.

Providing protection against large-scale changes to the system

Be extremely careful about threading. Dave’s current project is the highest performance system he has worked on—his trading system can process tens of thousands of transactions per second—and one of the key ways to achieve this was by keeping the core of the application single-threaded. As Nygard says, “The blocked threads antipattern is the proximate cause of most failures... leading to chain reactions and cascading failures.

The key to deploying any application in a reliable, consistent manner is constant practice: Use the same process to deploy to every environment, including production.

In general, a production-like environment has the following characteristics. • It should run the same operating system as the production system will. • It should have the same software installed as the production system will—and in particular, none of the development toolchain (such as compilers or IDEs) should be installed on it.

Finally, it is important to keep as few versions of your application in production as possible—try to limit it to two. Supporting multiple versions is painful, so keep the number of canaries to a minimum.

In every system, there comes a moment when a critical defect is discovered and has to be fixed as soon as possible. In this situation, the most important thing to bear in mind is: Do not, under any circumstances, subvert your process.

The idea is simply this: I take my pipeline and make the final step—deployment to production—automatic. That way, if a check-in passes all the automated tests, it gets deployed directly to production. In order for this not to cause breakages, your automated tests have to be fantastic—there should be automated unit tests, component tests, and acceptance tests (functional and nonfunctional) covering your entire application. You have to write all your tests—including acceptance tests—first, so that only when a story is complete will check-ins pass the acceptance tests.

The intuitive objection to continuous deployment is that it is too risky. But, as we have said before, more frequent releases lead to lower risk in putting out any particular release. This is true because the amount of change between releases goes down. So, if you release every change, the amount of risk is limited just to the risk inherent in that one change. Continuous deployment is a great way to reduce the risk of any particular release. Perhaps most importantly, continuous deployment forces you to do the right thing (as Fitz points out in his blog post). You can’t do it without automating your entire build, deploy, test, and release process. You can’t do it without a comprehensive, reliable set of automated tests. You can’t do it without writing system tests that run against a production-like environment. That’s why, even if you can’t actually release every set of changes that passes all your tests, you should aim to create a process that would let you do so if you choose to.

Even if you have good reasons for not releasing every change you make—and there are less such reasons than you might think—you should behave as if you were going to do so.

Continuously Releasing User-Installed Software

releasing rms

Splunk One of the killer tools to hit the IT operations world in recent years is Splunk. Splunk indexes log files and other textual data that includes timestamps (which most of the data sources we describe above can be configured to provide) across your whole data center. You can then perform real-time searches that let you pinpoint unusual events and do root-cause analysis of what’s going on. Splunk can even used as an operations dashboard, and be configured to send notifications.

Broadly, there are three approaches to managing state for tests. • Test isolation: Organize tests so that each test’s data is only visible to that test. • Adaptive tests: Each test is designed to evaluate its data environment and adapt its behavior to suit the data it sees. • Test sequencing: Tests are designed to run in a known sequence, each depending, for inputs, on the outputs of its predecessors. In general, we strongly recommend the first of these approaches. Isolating tests from one another makes them more flexible as well as, importantly, capable of being run in parallel to optimize test suite performance. Both of the other approaches are possible but, in our experience, don’t scale up well. As the suite of tests becomes larger and the interactions it embodies more complex, both of these strategies tend to result in failures that are very hard to detect and correct. Interactions between tests become increasingly obscure, and the cost of maintaining a working suite of tests begins to grow.

Data Management and the Deployment Pipeline Creating and managing data to use with automated tests can be a significant overhead. Let us take a step back for a moment. What is the focus of our testing? We test our application to assert that it possesses a variety of behavioral characteristics that we desire. We run unit tests to protect ourselves from the effects of inadvertently making a change that breaks our application. We run acceptance tests to assert that the application delivers the expected value to users. We perform capacity testing to assert that the application meets our capacity requirements. Perhaps we run a suite of integration tests to confirm that our application communicates correctly with services it depends on.

Commit testing is the first stage in the deployment pipeline. It is vital to the process that commit tests run quickly. The commit stage is the point at which developers are sitting waiting for a pass before moving on. Every 30 seconds added to this stage are costly. In addition to the outright performance of commit stage testing, commit tests are the primary defense against inadvertent changes to the system. The more these tests are tied to the specifics of the implementation, the worse they are at performing that role. The problem is that when you need to refactor the implementation of some aspect of your system, you want the test to protect you. If the tests are too tightly linked to the specifics of the implementation, you will find that making a small change in implementation results in a bigger change in the tests that surround it. Instead of defending the behavior of the system, and so facilitating necessary change, tests that are too tightly coupled to the specifics of the implementation will inhibit change. If you are forced to make significant changes to tests for relatively small changes in implementation, the tests are not performing effectively their role of executable specifications of behavior.

However, it is important that every time you branch, you recognize that there is a cost associated with it. That cost comes in increased risk, and the only way to minimize that risk is to be diligent in ensuring that any active branch, created for whatever reason, should be merged back to mainline daily or more frequently. Without this, the process can no longer be considered to be based on continuous integration.

As we have said, the only reasons to branch we can recommend without any caveats are for releases, for spiking, and in extremis when there is no other reasonable way to get your application to a point where it will be possible to make further changes by other methods.

Nevertheless, in our experience it is startlingly common to begin an IT program without a business case. That will likely lead to failure, because it is impossible to know what success looks like without a business case. You might as well be the Underpants Gnomes in South Park, whose strategy is 1. Collect underpants. 2. ? 3. Profit.

The most important part of an inception—the bit that ensures that the project has a chance of success—is getting all the stakeholders together face-to-face. That means developers, customers, operations people, and management. The conversations between these people, leading to a shared understanding of the problem to be solved and the way to solve it, are the real deliverables. The list above is designed to structure the conversations so that the important issues are discussed, risks are identified, and strategies to deal with them are put in place.