The DevOps Handbook: How to Create World-Class Agility, Reliability, & Security in Technology Organizations

by Gene Kim

Martin Fowler observes that, in general, a ten-minute build [and test process] is perfectly within reason. . . . [We first] do the compilation and run tests that are more localized unit tests with the database completely stubbed out. Such tests can run very fast, keeping within the ten minute guideline. However any bugs that involve larger scale interactions, particularly those involving the real database, won’t be found. The second stage build runs a different suite of tests [acceptance tests] that do hit the real database and involve more end-to-end behavior. This suite may take a couple of hours to run.17 Catch Errors as Early in Our Automated Testing as Possible A specific design goal of our automated test suite is to find errors as early in the testing phase as possible. This is why we run faster-running automated tests (e.g., unit tests) before slower-running automated tests (e.g., acceptance and integration tests), which are both run before any manual testing. Another corollary of this principle is that any errors should be found with the fastest category of testing possible. If most of our errors are found in our acceptance and integration tests, the feedback we provide to developers is orders of magnitude slower than with unit tests—and integration testing requires using scarce and complex integration test environments, which can only be used by one team at a time, further delaying feedback. Furthermore, not only are errors detected during integration testing diff...