The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

by Gene Kim

INTEGRATE PERFORMANCE TESTING INTO OUR TEST SUITE All too often, we discover that our application performs poorly during integration testing or after it has been deployed to production. Performance problems are often difficult to detect, such as when things slow down over time, going unnoticed until it is too late (e.g., database queries without an index). And many problems are difficult to solve, especially when they are caused by architectural decisions we made or unforeseen limitations of our networking, database, storage, or other systems. Our goal is to write and run automated performance tests that validate our performance across the entire application stack (code, database, storage, network, virtualization, etc.) as part of the deployment pipeline, so we detect problems early, when the fixes are cheapest and fastest. By understanding how our application and environments behave under a production-like load, we can do a far better job at capacity planning, as well as detecting conditions such as: When our database query times grow non-linearly (e.g., we forget to turn on database indexing and page load goes from one hundred milliseconds to thirty seconds). When a code change causes the number of database calls, storage use, or network traffic to increase ten-fold.