Test levels in Agile testing
In the past week Ulrich Freyer-Hirtz on the Agile Testing mailing list asked about different testing levels in an Agile project. He found the definition of unit, component, and acceptance tests appropriate based on several books. This discussion reminded me about the ambiguity that we face today in terms of testing. There have been several renaming attempts in the past few years. For example Dale Emery refers to a blog entry from 2004. Gojko Adzic made a similar renaming attempt. Let's take a step back, and see what all those names really want to tell us. Afterwards we will be able to make a more informed decision about names.
Test Levels
Why do we need to think in different test levels? I think we don't need to do that anymore. Back in the 1970′s when the distinction between programmers and testers had to do with different responsibilities most of the terms floating around today have been coined. Back then, it mattered whether you were testing on the unit level, on the component level, or on the system level. You had different missions for either of these levels.
In 2010 Mike Hill commented on my chapter in How to reduce the cost of software testing that he prefers the term microtest for unittests. The term unittest it seems has different meanings for different developers. Microtests make it transparent that we try to tests a little code as possible.
I like this term. It makes transparent that we aim to test as little as possible with these unittests. When I work with programmers, I often find the practice of unittesting producing tests focused on the functional requirements. The application with its database, mostly enterprise application servers, and any additional component is utilized. While this sort of tests is necessary as well, these are certainly no microtests. Microtests focus on one class of the production code – if possible. But what about that area of functional tests? And what about the remaining levels of testing, i.e. component, integration, system, sub-system, etc.? Let's take a look on other heuristics that might reveal insights about those. Enter the testing quadrants.
The Testing Quadrants
The testing quadrants as written down by Brian Marick provide a classification scheme for different tests as well as for the needs you might face. Rumor has it that most of the thoughts came actually from Cem Kaner, but Brian was the first to write them down, followed by Lisa Crispin and Janet Gregory in their Agile Testing book.
The testing quadrants describe a consultant's universal tool: four quadrants in a rectangular shape. The dimensions of the rectangle are technology-facing vs. business-facing on the x-axis, and team-support vs. product-critique on the y-axis. I think the term team-support does not ring immediately for certification-harmed testers. Yet, the dimensions describe quite well what helps a team, and makes a profound testing approach.
Microtests are found in the quadrant describing technology-facing tests that support the team. These clearly help drive forward the development. Over time they become change detectors, or checks as Michael Bolton would put it.
Functional tests usually lie in the quadrant with business-facing tests that support the team. These tests help the team, too. But they are written on a different level for a different purpose. They also help drive forward development, but are focused on a different level. Rather than making sure that you can exercise a network connection or connect to a database, these tests express business expectation and requirements. Some teams call them integration testing, acceptance tests, or system tests.
Whatever you call them, they address the issue that you don't necessarily know that you fulfilled the requirements for your customer. Acceptance tests may not reveal that information as well, but it informs you definitely that you fail to fulfill them in the future if one of them fails.
On the product critique side of the quadrants, performance, load, and the *ilities are found in the quadrant with technology-facing tests, while User Acceptance tests, usability testing, and the like are found on the business-facing quadrant.
With all these tests at my finger tips, what should I do in an Agile project? Enter risk analysis combined with the Testing Quadrants.
Risk analysis
In an Agile project, you face two constraints. On one hand, you want to serve the project as a tester. You want to provide feedback on time, to meet the two week iteration length, or whatever your time constraint there is. On the other hand you want to test the software thoroughly so that end users will not be harmed or confronted to a bug. So, you face time vs. test coverage – like in most traditional projects as well, but with a shorter timeframe.
So, you have to pick tests that you can fulfill now within the given timeframe. You want to make that as transparent as possible. So, depending on where your team currently is, you might be able to run tests for every quadrant, or you might be able to cover just the microtests and the functional tests.
But you have to make sure to include compensation for the tests that you currently cannot run in one iteration. This will take down your velocity, or you will have to plan from time to time for performance, load, or usability tests. However, over time you should be able to expand the tests that you can perform in one iteration, and fill these gaps. Over time you will gain a more balanced test set within your iteration.
Test Levels are an anti-pattern
Test levels are an anti-pattern. They are remains from a time long gone. On an Agile team everybody has the same responsibility to make the team successful. If you think about test levels, you have a more profound problem in your team. How come your team is not thinking as a team about issue? Why don't all team members contribute to the testing necessary? I claim thinking about test levels on an Agile project as an anti-pattern. When transitioning to Agile, you often start with explanations that testers are used to. So, that's where we start to classify different testing practices on the quadrants. But over time the team should make a mind-shift towards a Whole Team Approach to testing. If it's not, you should fix that problem, not the problem about levels or terms.
