Eradicating Non-Determinism in Tests

by Martin Fowler

An automated regression suite can play a vital role on a software project, valuable both for reducing defects in production and essential for evolutionary design. In talking with development teams I've often heard about the problem of non-deterministic tests—tests that sometimes pass and sometimes fail. Left uncontrolled, non-deterministic tests can completely destroy the value of an automated regression suite. In this article I outline how to deal with non-deterministic tests. Initially quarantine helps to reduce their damage to other tests, but you still have to fix them soon. Therefore I discuss treatments for the common causes for non-determinism: lack of isolation, asynchronous behavior, remote services, time, and resource leaks.

ebook

First published April 14, 2011

About the author

Martin Fowler (b. 1963) is a software engineer, Chief Scientist at ThoughtWorks, and an outspoken advocate for best practices in enterprise software design, particularly in regard to agile software development methodologies, including extreme programming.

Reviews

[Keith]

I'm not sure I understand enough about threaded programming in the first place to make full use of this, so I'm not rating it. It does seem like something I may turn back to if and when confronted with a real-world need, though.