September 30, 2008

The Antonym of Testing

Posted by Ben Simo

"... one usually encounters a definition such as, 'Testing is the process of confirming that a program is correct. It is the demonstration that errors are not present.' The main trouble with this definition is that it is totally wrong; in fact, it almost defines the antonym of testing."

- Glenford Myers,

Software Reliability: Principles & Practices, 1976

People keep telling me that testing is a validation activity -- that the purpose of testing is to validate that the software meets all the specifications, has no errors, meets performance SLAs, meets expectations of anonymous users, or some other lofty goal.

I read about testing processes designed to validate software. I use testing tools built to support validation. I listen to service companies pitch testing services to validate software. I read about testing metrics built on the assertion that software systems can be proved correct. I attend testing presentations explaining the presenters' best practices for validation.

The trouble is that we cannot prove software correct. We cannot prove the absence of bugs. We cannot test every possible state and input. We cannot evaluate every possible output. We cannot fully understand the desires of stakeholders. We cannot prove that customers will be happy. We cannot prove that a software product will solve the problems it was built to solve. If all this were possible, I suspect insurance companies would find a way to make a profit selling software quality insurance.

"If you think you can fully test a program without testing its response to every possible input, fine. Give us a list of your test cases. We can write a program that will pass all your tests but still fail spectacularly on an input you missed. If we can do this deliberately, our contention is that we or other programmers can do it accidentally."

- Cem Kaner, Jack Falk, and Hung Quoc Nguyen,
Testing Computer Software, Second Edition, 1999

Now, thirty-two years since Glenford Myers called testing to prove correctness the opposite of testing, we're surrounded by testing practices and tools based on proving correctness. The myth of proving correctness is alive and well.

Activities designed to try to prove correctness are the antonym of testing.

So if testing is not validation, what is testing? Testing is investigation; and communicating useful information about quality to decision makers.

"Testing is the process by which we explore and understand the status of the benefits and the risk associated with release of a software system."

- James Bach,
James Bach on Risk-Based Testing, STQE Magazine, Nov 1999

"Testing is done to find information. Critical decisions about the project or the product are made on the basis of that information."

- Cem Kaner, James Bach, Bret Pettichord,
Lessons Learned In Software Testing: A Context-Driven Approach, 2002

"A software tester’s job is to test software, find bugs, and report them so that they can be fixed. An effective software tester focuses on the software product itself and gathers empirical information regarding what it does and doesn’t do. This is a big job all by itself. The challenge is to provide accurate, comprehensive, and timely information, so managers can make informed decisions."

- Brett Pettichord,
Don't Become the Quality Police,, 2002

Once we admit that we cannot prove the software correct, we can refocus our efforts on finding useful quality-related information. Instead of pretending to assure quality or validate correctness, we can gather and communicate useful information. Investigate the software. Find information about threats to the quality of the systems under investigation. Communicate that information in terms that matter to stakeholders. Help managers make informed decisions.



October 01, 2008  
Anonymous wrote:

"Testing can never prove the absence of errors -- only their presence." by Steve McConnell in Code Complete: A Practical Handbook of Software Construction

October 02, 2008  
Catherine Powell wrote:

One thing I find interesting is that we call ourselves testers and expect people to understand that we're not validating anything. After all, at least most of my experience with testers outside of my chosen profession is the people who administer tests and judge the results (everything from AP tests in high school to thesis review boards to the guy on the other side of the interview table with the 10 page set of "does this code compile?" questions).

Yet we use the same name for something while asking people to understand that in THIS case tester means "information provider". It seems like we're overloading the word "tester".

Not, I should note, that this means I actually have a better word to offer up - I haven't heard one that really sounds right yet.

October 02, 2008  
Ben Simo wrote:

I like the term tester as a description of what I do. I don't like the terms quality assurance or engineer.

Testing is evaluating. As a tester, I evaluate. I perform experiments (aka tests) to gather information. I investigate.

I also evaluate the results of my testing. I judge which results should be communicated.

I'm also an advocate (think attorney) for the information I communicate. I often have to plead the case for fixing a bug or adding an enhancement.

Testers do many things and there are many titles that could apply to the many things we testers do.

We're detectives. We're investigators. We're evaluators. We're communicators. We're writers. We're critics. We're advocates. We're leaders. We're supporters. We're learners. We're teachers. We're forecasters.

We're testers.

October 05, 2008  
Joe wrote:

Excellent. Well said, Ben.

November 17, 2008  
Anonymous wrote:

Though testing is not able to prove the correctness of the software, it adds value by providing useful information on the following:

1) Existing (or resurfaced) problems or interesting situations where the software works in unexpected ways

2) The responsiveness/ capacity/ endurance of the software

3) The ease or difficulty with which the software can be manipulated into exposing hidden information/ functionality or vice versa

4) Advance information about the user's reaction

December 08, 2008  
Jaanvi wrote:

Its an excellent article Ben.
You have put it very nicely. Even I like calling myself a tester because it better explains the kind of work I am doing.

January 03, 2009  
Unknown wrote:
This comment has been removed by a blog administrator.