November 16, 2009

Talkin' 'bout test in a differrent light

Posted by Ben Simo

Pullin' out my big black book
Cause when I need a word defined that's where I look
So I move to the L's quick, fast, in a hurry
Threw on my specs, thought my vision was blurry
I look again but to my dismay
It was black and white with no room for grey
Ya see, a big "V" stood beyond my word
And yo that's when it hit me, that luv is a verb.

Words come easy but they don't mean much
When the words they're sayin' we can put trust in
We're talkin' 'bout love in a different light
And if we all learn to love it would be just right.

- DC Talk

The DC Talk song "Luv is a Verb" points out that love is something to be acted out. Real love is action, not just words and feelings. Love is expressed through action.

Like love, the word test is both a noun and a verb. Also like love, test requires action. Even the noun definitions for test describe action.



  • trying something out to find out about it
  • any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality, etc.
  • a set of questions or exercises evaluating skill or knowledge
  • the act of undergoing testing
  • the act of testing something

  • put to the test, as for its quality, or give experimental use to
  • test or examine for the presence of disease or infection
  • examine someone's knowledge of something
  • show a certain characteristic when tested
  • achieve a certain score or rating on a test
  • determine the presence or properties of (a substance)
  • undergo a test
from Princeton WordNet

While I don't think we'll find anyone that argues that test is not a verb, people involved in software development seem to use it primarily as a noun. I have nothing against the many things we create that we call tests. Our test cases, test code, test charters, and whatever test things we create can be useful tools -- but they are not the test.

Let's think about test in a different light.

Several years ago, Shrini Kulkarni challenged me in questioning whether there can be such a thing as an automated test. I don't think I disagreed with Shrini. I've not been one to trust testing to machines, but I've been a fan of automation throughout my testing career. I've automated many testing tasks, but not believed I can automate the testing itself.

Earlier this year, Michael Bolton told me of a distinction he was thinking about between checking and testing. While I had no disagreement with this distinction, I wasn't thrilled with the terms. I wanted something more descriptive. I thought Michael was making a distinction I had been trying to make: a distinction between validation and investigation. I've since come to understand that Michael is making a slightly different distinction. Michael has recently written a series of blog posts better describing the Checking vs. Testing distinction. Michael has limited the scope of checking to observations and decisions rules that can be executed without sapience -- without a brain-engaged human. If something requires human sapience, it is testing, not checking.

Yesterday, an insightful tester, Lanette Cream, made a nice attempt at defining test on her blog. In her latest revision, she defines test as follows.

A test is
an action
which produces discoveries
that can be used to evaluate product quality.

I like that this definition identifies action, discovery, and evaluation as being core to testing. However, I'm thinking of pushing, or rather constraining, this just a bit further.

What if we were to say that the evaluation is the action and discovery is the goal?

A test would then be the sapient part of validation or investigation -- the thinking and learning that cannot be automated. All those other things we do to test are really support activities that help us evaluate.

Test is not a document. Test is not code. Test is not executing a program. Test is not applying a procedural decision rule. Test is not anything that can be done by a machine. Test is the act of evaluating. Test requires sapience.

Test is thinking and learning that leads to discovery. We may test by evaluating existing data. We may test by running experiments that produce new data. We may take the output of automated checks to test. We may provide what we learn as input to coding new automated checks. The test is the action we perform in our minds.

This may come across as nitpicking vocabulary. That's not my intent. My goal is not to limit anyone's definition of test, but rather to shed a different light on what I believe sets testing apart from checking, and gives both checking and testing value.

If a check fails in a forest and no one is around to hear it, does it make a sound?

The true value of our checking and testing is in the mind of a sapient tester. What value is there in all the things we call checks and tests without a tester (whatever their role or title) evaluating information and learning?

Test is sapient evaluation that leads to discovery
I'm not quite comfortable with this. I want the emphasis to be on the sapient activity; and not generating and collecting data to support the thinking without ignoring that it is a necessary part of testing.

Regardless of where we shine the light or draw lines, let's keep in mind that test is a verb.

What do you think? Testing of my half-baked ideas is welcome and appreciated.



November 16, 2009  
David Locke wrote:

Bayesian/Gaussian neural nets only enforce known rules. Bayesian neural nets are prescriptivist.

Markovian neural nets actually learn when they discover a new hidden Markov node.

So what does that mean in terms of your discussion of testing as discovery? It means that using different statistical methods can impead the discovery of the discovery you test is intended to capture.

A test is but one sensor in a decision support network. If the fusion is Bayesian/Gaussian, you will not discover, you will just be informed relative to expectations set when you designed the test.

Take a careful look at your entire process.

November 16, 2009  
Calkelpdiver wrote:

Why don't we just called a spade a spade? In your usage Testing is "having or showing deep understanding and intelligent application of knowledge" in relation to making an evaluation of the item in question.

Sounds like some wise and sage advice to me. Very judicious and prudent, and just sensible. Sound rational and levelheaded?

But just for brevity let's say I was being 'sapient' in my application of thought on this subject.