May 3, 2008

Don't be fooled by the green lights

Posted by Ben Simo

If we're uncertain about the reliability and value of code, writing more code seems like a highly fallible and paradoxical way to resolve the uncertainty.
- Michael Bolton

There is a disturbing trend in software testing. This is a trend towards redefining test to be code and testing to be a coding activity.

In his book, Testing Object Oriented Systems, Robert Binder writes "Manual testing, of course, still plays a role. But testing is mainly about the development of an automated system to implement an application-specific test design." While this book contains a great deal of useful information about test design and test automation, I wholeheartedly disagree with this statement -- and I make a living developing test automation.

I find Bob Martin's statements about manual testing being immoral to be very disturbing. I know that he is referring to manual scripted testing, but that's not what I hear repeated. I can also think of some scripted testing that requires thinking manual testers and would be immorally expensive or dangerous to automate. (Context matters.)

I am concerned when I hear Ken Schwaber refer to "QA" as people doing incredible things in a "hopeless profession". (I've also heard him praise smart testers.) I am also concerned by QA people trying to defend their role by becoming process police.

All too often, I hear and read the words test and testing being tossed around with the assertion that all things called tests are the same, and are therefore interchangeable. If all things called testing are equal, then I would be the first to lobby for replacing skilled testers with developers that can create code called tests. TDD may be a great tool for helping developers build whatever they decide to build, but it is not the same a testing focused on providing stakeholders with information about value. These are very different things. (See What is Software Testing? for a sampling of testing diversity.)

Developers create. Testers critique. Good developers test what they create. The Agile and TDD emphasis on developers testing their own work is wonderful. I believe that a developer who is good at testing their own work is more valuable than one that is not so good at testing. However, the idea that code-centered TDD can replace value-centered testing by skilled testers is bad.

Testing is much more than exercising code. It is about finding and communicating useful information about value with limited time and resources. At the heart of good testing is a thinking person that questions the software and the people designing and building the software. The tools of testing are secondary.

It is difficult for a person to be both creator and critic. These require different skills and focus. In my experience, leaning too much towards create or critique hampers the other.

As a teenager, I created a software management program for floppy disk based computers. I was proud of my creation. My creation took advantage of new technology and had features that similar products did not have. I tested my creation. I refactored my code many times. The code was clean. The program was fast. I used my creation on a daily basis for over a year. I shared it with close friends. I thought I had created something really cool. I submitted my creation to a company that was soliciting programs for publication. My baby was rejected. It was not rejected due to being poorly designed, coded, or tested. It was rejected because it was deemed to not be of value to enough of the publisher's customers. I may have discovered this earlier if I had requested input from more than my closest friends.

Just like an American Idol contestant that can't sing, we can save ourselves time, money, and embarrassment if we solicit the input of good critics before the world is watching.

I want great developers that can create beautiful music on my development team. I also want a few Simon Cowells and James Bachs to let us know when we may be fooling ourselves.



May 04, 2008  
Cruisinqa wrote:

Good post. I agree in principal that manual testing shouldn't be replaced by automation or TDD-based developer tests. However, I have seen over the past few years fewer "skilled testers" that intelligently "question" the system and more testers that just create matricies with data variations (some of which make sense to test) which is then blindly executed and called testing. More time is put into creating and data mining for "low power" tests because that is what is convenient and brings up the test case count (which is all upper management cares about) and is all that someone who has a lack of domain knowledge can produce. I believe however there is a middle ground. If you know the type of TDD test cases that were run by the developers, you can steer the focus away from those areas and into areas that were not tested as thoroughly. For instance, interface points with other systems that can only be tested after the developer has moved the code to a managed envrionment. Also, if there are many data variations (some defined and others not as well-defined) then you can setup automated tests to hit those variations with "high volume automation". I have been trying to do this with my automation team and have so far been pretty successful. I have found that the technical challenge of the automation is not nearly as difficult as defining and managing the necessary test data.

May 06, 2008  
Shrini Kulkarni wrote:


Please bounce this off with Elisabeth Hendrickson and see how she sees TDD style tests vs human testing.

I was discussing with her other days about on a diff note about automation practices in IT world (GUI tool based) and her views on Automation in agile world with TDD covering the internals and FIT/FITness covering what agilists call as "Customer facing" tests.

So, on the contrary to popular belief, TDD is not end of testing world ...

I liked the way you brought distinction between testers and developers ... create vs critique.