May 15, 2007

Automating outside the box

Posted by Ben Simo

  1. any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality etc • the test was standardized on a large sample of students
  2. the act of testing something
  3. the act of undergoing testing • he survived the great test of battle
  4. trying something to find out about it

  1. the act of implementing the control of equipment with advanced technology; usually involving electronic hardware • automation replaces human workers by machines
  2. the condition of being automatically operated or controlled • automation increases productivity
  3. equipment used to achieve automatic control or operation • this factory floor is a showcase for automation and robotic equipment

What is test automation?

I just read some marketing literature from some leading test automation tool vendors. According to one of the vendors, their tool supports the following: Novice testers can create robust and easily maintainable tests that mimic real-life use of an application with a few mouse clicks; and the automation tool will troubleshoot errors without human intervention. (I'd give the actual text if it didn't give away the vendor. You may be guessing correctly as you read this.) If this is test automation, I want some. This is what the tool vendors are telling the executives that authorize spending large sums of money. And when such claims are believed, a very high standard is set for test automation. Most testers that implement these tools quickly learn that the automation nirvana promised by tool traffickers is not available at any price.

Wikipedia currently has a decent definition for test automation -- if you read the whole thing. It starts out with a classic definition of test automation...

Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, ... Commonly, test automation involves automating a manual process ...

There are many processes in testing that are good candidates for some form of automation if we do not try to remove the cognitive aspects of testing. Automation that retraces steps that have already been executed manually and reports "pass" or "fail" is unlikely to find bugs or help testers improve their understanding of a software system under test. The Wikipedia definition for test automation includes the following important aspect.

Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.
I believe that partial test automation is not just an important aspect. It is essential. It is not possible to replace all aspects of a thinking human being with a machine. Test automation that helps automate testing tasks is likely to be a greater benefit than attempts at complete automation.

Instead of trying to create end-to-end test execution automation, think of how a doctor uses medical tests to help diagnose a patient's problems. No blood test or x-ray can diagnose or heal a patient. Doctors use the information reported by these tests in making a cognitive diagnosis. Look for ways that automated execution can help gather data that is useful in diagnosing software.

Test execution automation can be very useful -- but it may not be the best place to start.

Test automation can also be useful in generating test data and test cases. I believe the potential for automated test generation is often overlooked. Wikipedia mentions test generation automation yet implies that it is more academic than practical.

One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.
Pairwise testing has become a fairly common implementation of test generation automation. Combinations generated by a pairwise or other orthogonal array data generation tool can even be used for creating tests for manual execution.

I ask you to challenge your assumptions about test automation. Think beyond regression testing. Think beyond test execution. Look for ways that automation can help make you more efficient and put your automation efforts there first.

If you are a toolsmith, talk to and watch manual testers work. Look for ways that tools can help them do their work. You may find that the most beneficial automation has nothing to do with your initial assumptions about test automation.



May 17, 2007  
Shrini Kulkarni wrote:

>> Commonly, test automation involves automating a manual process ...


Do you (have ever) agree to this? Does automation involve automating Manual process (testing) -- Is it possible to automated manual testing process?

Wikipedia seems to be wrong and incorrect. I would (I am sure you also) rephrase this as

Test Automation would involve some kind of programmatic (done by a machine) execution of some part of a test.

My definition of Test Automation:

Design: An act of translation of some question that one asks regarding a feature of an software application into some programmable instructions so that the question being asked is modeled with reasonable accuracy.
(courtesy: Michael Bolton)

in simple words translating a testing question into a program

Execution: Use of a machine (hence a program) to support any aspect of Testing with Testing at the center of whole scheme of things

(derived from the definitions by James and Michael)


May 17, 2007  
Shrini Kulkarni wrote:

>> model-based testing where a model of the system is used for test case generation....


I stronglt object your use of the word "model based" testing in this context.

I think you wanted to say [State Machine based] Model based testing.
Watch the words in bracket.

Note the difference between "Model" and "State Model"

I can use analogies like "OS" and "Linux" or "Car" and "Toyota Qualis", "Laptop", "Dell Laptop"

I would rephrase
"One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation,"


"Defining a state based model for a software (or a set of features of that software) and using that state machine based model for test case generation is an example of Automated Test case generation"


It is a time, we get some agreement on your terminiolgies and my objections around
[State Machine] Model based Testing...

James -- please (re)solve this conflict


May 17, 2007  
Ben Simo wrote:


Test automation can be used to automate specific manual processes -- or tasks. I do not believe that the entire testing process can be automated; and I don't think that's what is implied by the Wikipedia definition. That definition states that there are many processes in testing that may be candidates for automation. I don't think it says that all of testing can be automated.

The use of "model-based testing" in the definition came from Wikipedia. All testing is model-based and in the context I quoted, the term applies to model-based automation. State models are not the only way to model systems for automation, so I would not replace "model-based" with "state model".


May 18, 2007  
Shrini Kulkarni wrote:
This comment has been removed by the author.
May 18, 2007  
Shrini Kulkarni wrote:


I hope you are not getting annoyed with me for pulling you constantly on your use of "model based" ...

Let us argue so that each one of us understand MBT better and use the term "model based" more precisely than before ...

Look, I did not ask you to replace the word "Model based" with "State model" - instead all I am saying is "qualify - make it more specific" the word "Model based".


May 18, 2007  
Ben Simo wrote:

I hope you are not getting annoyed with me

I am not annoyed. :)

I did not ask you to replace the word "Model based" with "State model" - instead all I am saying is "qualify - make it more specific" the word "Model based".

Model-based test automation does not need to be limited to state models. State model based automation is the type of model based automation I've been doing lately, but its not the only way to model for automation. Data models can be used for automation. Combinatorial modeling is useful for automation (and manual execution). These types of automation are model-based but not state model based.

Ultimately all testing is based on models in the minds of testers.