February 26, 2007

Expecting the unexpected. Part 1

Posted by Ben Simo

One of the expectations for GUI test automation is unattended running of tests. However, this is often difficult to accomplish. Unexpected application behavior can stop an automated test in its tracks. Manual intervention is then required to run the script. Some automation tools offer run-time options to help the user prod the test along. Other tools require that the script or system under test be fixed before test execution can continue. The process of running a partial test, fixing the script (or waiting for an application fix), and then running another partial test to only find another script-stopping change can be time consuming. This process often takes longer than manual testing.

The problem is that scripted automation cannot adjust to application issues like a thinking manual tester. The automation script can only do what the scripter told it to expect. Some automation tools offer complex exception handling features that allow users to define expected unexpected behavior. There lies the problem: someone has to expect and code for the unexpected. There will always be unexpected unexpected behavior.

How can we create automation that can deal with the unexpected?