Posted by Ben Simo
Traditional regression test automation simply retraces test steps that have already been performed manually. This may find new bugs that show up in a previously tested path though an application but will not find bugs off the beaten path. In his book "Software Testing Techniques", Boris Beizer compares the eradication of software bugs to the extermination of insects.
Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.
- Boris Beizer
Software Testing Techniques
Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.
Test Automation Snake Oil
As with any automation, tests generated and executed by a MBTE are no better than the model provided by the human designer. If the test or model designer does not model something, that thing will not be tested. Good test design is essential to useful automation. Use model-based automation as an interactive part of the overall testing process. Use automation to collect information for human testers instead of attempting to replace them.
Use your brain. Do some exploratory modeling. Model the behavior you expect from the application and use the MBTE like a probe droid to go out and confirm your expectations. Then use the results of each test execution to refine your model for the next execution. Continually update your models improve your testing.
Go out and find new bugs.