Quality Frog - Questioning Software

Happy Mother's Day

2008-05-11T14:00:00.002-06:00

Aggravation Testing

2008-05-10T05:18:00.007-06:00

An example:

How Long Do I Have To Wait?

A few hours? I don't have hours. I am sitting in the car using borrowed WiFi from a campground. I had to seek out Internet access to use software that came on a CD. I finally find Internet access and now it says I may have to wait a several hours. Can I abort if it takes longer than I have? What happens if I lose my internet access while the firmware update is underway?

I'm already frustrated with this device. I'm already frustrated with the software. I was hoping that a firmware update might fix bugs and usability issues on the device itself. I have reached the tipping point. This thing is going back to the store.

Terrified by Improvisation

2008-05-04T09:12:00.000-06:00

[Improvisational comedy] involves people making very sophisticated decisions on the spur of the moment, without benefit of any kind of script or plot. That's what makes it so compelling -- and to be frank -- terrifying. ... What is terrifying about improv is the fact that it appears utterly random and chaotic. It seems as though you have to get up onstage and make everything up, right there on the spot. But the truth is that improv isn't random or chaotic at all. ... Improv is an art form governed by a set of rules... How good people's decisions are under the fast-moving, high-stress conditions of rapid cognition is a function of training, rules, and rehearsal.

- Malcolm Gladwell, Blink: The Power of Thinking Without Thinking

Now, reread the quote above and replace improv with exploratory testing. See a connection? Just as improvisational theater may appear to be random and chaotic (although entertaining) to the ignorant, exploratory testing can appear to be random and chaotic to those that have been taught to rely on scripts. Good improv and exploratory testing is neither. There are rules -- heuristics.

Heuristics are rules of thumb that help solve problems. In improvisational comedy, there are rules. These are not hard rules that guarantee comedy. These are rules that skilled improv actors can use to help keep things funny. Sometimes these rules don't work and actors have to adapt. And, because they aren't following a script, they can adapt when things don't work out. Some parts of improv are scripted. I am a fan of the television show Whose Line is it Anyway. Each comedy sketch in this show is given a structure (think charter) to direct the improvisation. This structure defines and restricts (think script) specific components of each sketch while leaving the bulk of the activity open to each actor to adapt to what happens as the sketch plays itself out. While we do not see it on screen, I suspect that a great deal of training, rules, and rehearsal went into the production of Whose Line. The shows did suffer from an occasional guest participant (usually a trained script actor) that was not as skilled at improv as the regulars. However, other guests (sometimes not actors) who understand the rules of improv have helped produce some of the funniest sketches.

Good exploratory testing works in the same way. Skilled exploratory testers set out with a charter -- a goal for each testing session. Skilled exploratory testers use heuristics to help them learn about the systems they test. Skilled exploratory testers practice.

Improvisation can have scripted aspects and rules that guide it. It is not chaotic and random. It is smart people using simple rules to make quick decisions and adapt to a changing environment under pressure.

Don't be fooled by the green lights

2008-05-03T13:55:00.007-06:00

If we're uncertain about the reliability and value of code, writing more code seems like a highly fallible and paradoxical way to resolve the uncertainty.
- Michael Bolton

There is a disturbing trend in software testing. This is a trend towards redefining test to be code and testing to be a coding activity.

In his book, Testing Object Oriented Systems, Robert Binder writes "Manual testing, of course, still plays a role. But testing is mainly about the development of an automated system to implement an application-specific test design." While this book contains a great deal of useful information about test design and test automation, I wholeheartedly disagree with this statement -- and I make a living developing test automation.

I find Bob Martin's statements about manual testing being immoral to be very disturbing. I know that he is referring to manual scripted testing, but that's not what I hear repeated. I can also think of some scripted testing that requires thinking manual testers and would be immorally expensive or dangerous to automate. (Context matters.)

I am concerned when I hear Ken Schwaber refer to "QA" as people doing incredible things in a "hopeless profession". (I've also heard him praise smart testers.) I am also concerned by QA people trying to defend their role by becoming process police.

All too often, I hear and read the words test and testing being tossed around with the assertion that all things called tests are the same, and are therefore interchangeable. If all things called testing are equal, then I would be the first to lobby for replacing skilled testers with developers that can create code called tests. TDD may be a great tool for helping developers build whatever they decide to build, but it is not the same a testing focused on providing stakeholders with information about value. These are very different things. (See What is Software Testing? for a sampling of testing diversity.)

Developers create. Testers critique. Good developers test what they create. The Agile and TDD emphasis on developers testing their own work is wonderful. I believe that a developer who is good at testing their own work is more valuable than one that is not so good at testing. However, the idea that code-centered TDD can replace value-centered testing by skilled testers is bad.

Testing is much more than exercising code. It is about finding and communicating useful information about value with limited time and resources. At the heart of good testing is a thinking person that questions the software and the people designing and building the software. The tools of testing are secondary.

It is difficult for a person to be both creator and critic. These require different skills and focus. In my experience, leaning too much towards create or critique hampers the other.

As a teenager, I created a software management program for floppy disk based computers. I was proud of my creation. My creation took advantage of new technology and had features that similar products did not have. I tested my creation. I refactored my code many times. The code was clean. The program was fast. I used my creation on a daily basis for over a year. I shared it with close friends. I thought I had created something really cool. I submitted my creation to a company that was soliciting programs for publication. My baby was rejected. It was not rejected due to being poorly designed, coded, or tested. It was rejected because it was deemed to not be of value to enough of the publisher's customers. I may have discovered this earlier if I had requested input from more than my closest friends.

Just like an American Idol contestant that can't sing, we can save ourselves time, money, and embarrassment if we solicit the input of good critics before the world is watching.

I want great developers that can create beautiful music on my development team. I also want a few Simon Cowells and James Bachs to let us know when we may be fooling ourselves.

A Good Practice

2008-04-04T17:20:00.000-06:00

The Association for Software Testing (AST) is a professional organization dedicated to advancing the understanding and practice of software testing. The AST provides forums for academics, students, and testing practitioners to discuss testing. AST does this through online forums, workshops, education programs, and conferences. The third annual Conference of the Association for Software Testing (CAST) provides a great forum for face-to-face conferring. This is not your typical conference where experts talk at the masses. This is the software testing conference that puts the confer back in conference.

Ever sit in a presentation about testing and think anything like the following?

Yeah that works for you but it'll never work in my situation.
What do you mean by X?
She must work with idiots.
How does he know what he says? I want to see data.
My management would never go for it.
What planet is he from?
You're full of it.

Your are not only free to think these things at CAST, but you are free to question presenters. Time is built into the program for facilitated discussion of every presentation.

If you'd like to become a better software tester, join AST and come to CAST.

If you'd like to meet and confer with peers from around the world, join AST and come to CAST.

If you'd like to meet and confer with testing experts, join AST and come to CAST.

If you'd like to be challenged, join AST and and come to CAST.

If you'd like to hear Gerald Weinberg talk about the past, present, and future of software testing: join AST and come to CAST.

If you can't afford those other testing conferences, join AST and come to CAST -- its about half the price of other conferences.

If you'd like to compete against other testers, join AST and come to CAST.

If you care about software testing, join AST and come to CAST.

Only through judgment and skill,
exercised cooperatively throughout the entire project,
are we able to do the right things at the right times
to effectively test our products.
context-driven-testing.com

While I am not a believer in best practices, I believe its a really good practice to participate in CAST 2008. See you in Toronto.

Retraining the unskilled to code software

2008-03-02T09:32:00.013-07:00

I stumbled across a 46 year old newspaper article about how automation is changing business. The following statement caught my attention.

"Unskilled workers can then be retrained to handle peripheral jobs in the EDP system such as coding, card punching and so on."
Computor Invasion Scares The Unskilled,
WINNIPEG FREE PRESS, 1962

Times sure have changed. If retraining unskilled workers to code software was a viable option, then they must have been coding some pretty simple software. Maybe "coding" really refers to the process of entering code designed by someone else. Maybe "coding" meant data entry.

Regardless of what "coding" meant in 1962, computers and the software we create today are more complex than they were 46 years ago. It is too bad that some who would never think of coding as unskilled work still seem to think of testing as work for the not-so-well-skilled. Good developers and testers are good thinkers.

"You've got to be smarter to run a company with a computor than without one. The information on which you will base your decisions comes at you faster. If you're going to take advantage of this, you have to think faster, more decisively and more clearly."
- George Aitken, Vice-President and Comptroller,
Great West Life Assurance Co, 1962
Computor Invasion Scares The Unskilled

CALL FOR PAPERS: Conference of the Association for Software Testing

2008-01-19T07:04:00.000-07:00

CALL FOR PAPERS
The 3rd Annual Conference of the Association of Software Testing (CAST) 2008

Toronto, Ontario, Canada, July 14-16, 2008

Beyond the Boundaries: Interdisciplinary Approaches to Software Testing

Keynote Presentation by Gerald M. Weinberg

The Association for Software Testing is pleased to announce its third annual conference (CAST 2008), to be held July 14-16. The meeting will be held in Toronto, Canada, a city which features enormous diversity in culture, businesses, educational institutions, and the arts. Toronto is the perfect location for a conference on this year’s theme: "Beyond the Boundaries: Interdisciplinary Approaches to Software Testing".

Interdisciplinary approaches draw from diversified branches of learning or practice, such that insights can be drawn upon and synthesized to influence a particular craft. The CAST 2008 Program Committee is now seeking papers that explain how one, two or more disciplines might assist with software testing.

Examples could include ways in which statistics and metrics combined with critical thinking can help software testers interpret performance test results; ways in which logical thinking combined with document design and modeling help testers better understand business requirements and execute functional tests; or ways in which research in human/computer interaction might influence usability testing.

Apropos of this theme, the Association is delighted to announce that the first of our keynote speakers will be Gerald M. Weinberg, presenting a talk entitled Lessons from Past to Carry into the Future. Fifty years ago, in 1958, Jerry established the very first separate software testing group, to aid in producing life-critical software for Project Mercury. Jerry will speak of many steps, done and not yet done, needed to complete the task of creating a true software testing profession.

Both academic research papers and industrial experience reports are welcome. The following (non-exclusive) list suggests topics of interest that the Committee would consider highly suitable for submission:

General systems (e.g. modeling, non-linearity, complexity)
Mathematics (e.g. probability, statistics, combinatorics / permutations, graphing, metrics, equivalence partitioning)
Epistemology (e.g. logic, lateral thinking, critical thinking, experiment design, decision making)
Cognitive science (e.g. biases, perception, descriptive decision making, human factors, dynamics of heuristics, learning)
Communication (e.g. rhetoric, document design, writing)
Visualization (e.g. graphical presentation of test results, display and presentation of test data)
Interdisciplinary approaches to teaching software testing

In addition to looking for papers that demonstrate an interdisciplinary approach to software testing, we’re looking for personal experience reports that clearly demonstrate skills and practices of seasoned software testing professionals. We’ll be looking for rich, diverse experiences and intriguing papers that illuminate the theme. If you have hands-on experience and a fascinating story to tell, contact us and we will assist you in evolving your tale so it will be ready to present at CAST.

CONFERENCE FORMAT

CAST is designed as a forum to stimulate discussions leading to innovation in software testing, and so is distinguished by significant interaction among presenters and attendees. Papers and experience reports accepted by the program committee are challenged, debated, and discussed by the conference attendees. We encourage and facilitate conversation by building flexibility into the schedule so that topics generating high energy can be explored more deeply without adversely disrupting the course of conference events. Trained facilitators will ensure that discussion sessions are appropriately structured and productive. Discussion sessions will have a recorder, and transcripts or summaries of the discussions will be made available to participants after the conference.

SUBMISSIONS

CAST 2008, although not associate with ACM, encourages authors to follow the ACM SIG Proceedings style, freely available at http://www.acm.org/sigs/publications/proceedings-templates.

We expect a typical submission to be between 4 to 6 pages long. All papers should be submitted electronically in PDF format via email to: CFP@associationforsoftwaretesting.org

Authors of accepted papers will receive complimentary registration to CAST 2008. Papers will be published in the conference proceedings. Authors will also be invited to submit their papers for inclusion in a future edition of the Journal of the Association of Software Testers (JAST).

IMPORTANT DATES

Monday Feb 4, 2008 : Deadline for abstract submission
Monday February 25, 2008 : Notification of acceptance/rejection to authors
Monday March 17, 2008 : Submission of revised paper integrating the reviewers’ comments
Friday April 4:, 2008 : End of the second period of reviewing
Monday April 28, 2008 : Final camera-ready papers due
July 14-16, 2008: Conference

CONFERENCE CONTACT

For further information about CAST 2008, please contact a member of the conference committee as listed below:

Sponsorship: Scott Barber, executive.director@associationforsoftwaretesting.org
General Conference Information: Michael Bolton, cast2008(“at” symbol)michaelbolton.net
Program: CFP@associationforsoftwaretesting.org

Evidence That Quality Has Everything To Do With Value

2008-01-14T23:54:00.000-07:00

“Quality is value to some person.”

Gerald Weinberg,
Quality Software Management – Systems Thinking

This morning, Jason Gorman's blog post title Proof That Value Has Little To Do With Quality? caught my attention. This title contradicts my definition of Quality. To me, Quality is all about value to stakeholders.

Quality is not about implementing the best development practices. Quality is not about writing solid code. Quality may not be about impressive features. Quality may have no relation to elegance. Quality may not even be reliable. Quality may be cheap or it may be expensive. Quality may be well planned or it may be haphazard.

Quality is all about value. Quality is about value to people that matter.

Jason references an interesting article about a web site that started as a learning exercise and "seems to come from the Anti-Perfectionist School of Design", yet is profiting its creator millions of dollars annually. In spite of many flaws, this web site is profitable because users find it valuable and users bring advertising dollars. I consider this to be a high quality web site in spite of its obvious flaws because it has value to people who matter. Instead of viewing this as an example of value having little to do with quality, I see this as a great example of Quality having everything to do with value. I suspect that Jason and I define Quality differently. This story is an example of how value sometimes has very little to do with all the other things we often call Quality.

Perhaps my thinking is too touchy-feely for those who think we need to measure and assure Quality through quantitative metrics and processes enforcement. By it's very nature, quality is subjective. Sometimes we can quantify the results of Quality: as in the $10 million in annual advertising profits. I suspect that some of you are subjectively estimating how better metrics and process might lead to better profits.

The real measure of Quality is a measure of value (not necessarily quantitative value) to those who matter.

"As professionals, we have no real control over the ultimate value of the software we create. And neither do our customers, or requirements analysts, or product owners, or whoever it is who's been charged with figuring out what the best use of the budget would be. It's all guesswork, like choosing lottery numbers or selecting which horse to bet on."
- Jason Gorman,
Proof That Value Has Little To Do With Quality?

While there is guesswork in determining who matters and what they value, it is not as random as selecting lottery numbers. Better understanding of who matters and what they value can help us reduce the guesswork. Ongoing dialog can bring better understanding. If Quality is nothing more than a lottery, then we might as well limit ourselves to BUFD and scripted manual testing.

Interactions, collaboration, and responding to our changing understandings can help us take control over Quality.

"Above all, listen to what your customers are telling you about Quality. ... Your customers are in a perfect position to tell you about Quality, because that's all they're really buying. They're not buying a product. They're buying your assurances that their expectations for that product will be met. ... Your customers may not have all the hard business facts. They may not be aware of your specs and your standards and your inspection reports ... They may not be able to give you a precise definition of Quality, but one thing's for certain -- they know it when they see it."

-John Guaspari,
I Know It When I See It: A Modern Fable About Quality

And, as Jason rightly points out, what satisfies users (and the business) today may not satisfy them tomorrow. Keep the dialog going.

Regular Expressions

2008-01-11T10:30:00.000-07:00

(bb|[^b]{2}); [Tt]hat is the \?\.

Regular expressions are great tools for testers. I have found them useful for describing GUI objects to GUI test automation tools. I have found them useful for automation results validation. I have found them useful for extracting data I care about from voluminous log files. I've also found them useful for manipulating data.

What are regular expressions? Regular expressions are patterns for finding text of interest. They are supported by many test tools, system utilities, text editors, and programming languages.

Regular expressions can include the following meta characters to define patterns.

^ Matches the beginning
$ Matches the end
. Matches any single character
* Matches zero or more occurrences of the preceding character
\ Escape character
? Matches zero or one occurrence of the preceding character
+ Matches one or more occurrences of the previous character
[ ] Defines a character class
[^ ] Defines an exclusion-based character class
\{ \} Matches a specific number or range of instances of the previous character
 Treats the expression between $ and $ as a group
| Or. Use to match one of many expressions
\< Matches the beginning of a word
\> Matches the end of a word
\b Word boundary
\B Not a word boundary

* Many tools do not support all meta characters

Here are some example regular expressions:

“frog”

Matches “frog”, “bullfrog”, and “tree frog”; but not “Frog”

“^Frog”

Matches “Froggy went a courting”, but not “Quality Frog”

“frog$”

Matches “frog”, “bullfrog”, and “tree frog”; but not “froggy” or “The frog sat on a log.”

“.at”

Matches “cat”, “rat”, “bat”, “goat”, and “gnat”

“20*5”

Matches “2005”, “20005”, “20000000000000000000000005”, “25”; but not “2ABC5” or “2006”

“Spee?d”

Matches “Sped” and “Speed”; but not “Speeed”

“20+5”

Matches “2005” and “20005”, but not “25”

“200[5-9]”

Matches “2005”, “2006”, “2007”, and “2009”; but not “2004”

“199[0-9]|200[0-9]”

Matches years 1990 through 2009.

“[0-9][0-9]*\.[0-9][0-9][^0-9]”

Matches “1.29”, “1.29%”, and “1234.55”; but not “1.299” or “.29”

“A[LKRSZ|C[AOT]|D[CE]|F[LM]|G[AU]|HI|I[ADLN]|K[SY]|LA|M[AFRHINOPST]|N[CDEHJMVY]|O[HKR]|P[ARW]|RI|S[CD]|T[NX]|UT|V[AIT]|W[AIVY]“

Matches any valid 2-letter US postal state or territory name abbreviation.

Want to learn more?

Take a look at my slides from last night's presentation to the Denver Mercury User Group. Check out Wikipedia. Or try a Google Search. If you ask bb|[^b]{2}, check out Think Geek.

Ha[p]{2}y T[ea]sting\.

Discombobulation Testing

2007-12-22T00:31:00.000-07:00

I do not like software that discombobulates users. I find that software error handling and reporting often confuses users more than it helps. Several months back, I created and wrote about a mnemonic of guideword heuristics that I have been using for testing error handling and reporting. I have found this useful in my own testing and have received some positive feedback from others that are using it. I created the following PowerPoint show to help demonstrate how I use the mnemonic. Take a look. It runs for just under seven minutes.

FAILURE.pps
(For those that don't have PowerPoint: try an experimental Flash version here.)

The next time you encounter an error, try the FAILURE heuristic.

F Does the error detecting, reporting, and handling function as expected?
A Is the error reported at the appropriate time in an appropriate manner?
I Is the impact to the user communicated?
L Is the error appropriately logged? ... or not logged if that is desired?
U Is the error appropriately reported in the user interface in terms that the user understands?
R Is the user told how to recover from the error condition?
E Does the error reporting evoke the desired emotions? Could changes in the error reporting and handling improve the users emotional response?

Merry Christmas, and Happy Testing!

Whose side are you on?

2007-12-16T15:35:00.000-07:00

Many marriages would be better if the husband and the wife clearly understood that they are on the same side.
- Zig Ziglar

Many software teams would be better if the testers and developers clearly understood that they are on the same side.

No one creates a team to battle their team members. Husbands and wives don't embark on life together so that they can spend their lives battling each other. Software development teams aren't formed to create conflict within a company. Yet, conflict happens. Partners become the victims of friendly fire.

When the going gets tough, the tough need to stop bickering long enough to realize that they are on the same team. Here are a few tips that can help keep a team together.

Keep the common mission in sight
Seek first to understand, then to be understood (Stephen Covey)
Help others overcome obstacles instead of assigning blame
Praise good work
Be honest and tactful

Although each team member may serve a different role, it is essential that teams value the input of each member and work together to accomplish the common mission. Testers and developers should not be enemies.

Solving Intractable Problems

2007-12-04T20:10:00.000-07:00

"A solution to a given problem is called optimal if one can prove that no better solution exists. Some skeptics might ask, Why should intuition rely on a rule of thumb instead of the optimal strategy? To solve a problem by optimization -- rather than by a rule of thumb -- implies both that an optimal solution exists and that a strategy exists to find it. Computers would seem to be the ideal tool for finding the best solution to a problem. Yet paradoxically, the advent of high-speed computers has opened our eyes to the fact that the best strategy often cannot be found." - Gerd Gigerenzer, Gut Feelings: The Intelligence of the Unconscious

An intractable problem is a problem for which there is no efficient means of solving. These aren't necessarily problems for which there is no solution. Instead, these are problems that take too long to analyze all the options.

One of the biggest challenges in testing software is to select useful tests from infinite options. Even finite sets of options can create intractable problems.

For example, state model-based test automation generally requires creation of an explicit finite model. Models are less complex than the real thing -- if they aren't, they are copies, not models. Model-based test automation can be used to generate and execute tests for many more paths through a computer program than people are willing or able to try. However, testing all paths in a model for a non-trivial program can easily become an intractable problem.

Gerd Gigerenzer demonstrates this with a challenge in his book Gut Feelings. Gigerenzer asks readers to find the shortest route to visit 50 cities starting and ending at the same city. Think you can find the solution? There are only 12 different routes to visit 5 cities. So how many combinations would you need to check to visit 50 cities? According to Gigerenzer, there are approximately 300,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 possible routes to visit 50 cities. Even with the help of computers, we do not have the time to calculate the best route.

Even relatively simple computer programs have significantly more data and path possibilities -- especially when we consider retracing paths as part of a larger path. There are potentially infinite path possibilities though even the simplest programs. (As I type each letter of this article into my computer, I am selecting an input from many more than 50 options.)

The problem of visiting 50 cities is solvable if we do not insist on finding the shortest route. Finding the shortest route and trying every possible path through a computer program are not really required to find a satisfactory solution.

Intractable problems like this require that we not consider all possibilities. Trying to consider all the options distracts us from what could be productive testing. Instead of considering all options, we only need to consider enough options to be satisfied. Otherwise, we will spend more time analyzing the possibilities than we do testing. Sometimes we just need to stop analyzing and go with our gut feelings.

"Gut feelings are based on surprisingly little information that makes them look untrustworthy in the eyes of our superego, which has internalized the credo that more is always better. Yet experiments demonstrate the amazing fact that less time and information can improve decisions. Less is more means there is some range of information, time, or alternatives where a smaller amount is better ..." - Gerd Gigerenzer, Gut Feelings: The Intelligence of the Unconscious

Gut feelings -- this unconscious thinking -- is based on heuristics, or rules of thumb. A heuristic is a problem solving device that helps narrow intractable problem into solvable problems. Heuristics are not laws. They do not apply to all situations. Two or more useful heuristics may even contradict one another. Gigerenzer states that the value of a heuristic is dependent on the context in which it is applied.

I have encountered many good testers that just seem to have a gift for breaking software. I once thought that this gift was something that could not be taught to others. I was wrong. Writers like, and including, Gerald Weinberg have shown me that teaching such problem solving is possible. Testing coaches James Bach and Michael Bolton have shown me that intuitive testing can be taught.

The secret to teaching such testing is to help good testers identify the heuristics that they use while testing. Once they are identified, they can be stated in such a way that they make sense to others. Then they can be named to make them easy to remember.*

As (or after) you test, think about why you do what you do and write it down. Don't think in terms of absolutes and programmable logic. Think in terms of rules of thumb. Then give each heuristic a name. Then share it.

If you are having trouble, here is a heuristic I've found useful for starting many things:

Start with what you recognize.

Hopefully starting will trigger many more ideas that are not obvious at the start.

* For some examples, look here for a bunch of links compiled by Brian Marick.