August 21, 2007

The Shopping List

Posted by Ben Simo

A couple nights ago, my son and I were in a store with a list of grocery items to purchase. My wife had given the list to our son and he was calling out a few items at a time. If he called out too many items, I'd forget them all and ask him to start over. When he called out something that I knew was nearby, we'd head in that direction and get that item. Darting back and forth may not be a very efficient way to shop, but it likely costs me less than systematically going through the store. When I go grocery shopping with an always-hungry 10 year old boy, it is very likely that we will buy twice as much stuff as is on our list.

Although he will claim that I always say "no", my son is skilled at talking his dad into buying things. In this trip to the store, he first asked for some expensive pastries knowing that I would say no -- like I always do. :) Then he proposed the cookies that I think he really wanted and gave me a well-thought-out argument as to why these were better and cheaper. The cookies found their way into our cart.

I wasn't very concerned about the accuracy of our selections because my wife was elsewhere in the store helping our daughter find some new gym pants for school. I knew that my selections would be tested before I paid for anything. Should anything not pass the inspection, I knew I could put it back and replace it with the correct item.

This trip to the store reminded me of a shopping trip a couple months back. My wife gave me a hand-written shopping list containing the text shown below.

DVD Lens Cleaner
Toilet Paper - get extra
canned dog & cat, kitty litter
Tea Tree Oil
Lined paper - college ruled
Mech. pencil 0.7
hamburger meat, chicken tenders tenderloin
cone shaped coffee filters
white vinegar 64oz - same isle ^
canned tuna fish
crackers - saltines + Ritz
Fig Newtons Maple Syrup
reg. coke + apple soda
Dairy section
coffee creamer, eggs, yogurt
variety juices from freezer section
plus lemonade
Pears - bartlett

As I worked my way though the list, I noticed that the level of detail specified for each item varied. Some things contained specifics while others did not. However, there is much implied in the list that is not written. How do I know that there is much implied? I've been married to Sophie for 16 years. I know my wife and she knows me. At least that's what I like to think.

She knows that she has to specify regular Coke because I'd buy Diet Coke if she didn't tell me otherwise. She also knows that I know that when she writes "apple soda", she is referring to a specific brand of apple flavored soda pop.

She did not specify what kind of toilet paper to buy but I have learned from experience which brands she will find to be acceptable. I had no idea how much was extra, so I ensured that we would not run out anytime soon.

She wrote "canned dog & cat", knowing that I would joke about not finding canned dog or cat meat in the store while understanding that I should get canned dog food and canned cat food.

She specifically told me what shape coffee filter to buy because she had recently bought a new coffee maker that requires a different shaped filter. Had she not told me which shape, I would have likely bought the wrong kind: the kind I had been buying for years. However, I discovered that there is more than one style and size of cone shaped coffee filter. I did my best to guess as to what size the new coffee maker used. I had not yet used the new coffee maker, but I had seen it. Then, before I left the store, I called home to confirm that I selected the right size.

From a conversation before I went to the store, I knew that the "canned tuna fish" was for the cats. I made an executive decision and bought tuna flavored cat food because it cost less. If I didn't know this, I might have thought that the tuna was to put on the crackers for our lunch.

I knew that "Fig Newtons" did not require that I buy that specific brand of fig bars. I also knew from experience that there were some brands I should not buy.

I am intelligent enough to know that the maple syrup is a separate item from the fig bars. However, I did not know if Sophie wanted real maple syrup or if maple flavored corn syrup would do. I bought both.

The list did not say anything about what kind of cheese I should buy but I knew what kind of cheeses we usually eat and made a guess.

I knew that "coffee creamer" came with many unwritten requirements that I have learned from shopping with my wife.

I knew what "variety juices" my family will drink and what they will not drink.

I knew that if I made a mistake in my selection of items, I might have to make another trip to the store. I was more careful in my selections than I am when Sophie is in the store. I double checked my list before I left the store.

I got most of the items right. The coffee filters fit the new coffee pot. I didn't get the vinegar right in spite of the more detailed directions about type, size, and location. And, I totally forgot the most important item on the list: the chocolate.

So, what do my shopping adventures have to do with software testing? Very much.

  1. Written requirements are only part of the story
  2. Explicit written requirements cannot fully communicate user needs
  3. Requirements may be communicated in passing
  4. Communication before, during, and after implementation is essential
  5. Familiarity improves the success rate -- and makes requirements definition easier
  6. Unit testing by implementers does not ensure success
  7. User acceptance testing earlier in the process costs less than waiting until the end
  8. Context frames everything

After nearly 16 years of marriage, I failed to completely meet expectations implementing something as simple as a shopping list.

And yet we expect people on software projects, that hardly know each other, to understand each other's desires.

What else can we learn from shopping lists?


August 18, 2007

Failure Usability

Posted by Ben Simo

One of my pet peeves about software is bad error messages. In my view, a bad error message is one that does not tell the user how the error impacts them and what they need to do in response to the error. Too many messages fail to communicate this information in terms that the software's user is going to understand. Too many of these messages are written for developers, not users.

There is a place for error logging in terms that help developers and testers troubleshoot and fix problems. This information is often best written to log files, not displayed in the user interface.

Pradeep Soundararajan and I recently discussed some of our experiences with error messages. You can listen to excerpts from this conversation using the link below.

After the above conversation, Pradeep walked me through an exercise that demonstrated a case in a popular office application in which I, the user, was not sufficiently informed that an error was occurring. It was obvious that the actions I was trying to perform were not functioning but the software gave me no clear indication of why it did not work as I expected.

Good testers recognize the need to include "negative testing" in their search for significant bugs. Testing for errors is a common part of functional testing. Let's go beyond functional testing of errors. Let's test errors for usability.

Here is an error testing mnemonic I created after our conversation.

  • Functional
  • Appropriate (or Apropos*)
  • Impact
  • Log
  • UI
  • Recovery
  • Emotions

Does the error detection and reporting function as required? Are errors not detected that should be detected? Are errors reported? Do error dialogs function as expected? Do the buttons work?

Appropriate (or Apropos*)
Are errors detected and reported in an accurate and timely manner for the intended audience? Are errors reported as soon as an error condition is met? Are warning messages displayed while there are enough resources to remedy the problem? Is a user allowed to waste time and effort only to be told that their work cannot be applied? Is the text of a message accurate? Does the text convey the situation to the intended audience? Is the error described in terms that will be understood by the intended audience?

Is the impact of the error sufficiently communicated to the user? Does the message contain too little information for the user to understand what occurred and how it impacts what they were attempting to do? Does the message contain extra information that distracts from communicating the impact?

Does technical information need to be logged for support, system administrators, developers, or testers? Will this log information be available if the user waits to contact support? Are log messages standardized to allow for automated information mining? Are logs detailed enough to facilitate troubleshooting? Are errors logged that add no value? Is there too much logging? Does excessive logging negatively impact performance and disk space? Does excessive logging complicate error investigation?

Are users given some indication that what they attempted to do failed? Are user interface messages worded for the intended audience? Are user interface error messages consistent with the look and feel of the software? Are error messages consistent with other activity that causes the same error elsewhere in the application? Are errors communicated in an efficient manner? Does a user need to click away excessive dialogs? Is this error best communicated as an error dialog? Is this error best communicated as text added to the window? Is this error best communicated audibly?

Does the error message tell the user how to recover from the error condition? Does the software facilitate recovery? If needed, is contact information provided? Is the user prompted through the recovery or left to figure it out on their own?

What emotions are likely to be raised by the error message? Does the information in the message add to a user's frustration or help quiet it? If a user is being told that they need to pay more to use a feature, does the message encourage them to upgrade or does it encourage them to find a competitor's product? Does the error message cause more confusion?

Here's an error message Pradeep gave me to help test the mnemonic. What faults can you find in the error message using the mnemonic?

The next time you see an error message -- or don't see one that you should -- don't stop at functionality. Check the rest of the FAILURE. (And look here for a PowerPoint show demonstrating the mnemonic.)

PS: While attempting to save this post, Blogger gave me the following error. This error fails many of the tests above. The second save button distracted me from the light grey error message. Trying again did not fix the problem. It was not clear if the error meant that my text had been saved or not. I finally had to copy the HTML of the post and paste it into a new Blogger session. Bad error reporting is easy to find. :)

* UDATE: 18 Aug 2007 @ 6:32 PM: Michael Bolton suggested that "Appropriate" would be more appropriate than "Apropos". After some consideration, I agree. Thanks Michael.


August 15, 2007

Excuses, Excuses

Posted by Ben Simo

I have heard a variety of responses from developers in response to bugs I report. Some are good, some bad, and some are just plain ugly. Here are a few handfuls.

  • That's strange.
  • How'd you do that?
  • It works on my machine.
  • I already fixed that. You'll have it in the next build.
  • No user would do that.
  • That's not how you're supposed to do that.
  • It's a data problem. Tell the users to fix the data.
  • That's a cool bug! Show me again.
  • I didn't touch that code.
  • It works as designed.
  • It works as coded. [Well, duh. What else would it do?]
  • That's not a bug, it's a feature.
  • I can't test everything.
  • Thank you.
... and my absolute favorite (this came from a development manager)

  • Don't judge it, just test it.

The difference between the good and bad is often the relationship between developer and tester. Testers need developers to create something to test. Respect your developers. Communicate with respect and help turn the ugly responses into good responses.

As I've heard James Bach say: Testers don't create quality. Developers create quality.

What's your favorite bug report response?


August 8, 2007

Things We Know

Posted by Ben Simo

 Charles Maxwell: Shark attacks helicopter I find it at work. I find it in online forums. I find it in books. I find it in papers. I find it in blogs. I find it at conferences.

I hear it from experts. I hear it from freshers. I hear it from friends. I hear it from managers. I sometimes even hear it come out of my own mouth.

It influences testers. It influences developers. It influences managers that influence testers and developers. It impacts customers.

It wastes time. It wastes money. It frustrates developers. It confuses executives. It demeans testers. It decreases quality in the name of improvement.

It permeates the practice of developing and testing software.

What is this ubiquitous it?

It is testing folklore.

It ain’t so much the things we don’t know that gets us in trouble. It’s the things we know that ain’t so.
- Artemus Ward

Here are some examples I pulled off the top of my head:
  • There are best practices
  • Tool vendors know those best practices
  • The right tools make good testing
  • Testers are the enemies of developers
  • Automated unit testing is the only testing we need
  • Written requirements are needed for testing
  • It is possible to document unambiguous requirements
  • Repeatability is maturity
  • Tests can be completely designed and scripted before execution
  • Testing is simple if guided by the right process
  • Quality can be tested into a product
  • Good manual testing can be replaced by automation
  • Automation is only good for regression testing
  • Test case counts are a good measure of test status
  • All web pages should load in under 6 seconds
  • Testers need to have development skills
  • Good testing can be pre-scripted to be executed by anyone that can follow directions
  • Boundaries are easy to identify
  • Most bugs occur at boundaries
  • Testing is easily outsourced to unintelligent people
  • Testing is easily outsourced to tools
  • Increased testing effort improves quality
What folklore do you encounter?

It is time to unlearn those things we know that ain't so. Challenge the folklore. Ask questions.

  • Who says so?
  • How do they know?
  • What are they missing?
  • Does it apply to my context?
  • Does it make sense?

Maybe its time to call Mythbusters.


August 4, 2007

Extreme Telecommuting

Posted by Ben Simo

"Ten years ago, there's no way this would have worked. Now there are hardly any barriers."

- Anthony Page

Many of us spend most of our days trapped in a cubical or windowless office. At times I have enjoyed the opportunity to telecommute from home. I've had some good and bad home offices over the years. I've worked with great views and I've worked in basements. I'm a bit envious of James Bach's new digs.

I have the pleasure of working from home one day a week. I look forward to this day because I don't have to deal with traffic, I can work in the comfort of my own home, and I can get work done with fewer interruptions.

Earlier this week, I came across a CNN story about telecommuters that don't work from home. These telecommuters work from wherever they want to be. They are working globetrotters. Today's technology makes it possible for many people to work from anywhere in the world. I think we are still some time away from this being an option for many employees. However, it may be a viable option for contract work. If work can be outsourced to anywhere in the world, why not a beach or mountain top?

"People ask me where I live, and I'm not sure what to say, I'm not sure where I live. I live in the world."
- Trygve Inda

If you could be an extreme telecommuter, from where would you work?