August 18, 2007

Failure Usability

Posted by Ben Simo

One of my pet peeves about software is bad error messages. In my view, a bad error message is one that does not tell the user how the error impacts them and what they need to do in response to the error. Too many messages fail to communicate this information in terms that the software's user is going to understand. Too many of these messages are written for developers, not users.

There is a place for error logging in terms that help developers and testers troubleshoot and fix problems. This information is often best written to log files, not displayed in the user interface.

Pradeep Soundararajan and I recently discussed some of our experiences with error messages. You can listen to excerpts from this conversation using the link below.

After the above conversation, Pradeep walked me through an exercise that demonstrated a case in a popular office application in which I, the user, was not sufficiently informed that an error was occurring. It was obvious that the actions I was trying to perform were not functioning but the software gave me no clear indication of why it did not work as I expected.

Good testers recognize the need to include "negative testing" in their search for significant bugs. Testing for errors is a common part of functional testing. Let's go beyond functional testing of errors. Let's test errors for usability.

Here is an error testing mnemonic I created after our conversation.

  • Functional
  • Appropriate (or Apropos*)
  • Impact
  • Log
  • UI
  • Recovery
  • Emotions

Does the error detection and reporting function as required? Are errors not detected that should be detected? Are errors reported? Do error dialogs function as expected? Do the buttons work?

Appropriate (or Apropos*)
Are errors detected and reported in an accurate and timely manner for the intended audience? Are errors reported as soon as an error condition is met? Are warning messages displayed while there are enough resources to remedy the problem? Is a user allowed to waste time and effort only to be told that their work cannot be applied? Is the text of a message accurate? Does the text convey the situation to the intended audience? Is the error described in terms that will be understood by the intended audience?

Is the impact of the error sufficiently communicated to the user? Does the message contain too little information for the user to understand what occurred and how it impacts what they were attempting to do? Does the message contain extra information that distracts from communicating the impact?

Does technical information need to be logged for support, system administrators, developers, or testers? Will this log information be available if the user waits to contact support? Are log messages standardized to allow for automated information mining? Are logs detailed enough to facilitate troubleshooting? Are errors logged that add no value? Is there too much logging? Does excessive logging negatively impact performance and disk space? Does excessive logging complicate error investigation?

Are users given some indication that what they attempted to do failed? Are user interface messages worded for the intended audience? Are user interface error messages consistent with the look and feel of the software? Are error messages consistent with other activity that causes the same error elsewhere in the application? Are errors communicated in an efficient manner? Does a user need to click away excessive dialogs? Is this error best communicated as an error dialog? Is this error best communicated as text added to the window? Is this error best communicated audibly?

Does the error message tell the user how to recover from the error condition? Does the software facilitate recovery? If needed, is contact information provided? Is the user prompted through the recovery or left to figure it out on their own?

What emotions are likely to be raised by the error message? Does the information in the message add to a user's frustration or help quiet it? If a user is being told that they need to pay more to use a feature, does the message encourage them to upgrade or does it encourage them to find a competitor's product? Does the error message cause more confusion?

Here's an error message Pradeep gave me to help test the mnemonic. What faults can you find in the error message using the mnemonic?

The next time you see an error message -- or don't see one that you should -- don't stop at functionality. Check the rest of the FAILURE. (And look here for a PowerPoint show demonstrating the mnemonic.)

PS: While attempting to save this post, Blogger gave me the following error. This error fails many of the tests above. The second save button distracted me from the light grey error message. Trying again did not fix the problem. It was not clear if the error meant that my text had been saved or not. I finally had to copy the HTML of the post and paste it into a new Blogger session. Bad error reporting is easy to find. :)

* UDATE: 18 Aug 2007 @ 6:32 PM: Michael Bolton suggested that "Appropriate" would be more appropriate than "Apropos". After some consideration, I agree. Thanks Michael.



August 18, 2007  
James Marcus Bach wrote:

The next step after creating a heuristic is to try using it on a project. Until you've tested it under live fire, you don't know much about whether it will help.

I'll try using it on that error message as you suggested:

F - I can't tell if the error message functions properly, just from seeing the message. For instance, you may have been doing something completely unlike adding a contact when the message came up. Or perhaps the user already is a premium customer. However, it appears that the URL may be misspelled.

A - The message could be timed right, assuming that it came up when the user tried to add a contact.

I - The impact appears to be reasonably communicated. I like how it lists the other features that are disabled (is that list correct, though?)

L - I don't know if anything about this is logged. It may not be important for an error like this to be logged.

U - It's written in reasonable English. It explains the problem (assuming that problem really exists). The UI is a little crude. That may not be a big deal, unless this is a major product.

R - The error message appears to show how to solve the problem (but is the URL correct?)

E - It doesn't seem to trigger negative emotions in me-- unless of course I actually paid to use this thing.

August 18, 2007  
Ben Simo wrote:

Thanks James. I have been using these heuristics for some time but just assembled them into a mnemonic. I will now try accessing the heuristics via the mnemonic.

I am very interested in whether or not it helps other people.

I applied the mnemonic to Pradeep's error message. I have written down most of my observations from my discussion about it with Pradeep. I will post my application of the mnemonic once more people have an opportunity to try it like you did.

Anyone else? Please try the mnemonic on the sample error and on real situations as you test.

Does the mnemonic help? Is it lacking?

August 19, 2007  
Shrini Kulkarni wrote:

Nice work Ben ---

I would attempt analysing future error messages with this ...

Few suggestions for your pnemonic ...

The letter F in FAILURE is not quite "functionality". I would have something like "E" - Event.
When application throws up message in response to change in some of its own or related systems change -it could be an error message or just some infomration or acknowldegement. When such a messge is displayed - first thing I would ask is - "What happened", "what you (application) are trying to tell me? Did I commit any mistake? Do you need anything from me? and so on ..."

Second part of dialog - that happens between application and user is - "what should I do now"
that is recovery part.

So an error message at the least should explain what happened and then explain what steps the user needs to take following the message notification...


August 19, 2007  
Ben Simo wrote:


The F in FAILURE is intended to be "Functional", as in functional testing of the condition that causes the error and functional testing of the error reporting. It goes beyond the event that causes the message to be reported and includes the functionality of the reporting mechanism: whether it be a dialog, a log entry, a change in flow, or something else.

The rest of the letters in FAILURE are more focused on usability questions.

Here's a very short version.

F Does the error detecting, reporting, and handling function as expected?

A Is the error reported at the appropriate time in an appropriate manner?

I Is the impact to the user communicated?

LIs the error appropriately logged? ... or not logged if that is desired?

U Is the error appropriately reported in the user interface in terms that the user understands?

R Is the user told how to recover from the error condition?

E Does the error reporting evoke the desired emotions? Could changes in the error reporting and handling improve the users emotional response?

Thanks for your comments. The input -- both good and bad -- of other testers helps me better communicate what I think about when I use the mnemonic.



August 19, 2007  
Ben Simo wrote:

... and yes, there is some overlap amongst the letters of the mnemonic.


August 19, 2007  
James Marcus Bach wrote:

I see no overlap in the letters of your mnemonic. They literally do not overlap. What does overlap is the meanings of the guidewords in your heuristic that is served by your mnemonic.

A mnemonic is a memory aid. But beneath your memory aid is a problem-solving aid, the words that suggest what to think about.

There is some overlap among the meanings of the words, but you have to ask yourself if each word contributes something unique an valuable. If that is true, don't be concerned with overlap.

August 20, 2007  
Ben Simo wrote:

Yep. The overlap is in the meanings -- not the letters themselves.

I think each one brings some meaning, emphasis, and/or point of view that the others do not. I have found that thinking about something from different angles is a good thing in that it helps me remember to think about things that I might otherwise miss. The overlap also helps me better think through the interaction between the parts of a system.

August 20, 2007  
Anonymous wrote:

This blog takes longer to load than any I've ever visited.

August 20, 2007  
Ben Simo wrote:

This blog takes longer to load than any I've ever visited.

Sorry about that. It was likely the embedded 5mb mp3 file in this post. I removed the embed and the audio is now available only as a link. Hopefully this helps. Please let me know if you encounter any other performance issues.

Thanks for the feedback.


August 22, 2007  
Unknown wrote:

I haven't had a chance to use it yet, but it's *really* easy to remember. I like it!

I don't know if you've been following my blog, but I've got a couple up that I've had a lot of success with and two more on the way when I get a chance to write them up.

It's odd, I didn't like the mnemonics of guideword heuristics until recently, then all of the sudden they've become really central to how I think about, do, and teach performance testing. Maybe it's another one of those things that's like riding a bike. ;)

December 04, 2007  
Anonymous wrote:

I got a problem with all the spelling errors on this blog. Even shown in your example error message. I think that correct spelling plays a major role in professionalism. Developers often get alot of errors, it is testers who should pick this up.

Microsoft Word should be able to assist you before you publish anything.


December 04, 2007  
Ben Simo wrote:

I agre that spellnig is importnat. I also udnerstnad that sometmies my figners mistyep tihngs. I try to fix tehm before I pubilsh but smoeitmes typos slip out – just like bgus into pordcution sotfwrae.

The example is an example error message for testing. It contains a spelling error. It contains grammatical errors. It contains confusing information. It is a bad error message. Pradeep created it an a testing exercise. It appears that you tested it for spelling. I hope you also test it for other problems.



December 01, 2009  
Jonas Söderström wrote:

Testing Error

F - Can not test, I dont know if the link is in their database

A - Pass, I came when I tried to access the link

I - Pass, no impact except for not reading the page

L - Can't test. This should be logged so that they can know which links people are trying to access but can't find.

U - Not so good. It looks very nasty and evil red. And it's very small. But it does draw attention to it by the annoying color.

R - Pass, It tells me to browse the page or search

E - A bit annoyed yes, and the color does not make it look any better

All in all it's a decent error message which could be made nicer and bigger.

September 01, 2011  
Kimball Robinson wrote:

Very good mnemonic. Under audience you might want to add a question such as this:

Localizable/Internationalization: Does the error display properly in other langauges, fonts, character sets, etc (when there is more text, or right-to-left text, etc)?

Another UI consideration: is the dialog consistent with the platform or operating system usability guidelines (such as for placement of ok/cancel buttons, etc?)

Another consideration: duration. I have seen error messages that disappear too quickly to read them. This might be obvious for self-timed items, but I've had situations when the message arrives while I am typing, and before I can stop, something I've typed dismisses the dialog. Hmm...