Bad Messages

Posted by Ben Simo

I regularly spend a great deal of time tracking down the root cause of software errors -- both on and off the job. Much of the investigation effort could have been avoided if I were not presented with incomplete or incorrect error messages. The text of error messages appears to be commonly overlooked by software developers and testers.

One of my early test automation development tasks was to fix the error messages in a test tool. The tool was used to validate the structure of data exchanged between computer systems. In nearly all cases, this tool displayed an error dialog window stating that the data was not valid. This tool gave the user (a tester) no information about what was wrong with the data. The lack of detail in these messages required that the user manually examine the data -- bit by bit -- with a protocol analyzer. This turned investigation of errors reported by an automation tool into a tedious manual task. The tool encountered a problem but did not report the source of the problem. I reviewed the code and added more detail to the error messages in this test tool. I also added logging of the data exchange in a format that made sense to human beings that could be reviewed without a protocol analyzer. The improved error messages and human-readable logging greatly decreased the troubleshooting and investigation time.

I am amazed at the poor error messages in test automation tools. Test automation tools and frameworks are likely to encounter errors in the software under test. I believe that this requires better error reporting and handling than many other tools. I am also disappointed in how difficult some of the automation tools make it to create scripts that run unattended.

One tool I use regularly often stops executing tests and displays a message like the following.

It took a great deal of effort to figure out how to coax the tool into returning error codes that could be handled in code instead of prompting a user during an automated test execution. The tool vendor's support personnel didn't seem to understand why it was a problem that I could not code their tool to handle an error without human intervention at run time. They kept telling me to fix the application under test to make the test tool's error dialog go away.

I also recently uninstalled a different test tool that had a horrible uninstall interface. It displayed a window with an uninstall progress bar that didn't move. After several minutes, the application displayed an "Install Complete" window underneath the progress window. The progress bar began to slowly move only after I clicked the "Finish" button on the buried completion window. Then it prompted me with something like the following.

Do I want to erase all files? What's the context? Of course, I don't want to erase all my files. I wonder how much the maker of this software spends on support calls about this message.

I used to receive numerous inquires from new computer users about the Windows "program has performed an illegal operation" error message. Many users thought this meant that they did something illegal and the police would soon be knocking on their door. People familiar with computers understand that this error message is about the program's interaction with the operating system and hardware; however this error message is misleading. This error message is designed for developers, not users. Good error messages tell the user what they need to know about the problem and what to do about it. Good error messages explain the problem but do not overwhelm the user with information that is useless to the intended user. This means that different kinds of applications require different kinds of error messages.

If you write software, please provide your users with accurate messages tailored to the user -- not developers. If you create test tools or automation frameworks, provide testers with information that is useful in determining what happened. Please.

People who write framework software should spend more time on useful error messages that show people why the error occurred and give a clue as to how to fix it.
- Eric M. Burke, We need better error messages

Want to practice creating better error messages? Try the Error Message Generator.