tag:blogger.com,1999:blog-4144824980987902052024-03-13T20:54:56.023-06:00Quality Frog - Questioning SoftwareBen Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.comBlogger113125tag:blogger.com,1999:blog-414482498098790205.post-89621750390044019132013-10-31T00:50:00.001-06:002013-11-16T22:20:35.883-07:00Is Healthcare.gov security now fixed?<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<div style="text-align: right;">
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small; vertical-align: baseline; white-space: pre-wrap;"><a href="http://blog.isthereaproblemhere.com/2013/10/is-healthcaregov-security-fixed.html">Cross-posted from Is There A Problem Here?</a></span></div>
<br />
<br />
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="vertical-align: baseline; white-space: pre-wrap;">I first attempted to use </span><a href="https://www.healthcare.gov/" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HealthCare.gov</span></a><span style="vertical-align: baseline; white-space: pre-wrap;"> to learn about options for covering my granddaughter, who is not covered by my employer-subsidized insurance. I encountered the same kinds of account creation issues others have reported, but I decided to turn on my web browser’s built-in developer tools to see if I might see details as to why form submissions were failing. I quickly discovered that the main browser window would often display a status other than what was actually occurring. For example, the form submission would fail to get a response from the server but the user interface would report that the form was submitted. Once I saw this behavioral mismatch between what was displayed in the browser and what was actually happening, I kept developer tools on as I used the site.</span></span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I do not consider using developer tools to watch data moving in and out of my own computer to be “hacking.” I have NOT “hacked” Healthcare.gov. I have only observed what is sent to my computer. I have NOT attempted to gain unauthorized access to Healthcare.gov accounts. Attempting to gain unauthorized access would be both unethical and illegal. Please don't try it.</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">While watching the interactions between my web </span><a href="http://en.wikipedia.org/wiki/Web_browser" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">browser</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and the Healthcare.gov servers, I saw information being sent to my computer that likely should not have been sent by the server. After I was told that Healthcare.gov will not take reports of security concerns, I started blogging them. </span></span></div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Then I came across a very serious issue.</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I discovered a design defect that subsequently led to me receiving a great deal of media attention. Little did I know that my findings would be mentioned in </span><a href="http://edition.cnn.com/TRANSCRIPTS/1310/30/cnr.02.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Wednesday's congressional hearings</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">:</span></span><br />
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span></div>
<br />
<div style="text-align: center;">
<object align="middle" classid="clsid:d27cdb6eae6d-11cf-96b8-444553540000" codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,0,0" height="500" id="cspan-video-player" width="410"><param name='allowScriptAccess' value='true'/><param name='movie' value='http://www.c-spanvideo.org/videoLibrary/assets/swf/CSPANPlayer.swf?clipid=4472876'/><param name='quality' value='high'/><param name='bgcolor' value='#ffffff'/><param name='allowFullScreen' value='true'/><param name='flashvars' value='system=http://www.c-spanvideo.org/common/services/flashXml.php?clipid=4472876&style=full'/><embed name='cspan-video-player' src='http://www.c-spanvideo.org/videoLibrary/assets/swf/CSPANPlayer.swf?clipid=4472876' allowScriptAccess='always' bgcolor='#ffffff' quality='high' allowFullScreen='true' type='application/x-shockwave-flash' pluginspage='http://www.macromedia.com/go/getflashplayer' flashvars='system=http://www.c-spanvideo.org/common/services/flashXml.php?clipid=4472876&style=full' align='middle' height='500' width='410'></embed></object>
</div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="font-family: inherit;"><a href="http://en.wikipedia.org/wiki/Anna_Eshoo" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><b>ESHOO</b></span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: On the issue of security, there was a security breach that arose recently, that I read about at any rate. And what I think is very important here, because the issue of privacy has been raised, and I think that that has been answered. Very importantly, there isn't any health information in these systems. But there is financial information, so my question to you is, has a security wall been built, and are you confident that it is there and that it will actually secure the financial information that applicants have to disclose?</span></span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="font-family: inherit;"><a href="http://en.wikipedia.org/wiki/Kathleen_Sebelius" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><b>SEBELIUS</b></span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: Yes, ma'am, I -- I would tell you that </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">there was not a breach</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">there was a blog by a sort of skilled hacker</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that if a certain series of incidents occurred, you could possibly get in and obtain somebody's personally identifiable...</span></span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(CROSSTALK) </span><span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>ESHOO</b></span><span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: But isn't that telling? Isn't that telling?</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>SEBELIUS</b></span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And we immediately corrected that problem</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, so there wasn't a -- it was a theoretical problem that was immediately fixed. I would tell you we are storing the minimum amount of data, because we think that's very important. </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The hub is not a data collector. It is actually using data centers at the IRS, at Homeland Security, at Social Security to verify information, but it stores none of that data,</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> so we don't want to be...</span><span style="background-color: white; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">..</span></span></div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: white; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: white; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Secretary Sebelius is correct: I did not breach or exploit any of the vulnerabilities that I reported on my blog. And it is nice that she thinks I’m “sort of skilled” as a hacker, when I’m actually a highly-experienced software tester.</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I identified a series of steps that could be easily automated to collect usernames, password reset codes, security questions, and email addresses from the system -- without any kind of authentication. </span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Attackers could use this information to go </span><a href="http://en.wikipedia.org/wiki/Phishing" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">phishing</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Exposing this information gives attackers sufficient information to gain trust and trick people into disclosing their security question answers.</span></span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If I were a malicious phisherman, I might send users email that directs them to a site masquerading as HealthCare.gov, and then ask victims to provide their security questions in order to revalidate their account. After collecting this information, I could then reset the password and access information the user provided to HealthCare.gov. </span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I found this issue last Thursday night (October 24th). I notified HealthCare.gov customer service immediately. The next morning, I found someone who could help pass information about my discovery to people within </span><a href="http://www.hhs.gov/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HHS</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><a href="http://www.cms.gov/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CMS</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> patched the most serious hole the same day, and made further changes on Monday before making a public statement about the issue.</span></span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="font-family: inherit;"><a href="http://swampland.time.com/2013/10/28/exclusive-password-reset-security-glitch-fixed-on-healthcare-gov/#ixzz2jG7vXNVa" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">“We are eliminating this theoretical vulnerability by preventing users from seeing the specific reset functionality when trying to reset their password... There is no public evidence that these design flaws were ever exploited to compromise user accounts."</span></a><span style="background-color: transparent; color: black; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> - Brian Cook, CMS Spokesman</span></span></div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: transparent; color: black; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">While I am </span><a href="http://blog.isthereaproblemhere.com/2013/10/appalled.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">appalled</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that the issue existed in the first place, I applaud the quick response. </span></span></div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Monday night, after CMS publicly confirmed the fix, I took a quick look at the new "fixed" password reset functionality.</span></div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I saw a couple of positive changes:</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The password reset code is no longer returned to the web browser.</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> This closes the biggest hole. Exploiting weaknesses in the password reset system will now require that password reset codes be obtained by intercepting email, or some other mechanism. </span></span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="2" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The system asks for security question answers and a new password before submitting the request. This slows down manual security question guessing attempts, but will have little impact on an automated attack.</span></div>
</li>
</ol>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I also saw that many potential security issues still exist:</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The system still </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">confirms whether a username or email address exists </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">in the error messages returned by the underlying services. Given that these are not public identifiers in the Insurance Marketplace, these should not be revealed. </span><span style="background-color: yellow; color: #cc0000; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(As of 11/07, this still exists.)</span></span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="2" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The system still </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transmits both the username and password reset code via email</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Email is generally not a secure means of communication. A more secure way to do this would be to send the user only half of the equation: the reset code; and then prompt the user for the username after they follow the reset link.</span></span><span style="background-color: transparent; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: yellow; color: #cc0000; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">(As of 11/08, this still exists.)</span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="3" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">password reset code still stays the same with each request </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(and is the same code used to initially activate the account). A more secure way to do this would be to change the code each time a password reset is requested or a password is reset; and in the case that a system contains sensitive information like this one does: put a time limit in which the reset code may be used.</span></span><span style="background-color: lime;"><span style="line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">(As of 11/13, this appears to have been fixed. Reset codes are changing and old ones don't work.)</span></span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="4" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When requesting a password reset, the </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">username and password reset code are still sent to 3rd party analytics companies</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Even if these companies can be trusted to not misuse the data, this likely violates their terms of use and privacy policies. And if these 3rd parties aren't expecting personal information, they may not protect it as they would protect personal information.</span><span style="background-color: transparent; color: #38761d; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: lime; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(As of 10/31, this appears to have been fixed.)</span></span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="5" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If any of the above (or other) issues lead to a username or password reset code being compromised, the security questions and email address associated with the account can still be retrieved from the system without authorization. </span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="6" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the unfortunate event that an account is compromised: </span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;"><ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">An attacker can </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">change the email address associated with the account without triggering notification </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">of the email change to the user. Once this is done, other account information can be changed without notifying the owner of the account.</span></span><span style="background-color: transparent; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: yellow; color: #cc0000; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">(As of 11/05, this still exists.)</span></div>
</li>
</ol>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="2" style="margin-bottom: 0pt; margin-top: 0pt;"><ol start="2" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">personal information used to validate a user's identity is returned to the browser </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">each time a user logs into a system. This data is both retained and returned to the browser when it should no longer be needed. Returning it to the browser each time a user logs into the system increases the potential damage should an account be compromised. This data includes the personal information the account owner provided to verify their identity -- sufficient information to steal another’s identify. </span><span style="background-color: transparent; color: #b45f06; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This also include all information entered into an insurance application for each person on the application (eg: names, DOB, SSNs, disability, pregnancy, finanical) and data retrieved from back-end systems (eg: employer, and income, and last paycheck details).</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></span><span style="background-color: transparent; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: yellow; color: #cc0000; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">(As of 11/13, the identify verification data is no longer returned to the browser; however, the personal information in the insurance application is still returned to the browser.)</span></div>
</li>
</ol>
</ol>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div style="background-attachment: scroll; background-image: none; background-position: 0px 0px; background-repeat: repeat repeat; border: 1px solid rgb(255, 153, 0); margin: 1em; padding: 1em;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-8wiE-mDUt3c/UnP7j2n5IuI/AAAAAAAA7Po/VN5cb9amy8g/s1600/Untitled.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a><a href="http://2.bp.blogspot.com/-8wiE-mDUt3c/UnP7j2n5IuI/AAAAAAAA7Po/VN5cb9amy8g/s1600/Untitled.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="400" src="http://2.bp.blogspot.com/-8wiE-mDUt3c/UnP7j2n5IuI/AAAAAAAA7Po/VN5cb9amy8g/s400/Untitled.jpg" width="315" /></a></div>
<b>Have you heard?</b><br />
<br />
<blockquote class="tr_bq">
<span style="font-family: inherit;"><a href="http://www.washingtonpost.com/blogs/the-switch/wp/2013/10/25/this-obamacare-contractor-doesnt-take-security-seriously-that-needs-to-change/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">SLAVITT</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: Our systems don't hold data. They just transport data through it.</span></span><br />
<br />
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span>
<span style="font-family: inherit;"><a href="http://en.wikipedia.org/wiki/Mike_Rogers_(Michigan_politician)" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ROGERS</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: You don't have to hold it to protect it.</span></span>
</blockquote>
<span style="font-family: inherit;"><b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span>
<br />
<br />
Both Secretary Kathleen Sebelius and Andy Slavitt, an executive VP at QSSI (the company tasked with fixing Healthcare.gov) have downplayed security concerns. They have suggested that personal information is not at risk because The Hub, the Healthcare.gov front end, does not store information; but rather, transports information. A system is only as secure as its weakest link. If front-end security is poor, then no amount of back-end security can protect information passing through the front end.<br />
<br />
Even if Healthcare.gov doesn't store information, it returns personal information to the browser. As outlined above, the data I once provided to verify my identity is sent to my computer each time I login to Healthcare.gov -- long after the identity verification has been completed. This information includes name, address, date of birth, phone number, and Social Security Number.<span style="color: #b45f06;"> It also returns data retrieved from back-end systems; including, employer, income, last paycheck details, and more.</span></div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Much of the work required to exploit these vulnerabilities can be automated. </span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>Many of these vulnerabilities are rather benign when considered individually. However, they quickly become more serious concerns if we consider how they may be combined and the exploits automated. </b></span></div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have discovered several additional vulnerabilities while using the site. </span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The email validation system demonstrated the same flaw as the password reset system: It </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">returned the activation code </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(which is the same as the password reset code) </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">to the browser</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; enabling one to create an account using an email address they do not own.</span><span style="background-color: transparent; color: #38761d; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: lime; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(As of 11/5, this appears to be fixed.)</span></span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="2" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">My </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">username and questionnaire answers were sent over the Internet without encryption </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">under an error condition that also led to my profile information not being displayed. </span></span><span style="background-color: transparent; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: yellow; color: #cc0000; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">(As of 11/08, this still exists.)</span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="3" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The system </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">returned Java stack traces to the browser</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; potentially revealing information about the internal workings or data of the system that could be exploited to find weaknesses in security.</span></span><span style="background-color: transparent; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: yellow; color: #cc0000; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">(As of 11/05, this still exists.)</span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<ol start="4" style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="color: black; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: inherit;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The </span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">security questions ask for things that are likely to be known by one's friends, family, or ex</span><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> -- and are the sorts of things many will post on Facebook or other social media. (This morning, HHS Secretary Sebelius referred to these questions as "personalized questions that can only be verified by you". They aren't.)</span></span><span style="background-color: transparent; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: yellow; color: #cc0000; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">(As of 11/08, this still exists.)</span></div>
</li>
</ol>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<div style="line-height: 1.15;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We can only speculate at what other security vulnerabilities might be found by someone willing to attempt to gain unauthorized access.</span></div>
<div style="line-height: 1.15;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
</div>
<div style="line-height: normal;">
<span style="font-family: inherit;"><br /><span style="color: black; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: black; vertical-align: baseline; white-space: pre-wrap;"></span></span></div>
<div style="background-attachment: scroll; background-image: none; background-position: 0px 0px; border: 1px solid rgb(255, 153, 0); margin: 1em; padding: 1em;">
<div style="line-height: normal;">
<b>But wait, there's more!</b></div>
<div style="line-height: normal;">
<br /></div>
<div style="line-height: normal;">
UPDATE 10/31:</div>
<div style="line-height: normal;">
<br /></div>
<ol style="line-height: normal;">
<li>Healthcare.gov returns the username for an account when given a user's real name and email address. <span style="background-color: lime; line-height: 18px; white-space: pre-wrap;">(This appeared to be fixed on 11/06, then reappeared on 11/08. As of 11/10, it appears to have been fixed again.)</span></li>
<li>Healthcare.gov returns the security questions for an account when given a username.<span style="line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: lime; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">(As of 11/11, this appears to have been fixed. I last say it on 11/08.)</span></li>
</ol>
<div style="line-height: normal;">
<a href="http://4.bp.blogspot.com/-bYqjsIts5Q4/UnP905Qe1-I/AAAAAAAA7P0/mZZKPKlHLdw/s1600/Fullscreen+capture+1112013+121407+PM.bmp.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a><a href="http://4.bp.blogspot.com/-bYqjsIts5Q4/UnP905Qe1-I/AAAAAAAA7P0/mZZKPKlHLdw/s1600/Fullscreen+capture+1112013+121407+PM.bmp.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="110" src="http://4.bp.blogspot.com/-bYqjsIts5Q4/UnP905Qe1-I/AAAAAAAA7P0/mZZKPKlHLdw/s320/Fullscreen+capture+1112013+121407+PM.bmp.jpg" width="320" /></a><br />
<br />
No other authentication is required. Although this doesn't provide an attacker with the password reset code, it exposes information that should be kept private and provides sufficient information to make phishing relatively easy.<br />
<br />
<br /></div>
</div>
<div style="line-height: 1.15;">
<span style="color: black; font-family: inherit; vertical-align: baseline; white-space: pre-wrap;"></span><br /></div>
<div style="line-height: normal;">
<span style="font-family: inherit;"><span style="color: black; vertical-align: baseline; white-space: pre-wrap;"></span><br /></span></div>
</div>
<b style="font-weight: normal;"><span style="font-family: inherit;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Conclusion</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I am very happy that the most egregious issue was immediately fixed. </span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Others issues remain.</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The vulnerabilities I've listed above are defects that should not make it to production. It doesn't take a security expert or “super hacker” to exploit these vulnerabilities.</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is basic web security. Most of these are the kinds of issues that competent web developers try to avoid; and in the rare case that they are created, are usually found by competent testers. </span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rather than individual incompetence, these issues might suggest a fractured development team -- where the developers building the components don't know how they are used in the system and therefore do not have sufficient situational awareness to understand the security implications of their decisions. Still, someone has to assemble the components and the system as a whole should be tested for security. Given that I don't know what's going on within the project, I can only speculate. I have; however, seen enough to be concerned.</span></div>
<span style="font-family: inherit;"><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The volume of users, the nature of the data presumed in the system, and the political attention all contribute to making HealthCare.gov a target of interest to attackers -- of higher interest than the typical web site. This demands a higher standard. This requires that security be made a priority throughout design, implementation, testing, and monitoring of the system.</span></div>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><span style="vertical-align: baseline; white-space: pre-wrap;"></span>
</span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: inherit; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I am still concerned about Healthcare.gov security. </span><span style="line-height: 1.15; white-space: pre-wrap;">It should concern all of us.</span><br />
<span style="line-height: 1.15; white-space: pre-wrap;"><br /></span></div>
</div>
<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-6287639982718272612013-10-29T12:42:00.003-06:002013-10-29T20:24:19.533-06:00Is there a problem with Healthcare.gov?I have discovered a number of issues with Healthcare.gov. I have blogged details on some of these on my other blog, <a href="http://isthereaproblemhere.com/">Is There A Problem Here?</a><br />
<div>
<br /></div>
<div>
These issues include, but are not limited to:</div>
<div>
<br />
1) The site creates more cookie data than it will accept. It returns HTTP 400 errors (displaying blank screens to the user) when the cookie data it generates gets larger than parts of the site are configured to accept.<br />
<br />
2) The site requires users create an account and verify identity and submit an application to get information about plan options. This creates a bottleneck that could have been avoided with different design.<br />
<br />
3) The client-side Javascript code I've reviewed contains some errors and is overly complex -- complex in a way that adds overhead and risk that makes current understanding and future maintenance of the code unnecessarily difficult.<br />
<br />
4) The site processed an application I did not submit -- and that I explicitly told it to not process.<br />
<br />
5) There are so many obvious security flaws that I doubt they took security seriously. This gives me reason to be concerned about security of parts I can't see. Some of the security issues I've seen are:<br />
<ul>
<li>Personal data sent unsecured over HTTP</li>
<li>Error messages that reveal the existence of usernames and email addresses in the system</li>
<li>Stack traces returned to the browser that reveal information about the internal system components</li>
<li>Usernames and password reset codes and questionnauire (not the application) answers sent to 3rd party analytics companies</li>
<li>Password reset codes returned to the browser</li>
<li>Email addresses associated with an account returned to the browser without authentication</li>
<li>An email validation system that returns the info to needed validate an email address to the browser -- enabling people to create accounts using others' email addresses</li>
</ul>
<div>
<br /></div>
If you want to see details, please visit my other blog at <a href="http://blog.isthereaproblemhere.com/search/label/Healthcare.gov">http://blog.isthereaproblemhere.com/search/label/Healthcare.gov</a></div>
<div>
<br />
Ben Simo</div>
<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com2tag:blogger.com,1999:blog-414482498098790205.post-67173917489224848012012-05-23T11:35:00.002-06:002012-05-23T11:39:48.716-06:00What is Performance?<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="ttp://www.flickr.com/photos/moto_club4ag/5631963557/in/photostream"><img border="0" src="http://1.bp.blogspot.com/-OAncuAY3lkg/T7z3wP94Q0I/AAAAAAAAp6o/1QUjBn0wuo4/s640/Slide5.JPG" /></a></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="http://4.bp.blogspot.com/-aSGl27gs6ro/T7z3w12TdzI/AAAAAAAAp6w/OcpVTHCris4/s1600/Slide6.JPG"><img border="0" src="http://4.bp.blogspot.com/-aSGl27gs6ro/T7z3w12TdzI/AAAAAAAAp6w/OcpVTHCris4/s640/Slide6.JPG" /></a></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<img border="0" src="http://3.bp.blogspot.com/-84Yl-hIvZRc/T7z3x4x_38I/AAAAAAAAp64/SK9n6ufIHsA/s640/Slide7.JPG" /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="http://3.bp.blogspot.com/-84Yl-hIvZRc/T7z3x4x_38I/AAAAAAAAp64/SK9n6ufIHsA/s1600/Slide7.JPG"><br /></a></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="http://2.bp.blogspot.com/-spcagxtCojY/T7z3y5enY0I/AAAAAAAAp7A/P6AQo6BAqNo/s1600/Slide8.JPG"><img border="0" src="http://2.bp.blogspot.com/-spcagxtCojY/T7z3y5enY0I/AAAAAAAAp7A/P6AQo6BAqNo/s640/Slide8.JPG" /></a></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<img border="0" src="http://1.bp.blogspot.com/-DqKKyN64ynA/T7z3zloS7kI/AAAAAAAAp7I/2CZ44P51MdY/s640/Slide9.JPG" /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="http://commons.wikimedia.org/wiki/File:Neon_Internet_Cafe_open_24_hours.jpg"><img border="0" src="http://4.bp.blogspot.com/-uOEYDQl6veQ/T7z3rbtsHII/AAAAAAAAp6I/U8giRfD1SIM/s640/Slide10.JPG" /></a></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="http://3.bp.blogspot.com/-kDessDHPFCo/T7z3tV_uiWI/AAAAAAAAp6Q/5SMUOphmYzo/s1600/Slide11.JPG"><img border="0" src="http://3.bp.blogspot.com/-kDessDHPFCo/T7z3tV_uiWI/AAAAAAAAp6Q/5SMUOphmYzo/s640/Slide11.JPG" /></a></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="http://4.bp.blogspot.com/-HULrsR_1yfg/T70SiJkWZDI/AAAAAAAAp9k/mpGjbrJF4E4/s1600/Slide20.JPG"><img border="0" src="http://4.bp.blogspot.com/-HULrsR_1yfg/T70SiJkWZDI/AAAAAAAAp9k/mpGjbrJF4E4/s640/Slide20.JPG" /></a></div>
<div style="text-align: center;">
<br /></div>
<br />As we design and test for performance, let's look beyond speed. Let's look beyond basic stability. Let's look at the many facets of performance. Consider in which ways your system needs to perform. How fast does it need to be? How long does it need to keep running? Does it need to grow? Does it need to be available at all times? How much can we spend? Can we make it faster?<br /><br />There are endless questions we could ask. Therefore, categorizing facets of performance and creating tests for each category can be helpful. However, let's not fail to look at the interaction between these categories.<br /><br /><div style="text-align: center;">
<a href="http://4.bp.blogspot.com/-td3NJqZv6RQ/T70c8-SoqiI/AAAAAAAAp-A/AMbdZ5z0Y0U/s1600/Fullscreen+capture+5232012+101802+AM.bmp.jpg"><img border="0" src="http://4.bp.blogspot.com/-td3NJqZv6RQ/T70c8-SoqiI/AAAAAAAAp-A/AMbdZ5z0Y0U/s640/Fullscreen+capture+5232012+101802+AM.bmp.jpg" /></a></div>
<br /><br />OFAT (One Factor At a Time) testing (as exampled in the above performance testing checklist excerpt) often fails to provide information related to the interaction between the categories. Let's do some MFAT (Multiple Factors At a Time) testing and analysis. Let's look at the system as a whole. Let's mix it up. Let's consider how these facets interact. Let's create test scenarios that include multiple facets.<br /><br /><div style="text-align: center;">
<a href="http://2.bp.blogspot.com/-Q6gTMV9XWY4/T7z9gsJhkMI/AAAAAAAAp9M/FjqwngD3yfY/s1600/Slide19.JPG"><img border="0" src="http://2.bp.blogspot.com/-Q6gTMV9XWY4/T7z9gsJhkMI/AAAAAAAAp9M/FjqwngD3yfY/s400/Slide19.JPG" /></a></div>
<br /><br /><br /><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com4tag:blogger.com,1999:blog-414482498098790205.post-17372361095737800242012-05-17T10:14:00.001-06:002012-05-17T10:37:22.142-06:00Time Machine: Devops, 45 Years Ago<div style="text-align: center;">
<a href="http://commons.wikimedia.org/wiki/File:1968_NCO_Computer_Operator.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" target="_blank"><img border="0" height="248" src="http://2.bp.blogspot.com/-Od6lbmN-7TM/T7Uk2ynirlI/AAAAAAAAppI/B2vykFHeBPg/s320/772px-1968_NCO_Computer_Operator.jpg" width="320" /></a>
<br />
<div class="separator" style="text-align: center;">
<i><br /></i></div>
<div class="separator" style="text-align: center;">
<i>"Two major inputs are required </i></div>
</div>
<div style="text-align: center;">
<i>to provide computer service: </i></div>
<div style="text-align: center;">
<i>equipment and manpower. </i></div>
<div style="text-align: center;">
<i><br /></i></div>
<div style="text-align: center;">
<i><b>For development</b>, </i></div>
<div style="text-align: center;">
<i>employees such as </i></div>
<div style="text-align: center;">
<i>programmers and systems analysts are needed;</i></div>
<div style="text-align: center;">
<i><b>for operation</b>, </i></div>
<div style="text-align: center;">
<i>the services of managers, operators, etc., are essential. </i></div>
<div style="text-align: center;">
<i><br /></i></div>
<div style="clear: both; text-align: center;">
<i><b><span style="font-size: large;">In practice the two activities usually coexist. </span></b></i></div>
<div style="text-align: center;">
<i><br /></i></div>
<div style="text-align: center;">
<i>Some sort of computer installation </i></div>
<div style="text-align: center;">
<i>is required </i></div>
<div style="text-align: center;">
<i>to <b>check out</b> development efforts. </i></div>
<div style="text-align: center;">
<i><br /></i></div>
<div style="text-align: center;">
<i>And almost every installation </i></div>
<div style="text-align: center;">
<i>engages in </i><i>continuing development </i></div>
<div style="text-align: center;">
<i>as old systems are modified and new ones begun."</i></div>
<div style="text-align: center;">
<i><br /></i></div>
<div style="text-align: center;">
- <a href="http://www.amazon.com/gp/product/0231083106/ref=as_li_ss_tl?ie=UTF8&tag=quali0a-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0231083106">William F. Sharpe, Economics of Computers, 1969</a> <img border="0" src="http://www.assoc-amazon.com/e/ir?t=quali0a-20&l=as2&o=1&a=0231083106" /></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The above suggests that <a href="http://en.wikipedia.org/wiki/DevOps" target="_blank">Devops</a> is not at all a new concept. Forty-five years ago, computer rental cost slightly more than the Devops staff required to use it. Non-personnel operations cost typically rounded out the final third of the total cost. Today, hardware is cheap and people are expensive.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
So, what did a Devops organization look like forty-five years ago? The table below presents some data gathered in a "nation-wide census of data processing personnel" in 1967.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.amazon.com/gp/product/0231083106/ref=as_li_ss_tl?ie=UTF8&tag=quali0a-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0231083106" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" target="_blank"><img border="0" src="http://2.bp.blogspot.com/--tb8Qze6GDc/T7UOlqV17LI/AAAAAAAApos/mi0AuxXTmf0/s800/256402753.jpg" width="90%" /></a>
</div>
<div style="text-align: center;">
<a href="http://www.amazon.com/gp/product/0231083106/ref=as_li_ss_tl?ie=UTF8&tag=quali0a-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0231083106" style="font-size: small;">William F. Sharpe, Economics of Computers, 1969</a><span style="font-size: x-small;"> </span><img alt="" border="0" height="1" src="http://www.assoc-amazon.com/e/ir?t=quali0a-20&l=as2&o=1&a=0231083106" style="border-bottom-style: none !important; border-color: initial !important; border-image: initial !important; border-left-style: none !important; border-right-style: none !important; border-top-style: none !important; border-width: initial !important; font-size: small; margin-bottom: 0px !important; margin-left: 0px !important; margin-right: 0px !important; margin-top: 0px !important;" width="1" /></div>
<br />
<br />
<div class="separator" style="text-align: center;">
<b><a href="http://www.thecostofliving.com/index.php?id=103" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" target="_blank"><img border="0" src="http://2.bp.blogspot.com/-uHpBTA2zxZE/T7UoJCXyEqI/AAAAAAAAppU/UbaRxP9Qpcs/s280/Fullscreen+capture+5172012+92658+AM.bmp.jpg" /></a></b></div>
<br />
<br />
<div style="text-align: center;">
How do these wages add up in today's dollars?</div>
<div style="text-align: center;">
<b><br /></b></div>
<div style="text-align: center;">
<b>Check out</b></div>
<div style="text-align: center;">
<b><a href="http://www.thecostofliving.com/index.php?id=103" target="_blank">The Co$t of Living</a>.</b>
<b style="font-weight: bold;"><br /></b></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
Compared to my experience over the past couple decades, this seems to be heavy on management -- both in cost and percent of personnel. The roles of computer operator and librarian are pretty much gone. Programming and analyst roles have since blurred. Software has replaced many things that people used to do. As the user base has moved from specialist operators to the general public, new testing and user experience design roles have evolved.</div>
<br />
What else has changed?<br />
<br />
What do you think devops will look like forty-five years into the future?<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-23508712442245596452012-05-16T15:00:00.000-06:002012-06-12T12:46:52.600-06:00The Right Stuff?<i>Update: I haz new job. I am no longer seeking work. (12 June 2012)</i>
<br />
<i><br /></i><br />
<div style="text-align: center;">
<i><span style="font-size: large;">You cannot be anything you want to be</span></i></div>
<div style="text-align: center;">
<i><span style="font-size: large;">-- but you can be a lot more of who you already are.</span></i></div>
<div style="text-align: center;">
<a href="http://ribbit.cc/q/OTom+Rath">- Tom Rath</a>, <a href="http://ribbit.cc/q/EStrengthsFinder+2.0">StrengthsFinder 2.0</a></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="http://grainypixels.blogspot.com/2009/05/self-portrait-2009.html"><img border="0" src="http://1.bp.blogspot.com/--glJ18eISGg/T7RRLd6w__I/AAAAAAAApmc/bNuCkW5e1Og/s640/20090515+(1).jpg" /></a></div>
<div style="text-align: center;">
<br /></div>
<br />
Given that I resigned a couple weeks ago and am <a href="http://www.questioningsoftware.com/2012/05/hire-ben-simo.html">looking for new work</a>, I've been asking myself a lot of questions about what I want to do next. Do I want to do detailed technical work? Do I want to lead people? Do I want to consult? Do I want to be someone's employee, or do I want to do short term contract work? Do I want to stay in Arizona, go back to Colorado, or go somewhere new? I've not yet decided. I am considering a number of options.<br />
<br />
<br />
What I do know is that I want to find work that is a good match for my strengths and mindset. When looking for new work for myself, and when I've been involved in hiring, matching strengths and mindset has been more important to me than matching specific technical skills. People who are enabled to exercise their strengths and philosophy tend to have the intrinsic motivation needed to update their skills as each situation demands.<br />
<br />
<br />
<div style="text-align: center;">
<i><span style="font-size: large;"><a href="http://flowchainsensei.wordpress.com/rightshifting/" target="_blank">Rightshifting</a>:</span></i></div>
<i></i><br />
<div style="text-align: center;">
<i><i><span style="font-size: large;">Improving the effectiveness </span></i></i></div>
<i>
<span style="font-size: large;"></span></i><br />
<div style="text-align: center;">
<i><span style="font-size: large;"><i><span style="font-size: large;">of knowledge-work businesses.</span></i></span></i></div>
<i><span style="font-size: large;">
</span></i> <br />
<br />
I am intrigued by the <a href="http://flowchainsensei.wordpress.com/rightshifting/">Rightshifting</a> ideas coming from Bob Marshall, the <a href="https://twitter.com/flowchainsensei">Flowchain Sensei</a>. The basic idea of the Rightshifting model is that the curve shown below indicates that the majority of organizations (peak of the curve) are ineffective compared to the few effective organizations on the right side of the curve. While organizations to the right of Bob's model tend to better match my personal mindset, finding work with such organizations is difficult -- there aren't many of them. My experience suggests that organizations further to the left can better benefit from what I have to offer than organizations to the right. However, if such organizations have no desire to improve their effectiveness, I am likely to get frustrated. Therefore, one more question I am asking myself is whether I prefer to be more of a hero in the midst of analytical dysfunction or more of a team member in a synergistic organization. I suspect my place is somewhere in the middle -- with an organization that is shifting right.<br />
<br />
<br />
<br />
<br />
<div style="text-align: center;">
<a href="http://www.authorstream.com/Presentation/flowchainsensei-107787-perspectives-rightshifting-flowchain-rightshift-rightshifting1-9a-science-technology-ppt-powerpoint/"> <img border="0" src="http://1.bp.blogspot.com/-68971WVcRKs/T7QGtxUOfMI/AAAAAAAApl4/QLQCUWIhmrI/s640/Fullscreen+capture+5162012+125511+PM.bmp.jpg" /></a></div>
<br />
<br />
<blockquote class="tr_bq">
Side Note: One way in which I think I may disagree with Bob Marshall's model is the assertion that the chaotic organizations on the left are less productive than the analytic organizations to their right. I suspect many of the majority (the analytical organizations) are just better at measuring (or pretending to measure) their productivity than the organizations to their left. What we label chaos is often order we don't yet understand well enough to describe. I'm also not convinced that organizations need to take a trip through the dehumanizing methods of many analytical organizations in order to become more productive.</blockquote>
<br />
<br />
<iframe align="right" frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="http://rcm.amazon.com/e/cm?lt1=_blank&bc1=FFFFFF&IS2=1&nou=1&bg1=FFFFFF&fc1=000000&lc1=0000FF&t=quali0a-20&o=1&p=8&l=as1&m=amazon&f=ifr&ref=tf_til&asins=159562015X" style="height: 240px; width: 120px;"></iframe>
I recently <a href="http://www.youtube.com/watch?v=o_Zub4RMfIo">watched a video</a> in which Bob Marshall recommended the <a href="http://www.amazon.com/dp/159562015X/ref=as_li_tf_til?tag=quali0a-20&camp=14573&creative=327641&linkCode=as1&creativeASIN=159562015X">Strengths Finder</a> book and the associated online assessment as a useful tool in discovering your strengths. I am typically skeptical of such assessments. I fear they tend to provoke the <a href="http://en.wikipedia.org/wiki/Forer_effect">Forer effect</a> in which people tend to demonstrate a <a href="http://en.wikipedia.org/wiki/Confirmation_bias">confirmation bias</a> and attribute accuracy to generalized information about themselves when it is presented as being specifically tailored for them. However, I decided to give <a href="http://www.amazon.com/dp/159562015X/ref=as_li_tf_til?tag=quali0a-20&camp=14573&creative=327641&linkCode=as1&creativeASIN=159562015X">Strengths Finder</a> a try. Despite my reservations and desire to critically evaluate the timed questionnaire as I completed it, I found the resulting top "themes" to be a surprisingly accurate description of my strengths. Perhaps this only demonstrates that even a skeptic like me can succumb to the Forer effect.<br />
<br />
According to Strengths Finder, my top five strengths are:<b><span style="color: #990000;"><br /></span></b><br />
<ol>
<li><b><span style="color: #990000;">Ideation</span></b></li>
<li><b><span style="color: #990000;">Command</span></b></li>
<li><b><span style="color: #990000;">Activator</span></b></li>
<li><b><span style="color: #990000;">Individualization</span></b></li>
<li><b><span style="color: #990000;">Relator</span></b></li>
</ol>
<br />
Whether this is based on a meaningful algorithm or is total BS, I found the results useful. (All models are wrong, but some are useful.) These five strengths seem to example what I believe can make a good <a href="http://context-driven-testing.com/">context-driven tester</a>; but then, I'm biased. :)<br />
<br />
So, back to what do I want in my next job or independent business venture: I want to exercise my strengths in an environment where I don't have to suffer <a href="http://en.wikipedia.org/wiki/Cognitive_dissonance">cognitive dissonance</a> in order to serve my employer or clients.<br />
<br />
In thinking about this, I created a mind map of the strengths from <a href="http://www.amazon.com/dp/159562015X/ref=as_li_tf_til?tag=quali0a-20&camp=14573&creative=327641&linkCode=as1&creativeASIN=159562015X">Strengths Finder</a> and things I like, as well as things I don't like. Following up on my desire to find a mindset match over simply matching skills, I've decided to ignore typical job hunting advice and share this.<br />
<br />
Think I might be a match for your organization? <a href="http://www.questioningsoftware.com/2012/05/hire-ben-simo.html">Hire me.</a> :)<br />
<br />
<iframe frameborder="0" height="500px" id="xmindshare_embedviewer" scrolling="no" src="http://www.xmind.net/share/_embed/QualityFrog/ben-simo/" width="100%"></iframe><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-90318556575988770902012-05-03T00:27:00.000-06:002012-05-03T00:27:50.711-06:00Target Tested ✔<div class="separator" style="clear: both; text-align: left;">
Computers empower us to create and manipulate things in ways that don't produce quite what we intended. This requires testing for what we don't expect.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
There is a problem with this advertisement from Target Australia. Can you spot it? </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-YbBNn2RNiIQ/T6Iil7M0XGI/AAAAAAAAo2o/xd7IzI2A8yo/s1600/Xfc9c.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-YbBNn2RNiIQ/T6Iil7M0XGI/AAAAAAAAo2o/xd7IzI2A8yo/s800/Xfc9c.jpg" width="90%" /></a></div>
<br /><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com4tag:blogger.com,1999:blog-414482498098790205.post-60246237532371938022011-04-17T11:24:00.000-06:002012-05-03T00:49:44.724-06:00Do the math!Sometimes numbers that make no sense are presented in ways that seem to make sense... or not. :)<br />
<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="390" src="http://www.youtube.com/embed/cgEuUzHYvOY" title="YouTube video player" width="480"></iframe><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-9864980502221554332010-08-25T16:47:00.004-06:002012-05-14T06:41:18.239-06:00uTest Interview<span class="Apple-style-span" style="font-family: Arial;"><span class="Apple-style-span">uTest invited me to be interviewed for <a href="http://blog.utest.com/testing-the-limits-with-ben-simo-part-i/2010/08/">Testing the Limits series</a> this month. They have graciously referred to me as "</span></span><span class="Apple-style-span" style="color: #333333; font-family: arial,helvetica,sans-serif; line-height: 22px;"><i>one of the most insightful and entertaining testers in the business</i>". Please give the interview a read. And if you have additional questions, please send them my way.</span><br />
<div>
<div>
<ul>
<li><span class="Apple-style-span" style="font-family: Arial;"><a href="http://blog.utest.com/testing-the-limits-with-ben-simo-part-i/2010/08/"><b><span class="Apple-style-span" style="font-size: x-large;">Part I</span></b></a><b><span class="Apple-style-span" style="font-size: x-large;">.</span></b><span class="Apple-style-span"> Worst bug, testing philosophy, defensive pessimism, certifications</span></span></li>
<li><span class="Apple-style-span" style="font-family: Arial;"><span class="Apple-style-span"><a href="http://blog.utest.com/testing-the-limits-with-ben-simo-part-ii/2010/08/"><b><span class="Apple-style-span" style="font-size: x-large;">Part II</span></b></a><b><span class="Apple-style-span" style="font-size: x-large;">. </span></b>Trusting automation, why Quality Assurance is a bad title, overstructured managemenet</span></span></li>
<li><span class="Apple-style-span" style="font-family: Arial;"><span class="Apple-style-span"><a href="http://blog.utest.com/testing-the-limits-with-ben-simo-part-iii/2010/08/"><b><span class="Apple-style-span" style="font-size: x-large;">Part III</span></b></a><b><span class="Apple-style-span" style="font-size: x-large;">.</span></b> How I got into testing, what would I be doing if there were no such thing as software</span></span></li>
</ul>
<div>
<span class="Apple-style-span" style="font-family: Arial;"><br />
</span></div>
</div>
</div><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-46331021908946363462010-02-19T17:13:00.000-07:002012-05-02T15:15:01.417-06:00What Sesame Street taught me about problem solving<object width="500" height="405"><param name="movie" value="http://www.youtube.com/v/dZWVW7jXt3I&hl=en_US&fs=1&rel=0&color1=0xe1600f&color2=0xfebd01&border=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/dZWVW7jXt3I&hl=en_US&fs=1&rel=0&color1=0xe1600f&color2=0xfebd01&border=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="500" height="405"></embed></object><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-77338874773732315312010-02-07T15:16:00.000-07:002012-05-02T15:16:42.156-06:00A checklist for buying a new computerBuying a computer has changed significantly in the past 25 years. Or not?<br />
<br />
<br />
<div style="text-align: center;">
<span style="font-size: x-small;">From Melbourne's <a href="http://www.theage.com.au/" target="_blank"><b><i>The Age</i></b></a>, 18 June 1984</span></div>
<div style="text-align: center;">
<a href="http://2.bp.blogspot.com/_Mp0d-dsENrg/S279lokDajI/AAAAAAAAiMw/AwJ8fmO5hbY/s1600-h/Fullscreen+capture+272010+103705+AM.bmp.jpg"><img alt="" border="0" src="http://2.bp.blogspot.com/_Mp0d-dsENrg/S279lokDajI/AAAAAAAAiMw/AwJ8fmO5hbY/s800/Fullscreen+capture+272010+103705+AM.bmp.jpg" /></a></div>
<div style="text-align: center;">
<span style="font-size: x-small;"></span><br />
<br />
<br />
<div style="text-align: left;">
<span style="font-size: x-small;"><span style="font-size: small;">Some features that are really important to me:</span></span></div>
<ul style="text-align: left;">
<li><span style="font-size: x-small;"><span style="font-size: small;">on-off switching <i style="color: #38761d;">(the most important feature)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">security against blow-up<i style="color: #134f5c;"> (because I fear burns and shrapnel from exploding computers)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">position of reset key <i style="color: #351c75;">(a reset key next to the shift key would be a pain)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">big yellow bus<i> <span style="color: #741b47;">(aren't computers good for education?)</span></i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">radiation shielding<span style="color: #990000;"> </span><i style="color: #990000;">(don't want to be sterilized while playing Pole Position)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">movable screen <i style="color: #783f04;">(I might want to take the screen home with me)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">ease of plug-in ROM installation <i style="color: #38761d;">(cartridges are better than tapes)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">recognize commands in u/l case <i style="color: #134f5c;">(I expect the computer to understand me -- EVEN WHEN I YELL AT IT!)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">hard disc option <i style="color: #0b5394;">(I prefer to save data to tape, but a hard disc would be a nice option)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">stand-alone capability of slaves <i style="color: #351c75;">(I want slaves that can work without being micro-managed)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">octave range <i style="color: #741b47;">(I need a computer that can sing soprano)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">RS-232C port <i style="color: #274e13;">(In case I want to interface it to my lawn sprinkler system)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">Paddles<i style="color: #660000;"> (to spank the computer when it misbehaves)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">key-word dictionary <i style="color: purple;">(to understand this check list)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">country of origin<i style="color: #cc0000;"> (can I get a machine made in the USA?)</i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">dynamic debugging tool<i> <span style="color: blue;">(certainly don't want a static debugging tool)</span></i></span></span></li>
<li><span style="font-size: x-small;"><span style="font-size: small;">Logo<i> <span style="color: #274e13;">(because I just love that little turtle)</span></i></span></span><br />
<span style="font-size: x-small;"> </span></li>
</ul>
</div><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-5227232037694926132010-01-07T06:19:00.000-07:002012-05-02T15:19:48.690-06:00Wait != Hesitate<a href="http://dilbert.com/strips/comic/2010-01-07/" title="Dilbert.com"><img alt="Dilbert.com" border="0" src="http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/70000/8000/500/78510/78510.strip.gif" /></a><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-85989466215283748482009-12-06T15:20:00.000-07:002012-05-02T15:22:44.481-06:00Metrics from where the sun don't shine<a href="http://dilbert.com/strips/comic/2008-05-08/" title="Dilbert.com"><img alt="Dilbert.com" border="0" src="http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/00000/5000/600/5652/5652.strip.gif" /></a><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-25073770359777062462009-12-05T15:23:00.000-07:002012-05-03T08:00:29.519-06:00Numbers don't lie, people do.<div style="text-align: center;">
<span style="font-style: italic;"><br /><br /><span style="font-size: large;">“despite its mathematical base, </span></span><br />
<span style="font-style: italic;"><span style="font-size: large;">statistics is as much an art as it is a science.</span></span><span style="font-size: large;"><br /></span><span style="font-size: large; font-style: italic;"> … Often the statistician must choose among methods,<br />a subjective process,<br />and find the one</span><br />
<span style="font-size: large; font-style: italic;"> that he will use to represent the facts. </span><span style="font-size: large;"><br /></span><span style="font-size: large; font-style: italic;">This suggests giving statistical material …<br />a very sharp second look</span><br />
<span style="font-size: large; font-style: italic;"> before accepting any of them. … </span><span style="font-size: large;"><br /></span><span style="font-size: large; font-style: italic;">But arbitrarily rejecting statistical methods </span><br />
<span style="font-size: large; font-style: italic;">makes no sense either.<br /><br />That is like refusing to read </span><br />
<span style="font-size: large; font-style: italic;">because writers sometimes<br />use words to hide facts and relationships<br />rather than to reveal them.”</span><br />
<br />
- Darrell Huff,<br />
How To Lie With Statistics, 1954<br />
<br />
<br />
<span style="font-size: large; font-style: italic;">“No doubt </span><br />
<span style="font-size: large; font-style: italic;">some graphics do distort</span><br />
<span style="font-size: large; font-style: italic;"> the underlying data,<br />making it hard for the viewer </span><br />
<span style="font-size: large; font-style: italic;">to learn the truth.<br />But data graphics are no different </span><br />
<span style="font-size: large; font-style: italic;">from words in this regard,<br />for any means of communication </span><br />
<span style="font-size: large; font-style: italic;">can be used to deceive.”</span><br />
<br />
- Edward Tufte,<br />
The Visual Display of Quantitative Information</div><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-13364731280460095862009-11-16T12:33:00.006-07:002015-05-26T02:01:00.275-06:00Talkin' 'bout test in a differrent light<div style="float: right;">
<a href="http://www.flickr.com/photos/qualityfrog/4108564256/"><img src="http://farm3.static.flickr.com/2619/4108564256_46edb4a670.jpg" style="height: 285px; width: 192px;" /></a></div>
<span style="font-style: italic;">Pullin' out my big black book</span><br />
<span style="font-style: italic;">Cause when I need a word defined that's where I look</span><br />
<span style="font-style: italic;">So I move to the L's quick, fast, in a hurry</span><br />
<span style="font-style: italic;">Threw on my specs, thought my vision was blurry</span><br />
<span style="font-style: italic;">I look again but to my dismay</span><br />
<span style="font-style: italic;">It was black and white with no room for grey</span><br />
<span style="font-style: italic;">Ya see, a big "V" stood beyond my word</span><br />
<span style="font-style: italic;">And yo that's when it hit me, that </span><span style="font-style: italic; font-weight: bold;">luv is a verb</span><span style="font-style: italic;">.</span><br />
<br />
<span style="font-style: italic;">Words come easy but they don't mean much</span><br />
<span style="font-style: italic;">When the words they're sayin' we can put trust in</span><br />
<span style="font-style: italic;">We're talkin' 'bout love in a different light</span><br />
<span style="font-style: italic;">And if we all learn to love it would be just right.</span><br />
<br />
- <a href="http://en.wikipedia.org/wiki/Dc_Talk">DC Talk</a><br />
<br />
<br />
The DC Talk song "<span style="font-style: italic;">Luv is a Verb</span>" points out that love is something to be acted out. Real love is action, not just words and feelings. Love is expressed through action.<br />
<br />
Like <span style="font-style: italic;">love</span>, the word <span style="font-style: italic;">test </span>is both a noun and a verb. Also like <span style="font-style: italic;">love</span>, <span style="font-style: italic;">test </span>requires action. Even the noun definitions for test describe action.<br />
<br />
<span style="font-weight: bold;"></span><br />
<blockquote>
<span style="font-weight: bold;">test</span><br />
<br />
<span style="font-size: 85%;"><span style="font-style: italic;">noun</span><br />
</span><br />
<ul>
<li><span style="font-size: 85%;">trying something out to find out about it</span></li>
<li><span style="font-size: 85%;">any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality, etc.</span></li>
<li><span style="font-size: 85%;">a set of questions or exercises evaluating skill or knowledge</span></li>
<li><span style="font-size: 85%;">the act of undergoing testing</span></li>
<li><span style="font-size: 85%;">the act of testing something</span></li>
</ul>
<span style="font-size: 85%;"><span style="font-style: italic;">verb</span><br />
</span><br />
<ul>
<li><span style="font-size: 85%;">put to the test, as for its quality, or give experimental use to</span></li>
<li><span style="font-size: 85%;">test or examine for the presence of disease or infection</span></li>
<li><span style="font-size: 85%;">examine someone's knowledge of something</span></li>
<li><span style="font-size: 85%;">show a certain characteristic when tested</span></li>
<li><span style="font-size: 85%;">achieve a certain score or rating on a test</span></li>
<li><span style="font-size: 85%;">determine the presence or properties of (a substance)</span></li>
<li><span style="font-size: 85%;">undergo a test<br />
</span></li>
</ul>
<div style="text-align: center;">
<span style="font-size: 78%;">from Princeton <a href="http://wordnetweb.princeton.edu/perl/webwn?s=test">WordNet</a></span></div>
</blockquote>
<br />
While I don't think we'll find anyone that argues that <span style="font-style: italic;">test </span>is not a verb, people involved in software development seem to use it primarily as a noun. I have nothing against the many things we create that we call tests. Our test cases, test code, test charters, and whatever test things we create can be useful tools -- but they are not the test.<br />
<br />
<br />
<span style="font-weight: bold;">Let's think about <span style="font-style: italic;">test </span>in a different light.</span><br />
<br />
Several years ago, <a href="http://shrinik.blogspot.com/">Shrini Kulkarni</a> challenged me in questioning whether there can be such a thing as an automated test. I don't think I disagreed with Shrini. I've not been one to trust testing to machines, but I've been a fan of automation throughout my testing career. I've automated many testing tasks, but not believed I can automate the testing itself.<br />
<br />
Earlier this year, <a href="http://www.developsense.com/2009/08/testing-vs-checking.html">Michael Bolton</a> told me of a distinction he was thinking about between <span style="font-style: italic;">checking</span> and <span style="font-style: italic;">testing</span>. While I had no disagreement with this distinction, I wasn't thrilled with the terms. I wanted something more descriptive. I thought Michael was making a distinction I had been trying to make: a distinction between <span style="font-style: italic;">validation</span> and <span style="font-style: italic;">investigation</span>. I've since come to understand that Michael is making a slightly different distinction. Michael has recently written a <a href="http://www.developsense.com/2009/08/testing-vs-checking.html">series of blog posts</a> better describing the <a href="http://www.developsense.com/2009/08/testing-vs-checking.html">Checking vs. Testing</a> distinction. Michael has <a href="http://www.developsense.com/2009/09/elements-of-testing-and-checking.html">limited the scope of <span style="font-style: italic;">checking</span></a> to <span style="font-weight: bold;">observations and decisions rules that can be executed without sapience</span> -- without a brain-engaged human. If something requires human sapience, it is <span style="font-style: italic;">testing</span>, not <span style="font-style: italic;">checking</span>.<br />
<br />
Yesterday, an insightful tester, <a href="http://blog.testyredhead.com/2009/11/15/what-is-a-test.aspx">Lanette Cream</a>, made a nice attempt at defining <span style="font-style: italic;">test </span>on her blog. In her latest revision, she defines <span style="font-style: italic;">test </span>as follows.<br />
<br />
<br />
<div style="text-align: center;">
<span style="font-style: italic;"></span><br />
<blockquote>
<span style="font-style: italic;">A test is</span><br />
<span style="font-style: italic;">an <span style="font-weight: bold;">action </span></span><br />
<span style="font-style: italic;">which produces <span style="font-weight: bold;">discoveries </span></span><br />
<span style="font-style: italic;">that can be used to <span style="font-weight: bold;">evaluate </span>product quality.</span></blockquote>
<span style="font-style: italic;"></span></div>
<br />
I like that this definition identifies action, discovery, and evaluation as being core to testing. However, I'm thinking of pushing, or rather constraining, this just a bit further.<br />
<br />
What if we were to say that the <span style="font-weight: bold;">evaluation </span>is the <span style="font-weight: bold;">action </span>and <span style="font-style: italic; font-weight: bold;"></span><span style="font-weight: bold;">discovery</span> is the <span style="font-weight: bold;">goal</span>?<br />
<br />
A <span style="font-style: italic;">test </span>would then be the sapient part of validation or investigation -- the thinking and learning that cannot be automated. All those other things we do to <span style="font-style: italic;">test</span> are really support activities that help us evaluate.<br />
<br />
<span style="font-style: italic;">Test </span>is not a document. <span style="font-style: italic;">Test </span>is not code. <span style="font-style: italic;">Test </span>is not executing a program. <span style="font-style: italic;">Test </span>is not applying a procedural decision rule. <span style="font-style: italic;">Test </span>is not anything that can be done by a machine. <span style="font-style: italic;">Test </span>is the act of evaluating. Test requires sapience.<br />
<br />
<span style="font-style: italic;">Test </span>is thinking and learning that leads to discovery. We may <span style="font-style: italic;">test </span>by evaluating existing data. We may <span style="font-style: italic;">test</span> by running experiments that produce new data. We may take the output of automated checks to <span style="font-style: italic;">test</span>. We may provide what we learn as input to coding new automated checks. The <span style="font-style: italic;">test </span>is the action we perform in our minds.<br />
<br />
This may come across as nitpicking vocabulary. That's not my intent. My goal is not to limit anyone's definition of test, but rather to shed a different light on what I believe sets <span style="font-style: italic;">testing </span>apart from <span style="font-style: italic;">checking</span>, and gives both <span style="font-style: italic;">checking </span>and <span style="font-style: italic;">testing </span>value.<br />
<br />
<i style="font-weight: bold;"></i><span style="font-style: italic; font-weight: bold;">If a check fails in a forest and no one is around to hear it, does it make a sound?</span><br />
<br />
The true value of our checking and testing is in the mind of a sapient tester. What value is there in all the things we call checks and tests without a tester (whatever their role or title) evaluating information and learning?<br />
<br />
<br />
<div style="text-align: center;">
<span style="font-weight: bold;"></span><br />
<blockquote style="font-style: italic;">
<span style="font-weight: bold;">Test is sapient evaluation that leads to discovery</span></blockquote>
<span style="font-weight: bold;">.</span></div>
I'm not quite comfortable with this. I want the emphasis to be on the sapient activity; and not generating and collecting data to support the thinking without ignoring that it is a necessary part of testing.<br />
<br />
Regardless of where we shine the light or draw lines, let's keep in mind that <span style="font-weight: bold;">test is a verb</span>.<br />
<br />
What do you think? <span style="font-style: italic;">Testing </span>of my half-baked ideas is welcome and appreciated.<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com2tag:blogger.com,1999:blog-414482498098790205.post-217744377230865602009-11-12T22:52:00.000-07:002012-05-03T00:30:59.709-06:00Comprehensive Understanding?<br />
<div style="text-align: center;">
<span style="font-style: italic; font-weight: bold;">"Anyone who feels</span><br />
<span style="font-style: italic; font-weight: bold;">that he</span><br />
<span style="font-style: italic; font-weight: bold;">can precisely define</span><br />
<span style="font-style: italic; font-weight: bold;">the boundaries of his profession</span><br />
<span style="font-style: italic; font-weight: bold;">either</span><br />
<span style="font-size: 21px; font-style: italic; font-weight: bold;">possesses a skill of little importance</span><br />
<span style="font-style: italic; font-weight: bold;">or</span><br />
<span style="font-style: italic; font-weight: bold;"><span style="font-size: 21px;">is incredibly naive</span>."</span></div>
<br />
<div style="text-align: center;">
- William F. Sharpe,<br />
The Economics of Computers, 1969</div><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-89207902548150613542009-11-12T08:31:00.000-07:002012-05-03T00:32:47.081-06:00Marginal value & cost of software<br />
<div class="mobile-photo" style="text-align: center;">
<span style="font-size: 16px;"><a href="http://2.bp.blogspot.com/_Mp0d-dsENrg/SvwkLRW2TcI/AAAAAAAAg64/UybScEvlnD4/s1600-h/IMAG0808-717655.jpg"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5403233428994149826" src="http://2.bp.blogspot.com/_Mp0d-dsENrg/SvwkLRW2TcI/AAAAAAAAg64/UybScEvlnD4/s400/IMAG0808-717655.jpg" /></a></span></div>
<div style="text-align: right;">
<span style="font-family: Arial; font-size: x-small; font-style: italic;">* Image from <span style="font-weight: bold;">the Economics of Computers</span>, by William F. Sharpe.</span></div>
<span style="font-family: Arial; font-size: 16px;"><br />Cost of additional copies approaches zero ONLY IF no maintenance or customer support costs are associated with sales of additional copies.<br /><br />If your quality stinks, expect that supposedly 99% profit copy you sold me to increase your costs.<br /><br />Quality improvement that reduces maintenance and support costs can have great value!</span><br /><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-42707739499310091922009-11-11T17:33:00.000-07:002012-05-03T00:34:25.231-06:00If scripted tests can't find bugs...<br />
<a href="http://twitter.com/shrinik/status/5625613966"><img align="right" alt="" border="0" id="BLOGGER_PHOTO_ID_5402995965385383650" src="http://4.bp.blogspot.com/_Mp0d-dsENrg/SvtMNEeqHuI/AAAAAAAAg6A/bJG8c2opfeo/s320/Fullscreen+capture+11112009+44215+PM.bmp.jpg" style="cursor: pointer; display: block; height: 174px; margin-bottom: 10px; margin-left: auto; margin-right: auto; margin-top: 0px; text-align: center; width: 320px;" /></a>Shrini Kulkarni (<a href="http://twitter.com/shrinik">@shrinik</a>) tweeted that a friend told him if exploratory testing finds bugs not found by scripted testing, it may be due to insufficient or incorrect test planning and review.<br />
<br />
Perhaps there aren't enough test cases. Perhaps testing techniques weren't applied properly. Perhaps the review of tests was done wrong.<br />
<br />
However, perhaps -- just perhaps -- people are fallible and can't completely understand or work through the complexity of their customer's problems and the technical implementation of a solution with finite knowledge, finances, and time.<br />
<br />
If it is reasonable to expect testers to design enough tests to exercise near-infinite paths with near-infinite data variations in a software system, then I think it should be reasonable to expect software designers and coders to anticipate everything that could go wrong and prevent all threats to the value of the software they produce -- all with finite finances and time.<br />
<br />
If this were reasonable, then it would be reasonable to skip testing altogether.<br />
<br />
In the real world, it is just as unreasonable to expect perfection from test case designers as it is to expect perfection from all the other people involved in developing software. Is it not?<br />
<br />
One of the things exploratory testing helps avoid is locking in our level of ignorance that exists at the start. An explorer can use information gained as they explore to help guide where they go and what they do next.<br /><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-20927687375104550922009-11-11T00:35:00.000-07:002012-05-03T00:43:52.279-06:00So you think you've achieved maturity?<br />
<blockquote>
<em><strong>Some people say that </strong><a href="http://en.wikipedia.org/wiki/Ada_%28programming_language%29"><strong>Ada</strong></a><strong> may be the last major </strong><a href="http://en.wikipedia.org/wiki/High-level_language"><strong>high level language</strong></a><strong> that will ever be developed</strong>, since <a href="http://en.wikipedia.org/wiki/Automatic_programming">automatic program generation techniques</a> may be available in the not too distant future. Thus it seems fitting that the last major programming language be named in honor of the <a href="http://en.wikipedia.org/wiki/Ada_Lovelace">first female prorammer</a>.</em></blockquote>
<div align="center">
- Richard Wiener and Richard Sincovec,</div>
<div align="center">
Programming in Ada, 1983</div>
<div align="center">
</div>
<br />
<br />
Arrogance combined with foolish optimism? :)<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-33435289746806351862009-08-05T00:37:00.000-06:002012-05-03T00:38:53.356-06:00Freedom & Responsibility CultureCool. Based on this slide deck, it appears that Netflix has a good understanding of developing and sustaining a corporate culture of Freedom & Responsibility.<br />
<div id="__ss_1798664" style="text-align: left; width: 425px;">
<a href="http://www.slideshare.net/reed2001/culture-1798664" style="display: block; font: 14px Helvetica,Arial,Sans-serif; margin: 12px 0 3px 0; text-decoration: underline;" title="Culture">Culture</a><object height="355" style="margin: 0px;" width="425"><param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=culture9-090801103430-phpapp02&stripped_title=culture-1798664" />
<param name="allowFullScreen" value="true"/>
<param name="allowScriptAccess" value="always"/>
<embed src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=culture9-090801103430-phpapp02&stripped_title=culture-1798664" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object><br />
<div style="font-family: tahoma,arial; font-size: 11px; height: 26px; padding-top: 2px;">
View more <a href="http://www.slideshare.net/" style="text-decoration: underline;">presentations</a> from <a href="http://www.slideshare.net/reed2001" style="text-decoration: underline;">reed2001</a>.</div>
</div><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-4276355564047037362009-05-07T00:40:00.000-06:002012-05-17T07:55:36.264-06:00F.A.I.L.U.R.E.F.A.I.L.U.R.E.<br />
<br />
A mnemonic for testing error handling & reporting.<br />
<div id="__ss_1403159" style="text-align: left; width: 425px;">
<a href="http://www.slideshare.net/joebensimo/failure-1403159?type=presentation" style="display: block; font: 14px Helvetica,Arial,Sans-serif; margin: 12px 0 3px 0; text-decoration: underline;" title="F.A.I.L.U.R.E.">F.A.I.L.U.R.E.</a><object height="355" style="margin: 0px;" width="425"><param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=failure-090507194930-phpapp02&stripped_title=failure-1403159" />
<param name="allowFullScreen" value="true"/>
<param name="allowScriptAccess" value="always"/>
<embed src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=failure-090507194930-phpapp02&stripped_title=failure-1403159" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object><br />
<div style="font-family: tahoma,arial; font-size: 11px; height: 26px; padding-top: 2px;">
View more <a href="http://www.slideshare.net/" style="text-decoration: underline;">presentations</a> from <a href="http://www.slideshare.net/joebensimo" style="text-decoration: underline;">Ben Simo</a>.</div>
</div><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0tag:blogger.com,1999:blog-414482498098790205.post-59999372881816587972009-02-15T18:12:00.005-07:002010-01-21T10:22:03.666-07:00Is There A Problem Here?<div style="float: right; padding-left: 10px;"><a href="http://2.bp.blogspot.com/_Mp0d-dsENrg/SZi_Cp6h1EI/AAAAAAAAT1o/bz3NJSm9CPM/s1600-h/frogsalad.jpg" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5303198613560742978" src="http://2.bp.blogspot.com/_Mp0d-dsENrg/SZi_Cp6h1EI/AAAAAAAAT1o/bz3NJSm9CPM/s400/frogsalad.jpg" style="cursor: pointer; display: block; height: 242px; margin: 0px auto 10px; text-align: center; width: 322px;" /></a><br />
</div>Over the years, I have collected a number of examples of software failures in the wild -- some I've encountered myself, some were shared by others. I've had intentions to create a blog for sharing these software failures, and new ones as they are discovered, with hope that software designers, developers, and testers can discuss and learn from them. I have finally launched that blog. It is titled <span style="color: #cc0000; font-style: italic;">Is There A Problem Here?</span><br />
<br />
I invite you to visit the blog and contribute at <a href="http://isthereaproblemhere.com/">http://IsThereAProblemHere.com</a>.<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com6tag:blogger.com,1999:blog-414482498098790205.post-59386259750159551742009-01-05T22:56:00.009-07:002010-01-21T10:23:51.468-07:00I'm helping you. I'm helping you.<a href="http://3.bp.blogspot.com/_Mp0d-dsENrg/SWL6iwEfgII/AAAAAAAASbU/dHZ17fFi4z0/s1600-h/100_2583.JPG" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5288064387412623490" src="http://3.bp.blogspot.com/_Mp0d-dsENrg/SWL6iwEfgII/AAAAAAAASbU/dHZ17fFi4z0/s320/100_2583.JPG" style="cursor: pointer; float: right; height: 320px; margin: 0pt 0pt 10px 10px; width: 240px;" /></a><br />
A few months ago, I enlisted my 11 year old son to help me with some work around the house. After a short while, he was doing something other than what I had asked him to do.<br />
<br />
I told him, "You're not helping me."<br />
<br />
"But I am helping you.", he replied.<br />
<br />
"No you're not."<br />
<br />
"I'm helping you. I'm helping you.", he shot back. He was frustrated. He really thought he was helping me; and I was putting down his work. I was frustrated too. From my view, his <span style="font-style: italic;">helping </span>was creating more work for me. I did not feel helped.<br />
<br />
Then it hit me. I've heard this argument before -- from software testers.<br />
<br />
I've seen testers, and test managers, attempt to justify their work by telling team members and stakeholders <span style="font-style: italic;"></span> "I'm helping you. I'm helping you." We QA and tester people develop metrics and reports to help us demonstrate how helpful we are. We talk about our quality assurance and testing processes. We talk about all the test cases we develop and execute. We like to show off our test automation that spits out impressive color-coded results.<br />
<br />
However, we still encounter unhappy team members and stakeholders. We develop adversarial relationships with developers. We have to explain ourselves to project leads that question the value of our testing. We hear people tell us we're not helping and we keep saying "I'm helping you. I'm helping you."<br />
<br />
Maybe, just like my son, we're not giving our stakeholders what they need. Maybe we aren't really helping. So instead of shooting back the "I'm helping you." line, we can stop and listen. Find out what our stakeholders want from us. Listen and ask clarifying questions to better understand how we can help.<br />
<br />
I'm not advocating that we just give in and do whatever we're asked without defending our positions. However, we can be willing to adjust our positions to better serve our stakeholders. (Joining an overly optimistic rush to release poor quality software usually doesn't serve them.) If there is disagreement, work to resolve it. Sometimes we may need to educate others on our areas of expertise. Yet we testers also need to respect others' roles and expertise. Listen and learn.<br />
<br />
So, the next time you feel like screaming "I'm helping you. I'm helping you.", try to better understand how you can help before turning up your defenses.<br />
<br />
Serve your stakeholders.<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com7tag:blogger.com,1999:blog-414482498098790205.post-13151381212637136012008-09-30T22:23:00.002-06:002010-01-21T10:27:55.142-07:00The Antonym of Testing<a href="http://1.bp.blogspot.com/_Mp0d-dsENrg/SOLfFSVvNDI/AAAAAAAAMss/rrhCCBb3MZ0/s1600-h/101B6291.JPG" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5252005397382771762" src="http://1.bp.blogspot.com/_Mp0d-dsENrg/SOLfFSVvNDI/AAAAAAAAMss/rrhCCBb3MZ0/s400/101B6291.JPG" style="cursor: pointer; float: right; height: 252px; margin: 0pt 0pt 10px 10px; width: 335px;" /></a><span style="font-style: italic;"><span style="font-family: georgia;"></span></span><br />
<blockquote><span style="font-style: italic;"><span style="font-family: georgia;">"... one usually encounters a definition such as, 'Testing is the process of confirming that a program is correct. It is the demonstration that errors are not present.' The main trouble with this definition is that it is totally wrong; in fact, it almost defines the antonym of testing."</span></span><br />
<span style="font-family: georgia;"><br />
- Glenford Myers, </span><br />
<a href="http://www.amazon.com/gp/product/0471627658?ie=UTF8&tag=qualfrog-20&linkCode=xm2&camp=1789&creativeASIN=0471627658" target="_QFTesting"><span style="font-family: georgia;">Software Reliability: Principles & Practices</span></a><span style="font-family: georgia;">, 1976</span><br />
</blockquote><span style="font-family: georgia;"><br />
</span>People keep telling me that testing is a <a href="http://en.wikipedia.org/wiki/Verification_and_Validation_%28software%29"><span style="color: blue;">validation</span></a> activity -- that the purpose of testing is to validate that the software meets all the specifications, has no errors, meets performance SLAs, meets expectations of anonymous users, or some other lofty goal.<br />
<br />
I read about testing processes designed to validate software. I use testing tools built to support validation. I listen to service companies pitch testing services to validate software. I read about testing metrics built on the assertion that software systems can be proved correct. I attend testing presentations explaining the presenters' best practices for validation.<br />
<br />
The trouble is that we cannot prove software correct. We cannot prove the absence of bugs. We cannot test every possible state and input. We cannot evaluate every possible output. We cannot fully understand the desires of stakeholders. We cannot prove that customers will be happy. We cannot prove that a software product will solve the problems it was built to solve. If all this were possible, I suspect insurance companies would find a way to make a profit selling software quality insurance.<br />
<br />
<blockquote>"If you think you can fully test a program without testing its response to every possible input, fine. Give us a list of your test cases. We can write a program that will pass all your tests but still fail spectacularly on an input you missed. If we can do this deliberately, our contention is that we or other programmers can do it accidentally."<br />
<br />
- Cem Kaner, Jack Falk, and Hung Quoc Nguyen,<br />
<a href="http://www.amazon.com/gp/product/0471358460?ie=UTF8&tag=qualfrog-20&linkCode=xm2&camp=1789&creativeASIN=0471358460">Testing Computer Software, Second Edition,</a> 1999<br />
</blockquote><br />
Now, thirty-two years since Glenford Myers called testing to prove correctness the opposite of testing, we're surrounded by testing practices and tools based on proving correctness. The myth of proving correctness is alive and well.<br />
<br />
<span style="font-weight: bold;">Activities designed to try to prove correctness are the antonym of testing. </span><br />
<br />
<span style="font-weight: bold;">So if testing is not validation, what is testing?</span> Testing is investigation; and communicating useful information about quality to decision makers.<br />
<br />
<blockquote><span style="font-style: italic;">"Testing is the process by which we explore and understand the status of the benefits and the risk associated with release of a software system."</span><br />
<br />
- James Bach,<br />
James Bach on Risk-Based Testing, STQE Magazine, Nov 1999 <br />
<br />
<br />
<span style="font-style: italic;">"Testing is done to find information. Critical decisions about the project or the product are made on the basis of that information."</span><br />
<br />
- Cem Kaner, James Bach, Bret Pettichord,<br />
<a href="http://www.amazon.com/gp/product/0471081124?ie=UTF8&tag=qualfrog-20&linkCode=xm2&camp=1789&creativeASIN=0471081124">Lessons Learned In Software Testing: A Context-Driven Approach</a>, 2002<br />
<br />
<br />
<span style="font-style: italic;">"A software tester’s job is to test software, find bugs, and report them so that they can be fixed. An effective software tester focuses on the software product itself and gathers empirical information regarding what it does and doesn’t do. This is a big job all by itself. The challenge is to provide accurate, comprehensive, and timely information, so managers can make informed decisions."</span><br />
<br />
- Brett Pettichord,<br />
<a href="http://www.stickyminds.com/sitewide.asp?ObjectId=3543&ObjectType=COL&Function=edetail">Don't Become the Quality Police, StickyMinds.com</a>, 2002<br />
</blockquote><br />
<br />
Once we admit that we cannot prove the software correct, we can refocus our efforts on finding useful quality-related information. Instead of pretending to assure quality or validate correctness, we can gather and communicate useful information. Investigate the software. Find information about threats to the quality of the systems under investigation. Communicate that information in terms that matter to stakeholders. Help managers make informed decisions.<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com7tag:blogger.com,1999:blog-414482498098790205.post-61839066511403601412008-07-26T22:34:00.007-06:002010-01-21T10:32:37.767-07:00Pause at the Pump<ul><li><span style="color: #000099;">My fuel gauge is on empty.</span></li>
</ul><ul><li><span style="color: #000099;">I don't want to stop, but pull into the gas station.</span></li>
</ul><ul><li><span style="color: #000099;">I tell the children to stay in the car.</span></li>
</ul><ul><li><span style="color: #000099;">I get out of the car.</span></li>
</ul><ul><li><span style="color: #000099;">The children are asking me questions from inside the car that I can't hear well enough to understand.</span></li>
</ul><ul><li><span style="color: #000099;">I swipe my credit card in the pump's card reader.<br />
</span></li>
</ul><span style="color: #000099;"> </span><br />
<ul><li><span style="color: #000099;">The pump responds by prompting me to "<span style="color: #993300;">SELECT WINDOW OR OUTSIDE</span>".</span></li>
</ul><ul><li><span style="color: #000099;">The children are still talking to me. I still don't understand.</span></li>
</ul><ul><li><span style="color: #000099;">I pause and stare at the keypad.</span></li>
</ul><br />
<br />
<a href="http://2.bp.blogspot.com/_Mp0d-dsENrg/SIv67TvjC_I/AAAAAAAAGwo/cQeHLxSX-hI/s1600-h/IMAG0561.jpg" style="color: #000099;"><img alt="" border="0" src="http://2.bp.blogspot.com/_Mp0d-dsENrg/SIv67TvjC_I/AAAAAAAAGwo/cQeHLxSX-hI/s400/IMAG0561.jpg" style="margin: 0px 10px 10px 0px;" /></a><br />
<br />
<br />
<span style="color: #000099; font-weight: bold;">Is there a problem here?</span><br />
<br />
<ul><li><span style="color: #000099;">I pause to think for a moment.</span></li>
</ul><ul><li><span style="color: #000099;">I hear the children asking me questions that I still don't understand through the closed car windows.</span></li>
</ul><ul><li><span style="color: #000099;">I scan the keypad again.</span></li>
</ul><ul><li><span style="color: #000099;">I cant find the "<span style="color: #993300;">OUTSIDE</span>" button.</span></li>
</ul><br />
<br />
<span style="font-weight: bold;">Recognizing Bugs</span><br />
<br />
At <a href="http://www.cast2008.org/">CAST</a> last week, <a href="http://testertested.blogspot.com/">Pradeep Soundararajan</a> gave a Lighting Talk about the importance of testers being able to recognize a bug. Tests may be of little use if the tester doesn't recognize the bugs triggered by the test.<br />
<br />
Sometimes bugs are obvious. Sometimes bugs are not clearly violations of requirements documents. This is especially true when it comes to <a href="http://en.wikipedia.org/wiki/Human-computer_interaction">human computer interaction</a> problems.<br />
<br />
Requirements are not always clear and objective.<br />
<br />
So, do you recognize why I may have paused when I read the prompt on the gasoline pump? It wasn't the price of the $4 per gallon fuel. It wasn't because I had to think about how I wanted to pay.<br />
<br />
I paused because I didn't see a button labeled "OUTSIDE". Plus, I already swiped my credit card indicating that I wanted to pay at the pump. And even if I were to pay at the cashier window, I would still be outside.<br />
<br />
I wonder how many minutes are wasted each month prompting customers to select where they want to pay after they have swiped their credit card. I wonder how many other people pause and read twice in search of the button to indicate that they want to pay outside.<br />
<br />
The developers and testers of the software in this pump may have not recognized this problem. Maybe the makers executed test scripts -- either automated or manual -- and were blind to the problem. Or maybe they didn't deem it important enough to change.<br />
<br />
Sometimes familiarity with the technical details of a system can hide problems that are obvious to those that don't know the technology, the requirements documents, and the test scripts. As testers it is important that we be careful not to let our familiarity with a system make us blind to to bugs -- things that bug our users.<br />
<br />
Will you recognize a problem if you see it?<div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com6tag:blogger.com,1999:blog-414482498098790205.post-40134067144206932302008-05-20T17:24:00.006-06:002012-05-17T07:56:09.906-06:00Is There A Problem Here?<a href="http://3.bp.blogspot.com/_Mp0d-dsENrg/SDNeJDftPQI/AAAAAAAADUY/tPJdt_Ly2dU/s1600-h/MSN_firefox.jpg"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5202605504193182978" src="http://3.bp.blogspot.com/_Mp0d-dsENrg/SDNeJDftPQI/AAAAAAAADUY/tPJdt_Ly2dU/s400/MSN_firefox.jpg" style="cursor: pointer; margin: 0pt 10px 10px 0pt;" /></a><br />
<br />
<span style="font-size: 130%; font-style: italic;"><span style="font-weight: bold;"></span></span><span style="font-size: 130%; font-style: italic;"><span style="font-weight: bold;"></span></span><br />
<blockquote style="font-family: arial;">
<span style="color: #3333ff; font-size: 130%; font-style: italic;"><span style="font-weight: bold;">msn </span></span><span style="color: #3333ff;">video</span><br />
<br />
<span style="color: #3333ff;">To use this product, you need to install free software</span><br />
<hr />
<span style="color: #333333; font-size: 85%;">This product requires Microsoft Internet Explorer 6 with Microsoft Media Players 10 and Macromedia Flash 6 or higher versions, or Mozilla Firefox 1.5 with Macromedia Flash 8, or Safari 2.0.4 with Macromedia Flash 8. To download these free software applications, click the links below and follow the on-screen instructions.<br /><br /><span style="font-weight: bold;">Step 1: </span>download firefox 1.5</span><span style="color: #333333; font-size: 85%;"><span style="font-size: 78%;">download firefox 1.5</span><br /><br /><span style="font-weight: bold;">Step 2: </span>Download Macromedia Flash Player</span><span style="font-size: 85%;"><span style="color: #333333; font-size: 78%;">Macromedia Flash player is free to download.<br />If still having problems, uninstall Flash and then re-install Flash.</span><br /><br /><span style="color: #993399;">Once the installations are complete, reload this page.</span></span><span style="font-size: 85%;"></span><br />
<blockquote>
</blockquote>
</blockquote><div class="blogger-post-footer"><br/><img src="http://blog.qualityfrog.com/frog/qfrog-tiny.gif"/><br/><a href="mailto:ben@qualityfrog.com?subject=Questioning%20Software">Ben Simo</a><br/><a href="http://www.QuestioningSoftware.com">QuestioningSoftware.com</a><br/>
<hr/>
<a href="http://www.cast2009.org"><img src="http://lh3.ggpht.com/_Mp0d-dsENrg/SbwakO5GHuI/AAAAAAAAUd8/GjsCcF_X8K4/s800/CAST_WebBanner.jpg" /></a></div>Ben Simohttp://www.blogger.com/profile/11448600123169359955noreply@blogger.com0