May 23, 2012

What is Performance?

Posted by Ben Simo

As we design and test for performance, let's look beyond speed. Let's look beyond basic stability. Let's look at the many facets of performance. Consider in which ways your system needs to perform. How fast does it need to be? How long does it need to keep running? Does it need to grow? Does it need to be available at all times? How much can we spend? Can we make it faster?

There are endless questions we could ask. Therefore, categorizing facets of performance and creating tests for each category can be helpful. However, let's not fail to look at the interaction between these categories.

OFAT (One Factor At a Time) testing (as exampled in the above performance testing checklist excerpt) often fails to provide information related to the interaction between the categories. Let's do some MFAT (Multiple Factors At a Time) testing and analysis. Let's look at the system as a whole. Let's mix it up. Let's consider how these facets interact. Let's create test scenarios that include multiple facets.


May 17, 2012

Time Machine: Devops, 45 Years Ago

Posted by Ben Simo

"Two major inputs are required 
to provide computer service: 
equipment and manpower. 

For development
employees such as 
programmers and systems analysts are needed;
for operation
the services of managers, operators, etc., are essential. 

In practice the two activities usually coexist. 

Some sort of computer installation 
is required 
to check out development efforts. 

And almost every installation 
engages in continuing development 
as old systems are modified and new ones begun."

The above suggests that Devops is not at all a new concept. Forty-five years ago, computer rental cost slightly more than the Devops staff required to use it. Non-personnel operations cost typically rounded out the final third of the total cost. Today, hardware is cheap and people are expensive.

So, what did a Devops organization look like forty-five years ago?  The table below presents some data gathered in a "nation-wide census of data processing personnel" in 1967.

How do these wages add up in today's dollars?

Check out

Compared to my experience over the past couple decades, this seems to be heavy on management -- both in cost and percent of personnel. The roles of computer operator and librarian are pretty much gone. Programming and analyst roles have since blurred. Software has replaced many things that people used to do.  As the user base has moved from specialist operators to the general public, new testing and user experience design roles have evolved.

What else has changed?

What do you think devops will look like forty-five years into the future?


May 16, 2012

The Right Stuff?

Posted by Ben Simo

Update: I haz new job. I am no longer seeking work. (12 June 2012)

You cannot be anything you want to be
-- but you can be a lot more of who you already are.

Given that I resigned a couple weeks ago and am looking for new work, I've been asking myself a lot of questions about what I want to do next. Do I want to do detailed technical work? Do I want to lead people? Do I want to consult? Do I want to be someone's employee, or do I want to do short term contract work? Do I want to stay in Arizona, go back to Colorado, or go somewhere new? I've not yet decided. I am considering a number of options.

What I do know is that I want to find work that is a good match for my strengths and mindset. When looking for new work for myself, and when I've been involved in hiring, matching strengths and mindset has been more important to me than matching specific technical skills. People who are enabled to exercise their strengths and philosophy tend to have the intrinsic motivation needed to update their skills as each situation demands.

Improving the effectiveness 

of knowledge-work businesses.

I am intrigued by the Rightshifting ideas coming from Bob Marshall, the Flowchain Sensei. The basic idea of the Rightshifting model is that the curve shown below indicates that the majority of organizations (peak of the curve) are ineffective compared to the few effective organizations on the right side of the curve. While organizations to the right of Bob's model tend to better match my personal mindset, finding work with such organizations is difficult -- there aren't many of them. My experience suggests that organizations further to the left can better benefit from what I have to offer than organizations to the right. However, if such organizations have no desire to improve their effectiveness, I am likely to get frustrated. Therefore, one more question I am asking myself is whether I prefer to be more of a hero in the midst of analytical dysfunction or more of a team member in a synergistic organization. I suspect my place is somewhere in the middle -- with an organization that is shifting right.

Side Note: One way in which I think I may disagree with Bob Marshall's model is the assertion that the chaotic organizations on the left are less productive than the analytic organizations to their right. I suspect many of the majority (the analytical organizations) are just better at measuring (or pretending to measure) their productivity than the organizations to their left. What we label chaos is often order we don't yet understand well enough to describe. I'm also not convinced that organizations need to take a trip through the dehumanizing methods of many analytical organizations in order to become more productive.

I recently watched a video in which Bob Marshall recommended the Strengths Finder book and the associated online assessment as a useful tool in discovering your strengths. I am typically skeptical of such assessments. I fear they tend to provoke the Forer effect in which people tend to demonstrate a confirmation bias and attribute accuracy to generalized information about themselves when it is presented as being specifically tailored for them. However, I decided to give Strengths Finder a try. Despite my reservations and desire to critically evaluate the timed questionnaire as I completed it, I found the resulting top "themes" to be a surprisingly accurate description of my strengths. Perhaps this only demonstrates that even a skeptic like me can succumb to the Forer effect.

According to Strengths Finder, my top five strengths are:

  1. Ideation
  2. Command
  3. Activator
  4. Individualization
  5. Relator

Whether this is based on a meaningful algorithm or is total BS, I found the results useful. (All models are wrong, but some are useful.)  These five strengths seem to example what I believe can make a good context-driven tester; but then, I'm biased. :)

So, back to what do I want in my next job or independent business venture: I want to exercise my strengths in an environment where I don't have to suffer cognitive dissonance in order to serve my employer or clients.

In thinking about this, I created a mind map of the strengths from Strengths Finder and things I like, as well as things I don't like. Following up on my desire to find a mindset match over simply matching skills, I've decided to ignore typical job hunting advice and share this.

Think I might be a match for your organization? Hire me. :)


May 3, 2012

Hire Ben Simo!

Posted by Ben Simo

Update: I haz new job. I am no longer seeking work. (12 June 2012)

I resigned this week.  

Therefore, I am now available and looking for new testing work.

I prefer to stay in the Phoenix, Arizona, area but may be willing to move if the opportunity is right. (I dislike the disruption of moving more than I dislike the idea of living elsewhere.)

I am an amphibious time-traveling context-driven cyborg software tester.

What does this mean?

I am amphibious. Although I am a software tester first, I’ve spent much of my career straddling roles and environments. It is common for testers to view me as a very technical person while programmers view me as one more focused on value. I check existing beliefs and test to find new information. I script and I explore. I test to help understand whether the right thing is being done and I check to help verify that what is being done is being done correctly. I've often found myself playing the role of liaison between techies and regular people -- helping disparate stakeholders understand one another. 

I am a time-traveler. However, my time machine only goes forward. Over 20 years ago, I fell into a testing role with the illustrious title of “data collector”. I was one of a few hired to execute test cases created by others and report the results. However, from day one, things didn’t go as planned. Executing the test cases required deeper understanding than was communicated in the documentation. I quickly learned that testing can be fun and mentally challenging work. In these past 20 years, I've seen many technologies and practices come and go, and then come back again. One thing remains constant: the need to test whatever it is that us fallible humans try to create in the virtual worlds inside our machines -- virtual worlds that have real impact on real people in the real world.

I am context driven. This means I reject the concept of best practices. Instead, I try to first understand, and then select practices that fit each situation. I adapt my testing techniques and practices to the context. While my context-driven worldview is often at odds with those of others with whom I work, I recognize that people are the most important part of any context, and therefore try to find ways that I can contribute within whatever environment I find myself.

I am a cyborg.  I have both human and artificial parts. No, I am not referring to the hardware used to reconstruct my wrist last year. I like to practice sapient brain-engaged testing and enhance it with automation. Rather than try to replace sapient testing, I use automation to enhance and extend my capabilities as a tester.

I am a tester.  Most of all, I am a tester. While I have many other skills, I generally use them to support my testing and help others test well. I like to develop software that helps solve testing problems, but I don't want to spend the majority of my time with code. I like people more than I like code. I enjoy getting to know stakeholders and learning what matters to them. I enjoy exploring software in search of threats to value. I enjoy communicating and helping apply my discoveries to make things better.

View Ben Simo's profile on LinkedIn

Want to know more about me?
  • Check out my profile on Linked In.
  • Read this blog.
  • Read my interview with UTest: Part 1, Part 2, Part 3.
  • Follow me on Twitter.
  • Send me an email.
  • Check out the experimental mind map below. This highlights some of my experience; and also contains a bunch of details that I typically think of as resume clutter.


Target Tested ✔

Posted by Ben Simo

Computers empower us to create and manipulate things in ways that don't produce quite what we intended. This requires testing for what we don't expect.

There is a problem with this advertisement from Target Australia. Can you spot it?