October 16, 2007

Problems: So What's On All Those Sticky Notes?

Posted by Ben Simo

In my previous post about the Agile Alliance Functional Testing Tools Workshop , I wrote the following:

After reviewing existing tools used by agile teams: we identified software testing issues that have been solved (yellow), those that have been partially solved (orange), and those that have not been solved (pink). As I recollect, most of the solved issues were technical problems and most of the unsolved problems were people problems. Many of the partially solved problems were those for which I believe we have technical solutions but have not yet been integrated and presented in ways that best support people.
In case you are wondering what problems we wrote down on these notes, Frank Maurer kindly transcribed them for the workshop participants and I have posted them below.

Looking at this list reminds me of the traffic safety problem lists I made in the defensive driving classes I used to teach. As unsafe driving practices were brought up by students, I would add them to a list on the whiteboard. I then asked the students to identify whether each item on my list was primarily due to driver skill or driver attitude. The students usually blamed most of the problems on driver attitude.

Skill and attitude play important roles in software development and testing. Team members need both. Brian Marick addresses this in his guiding values of discipline, skill, ease, and joy. Instead of looking at the functional testing problems as skill or attitude problems, I looked at them as man or machine problems. I asked myself if each problem appeared to be mainly a human problem or a technical problem.

Software development and testing involves a mix of people and technical problems. Interfacing people with people and people with technology is often harder than interfacing technology with technology. Most of the identified problems have both technical and human aspects to the problem and possible solutions. Some are due to the nature of people or the nature of software and will likely never be completely solved.

I find that identifying whether a problem is primarily a people or technology problem helps me identify possible solutions. I quickly scanned this list and identified whether I thought things were primarily human or technical issues. My notes are included to the right of each item. I don't necessarily interpret each item as its author (a human communication problem), and I do not necessarily agree that each item is a problem or belongs in the specified group.

As you review this list, ask yourself if the problem and possible solutions are grounded in people or technology.

Unsolved Define Test Human
Unsolved Organizing large sets of Tests/Expect. Actions/Examples for a large, complex system so you can wrap your head around the whole thing. Human
Unsolved Having tests survive handoffs. Project team -> op support -> proj team. Human and technical
UnsolvedGetting people to care Human
Unsolved Transferability of ubiquitous language to other projects Human
Unsolved Write SW that is understood Human
Unsolved Reducing Uncertainty Human
Unsolved Limitations of natural language Human
Unsolved How would we act if we really believed code was an asset? Mostly Human
Unsolved Allowing Customers to articulate their expectations in a format/tool/way that is comfortable for them Mostly Human
Unsolved Multi-model specification text + table + graphic in one test. Mostly Technical
Unsolved Domain experts Human
Unsolved Fully Automated regr. That does not reduce dev.velocity. Mostly Technical
Unsolved Generate a domain model from tests. Human and technical
Unsolved Testing usability as part of acceptance testing in incremental development. Human
Unsolved Common language to express GUI based tests. Mostly Human
Unsolved Conveying "experts" perspective to majority of development team. Human
Unsolved Automated Software Development Technical (Machines aren't creative)
Unsolved Functional tests that can be easily re-used later in lifecycle. Mostly Technical
Unsolved Test business requirements independent of current interaction/Api design. Mostly Human
Unsolved Composing tests into useful larger tests Technical and Human
Unsolved Test first performance Human
Unsolved Getting BA to write the tests Mostly Human
Unsolved Having customer to be able to write test and enjoy it. Mostly Human
Unsolved Different test notations for different user groups. Human problem, technical solution?
Unsolved Acceptance/Functional Tests good for communication and automation. Human problem, technical solution?
Unsolved Model the time domain " and 3 months later an email. Mostly Technical? (Don't want execution to take 3 months.)
Unsolved Change touch-point dynamically. ?
Unsolved Terminology (Test or not a Test?) Human

Partially Solved Understand what has not been tested. Human and Technical
Partially Solved Trace tests into project management tools Human and Technical
Partially Solved Getting buy-in for need to automate.(docs,tests,specs) Human and Technical
Partially Solved Accurately & completely communicating requirements. Human problem, partial technical solutions
Partially Solved Satisfying every role's need/desire to be at center, in control. Human
Partially Solved Having functional tests specify requirement specifications. Human and Technical
Partially Solved Sustain a productive conversation with all stake holders. Human problem, partial technical solution
Partially Solved How do we get across what the project would feel like if things were going well. Human
Partially Solved Write robust (U.I) tests that are not brittle. Mostly Technical
Partially Solved Fragile tests. Mostly Technical
Partially Solved Valuing individuals and interactions over processes and tools. Human
Partially Solved Describing customer intent. Human
Partially Solved Executable (as tests) Models (as specifications) Human and technical
Partially Solved Test partitioning Human and technical
Partially Solved Tests as support artifacts. Human and technical
Partially Solved Test Generation Automation. Human and technical
Partially Solved Running tests parallely ( Fast feedback) Mostly technical
Partially Solved Reconciling preferred style of abstraction. Human and technical
Partially Solved Dealing with size. Human
Partially Solved Finding the right words in which to write a test. Human
Partially Solved Express requirement in the domain language, graphical, word based , table based. Mostly Technical
Partially Solved Functional Test driven development..not just for agilists. How to sell to waterfallists? Mostly Human
Partially Solved How to Integrate tools? Mostly Technical
Partially Solved Common test case format. Human and Technical
Partially Solved Cooperation and collaboration between tool developers (tool stack, tool platform). Human and Technical
Partially Solved Change from one notation to another. (graphic -> tabular) Mostly technical
Partially Solved Test/Example -> model (generated model based tests) Mostly technical
Partially Solved IDE for testers and BAs Human and technical
Partially Solved Get all roles actively involved Mostly Human
Partially Solved View specs/examples/tests, differently for different roles. Mostly Technical
Partially Solved Different editors for different roles? Mostly Technical
Partially Solved Super-power IDE Technical solution to human problems?
Partially Solved Test Refactoring Human and technical
Partially Solved Refactoring tests Human and technical
Partially Solved Prioritize and execute tests to get faster feedback. Mostly Human
Partially Solved Describe a test at an appropriate level of abstraction Mostly Technical
Partially Solved Choosing what to automate (when you can't automate everything) Mostly Technical
Partially Solved Tools to support exploratory testing Human and Technical
Partially Solved Reusable test artifacts (poor modularity cohesion) Mostly Technical
Partially Solved Build community with BAs Mostly Human
Partially Solved Allow for refactoring from /to code <-> tests <-> req'ts Mostly Technical
Partially Solved Setup a wiki to discuss smaller problem solving. People
Partially Solved Capture war stories + testimonials + experiences. Human, partial technical solution
Partially Solved Test Maintenance Human and Technical
Partially Solved Book: Patterns of Self testing software ?
Partially Solved Shared vocabulary around parts of a functional testing solution ("Fixture",etc) Human
Partially Solved Ensure adequate test coverage. Mostly Technical
Partially Solved Communication of what has been tested Human and technical
Partially Solved Communication using the ubiquitous language. Human

Solved Express automated/able tests in tables Mostly Technical
Solved Correctly implementing programmer intent Mostly Technical
Solved Deliver SW to test that doesn’t crash immediately Mostly Technical
Solved Provide traceability between Story or Requirement and accpetance/functional test Mostly Technical
Solved Gui Testing - functional testing is more than GUI testing Technical
Solved Data-Driven Testing Technical
Solved Driving Apps Technical
Solved Integrating test executors/drivers with build process Technical
Solved Edit Fit tests from eclipse Technical
Solved Report Results Technical to report, human to be understood
Solved Express expectations in code Human and technical
Solved Unit testing Mostly Technical

I think we can solve the technical issues and use technical solutions to help people manage some of the people problems. Applying discipline and skill to solve the technical problems may help add to testers' ease and joy.

Where do you think the solutions lie? Have a solution? Please share it.


1 Comment:

May 06, 2008  
Van der Hoorn wrote:

Interesting list.

Maybe you're interested to know that there is an article on 'Test first performance' by IBM, written by Johnson et al. (2007) called "Incorporating Performance Testing in Test-Driven Development". They introduce Test first performance (TFP).