5 Effective Ways for Usability Testing to Play Nice with Agile
Jeff Gothelf | June 15th, 2011Usability testing has been a fundamental tool in the UX arsenal for decades now. The value of actually meeting your customers and letting them experience your product makes a significant impact to the shape of that product. In it’s most formal version, testing can be a multi-day, multi-thousand $/€ process that delivers final analysis days if not weeks later. With many organizations moving to an Agile philosophy and methodology, UX practitioners are finding it difficult to integrate formal usability testing into this faster-paced, iterative approach to software development.
See? Lions and zebras can get along. So, too, can Agile and Usability Testing.
The main challenge is time. The need to keep the teams moving forward, coupled with typical two-week development cycles between releases (aka sprints) doesn’t provide an obvious point to insert testing. Indeed, fitting a formal usability testing cycle into a series of two week sprints is challenging if not impossible. Instead, consider the following techniques to help get that critical customer feedback and still keep your team’s momentum.
1. Start small.
Jakob Nielsen has proven through scientific method that 5-8 is the optimal number of participants for qualitative tests. Consider testing only 3 people each time instead. The goal of agile usability testing is to clear the boulders from the road – not to provide pixel-level design direction. Three participants will, very quickly, validate (or not) your product’s core workflows and give your team a sense of where to focus next.
In addition, three participants take less time to test than 5+. Whereas a traditional rubric would have a team tied up for a day or two, a 3-person study can be completed in a morning. This is especially helpful if you don’t have a dedicated research staff. Your UX generalists can test, get feedback quickly and be back optimizing the design within one day.
Finally, three participants are cheaper. There is greater value in testing multiple iterations on more customers than one design on many customers. It allows you to move forward more quickly and keep your Agile team’s momentum.
This approach assumes you are performing moderated usability testing in the context of a lab, work site or some other location. As an additional data point, consider using remote usability testing as a way to provide additional perspective on your findings.
2. Test *every* week.
Pick a day (preferably not Monday or Friday) and make that your “testing day.” Every week, on that day, you’re in the lab, coffee shop, or customer site testing. Make it clear to your teams that this is testing day. Put recurring meeting invitations on your calendar and forward out to stakeholders. The night before testing day, send out a reminder with a participant schedule.
The goal is to make that day synonymous with testing. Teams should stretch to get work into that day’s test and use the cadence to drive productivity.
What ends up happening is the organization becomes accustomed to a steady stream of qualitative insight coming from the UX and product teams. Testing day becomes a target for quick-turnaround validation and the team becomes almost reliant on that stream to ensure the quick decisions made at iteration planning line up with business and user goals.
In weeks where testing day gets de-prioritized (hey, it happens), a concurrent micro-study using an automated online tool can maintain the flow of insight until you get back on track.
3. Show them whatever you have ready.
Customers will react to whatever you put in front of them. Yes, the reaction will differ based on what that item is or it’s fidelity but you’ll get *some* feedback — and that’s much better than nothing.
If you have code in production or a qa environment, show it. If you have wireframes or mockups, show them. If you can string together a series of mockups in fireworks, visio or even PowerPoint, do that and show it to real customers. A napkin sketch is also viable. Anything that will get your ideas in front of users will provide value. Even if you have nothing, it’s still worth sitting down with a customer and talking about their pain points.
4. Invite everyone (or bring the testing to them).
*Note: this tip assumes you have some kind of lab onsite. If you don’t, and typically do remote usability testing, guerrilla testing or site visits, point #5 below is most crucial to your success.
Key to getting the findings from these weekly tests into the iteration is buy-in from the rest of your scrum team and the organization as a whole. The most effective way to do this is to get the engineers, product managers and stakeholders to the testing sessions.
Most testing software broadcasts over your local network and if yours doesn’t you can always use Webex or GoToMeeting to broadcast. Setup multiple viewing areas convenient to your team. Ensure they know where to go and how to turn viewing on.
Another option is to enable viewing at each person’s desk. Again, most usability and general screensharing software allows this. This is most convenient as it doesn’t require the team to go anywhere to participate.
After viewing one or two customers the team gets a sense of where the problem areas are leaving you free of the responsibility of convincing them later. In addition, most team mates start to like the immediacy of the feedback on their work and start attending regularly without much reminding.
5. Report back quickly.
This one is critical. Turn feedback around to your team and to the organization as quickly as possible. No need for a fancy report or designed layouts. Type up a quick executive summary followed by recommendations for the pain points discovered and send out via email. Many tools offer some kind of findings visualization that allows you to, very quickly, generate a scatter plot or heat map of user activity. Ideally you should keep a copy of the findings in a commonly accessible place like a wiki or shared folder.
The goal here is twofold. One, you are getting customer feedback to the team so they can adjust their work accordingly. The other goal is to tell the broader organization that, despite moving quickly, you’re validating your approach with real people.
*5.5 – Use an outside recruiter.
Getting the right customers to your studies is critical to the validity of your findings Keeping up the pace of recruiting participants to your studies on a weekly cadence, however, is a time consuming endeavor. It is highly recommended that you find a resource external to your team to perform this task. If you can afford it, hire a recruiter who specializes in this task. If not, minimize time on task for this by spreading the work across the team using a shared calendar.
Agile development practices pose some challenges to traditional usability testing methods. However, with a bit of modification and a focus on outcomes as opposed to output, getting to validated learnings can be successful and rewarding.
What’s been your experience? Share your feedback in the comments.
This is the first guest post on our blog, and we plan to do more. Interested in writing a post for us? Please contact loucas@usabilla.com
Follow me on Twitter
June 16th, 2011 at 6:06 PM
Great advice — especially focusing on outcomes — not output in the Agile context. Responding to change, rather than following a plan…
We have revolving-door UX sessions weekly with user surrogates face to face. We have 10 people who are committed to 15 minute sessons on the half hour every monday. That way, we can divvy up the projects and each of the 3 scrum teams gets at least 3 sessons. We also swap our participants around to avoid fatiguing them with the same project. Generally, we are validating ixd concepts with Balsamiq click-thrus that represent our “happy path hypothesis” We record those sessions with Silverback. The UX person gives the report out immediately when the sessions are done and archives the recorded sessions so that we can do further analysis later.
We also do longer bi-monthly sessions with Real Users (customers who have signed up to be “UX Lab Rats”. These are 45 minute sessions — remote testing. I use WebEx and share an application — live code or a prototype. We record it — but also encourage developers and product managers to dial in (stay mute) and observe real-time.
This program has been the most influential in getting people to understand that what UX and UCD is all about — people using the stuff that we are building to solve their problems.
June 16th, 2011 at 7:03 PM
This is great stuff. Quick and effective usability feedback is so important yet many people seem to forget to do it at all. Your product and approach sounds very relevant within an agile project. There is of course room for different types of usability test, including things like our own Kupima product and other similar services, which can augment and enrich the feedback you would get from something like Usabilla. The great thing about the web is that nobody is forced to just do things one way – variety of approach can be a very useful strategy to adopt.
June 17th, 2011 at 8:26 AM
Really interesting article. We’re currently going through this process, integrating usability testing into this process.
We’re working with an agency who are delivering the high-fidelity Axure prototypes according to the scenarios developed to test function and flow. One of the biggest time stealers in this actually seems to be the production of the prototype and all the different screens to meet the scenario requirements.
As usability testing (especially with a lab) is expensive you want to be getting the most out of each session which is why we’re working with hi-fidelity mockups.
We’ve worked with our agency (who have been great) to streamline this and make it more manageable but it would be interesting to see what you recommend for streamlining high fidelity prototype production to lead into usability testing?
June 17th, 2011 at 10:17 PM
In response to Graham – ditch the high fi prototype, test whatever you have. See Jeff’s text above.
June 18th, 2011 at 9:50 PM
I like the testing approach. I mean it is obviously different if the developer is dog-fooding it, as they are going to make routines and interfaces which suit them best, so getting outside opinions, ones from outside the customer base, and outside the organization, count.
June 21st, 2011 at 7:31 AM
I think this post has very little about using Agile with Usability testing – I think this isn’t a good post by the standards of this blog.
1) You can use remote usability testing
2) Do it so it fits in with your Scrum/Agile cycle
This seems very obvious to me. Am I the only one who thinks this?
The ideas seem also very similar to tips I have previously read post here: http://goo.gl/YrfS6
June 21st, 2011 at 10:36 AM
@Adrienne
Remote usability is definitely one of the tools of trade for rapid tests. There’s no such thing as THE usability testing tool, so depending on what you’d like to learn you pick a tool (offline, online, functional, qualitative, quantitative, etc). What I personally like about this blogpost is a broader take on how to fit tests into development cycles. Quite some organizations I meet are struggling with this theme.
Take a look at this video, which is a nice example of Mindflash (http://mindflash.com), which includes both qualitative offline tests and remote usability test (including Usabilla tests) in their development cycles: http://uxweek.com/2010/videos/video-2010-paula-wellings-cameron-gray
June 21st, 2011 at 11:18 AM
I have to agree with Adrienne, not the best post I’ve seen on the topic – I would like to see a lot more detail. This post doesn’t tell me much at all. I think the Smurf one was much better! I guess there is some similarity between the posts too.
Paul, you make a good comment though, I know a lot of organisations who simply don’t see a way to fit in their testing. Quick, simple tools make it more realistic for them to test on a regular basis. Old fashioned testing is simply too time consuming and expensive. I would love to know more about your experiences with large organisations.
June 29th, 2011 at 2:02 PM
[...] skulle kort sagt testat oftare och mer. Usabilla publicerade nyligen en artikel om testning i agila projekt där det pratas om att testa varje vecka. Man låter en dag i veckan vara test-dagen och gör [...]
July 6th, 2011 at 3:22 PM
[...] via: http://blog.usabilla.com/ [...]
July 14th, 2011 at 7:44 AM
Fundamental to success with agile practices being successful is continuous testing. While continuous integration is something most Agile practitioners adhere to continuous testing is equally if not more important to ensure there is continuous delivery of solutions that meet the business requirements of cost, time and quality.
July 27th, 2011 at 2:46 PM
Pretty nice post. You can write in more details but still I liked your article. Thanks for sharing.
July 28th, 2011 at 9:13 PM
[...] 5 Effective Ways for Usability Testing to Play Nice with Agile (usabilla.com) [...]
November 17th, 2011 at 12:47 PM
[...] http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/ [...]