The Test Environment
Here is how to prepare your test environment:
– Formal testing labs will typically have set ups like two-way mirrors, or closed circuit video monitors, as well as cameras often capturing both the screen and the user from the front. Just know that while that is great if you have it, you do not need these things to have an extremely useful and valuable test.I can’t count how many prototypes I aretested at a tiny table at Starbucks just big enough for the laptop, with three chairs around the table. In fact, in some ways this is preferable to the testing lab because the user feels a lot less like a lab rat.
– The other environment that works terrific is your customer is office.
It is time consuming to go there, but often even 30 minutes in their office will tell you a lot, and they are “master of their domain” and often very talkative. Also, all the cues are there to remind them of how they might use the product. You can also learn from seeing what their office looks like. How big is their monitor? How fast is their computer and network connectivity? How do they communicate with their colleagues on their work tasks?
– There are tools for doing this type of testing remotely, but while you can see where their mouse is, and what the user is clicking on, it is not the same as looking at the person is eyes and body language, so again while the more testing the better, to me this is not a substitute for face-to-face testing.
– As product manager, you need to make sure you are at every single test.
Do not delegate this. Real value comes from experiencing as many users as possible, first hand, interacting with and responding to your ideas. Even if you use an outside firm to arrange and administer the tests, you need to be there with them during the testing. No one else knows your product as well as you do, and you will have insights from watching the slightest hesitation or confused look, or the nuance of a question that reveals that they don’t understand the product model or a particular feature. What gets summarized for you by a proctor will probably miss 4 or 5 key insights.
– Some people believe that the product manager (and the interaction designer) are too close to the product to do this type of testing objectively, and they may either get their feelings hurt, or only hear what they want to hear. My view is that the good product managers and interaction designers get past this very quickly. They know they will get the product wrong initially, and that almost nobody gets it right the first time, and they know that learning from these tests is the fastest path to a successful product.
– Ideally, you should have one person administer the tests, and another person taking notes. It is very useful to have someone to debrief with afterwards to make sure you both saw the same things and came to the same conclusions. That said, if it is just you and your laptop, and you are got a ready and willing target user in front of you, do it. It is all good.
– If you as product manager have a user researcher or usability engineer along with you, let him or her administer the test and you take notes.
Otherwise, you will probably be the one that administers. It is great to invite others on the product team to be your note taker. Most often it will probably be the interaction designer, but the visual designer, developers and especially managers are all useful and they will get a lot out of it.
Testing Your Prototype
Now that you are got your prototype ready, you are lined up your test subjects, and you are prepared the tasks and questions, here are a set of tips and techniques for administering the actual testing:
– When you first sit down with the test subject, make sure to tell him or her that this is just a prototype, it is a very early product idea, and it is not real, they won’t be hurting your feelings, and you are testing the ideas in the prototype and most importantly you are not testing her. She can’t pass or fail ?only the prototype can pass or fail.
– Also, you should greet the person warmly and offer a coffee or something, but the sooner you get to the prototype the better. Tell them you will chat with them about this after they test the prototype, but you want to get their untainted impressions first. Realize that the more you chat beforehand, the more clues you are giving away and the less of a true first impression your test subject can provide. If more than 5 minutes goes by without the user starting in on the prototype you are talking too much.
– When testing, you will want to do everything you can to keep your users in ose mode?and out of critique mode.? What matters is whether users can easily do the tasks they need to do, and whether they value the product. It really does not matter if the user thinks something on the page is ugly or should be moved or changed. Sometimes misguided testers will ask users questions like chat three things on the page would you change? To me, unless that user happens to be an interaction designer, I will not really interested. If users knew what they really wanted, software would be a lot easier to create. So watch what they do more than what they say.
– During the testing, the main skill you have to learn is to keep quiet.
Normally when we see someone struggle most of us have the urge to help the person out. You need to suppress that urge. You have to turn into a horrible conversationalist. Get comfortable with silence.
– There are three important cases you are looking for: the user got through the task with no problem at all and no help; the user struggled and moaned a bit but he eventually got through it; he got so frustrated he gave up. Sometimes people will give up pretty quick, so you may need to encourage them to keep trying a bit longer, but if he gets to the point that you believe he would truly leave the site and go to a competitor, then that is when you note that he truly gave up.
– In general, you will want to avoid giving any help or leading the witness in any way. If you see the user scrolling the page up and down clearly looking for something, it is ok to ask the user what specifically he is looking for, as that is very valuable to you. Some people ask users to keep a running narration of what they are thinking, but I find this tends to put people in critique mode, as it is not a natural behavior.
– Act like a parrot. This helps in many ways. First, it helps avoid leading. If they are quiet and you really can’t stand it because you are uncomfortable, tell them what they are doing: i see that you are looking at the list on the right.?This will prompt them to tell you what they are trying to do, looking for, whatever. If they ask you a question, rather than giving a leading answer you can play the question back to them, will clicking on this make a new entry you are wondering?if clicking on this will make a new entry and they will usually take it from there because they will want to answer your question:
yeah, I think it will.? Parroting also helps avoid leading value judgments. If you have the urge to say great!? instead you can say, you created a new entry.?Finally, parroting key points also helps your note-taker because they have more time to write down important things.
– Fundamentally, you are trying to get an understanding of how your target users think about this problem, and to identify places in your prototype where the model the software presents is inconsistent or incompatible with how the user is thinking about the problem. That is what it means to be counter-intuitive. Fortunately, when you spot this it is usually not hard to fix, and can be a big win for your product.
– You will find that you can tell a great deal from body language and tone. It is painfully obvious when they don’t like your ideas, and it is also clear when they genuinely do. They will almost always ask for an e-mail when the product is out if they like what they see, and if they really like it, they will try to get it early from you. Recently I attended some prototype testing with one of my clients in Berlin, and even though I don’t speak German, it was obvious what the issues were, and which ideas worked well and which ones didn’t.
Updating the Prototype
The whole idea of this testing is to identify what you need to fix in the prototype to make it more useful and more usable. So as fast as possible, you will want to work to correct the problems:
– Some people believe you have to freeze the prototype, the tasks and the questions for a complete round of testing (generally 6-8 users) before drawing any conclusions. I don’t think that is true, and have found that you can significantly accelerate this process of getting to a good product faster by responding more quickly to the feedback.
– You don’t have to be hit on the head by 8 users in a row to know you need to fix something big. So go ahead and fix it when you believe you are identified a problem, even if that is after only 2 or 3 users.
The harder question is when you are done.Generally, if you can get through 6 consecutive users where they understand and appreciate the value, and they can get through the key tasks, you are in good shape and you are done your job.
– You might determine that you just aren’t able to get people interested in this problem, or figure out a way to make this usable enough that your target users can realize that value. In this case, you may decide to stop there and put the idea on the shelf. Some product managers consider this a big failure. I view it as saving the company the wasted cost of building and shipping this product, plus the opportunity cost of what your engineering team could be building instead.
This whole process might sound complicated or difficult, but the remarkable thing is just how easy and effective it actually is. The best way to prove this to yourself is to take your laptop with your product or prototype on it to someone that hasn’t seen it yet and just give it a try.