Pilot Usability Study
From CS 160 Fall 2008
Due: before class on November 17th, 2008
The goal of this assignment is to learn how to perform a simple usability test and to incorporate the results of the test into design changes in your prototype. In the real-world, this kind of "pilot" study would be used to redesign your experiment before running the study with a larger pool of participants. This assignment will be performed individually, with each group member contributing their own unique assessment with a unique user.
You will be performing this test using the latest version of your interactive prototype. As you are working on the pilot study you should also be working to add more functionality to your prototype based on the feedback you received in your interactive prototype presentation. We will expect you to describe all the functionality your have implemented and/or improved since submitting your interactive prototype.
Each group member will need to find one unique participant (i.e., volunteers who are not in this class and who aren't used by another member of your group) to work through your benchmark tasks. Remember, it must be voluntary. You should get the participants to sign an informed consent form and obtain other demographic information (e.g., age, sex, education level, major, experience with your type of tasks and application, etc.) Its best to use subjects who didn’t participate in your lo-fi tests, but they are acceptable if you can’t find others.
Your test will use three tasks. They should include 1 easy task, 1 medium task, and 1 difficult task. These tasks should give good coverage of your interface; if they don't then this is a good chance to redesign your tasks.
Measures and Observations
Although it will be hard to get statistically significant bottom-line data with only one participant and a rough prototype, you should measure some important dependent variables to get a feel for how it is done (i.e., task time, # of errors, etc.). You can also compare these with the measures recorded by your other group members.
You should concentrate on process data. For example, you should instruct your participant to think aloud. You should make a log of critical incidents (both positive and negative events). For example, the user might make a mistake and you notice it or they might see something they like and say "cool". Set up a clock that only the observers can see (one or more of you should observe), and write down a log containing the time and what happened at that time when a critical incident occurred.
If you happen to have access to a video camera, it is fine to use it (but make sure your subjects consent) -- note the time that you start taping so that you can find your critical incidents later on tape. You may wish to use a tape recorder if you don't have a video camera.
You will give the participant a short demo of the system. Do not show them exactly how to perform your tasks. Just show how the system works in general and give an example of something specific that is different enough from your benchmark tasks. You should write-up a script of your demo and follow the same script with each participant.
The participant will then be given task directions for the first task that tells them what they are trying to achieve, not how to do it. When they are finished, you will give them the directions for the next task and so on. Each participant will perform all 3 tasks. You will want to keep the data separate for each task and participant.
You must report your results (values of dependent variables, summary statistics, and summaries of the process data) and in the "Discussion" section you should draw some conclusions with respect to your interface prototype. You should also say how your system should change if those results hold with a larger user population. This should be the most important part of the write-up. We want to understand how you would fix your system as a result of what you observed.
This is an individual assignment and each group member will need to submit their own unique writeup. This should be turned in on paper (in class on the due date) and on the Wiki, should follow this outline with separate sections for the top-level items (number of pages per section are approximate). It should be about 5 pages, plus appendices and sketches that describe what you did.
Introduction (5 points)
- Introduce the system being evaluated (1 paragraph)
- State the purpose and rationale of the experiment (1 paragraph)
Implementation and Improvements(15 points)
- Describe all the functionality you have implemented and/or improved since submitting your interactive prototype (1 page).
Method (10 points)
- Participant (who -- demographics -- and how were they selected) (1 paragraph)
- Apparatus (describe the equipment you used and where) (1 paragraph)
- Tasks (1/2 page) [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
- Procedure (1 page) describe what you did and how
Test Measures (5 points)
- Describe what you measured and why (1/2 page)
Results (10 points)
- Results of the tests (1 page)
Discussion (15 points)
- What you learned from the pilot run (1 page) what you might change for the "real" experiment? what you might change in your interface from these results alone? If you'd like, you may include results and assumptions from other group members' tests here as well.
Appendices (5 points)
- Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
- Raw data (i.e., entire merged critical incident logs)
To upload images to the wiki, first create a link for the image of the form [[Image:image_name.jpg]] (replacing image_name.jpg with a unique image name for use by the server). This will create a link you can follow that will then allow you to upload the image. Alternatively, you can use the "Upload file" link in the toolbox to upload the image first, and then subsequently create a link to it on your wiki page.
Hand in Printout in Class
Print your assignment and hand it in at the begining of class on April 15th.
Add Link to Your Group's Page
Edit your group's page to add a link to a new wiki page for this assignment. The wiki syntax should look like this:
[[PilotStudy-Group:ExampleGroup-FirstNameLastName|Pilot Study (FirstName LastName)]]
Again replace ExampleGroup with your group's name and FirstNameLastName with your name. Look at Group:ExampleGroup for an example. Then click on the link and enter the information about your assignment. Be sure to clearly address everything mentioned in the writing guidelines above. Writeups for all group members should be posted here.
Add Link to Your Finished Assignment
One you are finished editing the page, add a link to it here with your group name as the title link. The wiki syntax will look like this: *[[PilotStudy-Group:ExampleGroup|Group:ExampleGroup]]. Hit the edit button for this section to see how I created the link for the ExampleGroup.