From CS 160 Fall 2008

Jonathan Fong's Pilot Usability Study assignment.

Contents 1 Introduction 2 Implementation and Improvements 2.1 Help System 2.2 Implementation of interface to choose utensils 2.3 Scoring 3 Method 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.3.1 Easy: Complete the tutorial 3.3.2 Medium: Successfully complete an easy meal (pie and/or salad) 3.3.3 Hard: Successfully complete a hard meal (steak and potatoes) 3.4 Procedure 4 Test Measures 4.1 Completion Time 4.2 Number of incorrect clicks 4.3 Critical Incidents / Process Data 5 Results 5.1 Easy Task 5.2 Medium Task 5.3 Hard Task 5.4 Summary 5.5 Process data 6 Discussion 7 Appendices 7.1 Equipment and Materials 7.2 Raw data 7.2.1 Easy Task: Tutorial 7.2.2 Medium Task: Salad and Pie 7.2.3 Hard Task: Steak

Introduction

This Pilot Usability Study experiment is our attempt to quantitatively study the effectiveness of certain features in our game, and their effect on the user experience. The rationale is that

1. the quantitative data collected can be compared to other usability studies
2. the critical incidents will give qualitative feedback on the user experince

The timing the user's interactions will correspond to speed of understanding; negative incidents suggest badly implemented features (and vice versa).

Implementation and Improvements

Help System

• Tutorial: The user has the option of doing the "tutorial" which guides him/her through the gameplay interface. It is assumed all users will complete this before playing the game "for real."

Key aspects of this tutorial are:

A) pointing out the interface elements:

B) Improved "coach" prompts:

• The prompts are:
  • Let's get started learning how the game works.
  • Your goal is to complete your meal while maintaining proper manners.
  • To do that, you use your mouse to interact with this plate of food.
  • You can see how much food is left on your plate with this bar.
  • To use the utensil in your left hand, left click on the plate. Try it now!
  • Then, to use the utensil in your right hand, right click on the plate. Give it a try.
  • Good. Remember that this is true no matter what utensil you are holding.
  • As you can see, you've cut the steak on your plate and a piece has come off
  • Use your fork on this piece to grab it so you can eat it! (Left click)
  • You've grabbed the food. Now you may eat it by left clicking on your mouth.
  • Great. This is how eating usually goes. Sometimes you use different motions for eating
  • For example, you don't need to cut a salad. You just poke it with a fork and eat it
  • Or when eating pie, you cut it with your fork, and then you can poke the cut piece and eat it.
  • Sometimes you'll have to pass some items across the table to other people.
  • When that happens, just left click on the arrow that appears to complete that action. Try it now.
  • Remember that if you take too long to do this, people will get fed up and so will I.
  • There's one more "distraction" you should know about. When you're sitting somewhere for a long time...
  • Sometimes you slouch. Sit up straight, or I could notice and become upset!
  • To fix a slouch, just left click anywhere on yourself.
  • Good. That's pretty much it.
  • One last thing. To pause the game, just click this button. Try it out.
  • In general, just follow my instructions during a meal. Watch out if I get mad - like this face.
  • You've now completed the tutorial. Go and play the game now!

Implementation of interface to choose utensils

Based on feedback from our interactive prototype, we implemented this feature to teach users about which utensils should be used with different kind of dishes.

• Interface with options of which utensil:

• Interface with a utensil selected:

• Proper utensils:
  • Salad: Salad fork (outer fork on the left) goes in the right hand
  • Steak and potatoes: Dinner fork (inner fork on the left) goes in the left hand, dinner knife (inner knife on the right) goes in the right hand
  • Pie: Dessert fork (outer fork on the left) goes in the right hand

Scoring

• Time feedback: tracks and tells the user how long he/she took to complete the meal
• Rank: based on performance during the meal, users are ranked to give qualitative feedback

Method

Participant

• Who: Recent UC Berkeley graduate
• Demographics
  • Age: mid-20's
  • Education level: University-educated (Bachelor's)
  • Major: Cognitive Science (with minor in Education Studies)
  • Experience (general): technology-saavy; enjoys computer games, but not an avid player
  • Experience with system being tested: First experience with the game
• Selection Criteria
  • a new user, who had no prior experience with the game
  • Age: 18-29, our intended target audience
  • minimal technology-literacy: using a computer and mouse

Apparatus

• Timing device: Casio wristwatch (See Appendix A-2)
• Computer: Lenovo, X61t (Tablet) (See Appendix A-3)
• Mouse: Logitech MX510 (See Appendix A-4)
• Where
  • Location: Apartment, dining room table
  • Time: Evening
  • Neighborhood: Northside (i.e. quiet/peaceful)
  • Environment: No distractions (e.g. TV, radio, other people)
  • Lighting: Overhead ceiling fan with halogen lights

Tasks

(modified from the write-up for the Interactive Prototype assignment)

Easy: Complete the tutorial

Task summary: Find and complete the newly implemented tutorial, which demonstrates the interface to the user.

Medium: Successfully complete an easy meal (pie and/or salad)

Task summary: The player is presented with a plate of food and must eat it. The "pie" and "salad" meals do not require "complicated" clicking. All the passing disturbances and posture are present as normal.

For the pie:

1. Use the fork to cut a piece of pie (left click)
2. Use the fork to stab the newly-cut piece of pie (left click)
3. Click on the avatar's mouth. (left click)
4. Repeat until pie has been entirely consumed.

For the salad:

1. Use the fork to stab a piece of salad (left click)
2. Click on the avatar's mouth. (left click)
3. Repeat until salad has been entirely consumed.

Hard: Successfully complete a hard meal (steak and potatoes)

Task summary: The hard task has the same objective as the medium task: the user must consume all the food on their plate. However, the interface for the steak and potatoes requires more clicking (to cut, then stab).

1. Use the fork to stab the steak (thus holding it in place). (left click)
2. Use the knife to cut a piece off the steak. (right click)
3. Use the fork to stab the piece of steak (left click)
4. Click on the avatar's mouth. (left click)
5. Repeat until steak has been entirely consumed.

Procedure

1. Greet and thank the user for participating
2. Have user sign the consent form
3. Have user complete tutorial.
4. Measure dependent variables and critical incidents:
   1. completion time
   2. number of incorrect clicks
5. Have user complete the pie or salad meal
6. Measure dependent variables and critical incidents:
   1. completion time
   2. number of incorrect clicks
7. Have user complete the "steak and potatoes" meal
8. Measure dependent variables and critical incidents:
   1. completion time
   2. number of incorrect clicks
9. Thank user again for participating

Test Measures

Completion Time

Total completion time was measured to establish a data-point for the completion time of a technologically-saavy user. This can be then compared to the completion time of users from different demographics to find correlation. Thus, I am assuming this is a dependent variable.

Number of incorrect clicks

When the user clicks incorrectly, this is not as serious as a critical incident, but is still an indication of a misunderstanding of the interface. Like the completion time, this is a dependent variable.

Critical Incidents / Process Data

As we've been taught, these tell us where there are problems in the interface or where something is implemented particularly well. Because this usability study is limited to one participant, I focused on collecting and analyzing this data rather than give significant weight to the two dependent variables listed above.

Results

Easy Task

• Completion time: 3 minutes
• Number of bad clicks: 7

Medium Task

• Salad
  • Completion time: 1 minute, 30 seconds
  • Number of bad clicks: 1
• Pie
  • Completion time: 58 seconds
  • Number of bad clicks: 0

Hard Task

• Pie
  • Completion time: 1 minute, 50 seconds
  • Number of bad clicks: 4

Summary

• Total completion time: 6 minutes, 18 seconds
• Total bad clicks: 12

Process data

• The user appeared bored and impatient many times
  • Throughout the tutorial, waiting for the new prompts from the coach
  • At every meal, after following the eating procedure well, the user ended up eating too fast
• The ranking at the end of the meal made the user happy: being rated a "connoisseur" was a compliment and the user was proud of it
• The user never made the same mistake (e.g. bad click for the same reason) twice (other than eating too fast, which will not be considered a mistake)
• Discussion of context of the bad click incidents will be discussed in the Discussion section of this write-up.

Discussion

Compared to our low-fidelity testing, the user was much clearer about the interface and what to do because the tutorial gave a simple walkthrough. This tells me that the tutorial is very useful and that any time/effort spent in making this aspect of the game clear is worth the investment. The user thought the tutorial was slow, though, because it was restricted to a set pace. He tried to click to go to the next step, which is common so that advanced users can move faster. This is not in our tutorial, but Nielson's "Flexibility and Efficiency of Use" (H2-7) heuristic dictates that we should implement it.

The user had trouble stabbing the piece of steak in the tutorial. While in later tasks, the user did not have trouble, the severity of the initial incident (6 bad clicks) indicates the user deeply misunderstood. The cursor icon when the user clicks but misses the piece of steak is the "fork and knife" cursor (Figure 4-5). This gives the affordance that the user is in a mode where the fork and/or knife can be used. Unfortunately, this is inaccurate for stabbing cut pieces of food. As implemented in the version tested, the cursor will change to a regular cursor (Figure 4-7) to indicate the piece can be stabbed. Once the user realized this, he didn't say anything, and adapted, but this should still be changed in future versions. It is probably a violation of Nielson's "Consistency and Standards" (H2-5) heuristic.

The piece of pie that was cut from the main piece was hard for the user to find. All cut pieces should be moved to a clear part of the plate so that the white color of the plate will make the piece easier to spot.

Hard to see the cut pie piece

Extra outlining/color added to indicate the piece (for purposes of this write-up/discussion only, not a proposed solution)

The user frequently ate too fast. Because there is nothing else to do, the user keeps on eating the food, then gets confused and slightly upset when the coach tells him he is eating too fast. The idea of pacing should be further developed. Perhaps when the Food Remaining Bar is implemented, the user will be able to get visual feedback on his pacing, and thus keep pace better. Future experiments for this feature would be set up so that the number of times the user eats too fast is the dependent variable, and the presence of the Color-Codes Food Remaining Bar is the independent variable. The null hypothesis would be that the Color-Codes Food Remaining Bar has no affect on the number of times the user eats too fast.

The user was proud of his achievement of the "Connoisseur" rating. This was glad to see. In designing motivations to play the game, we hoped that getting a high score would be incentive. The user's reaction indicates that it is. However, the user appeared to get bored, so we might not be increasing the Zone of Proximal Development (ZPD) fast enough. Further experiments should test different levels of difficulty to see if we can increase the difficulty faster without affecting the playability.

Appendices

Equipment and Materials

A-1: Consent Form

A-2: Timing Device

A-3: Laptop

A-4: Mouse

• Appendix A-1: Consent Form
• Appendix A-2: Timing device: Casio wrist-watch, Model 2688, GW-330A
• Appendix A-3: Computer: Lenovo X61t, SXGA+ model
  • Form-factor: laptop mode
  • Screen brightness: full
  • Screen Resolution: SXGA+ (1280x760)
• Appendix A-4: Logitech MX-610, cordless laser mouse
• Appendix A-5: Task instructions

Raw data

Easy Task: Tutorial

• 0:10 1 bad click (user tried to click to get next message from coach, sees it doesn't work)
• 0:20 "Yea yea.... going a bit slow..." (user bored/impatient waiting for new message from the coach)
• 0:40 "lol, good sound effect" (user amused)
• 1:05-1:15 6 bad clicks (user did not know how to stab the piece of steak: cursor was not on it)
• 1:40-1:50 tried to click the salad and pie to practice (because of what the coach said)
• 2:20 (coach said "sit up", but before coach said how) "how do I sit up???"
• 3:00 "okay.. that was easy"

Medium Task: Salad and Pie

• Salad
  • 0:06 Tried spoon above plate first. Reaction when wrong: "But that's supposed to be the utensil for dessert!"
  • 0:32 (after a piece of pie was cut off) "where did my piece go?" (because it was placed on top the whole piece, and hard to see)
  • 0:47 1 bad click trying to click the cut off piece
  • 0:54 (coach just told user he is eating too fast) "oh... okay..." (user a bit bored)
  • 1:30 (user just completed pie) "yay. Connoisseur! cool" (user proud of achievement)

• Pie
  • 0:22 (coach just told user he is eating too fast) "oh yea..." (user remembers, and waits a bit impatient)
  • 0:30 (two arrows come up to pass salt and pepper) "lol, how can I be passing the same thing both ways?!" (user a bit amused, but considers this illogical)
  • 0:58 (user just completed salad) "yay. Connoisseur again! " (user proud of achievement)

Hard Task: Steak

• 0:02 (in the utensil selection screen) "why is my steak raw?" (user being critical, a bit confused)
• 0:17 1 bad click (picks the outer fork) "I thought the rule was 'work outside in'?!", then clicks dinner fork.
• 0:41 1 bad click (clicked left to cut)
• 0:52 1 bad click (clicked the mouth to eat: a piece was cut, but not yet on the fork).
• 1:10 (coach just told user he is eating too fast) "again?" (user is impatient)
• 1:35 1 bad click (user cut the last piece of steak, but wanted to try to eat the potato first)
• 1:44 (user just ate the last piece of steak) "oh, I thought the meal was steak *and potatoes*??).
• 1:50 "connoisseur again" (user not as proud as before, as if the victory was too easy)