PilotStudy-Group:Group Ate-JacekWnuk
From CS 160 Fall 2008
Jacek Wnuk's Pilot Usability Study writeup.
Contents |
Introduction
(5 pts) Introduce the system being evaluated (1 paragraph); State the purpose and rationale of the experiment (1 paragraph)
The system being evaluated is the next iteration of the interactive prototype for Formal Logic, a "classy dining simulator" focused on teaching the player etiquette while still being fun. The game focuses on several elements of dining etiquette, namely: the selecting of the correct utensils for the meal at hand, the eating of the meal itself, and all the minor challenges that a real dinner-goer may face, namely passing items around the table, fixing one's slouching posture, and making sure one does not eat too quickly.
Implementation and Improvements
(15 pts) Describe all the functionality you have implemented and/or improved since submitting your interactive prototype (1 page).
The interactive prototype was originally intended to be done in Flash, but due to the fact that nobody in the group itself knew how to use flash, we decided that rather than face the steep learning curve of Flash, we would go with developing the project in C# using .NET. This did, however, give us steep limitations as to what we could do with animations and made it rather difficult to work with the graphical elements.
The main concerns in the Heuristic Evaluation were that (a) the game wasn't all that educational, and (b) that the game's controls were somewhat unintuitive and hard to understand immediately. The first concern, though it is a valid one, we discussed and decided it didn't merit that much further investigation considering that this was what we already had and that anything radically different would require a complete 180 degree change and require us to start from square one.
Utensil Cursor
As for the unintuitiveness of controls, we have indeed done a lot to address this. Initially the game's fork-knife utensil cursor (displayed when hovering over food) has lost some of its confusing nature. In the Lo-Fi prototype, the cursor had been merely an idea, and was just a crossed fork-knife combination on a post-it note. This evolved into the below icon for the interactive prototype. Finally we discovered that the order of the fork and knife was the opposite from the head-on view AS WELL as the actual controls (as in, the fork was on the left in the cursor icon, but it was on the right side in the actual gameplay, and was controlled by a right click). This was initially remedied by flipping the icon. Eventually we flipped it back and settled on a left-click-controls-object-in-left-hand while right-click-controls-object-in-right-hand setup, which works well also for continuity for the utensil-selection screen.
Utensil-Selection
This actually adresses (a). It was conceived of as an idea right from the top of the idea-planning process. This is the utensil selection portion we had always intended to implement in the game. Clearly it adds to the etiquette-learning value by teaching one of the fundamentally most confusing elements of etiquette. The interface uses a buzzer if the user chooses the incorrect utensil, and if there's a left-hand utensil as well, it goes onto that after having selected the right-hand one already.
Salads & Pies
Finally, the game has the option of other meals besides steak/potatoes. This also means different control schemes; the salad does not need cutting at all, and the pie does but the cutting is done with the fork (so clicking with the fork first cuts it, then clicking again sticks it on the fork)
Tutorial
Last but not least, a Tutorial section was implemented to help with the still potentially confusing controls. The tutorial goes through everything step by step, slowly, so the user can understand what he/she needs to do when it comes time for the actual game. It is selectable from the game's main menu and presents the user with an example main game screen in which the tasks are laid out one at a time.
Passing Occurance
The "pass the salt/pepper/bread" events used to happen more often and some students commented on it being unrealistic, so this was changed to occur less frequently.
Method
Participant
(10 pts) Participant (who -- demographics -- and how were they selected) (1 paragraph)
My participant is a UC Berkeley undergraduate student majoring in Economics. He worked last summer at Washington Mutual. He was mostly chosen for the closeness to the target audience. He loves food, particularly Mexican. He travels to Vietnam on occasion and admits to being somewhat confused about food etiquette at times. He is pretty familiar with computers and isn't really a big gamer, but plays Warcraft III on occasion. It was his first experience with the game.
Apparatus
(describe the equipment you used and where) (1 paragraph)
 Sony Vaio CR |
 Logitech USB Laser Mouse |
 Windows Vista Home Premium |
My laptop (Sony Vaio running Windows Vista) with a Logitech mouse on my relatively empty desk in my apartment were the main equipment used in this experiment.
There were no other distractions in the room; nobody else was in the room at the time and any and all music was turned off prior to testing.
Tasks
(1/2 page) [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
These are the same tasks we used for our Interactive Prototype, with some cutting down for more readability and with the first one changed from passing salt/pepper/bread to getting through the tutorial, which I felt was a more separate and reasonable first step, and something that hadn't been evaluated properly as of yet. For more details see our Interactive Prototype writeup.
Easy: Getting through the tutorial
The tutorial is just like what it sounds. It is selectable from the game's main menu and presents the user with an example main game screen. It slowly explains the game's elements and makes sure the user understands what he/she must do by having them actually perform the actions, but in a much slower and less stressful environment.
Medium: Successfully complete steak & potatoes meal without distractions
Task summary: The player is presented with a plate of food and must eat it using the affordances provided by our input mechanism (the mouse.) Unlike the Hard version of this task, there are no additional distractions; the goal is simply to wolf down your food. The user, in this version of the interactive prototype, may eat a salad, steak & potatoes, or pie, though I chose the steak & potatoes meal because it is the most difficult of the 3. The user must:
- Use the fork to stab the steak (thus holding it in place). (left click)
- Use the knife to cut a piece off the steak. (right click)
- Use the fork to stab the piece of steak (left click)
- Click on the avatar's mouth. (left click)
- Repeat until steak has been entirely consumed.
Hard: Successfully complete meal while dealing with environmental challenges
The hard task has the same objective as the medium task: the user must consume all the food on their plate. However, the hard task also introduces various obstacles representing what a diner might encounter. The obstacles we are including in this prototype are:
- Posture, which is fixed with a click
- Eating too quickly, which is fixed by waiting until your instructor says it's ok
- Passing bread & salt/pepper (see Easy step)
The overarching superobjective of instructor happiness is reflected by your instructor's feedback area in the upper-right corner. Your performance on other tasks is reflected in this supertask - a poor job in eating food for whatever reason will result in a negative reaction from your instructor.
Procedure
(1 page) describe what you did and how
- First, I greeted the user and thanked him for participating, explaining somewhat the importance of the process.
- Then I had the user agree and sign the consent form.
- I asked the user if I could videotape him during the process\
- I had the user sit down at my laptop with just the shortcut to the program on the desktop
- I had the user complete the tutorial
- I noted any issues the user voiced
- I noted problematic clicks and confusing sections
- I had the user complete the steak & potato meal ignoring any distractions
- I noted any issues the user voiced
- I noted problematic clicks and confusing sections
- I had the user complete the steak & potato meal with all distractions, trying to get a high score
- I noted any issues the user voiced
- I noted problematic clicks and confusing sections
- I allowed the user to try running through the pie and salad meals
- I noted any issues the user voiced
- I noted problematic clicks and confusing sections
- I thanked the user again for his time and sent him on his way.
Test Measures
(5 pts) Describe what you measured and why (1/2 page)
Critical Incidents / Process Data
For the most part, I was on the lookout for any critical incidents, as these are the best ways to spot a problem with an interface. Most other variables are dependent from this.
Comments on Interface
I recorded the user's comments on the interface, noting in particular if he said something might be confusing for instance, even if he himself had not had a significant issue with the interface element.
Incorrect Clicks
A dependent variable of critical incidents, I noted concentrations of incorrect clicks in specific areas. These might indicate a lack of understanding still of the interface.
Time
The tutorial is self-paced and so does not need a timer, though the sections in which the user had to click had my attention as to roughly how long it took the user to click. The game itelf is timed in the corner, allowing me to guage roughly how long different sections were taking.
Results
(10 pts) Results of the tests (1 page)
Easy: Getting through the tutorial
The user was somewhat frustrated with the tutorial, and repeatedly clicked around to try to make the tutorial go faster. The total of illegal clicks was 7. The user briefly didn't know what to do at the parts where he actually had to click on the meat, as he had gotten used to not being able to do anything as the instructor kept talking (he said this aloud). It took him 1 extra click to understand cutting with right click, since, in his words, it is "kind of unusual" as a control scheme. He also missed the first time he tried to pick up the meat chunk. Other than that, the tutorial was completed. The user was not very excited about the tutorial the whole way through.
Medium: Successfully complete steak & potatoes meal without distractions
After the tutorial, the user found this task to be relatively easy (the user specifically mentioned the tutorial). The utensil-selection went by without a hitch, though he briefly hesitated over which knife to use. He only had 2 extraneous clicks, one out curiosity to see what would happen and another when he accidentally missed the meat chunk. The user found the task very amusing, since he purposefully got to ignore the instructor's advice regarding passing salt/pepper/bread, and ate as fast as he liked. Interestingly, toward the end when he had gotten in the routine of eating, he actually did correct his posture almost reflexively in the game and did pass one of the bread arrows. He was very amused when the game called him a Ruffian at the end for doing so poorly etiquette-wise.
For review, eating was broken up into:
- Use the fork to stab the steak (thus holding it in place). (left click)
- Use the knife to cut a piece off the steak. (right click)
- Use the fork to stab the piece of steak (left click)
- Click on the avatar's mouth. (left click)
- Repeat until steak has been entirely consumed.
All of these were satisfactorily completed without any major incident.
Hard: Successfully complete meal while dealing with environmental challenges
The hard task has the same objective as the medium task: the user must consume all the food on their plate. The user did not have much trouble adapting to the added challenges in the game, perhaps surprisingly. He said that he already "kind of knew what to expect" from playing it once on medium without caring for the other objectives, which does make sense as he did experimentally click the pass bread once and did straighten the posture. Thus he had no real problem with the following of the 3 extra tasks:
- Posture, which is fixed with a click
- Passing bread & salt/pepper (see Easy step)
but he did have an issue with:
- Eating too quickly, which is fixed by waiting until your instructor says it's ok
His problem with this one was that he became very confused as to when it was okay to eat again after having been criticized for eating too quickly. He did not notice that the food remaining bar turned back to blue after being red when he ate too quickly and had just assumed that it had turned red because there wasn't much food left. When other items popped up after he ate too fast, such as a posture fix, he was no longer sure if he had to wait, because the instructor was smiling again and was no longer talking about the food speed. When the instructor finally did say it, the user realized it was okay, but mused aloud that it was very confusing, especially after another warning popup quickly erased the 'okay to eat again' message.
The overarching superobjective of instructor happiness ended with a grade of Yuppie for the user, to which he gave a slightly confused "yay," since he knew it was higher, clearly, than ruffian, but he wasn't sure if it was entirely how good or bad it was either.
Extra: Pie and Salad
The user didn't have a major problem with the salad or pie eating part, but was confused for the utensil selection for the pie, noting that he believed it to be incorrect (turns out he was correct in saying that, but his solution was also incorrect).
Discussion
(15 pts) What you learned from the pilot run (1 page) what you might change for the "real" experiment? what you might change in your interface from these results alone? If you'd like, you may include results and assumptions from other group members' tests here as well.
Easy: Getting through the tutorial
The fact that the user kept clicking to try to advance the tutorial shows us that we need to implement some sort of balancer for the speed of the tutorial. Either we make the tutorial itself shorter (not recommended so we don't alienate possible players who may not be as proficient with computers), the "Flexibility and efficiency of use" heuristic dictates that we should have a button that can skip to the next blurb in the tutorial to provide shortcuts for more advanced users to get to the meaty how-to-play information.
Medium: Successfully complete steak & potatoes meal without distractions
According to the diagram above, our layout for choosing actually does not need the second knife, so the 2-knife mixup probably won't happen. We do, however, need another fork (J), which may contribute to confusion for that section.
The only really major eating-centric issue after that is that the meat chunks are a little hard to click on after they are cut. This issue can be solved easily by just making that whole general area (the entire plate perhaps, or just the meat) clickable for this situation. This would not be very difficult.
One item I want implemented but that we have not yet implemented was a more complicated end dialogue, counting down how many times you failed or succeeded at certain tasks. While the user was amused by the Ruffian name, it might be useful for Error Prevention's sake to show exactly what the user needs to work on for the next run through the game.
For review, eating was broken up into:
- Use the fork to stab the steak (thus holding it in place). (left click)
- Use the knife to cut a piece off the steak. (right click)
- Use the fork to stab the piece of steak (left click)
- Click on the avatar's mouth. (left click)
- Repeat until steak has been entirely consumed.
All of these were satisfactorily completed without any major incident.
Hard: Successfully complete meal while dealing with environmental challenges
The hard task has the same objective as the medium task: the user must consume all the food on their plate. The user did not have any really major issues with the Hard task, meaning the tutorial must be fine at explaining the following:
- Posture, which is fixed with a click
- Passing bread & salt/pepper (see Easy step)
but the tutorial does not do a good enough job explaining:
- Eating too quickly, which is fixed by waiting until your instructor says it's ok
His problem with this one was that he became very confused as to when it was okay to eat again after having been criticized for eating too quickly. He did not notice that the food remaining bar turned back to blue after being red when he ate too quickly and had just assumed that it had turned red because there wasn't much food left. For this reason it might be a good idea to reconsider the color-coded food bar, mockup below. This was not implemented mainly due to difficulty in coding it, but it may be a worthwile investment.
On the other hand, another major reason why it's unclear is that often one instructor warning will be replaced by another. Sometimes someone will ask for salt as you begin eating too quickly, etc., and then the 'eating too quickly' message is blocked somewhat, or, even worse, the 'begin eating again' message is blocked. Thus several messages should be up there at once, possibly with a possibility to scroll.
That the end result was Yuppie is not entirely acceptable either - for Error Prevention's sake, there should be a list giving feedback as to what happened, much like our original idea.
Extra: Pie and Salad
The user though that there should be a dessert fork up top near where our teaspoon is in the layout, but apparently it turns out that the above diagram's [J] is where the dessert fork is meant to be located. For the meantime, the user did end up using the salad fork for the pie (as we have marked correct). This is clearly an issue and will be fixed before the next prototype.
Appendices
(5 pts) Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions); Raw data (i.e., entire merged critical incident logs)
Consent Form
Instructions
The instructions were part of the tutorial. No other instructions are necessary.
Script
I did not script this one - I felt that more could be learned from a more improvised approach, looking into what appears to be an issue for that particular person and making it very personal. Having a script would create a feeling of stiffness that would be counter-productive, I believe, to making the user comfortable enough the criticize work.
Raw Data
Here are the Extraneous Clicks and Notable Quotes sections of my notes. For the Critical Incidents please refer to the Results section.
Easy: Getting through the tutorial
Extraneous clicks: 7
Notable Quotes:
- "Uh... how long is this gonna go for"
- "can I skip this?"
- "I guess it's because, the controls are, uhh, kind of unusual I guess"
Medium: Successfully complete steak & potatoes meal without distractions
Extraneous clicks: 2
Notable Quotes:
- "Umm, I don't remember this in the tutorial... which knife is it?"
- "So I just get to ignore him? Nice."
- "Oh hey, I just made him sit up straight"
- "Heh, apparently I'm a Ruffian"
Hard: Successfully complete meal while dealing with environmental challenges
Extraneous clicks: 1
Notable Quotes:
- "Uh, wait, can I eat now then?"
- "Yay, I'm not Ruffian... is Yuppie good then?"
Extra: Pie and Salad
Extraneous clicks: N/A
Notable Quotes:
- "Shouldn't there be a little fork up there next to the little spoon?"
