PilotStudy-Group:Group Ate-JoshuaKwan

From CS 160 Fall 2008

Joshua Kwan's Pilot Usability Study writeup.

Contents 1 Introduction 2 Implementation and Improvements 3 Method 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.4 Procedure 4 Test Measures 5 Results 6 Discussion 6.1 Changes we will make 6.2 Changes requiring discussion 7 Appendices

Introduction

The system being evaluated is, in its own words, a "classy dining simulator" intended to teach people about what considerations go into merely participating in a formal dining situation. Specifically, within the game, the player learns to pick what utensils go with which meal, and then eat the meal using these utensils. During the meal, the player has to deal with several obstacles that risk to embarass him; specifically, the obligation to respond to requests to pass things around the table such as bread or condiments; the obligation to keep a good posture and not slouch; and the obligation to not eat excessively quickly.

The experiment is to find out whether the game, as it stands, is at least understandable and not confusing. Moreover, it should be able to impart some modicum of education or, at the very least, pure entertainment. As described below, we no longer expect this game to be of strong educational value with teachings that can be immediately put to use in real formal dining situations. It is better suited as something that raises awareness about formal dining knowledge.

Implementation and Improvements

The interactive prototype was the culmination of a series of decisions that left us a little short of our initial lofty goal for the project. It featured a UI written in .NET using C#, while we had originally intended to write it in Flash. Due to lack of time devoted to development and a learning curve (none of us knew Flash), we decided after the Interactive Prototype stage to continue with the C# version. As such, we also had to give up many interactivity goals such as fluid animation and certain image effects that would require so-called custom controls in .NET, a complex undertaking.

There were also concerns across the board, with feedback from ourselves, the professor, and our heuristic evaluation group, that in the end, the featureset that this game provided didn't really teach people about how to eat in a fancy restaurant. It was simply a kneejerk-reaction type game that had dining as a setting. To change this would require a fundamental rethinking of our game mechanics and the loss of many hours of work and experimental data.

Thus, we worked with what we had. Things like this often happen in real software development, so we recognized this as a necessary challenge. As a result, we tried to incorporate as many of the original ideas as were feasible into the .NET version. Here is the list of changes from the first version:

Major changes

• A level selection screen has been added, permitting you to select what part of a meal you want to eat in the environment. Each has its particular method of eating:
  • You eat pie with only a fork, and you use it to cut the pie as well as stab it and eat it.
  • You eat steak with a fork and knife; you must stab the steak, then cut it, then grab the cut piece with your fork and eat it.
  • You eat salad with only a fork, and it doesn't need to be cut so you can just stab it, and eat it.
• A utensil selection screen has been added to test the player on what utensils to choose for a particular meal.
  • Pie and salad require just the small fork in the right hand.
  • Steak requires the big fork in the left hand and the big knife in the right hand.
• A tutorial screen has been added to teach the player about how to play the game. There isn't, however, a tutorial screen for the utensils selection part.

Minor changes

• The occurrence of random "pass the bread/salt/pepper" events has been fixed. It used to be a little excessive, and the algorithm needed changing.
• Many people were confused because the third-person view of the game would cause confusion over what button to press to use the utensil in the player's right hand, and what button to press for the left-hand utensil. We decided to settle on an easy mnemonic:
  • left mouse click = the avatar's left hand.
  • right mouse click = the avatar's right hand. This is a little less confusing than it would have been before because the user is forced to put specific utensils into specific hands prior to the game starting.

Method

Participant

My test participant is Nick, who is a UC Berkeley alumnus in Linguistics and is currently working for Truveo in San Francisco. Though he works for a technology company, he's not really a huge gamer or power user per se (though he has dabbled in World of Warcraft before.) He is right handed. I didn't have any particular reason to select him; I guess he is an average person and is a potential part of the game's audience. He likes food!

Apparatus

I just put my laptop down in the living room. A real mouse (not a trackpad) was available.

Tasks

The only task I chose to change is utensil selection, which is the new major task that is available. It replaces the medium task. The task list is reproduced here in short form; to see the full explanations for the two unchanged ones, check our Interactive Prototype page.

• Easy: Passing Salt or Bread around the table
  • Task summary: Every 30 seconds or so (this time depends on the stage the player is at), the player is presented with an arrow containing either a Salt & Pepper icon or a Bread icon, and must pass it using our input mechanism (the mouse).
• Medium: Selecting utensils for the Steak Meal
  • Task summary: The player is presented with a set of utensils that can be eaten with the steak meal. With the mouse, the player must click the appropriate utensil for each hand (assuming a right-handed avatar.)
• Hard: Successfully complete meal while dealing with environmental challenges
  • Task summary: The player is presented with a plate of food and must eat it using the affordances provided by our input mechanism (the mouse.) During the meal, there are three types of things that must be dealt with: posture, eating speed, and passing bread/condiments around.

Procedure

(1 page) describe what you did and how

I'm not sure why this requires a whole page. I sat Nick down and asked him to try playing the tutorial, then playing the steak/potatoes meal, then trying the salad meal, and noting down everything and gathering feedback during the test and afterward. I didn't give Nick any instructions, I just told him the basic premise of the game and told him to get started.

Test Measures

I counted the number of times Nick got really confused. If I felt that his understanding could improve drastically by explaining a very small thing, I would do that and repeat the task and see how much it helped. That way, it would be clear what kind of documentation/tutorial improvement could be done in code to make that happen automatically the next time someone tried the game. If a major disconnect occurred, I would just write down exactly what happened.

Results

Nick started the tutorial and gave me a running commentary. One of the big things he was upset with is that he was expecting the tutorial to be clicked through, not on a timer. Kevin's test participant also noticed this, so this will be a high-priority change for the next iteration of the prototype.

There were no other major incidents during the tutorial phase. A minor incident is that Nick tried to poke the example salad during the explanation that different meals require different motions. What we wanted to do was not hold the player's hand through figuring out how to eat the salad, and let him/her figure it out. We actually didn't storyboard that part of the tutorial, I just thought it would be useful when I was coding it up.

I then got Nick to play the steak/potatoes stage of the game. I did this because I hoped he would remember being taught how to eat the steak with the mouse in the tutorial that he had just been through. The main part of the game was the site of several incidents during the experiment:

• He forgot the "stab steak, then cut it, then poke the piece and eat it" mechanic and immediately tried to cut the steak with the knife. The "incorrect" feedback noise immediately alerted him that something was wrong, so he eventually remembered how it was done. In the tutorial, we should probably make it clear that this is something that needs to be remembered.
• He didn't notice when the instructor told him to slow down. He noticed the progress bar turning red but didn't know what to do about it.
• He didn't notice when the instructor said he was slouching. He actually fixed the slouch by accident by trying to eat his last piece of food. This is a regression from the interactive prototype, because it used to be that you would left-click to fix a slouch and right-click to eat what's on the fork. Now the left-click action is overloaded to do both, and fixing the slouch takes precedence. We need to figure this out for the next iteration.
• He was confused by the potato left on the plate and tried to eat it, except it's not actually clickable/edible. We should change the images to only have one kind of food on the plate.

The one thing that did go very well is that he was able to pass things across the table with ease (i.e. he noticed the arrows immediately and acted on them.)

This trial was marred by Nick not noticing what was going on with the instructor's messages (I did ask him whether he noticed the instructor saying anything.) He suggested at this point that "urgent" messages be colored differently, i.e. the speech bubble should change color. We can do this easily and it may have a significant effect. I then moved on and got Nick to try the salad.

During the salad trial I noticed that it was much more successful now that he was attentive to the instructor's messages. Nick exposed a bug during this trial; the eater had the fork and knife in his hands from the steak/potatoes level. When the salad was stabbed, the image was correct, with the fork in the right hand instead of the left. This is easy to fix and will be rectified for the next version. Also, Nick mentioned in a post-experiment discussion that he would not known what to do with the salad had he not been instructed how to during the tutorial. Perhaps it should be made more intuitive, or we can agree that this a "discovery" element of the game and just keep it that way.

Discussion

In this section I'll just summarize the changes we are committing to make based on the result of Nick's experience and the changes that are under consideration but still need discussion. Most of the "discussion"-type elements in prose are listed above in the Results section.

Changes we will make

• Improve message feedback and turn the speech bubble red for "urgent" messages where the player must act on something within a limited period of time. Also, improve the tutorial to stress that the instructor's messages are always meaningful. I believe that Nick didn't think the "instructor" would be present in the actual meals, and didn't look out for his messages.
• Fix the bug where the wrong utensils are being shown.
• Change the tutorial to be a click-through tutorial instead of on a 7-second timer for every message. I believe that click-through tutorials represent a design pattern that is common in computing these days.

Changes requiring discussion

• Decide whether we should show the player how to eat salad/pie during the tutorial, instead of just demonstrating the possibility of different meals and different eating motions.

Appendices

No demo script was used in this case. I felt it was very informal and formalizing the process would have made the participant behave differently.