From CS 160 Fall 2008

Due: before class on November 17th, 2008

Contents 1 Introduction 2 Implementation and Improvements 2.1 Tutorial 2.2 Meal Selection 2.3 Utensil Selection Screen 2.4 Eating Pacing 3 Method 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.3.1 Original Tasks 3.3.2 New Tasks 3.4 Procedure 3.4.1 Greeting 3.4.2 Observation 3.4.3 Questions & Answers 3.4.4 Outline of Procedure 4 Test Measures 4.1 Metrics 4.2 Additional Measures 5 Results 5.1 Dependent Variables 5.2 Summary of the Process Data 5.2.1 Task 1 5.2.2 Task 2 5.2.3 Task 3 5.2.4 General Feedback 6 Discussion 7 Appendices (5 points) 7.1 Materials 7.2 Raw Data 7.2.1 Task 1 7.2.2 Task 2 7.2.3 Task 3

Introduction

In today's day and age, many young Americans are unfamiliar with proper dining etiquette. With the availability of fast food restaurants, children get less exposure to what is considered proper dinning manners and behaviors. There are, however, still a number of ways in which children and teenagers can get trained in dining etiquette, but many of those options are either expensive or time consuming. What our team has come up with is a fun, entertaining way to learning dining etiquette without the high costs associated with formal etiquette courses. By having players interact with a goofy avatar, we hope to teach our users dining skills that they can exercise in the real world.

In the last two weeks, our team has come up with a new release of Formal Logic. Compared to our interactive prototype, our new version of Formal Logic is more comprehensive and incorporates many of the suggestions that we received from the heuristic evaluation of our game. In order to test how well we have implemented these changes and more importantly how users will respond to them, each of our team members performed a pilot study. The results of this study will help us identify areas or features that still require more work. Additionally, we can use our participants' feedback to make further improvements to our game

Implementation and Improvements

Taking into consideration the comments and suggestions we received from the heuristic evaluation assignment, our group made a number of changes to our serious game. Descriptions of each of the changes are below.

Tutorial

First and foremost, we created a tutorial for our game, Formal Logic. The tutorial was created to familiarize the player with the game objective and controls. The objective of our game is pretty simple and straightforward, but we had to point out several key features of our game that may not be apparent: having distractions at the table (i.e. passing salt and pepper) and keeping the avatar from slouching. Additionally, the player controls are not intuitive in gaming standards. The use of the left and right mouse buttons to control the avatar's left and right hands respectively is something players may not grasp right away.

Meal Selection

In our interactive prototype, the player was restricted to eating steak. That made for a very uninteresting game, so we incorporated two additional meals: a salad and dessert (pie). When the player first starts the game, he must choose his course. Giving the player this flexibility not only makes the game more interesting, it broadens the scope of the game. The player now learns about how to eat a salad and dessert properly.

We took a very modular approach when we implemented the game. Each of the plates are stored as separate image files, so it is easy for us to scale the game to include additional courses.

Utensil Selection Screen

One of our targets in this release was to include a utensil selection screen, which requires the user to select the utensils for his chosen course. We stuck with a first person view of the table as shown in the image below. With the first person perspective of the table and utensils, the player is better able to relate the game with real world experiences.

The player is asked to select a utensil for his right and left hands (or just his right, depending on the selected course). When the correct utensil is chosen, the utensil is colored in gold to indicate that the right utensil has been selected. Only after the correct utensils are chosen is the player allowed to continue with the game.

Eating Pacing

One comment that we received from the Heuristic Evaluation was that the player tended to respond to the game much faster than the pacing meter allowed. Thus, the instructor would continually tell the player to "slow down" his pace. As a result, we made the eating pace a little faster in order to accommodate our users. Also, this prevents the users from getting bored while waiting for the instructor to tell him to "proceed eating."

Method

Participant

In order to vary our participant pool, I interviewed a friend with little gaming experience but adequate knowledge of dining etiquette. He did not participant in any of our previous interviews, but was aware that we are designing a game for this class. Information about this participant is below.

• Age: 21
• Sex: Male
• Education: Berkeley undergraduate, majoring in Chemical Biology
• Gaming Experience: Novice. The participant had previous experience playing games in middle school, but little to none now.
• Computer Literacy: Average. Like most college students, my participant knows how to work a word processor, spreadsheets, and create power points. He can search the web and has very little programming experience.
• Experience with Proper Dining Etiquette: Adequate. He has experience eating in upscale restaurants and understands what to do and not to do in formal settings. In general, he is very proper and well mannered.
• Other notes: Because this participant is a good friend of mine, he was trusted to give honest feedback about the game. This assumption proved to be correct, as he didn't refrain from talking aloud during the interview and gave his fair opinion of the game.

Apparatus

Five items were used to perform the evaluation:

• Waiver: A consent waiver was the first thing that was presented to the participant. By signing the waiver, the participant granted me permission to use his comments and feedback in this report. Also, the waiver indicated that the participant's identity would remain anonymous and not be used against him in any way or form.
• Timer: A cell phone timer (on the Glyde) was used to record the time needed for the participant to complete each task.
• Laptop: A Dell Inspiron 600m was used by the participant to play the game
• Notebook and pen: Because the one and only laptop available was being used by the participant, I had to take notes with a pen and paper.

Tasks

Because we incorporated two new features in the game (the tutorial and the utensil selection stage), two of the three tasks were changed from those that were used in our low fidelity prototype interviews.

Original Tasks

1. Pass the salt and pepper (Easy): The user was required to recognize when salt and pepper were requested from another diner and take the appropriate action in passing the condiments
2. Complete a meal (Medium): Our medium difficulty task was to complete a meal without distractions. This tested the player's understanding of the game controls.
3. Complete a meal with distractions (Hard): Finally, our participant was asked to finish a meal with distractions. The distractions included passing salt and pepper and correcting the avatar's posture.

New Tasks

1. Complete the Tutorial (Easy): To help participants learn the game controls, the first task was to complete the tutorial. The tutorial described the objective of the game and had instructions on how to interact with the interface.
2. Select Utensils for a Meal (Medium): The utensil selection screen was a new feature that was implemented in this version. It had not been tested before, so it was a good task to have. Also, selecting utensils for a course is not a very difficult task but does require some prior knowledge of dining etiquette. Thus, it was deemed a Medium level task.
3. Complete a Meal with Distractions (Hard): This task remained the same (and unchanged) from the low fidelity prototype interviews.

Procedure

Greeting

When the participant arrived, I greeted him and asked him to complete a waiver stating that his identity would not be disclosed and furthermore, his opinions would remain anonymous. I then proceeded to explain the purpose of the game and who are our target audience is. Once our interviewee had a good grasp of what we are trying to achieve with the game, I described the overall objective (to complete a meal with distractions) and explained his three tasks in further detail. Lastly, I asked the participant to think aloud if he was comfortable doing so.

Observation

When the participant was ready, I had him attempt each of the three usability tasks while I recorded his actions. I made special note of each of the measures described in the Test Measures section.

Questions & Answers

I followed the observation section with a Q&A. Questions were presented to the participant one at a time and his responses were recorded on my laptop. After all of the questions were answered and documented, I asked if our participant had any further suggestions or feedback.

Outline of Procedure

An outline of the procedure is as follows:

• Greeting and Introduction
  • Sign waiver
  • Describe learning objective and target audience
  • Explain game objectives
    • Review three usability tasks in detail
  • Ask the player to think aloud

• Observation
  • Monitor player's interactions with the game
  • Record player's reactions to the game
    • Note any difficulties the player encountered

• Questions & Answers
  • Ask questions
  • Record player's responses and feedback
  • Discuss improvements

Test Measures

Metrics

The following metrics were used in the evaluation to gauge the effectiveness of the interface in helping the user complete his tasks:

1. Time needed to complete a task
2. Number of positive comments made during game play
3. Number of negative comments made during game play
4. Number of mistakes made

The time that the participant takes to complete a task is a measure of how well our interface is . An easier task should take less time, whereas one that is unclear or hard to complete will take more time. The completion time can also be used as a gauge of how intuitive the interface is. A complicated or poorly designed interface will require the user to spend more time figuring out user controls and tasks.

Positive comments, such as "cool" or "that was neat," highlight the parts of the game that draw the player's attention.

On the other hand, negative comments help to reveal flaws in the game design. If a lot of negative comments are recorded, we may need to rethink part or all of our interface. Negative comments also help us identify where we need to make improvements in the game.

By recording the number of mistakes or errors that the participant makes, we can get a sense of what aspects of the interface are still confusing or unclear.

Additional Measures

The following measures were also logged during the interview:

• Difficulty or ease in completing each tasks
• General feedback on the game and interface

Results

*Results of the tests (1 page)

Dependent Variables

Results

	Time to Complete Task	# Positive Events	# Negative Events	# Mistakes Made
Task 1	~4 Min.	0	2	2
Task 2	~30 Sec.	1	0	1
Task 3	~40 Sec.	0	0	2

Summary of the Process Data

Task 1

Of the three tasks, the first one took the longest simply because the participant must work at the tutorial's pace. There is not an option to proceed to the next step. Instead, the player must wait for further instructions. On that note, one of the negative comments was that there was no option to skip a step or fast forward through a part of the tutorial.

One of the mistakes that the player made was to click on the "Pause" button. We disabled the said button because it didn't add any functionality in the tutorial, however it appears that that only confuses the player more.

The other thing that confused the participant was closing or ending the tutorial. There is no clear "Exit" or "Quit" button. Instead, the player needs to close the window in order to return to the main menu. This wasn't intuitive.

On the plus side, the tutorial seemed to have helped the participant learn the game controls. Compared to the players in our low fidelity prototype interviews, this participant had a guide who walked him through how to use the controls.

Task 2

The utensil selection task was pretty straightforward. The participant chose to eat a steak and potato course and as a result had to choose a steak knife and fork. The one mistake that the participant made was selecting the salad for in place of the dinner fork. Also, the one positive comment came when the participant first saw the raw steak and potato on the plate. He was rather intrigued by our image selection/choice.

Task 3

Having completed the tutorial, the participant was able to complete an entire meal of steak and potatoes with ease. He didn't make any further comments while he performed the task but did make two minor mistakes that were not his fault but worth noting. The mistakes were really one and the same. After cutting the steak, the participant misclicked the cut piece of meat. He performed the right action by left clicking on the piece to use the fork but the clicks did not register. This was perhaps caused by a coding or interface issue.

General Feedback

When asked for further feedback and suggestions, the participant touched on a few points. Those points are outlined below.

• Game Controls
  • Difficult to figure out game controls. Left hand should operate avatar's left hand (i.e. using a keyboard). It was awkward/weird to use only your right hand to control the left/right hands of the avatar
• Difficulty
  • Appropriate for little kids... but he didn't pick the right utensil
  • Suggestion: Offer different difficulty levels (i.e. easy, medium, hard)
• Dining Etiquette
  • The game doesn't really seem to teach the user about dining etiquette or at least not to the extent that the participant expected
  • Suggestion: Enforce a penalty or feedback for picking the wrong utensil

Discussion

Throughout the interview, I kept several questions in mind and posed them to the participant at the end of the interview. The questions and a summary of the participant's responses are below.

• Clarity of Game Play: Is there a steep learning curve? How difficult is it for the player to learn the controls?
  • The tutorial provided a good overview of the game controls. If it weren't for that, however, the controls would have been confusing. In particular, it's not very intuitive to use the left and right mouse clicks to control the avatar's hands. Instead, the participant expected to actually use his left and right hands (perhaps on the keyboard) to control the utensils.
• Difficulty Level: Was the game too easy/difficult for the player? Were there specific tasks that the player struggled with?
  • Adding difficulty levels would really enhance the game play
  • With the current implementation, the game gets tedious and boring rather quickly
• Learning dining etiquette: Does the game meet its goal of teaching dining etiquette? If not, what improvements can we make to meet that goal?
  • The game didn't really meet the participant's expectations in terms of learning dining etiquette. Although the game was fun to play, he didn't find that he learned anything new. His one suggestion was to incorporate educational facts in the game.

The pilot run showed that although we attempted to fix or clear up many of the issues found in our interactive prototype, there are still a lot of features to be desired. In particular, the tutorial requires more work. We can and should include an option for the player to skip ahead to the next step or skip certain parts of the instructions. With the current implementation, the user is required to sit and stare at the instructions as they are presented. Likewise, if the player wants to exit the tutorial there should be a clear button to "Quit" the tutorial. Another improvement that can be made is to incorporate feedback into the utensil selection part of the game. One comment from the participant was that no feedback or prompt was given when the wrong utensil was chosen. What we can do in this case is provide a reason as to why the chosen utensil was not correct.

Another area of improvement is in fixing the accuracy of the mouse clicks. On several occasions, as mentioned above, the participant "misclicked" his steak because either the game didn't register his actions correctly or the food pieces are too small to click on. One way to fix this problem is to make the plate and food larger so that there won't be "misclicks."

As far as educating the player on dining etiquette is concerned, we may want to consider incorporating educational facts in the game as the participant suggested. With the way the game is currently structured, the player is simply performing mouse clicks.

Appendices (5 points)

Materials

*Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)

Raw Data

Notes from the interview:

Task 1

0:04 - *Chuckle* [about the pictures used] (Negative Event)

0:34 - "The instructions are kind of slow." (Negative Event)

0:40 - "Haha, he has a funny face!"

1:25 - "Why can't I click on the Pause button?" (Mistake)

2:10 - "Oh, wine!" Tried clicking on the wine glass but solicited no response. (Mistake)

Task 2

0:02 - "Oh cool!" (Positive Event)

0:14 - "What?! Ohh... that's the salad fork." (Mistake)

Task 3

0:15 - "Uhhh... why isn't he picking up the meat?" (Mistake)

0:17 - "I still can't click on the meat." (Mistake)