From CS 160 Fall 2008

Contents 1 Introduction 2 Implementation and Improvements 2.1 Settings Screen 2.2 Gameplay Screen 2.3 Upgrade Screen 2.4 Highscores 3 Methods 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.4 Procedure 4 Test Measures 5 Results 6 Discussion 7 Appendices 7.1 Demo Script 7.2 Task Instructions 7.3 Raw Data

Introduction

The system being evaluated is Defend Your Body, a serious game that aims to improve its audience’s eating habits through feedback provided during gameplay. The game is a strategy type game, and the main goal of gameplay is to defend a human body from taking in an excessive amount of food, while helping it consume a healthy diet. Extra features such as upgrading the body’s organs help familiarize the player with the functions of these organs, while enhancing gameplay by aiding in tackling the increased difficulty of each new wave. In addition, the settings feature allows the user to enter his or her age, height, weight, and level of physical activity; these stats are then integrated into gameplay such that the player is given a realistic view of what nutrient levels are required for his or her body type. Lastly, the highscore feature allows the users to compete with other players online.

Since many college students lack the knowledge of keeping track of a healthy diet, our game is designed to simulate choosing the healthier options and eating the right portions of food (i.e. balanced diet). Our goal is to provide an entertaining but informative tool for users to learn about creating a healthy diet, and we want our users to not only become more familiar with regulating a balanced daily intake of nutrients, but also be able to make quick and subconscious decisions on which combinations of foods will satisfy the daily requirement of nutrients. Since most college students find games amusing, we believe that the entertainment factor of the game will make learning about eating well enjoyable, and allow players to be exposed to information that they probably wouldn’t otherwise seek out. Since a healthy or unhealthy diet could have lasting effects on a human body for many years down the road, ignoring the habit of keeping track of what we eat could have severe consequences. We believe with this game, our target users will be able make better decisions about what they eat by just playing the game a few times a week.

Implementation and Improvements

We made many changes and improvements to our prototype since our last iteration. Below are the corresponding screens and the changes we made to each one.

Settings Screen

We removed the unnecessary check box next to the “Mouse” text, since the user could no longer choose between using the mouse and keyboard. In addition, we changed the mouse sensitivity bar from a continuous slider to an incremental one that varied from 1 to 5; this way the user could choose between specific mouse sensitivities.

Also, we changed the interface of age/weight/height values from sliders to text boxes (with integrated limits), so that the user could just easily type in their physical stats. We also changed the “physical activity” dropdown menu from walking/biking/etc. options to “hours of physical activity per week”, as this made more sense and was more intuitive for our user to select a representative level of physical activity.

Finally, we integrated our settings screen into our gameplay (in our last implementation this was one of our wizard of oz techniques). Thus, the physical characteristics entered and saved by the user will be applied to nutrient levels (i.e. daily recommended values) needed during each wave. In order to do this, we created default values for each of the nutrients, and then combined them with multipliers through condition checks on each physical characteristic (age/weight/height/physical activity). For example, if our default values for calories/fat/carbs/protein were 2300/60/300/50, we would multiply these values by 1.1 if they were under age 30, multiply the resulting value by another 1.1 if they weighed over 200lbs, etc.

Gameplay Screen

There were many changes/improvements made to our gameplay screen.

• Since the group that did our heuristic evaluation noted that the screen was cluttered with food sometimes, we created a “Delete” and “Absorb” mode for the mouse, displayed on the left side of the screen. The user can toggle between “Delete” and “Absorb” by pressing the Shift or Ctrl button – when Shift is pressed, the mouse will change into “Absorb” mode; Ctrl will put the mouse in “Delete” mode. When the mouse is in “Delete” mode, it will act normally and mouse clicks will attack and eliminate food from the stomach. When the mouse is in “Absorb” mode, clicking on any food will “absorb” it – in other words, the nutrients of that food will be added to the wave totals. In essence, “absorb” creates the same effect as letting the food pass through the body.
• When you mouse-over a food, the additional nutrients that the food will provide is displayed in both bar and number form now. Also the display of this information will last for about 2 seconds, so that the user doesn’t have to continually hover the mouse over the food to read the values.
• We changed the fat/carb/protein nutrient bars to numbers instead of percents. This makes it easier for the user to understand exactly how many grams of each nutrient is needed per day.
• We implemented the functionality of multiple waves with increasing difficulty. In essence, each successive wave will a few more food, the food will have more life, and the food will move faster through the stomach. This increasing difficulty is what makes the game more challenging for the player, and to encourage them to play better.
• We implemented a mechanism for deciding when the player loses life. Originally, the player lost life whenever he or she clicked on a “bad” food. However, we realized that there are no absolute “good” or “bad” foods, so we made the player lose life based our how far away they undershoot or overshoot their daily nutrient goal. For example, once a player overshoots the daily required calories (i.e. 2300) in the middle of a wave, any additional food that is absorbed or passes through the body will take away one life. Moreover, if the player doesn’t get enough of a nutrient by the end of a wave, life will be subtracted depending on how far the player undershot the goal.

Upgrade Screen

• On mouse rollover of the images of the organs, now the in-game effects of the organ will be displayed as well as organ’s function in real life.
• We implemented two scores: one to display to the total score so far, and one to display the upgrade points that the player can spend on upgrading organs. The total score will never be decremented, while the upgrade score will be decremented whenever the player choose to upgrade an organ.
• We added a “repair health” upgrade that allows the player to spend points to regain 1 life. This is to make the player more careful about not losing any life during gameplay.

Highscores

Originally, our highscore screen displayed all scored in non-sorted order. Our new implementation displays only the top 10 scores automatically sorted in descending order. This makes viewing the top scores quicker, and it is easier to see who is the best player.

Methods

Participant

Age: 21
Sex: Male
Education Level: currently in college
Major: Mechanical Engineering
Experience with serious games: low-medium

I selected this participant randomly from campus. The participant was selected based on his desire to eat healthier but lack of knowledge of healthy eating habits. Also, his age and occupation (student) fell within our target user group, and his low to medium experience with serious games meant that I probably wouldn’t need excessive time to familiarize him to our prototype.

Apparatus

The only equipment I use was a laptop with internet connection, a voice recorder (to aid in recording results), and a pencil and paper to take notes. Since our flash game was linked to online, accessing the internet from my laptop allowed the participant to access the game. The test was conducted in the first floor of Moffit Library.

Tasks

Playing the game (hard):
This task involves performing and demonstrating familiarity with the gameplay. This is done in the main gameplay screen, in which waves of foods will enter the stomach; the player must selectively allow a healthy combination of foods to pass through the stomach, so as to get the nutrient intake as close to the daily recommended values as possible. Once a food exits the stomach (through the bottom of the screen), it will be absorbed by the intestines and its calorie, fat, carbohydrates, and protein amounts will be added to the total nutrition for that wave. If too much or too little food passes through the stomach, the health bar below the heart will go down, and once it hits zero, the game has ended. Since each wave represents a day, the goal of each wave is to allow the desired daily values of the respective nutrients into the body. Additionally, the stomach, liver, and heart can be upgraded to aid in gameplay (upgrading these organs is our next task).

Upgrading organs (medium):
At the end of each wave, the gameplay screen will switch to the upgrade screen. This screen contains the pictures of three organs—heart, liver, and stomach—that can be upgraded by allocating points (acquired during gamepaly) to those organs. When the mouse rolls over the picture of each organ, text in the upper right of the screen will display information about the organ and its effects in the game. Upgrading the organ is categorized as a moderate task because the player can't just upgrade whichever organ that is shown on the screen; each organ has a different, positive effect on gameplay.

Submitting a high score (easy):
A player can submit a high score either by ending the game from the menu (situated in the gameplay and upgrade screens) or by dying (losing all health). When either case happens, a screen will pop up prompting the player to submit a high score. The actual process of submitting the score is done by typing a nickname or initials and clicking submit button. Highscores can be viewed from the main menu.

Procedure

The first step was to set up a time and place to conduct the test. Since the test was conducted on only one participant, it was not difficult to arrange a setting to perform the test; in our case, we chose the first floor of Moffit Library. This place was quiet enough to allow me and the participant to hear each other, but not so formal as to make the participant feel nervous or pressured.

After the customary greetings, I introduced the participant to the premise of the test by explaining to him the concept of a serious game, and detailing the theme and genre of the serious game that my group has been working on. After the participant was comfortable with what the purpose of the test, I explained to him that I would be presenting him with three representative tasks that would require comprehensive knowledge of the user interface (I did not actually present the tasks at this moment). I then proceeded to demo the prototype that the participant would be testing, showing him examples of accessing the various screens, but making sure not to complete any of the representative tasks that I would be presenting to him. Since our instruction video was also a wizard of oz technique, I also explained to the participant how the gameplay mechanics worked.

After that, I presented the participant with the three representative tasks one by one, in the order: playing the game, upgrading organs, submitting a high score. For each task, I set up a voice recorder and timer and wrote down all critical incidents as well as other feedback that was provided as the participant was trying to complete the task. For each critical incident, I wrote down the time that the incident started, the duration of the critical incident, and a short description of the incident. In addition, encouraging the participant to think aloud and not stress out about not being able to complete a task (before I presented the tasks) helped me in getting more feedback.

After the three tasks were presented, I debriefed the participant by asking him what he thought of each of the representation tasks, what he thought of the interface in general, if he had much fun playing the game, what he learned, and any other comments. This additional feedback supplemented my log of critical incidents. After the participant had no more comments or questions, I thanked him for his time, and briefly took some time to go over and reorganize the notes that I took during the test.

Test Measures

• Response time from navigating the main menu screen to starting the gameplay: I wanted to see if the participant felt the need to click on “instructions” or “settings” before starting the game, since the interface is designed for these buttons to be clicked first.
• Time it takes for the participant to absorb their first food – I wanted to see how long it took the participant to realize that he had to absorb food in order get closer to the nutrient requirements
• Time it takes between the participant performing a mouse-rollover on a food and actually choosing to absorb or eliminate it – This is to test to if the participant actually recognizes that the nutrients of each food have to be considered, or if the participant just clicks blindly
• How many food (if any) passed through the stomach – This was measured to see if the level of difficulty of the game was too easy or too hard. If no food passed through, it was too easy; if many foods were unintentionally being allowed to exit the stomach, then the game might be a little hard.
• How many life the participant lost on the first (and subsequent) waves – This is another measure of how easy or difficult our game is
• Time spent figuring out which organs to upgrade – This was measured to determine if the participant actually thought about which organs to upgrade, and took time to read the organs’ in-game and real life functions
• Time it took for the user to figure out how to submit a high score – This was measured to see if the participant understood the two different ways that a player could submit a high score (in other words, demonstrating basic familiarity of the menu interface).

Results

Even though the participant didn’t have any significant amount of exposure to serious/educational games, he demonstrated relative ease and confidence with navigating through the interface once I finished my demo. For example, when I presented the participant with the first task, he dove immediately into accessing the instruction video and changing the settings, spending almost no time to stop for a second and consider what he had to do. In addition, the participant had no problem understanding and entering in his physical characteristics into the settings screen, but he did mention that the text boxes were slightly misaligned with the corresponding text.

The participant spent around one minute and thirty seconds (1:30) navigating around the main menu screen before starting the actual game, which is actually the desired behavior since we wanted our user to change/save the settings first (for the first task). After starting the game, it again took very little time for the participant to figure out what was going on; while at first he rapidly clicked on every food that was coming into the stomach, 10 seconds into the gameplay he realized he had to fill up the nutrient bars at the top left, switched to “absorb” mode and started absorbed his own food. However, I also noticed that he would switch off between brief periods of “absorb” and “delete” mode, deleting food indiscriminately for a few seconds, and then absorbing food indiscriminately (i.e. even the hamburgers, beer) for a few seconds. Little time (0-2 seconds) was actually spent looking at the nutrition values of a particular food (triggered by hovering the mouse over the food).

When accessing the upgrades, the participant was a little confused at first, because he couldn’t figure out what each of the organs did (the mouse-rollover event had a slight delay before displaying the organ info). However, he eventually found out how to access the organ info (after about 5 seconds of idle wondering), and quickly skimmed through each of the organ descriptions. On suggestion he provided at this time was to separate the real-life and in-game descriptions of what the each organ does. He also noted some formatting (spacing) issues with the pop-up text. He then proceeded to randomly upgrade the organs without thinking about what organ or what combination of upgraded organs would lead to the highest chance of survival.

Also, in the three waves that the participant played, not a single food passed through (i.e. exited) the stomach, and the participant never lost a single life, demonstrating a clear mismatch of difficulty and skill level between game and player. Even by the third wave, when the food was moving faster and required more clicks to eliminate, the participant’s rapid clicking speed combined with the food’s long path through the stomach meant that any and all unwanted food was eliminated before it could exit the stomach.

During debriefing, the participant noted that the concept and general idea of the game was interesting, but that it needed more compelling features to attract players to play for more than a few waves. In particular, he mentioned that all food exhibited pretty much the same behavior, and that as soon as the novelty of the game wore off, there will not enough randomness and variation to maintain the player’s interest. He suggested increasing the difficulty level o the game, as well as adding special effects to certain foods (i.e. some foods explode when clicked on). In addition, the participant noted that the nutrition values of even some of the “healthier” foods were not very balanced (in fact, some of the nutrition values we had entered were completely wrong), and that this made it hard to exactly reach the daily nutrient goals. For this problem, he suggested to provide a wider and more diverse variety of foods.

Discussion

After conducting the testing session, I learned the importance of gameplay capturing the players’ interest and affecting their desire to replay the game. As was noticed during the test session (as well as mentioned by the participant), the lack of variation in gameplay made the game repetitive and uninteresting after the novelty factor of the game wore off. To combat this problem and make the game more interesting, in the next iteration I would probably add additional effects to different foods (i.e. some foods split apart, some poison the stomach), as well as add special foods that have random effects of even “boss” foods (foods that have lots of hp). In addition the upgrades could be more interesting, as we could introduce special weapons to attack the food instead of using the mouse click every single time.

In terms of the actual interface of game, most of the critical incidents that occurred during testing were only minor problems, and our overall interface proved to be pretty sound. One problem that I did notice during gameplay (the first task) was that the participant would switch off between absorb mode and delete mode for short bursts of time; all food in these short burst would either by (all) absorbed or (all) eliminated. Since the goal of our gameplay is to allow the user to selectively absorb particular foods, and not to spend five seconds absorbing and another five seconds deleting, a possible solution to convey this purpose more clearly would be to make it take much less time to absorb a food (i.e. one click) than to eliminate it (many clicks). This way, the player can focus most of his energy on eliminating unwanted food, which is the main aspect of gameplay, and quickly absorb food with little effort.

Another relatively significant problem was the difficulty level of the game – it was simply too easy to survive through each wave, providing little challenge for the player. This problem relates to our first one in that without an urgent pressure to survive, the player is not aroused to play the game with a heightened attention, and thus loses interest quickly. In order to bring the danger aspect to the game, we could simply make the food have more life and move faster through the stomach, but this would not increase the fun factor of the game by too much. Instead, one better solution is to, again, attribute certain special effects to different foods in the game; for example, some foods could take a more direct path through the stomach, some foods could damage the body every second it is in the stomach (i.e. poison damage), and some food could have super armor and take many clicks to eliminate.

Other minor interface problems included the text boxes the in settings screen not being lined up (which could be easily fixed) and the in-game and real-life info of the upgrade organs being clumped together (which we will separate in or next iteration – thanks to the suggestion of the participant). Other than that, I did not notice the participant had any difficulty with navigating our game’s interface.

In the end, most of our usability problems that arose in this iteration related to gameplay. Thus, being able to test an interactive prototype with a participant was very helpful in allowing us to see what was lacking in our gameplay mechanism, a process that was hard to emulate and evaluate with a lo-fi prototype. Of course, a real usability study would require more than just one user and a longer window of time to test the lasting effects (if any) that the game has on our players. Since our main goal is to improve the eating habits of health-conscious college students (which is a pretty wide demographic), the small differences between each person in our target user group could produce results that varied from those that I obtained from my one participant. In addition, to observe the actual benefits of the game on eating habits, we would have to monitor how often our test users replayed the game, and check their diets periodically to see if they changed over time. The fact that this pilot usability study was only a one-time test on one participant provides only limited information, as the results could potentially be an outlier. Nevertheless, this test provided a valuable insight to how much interest a player would actually take to our game.

Appendices

Demo Script

This experiment is for our CS160 User Interface class. The theme for this semester is to design a serious game, in which we go through the phases of brainstorming, selecting and interviewing a target user group, creating and testing a prototype, and finally implementing the entire product. We are currently in the phase of conducting pilot usability tests of an interactive prototype, and you have been selected as one the testers (whose identities will not be revealed). Based on your feedback, we will be able to make changes and improve upon our prototype to make our design more appealing.

Before you start this test, we want you to know that any difficulties or inconveniences you have with our prototype is not your fault, but the fault of our design. We will not be allowed to speak during the testing (as not to bias our testers), but feel free to voice your questions, thoughts, and suggestions while you are testing our prototype.

As you can see, the prototype mainly revolves around the use of the mouse to click various buttons that will bring you to different screens of the interface. For example, there is a settings screen, where you can input your physical characteristics, an instructions screen, where you can play an instruction video, and a gameplay screen, where most of the game playing takes place. Fortunately, our interface is almost completely implemented, so you will have access to pretty much all the functionalities of the game. The goal of our design is to see if our users can easily perform specific tasks. Thus, we will present you with three separate tasks that we want you to complete. We will present the tasks to you one at a time, and we want you to show us you know how to complete them by using our interface. Don’t feel bad if you are having trouble completing a task-that would be our fault for creating a poor design.

Any questions?

Task Instructions

1. Play the game (hard) - show us you know how to play the game
2. Upgrade organs (moderate) - upgrade at least one of the body organs in the game
3. Submit a high score (easy) - submit your in-game score

Raw Data

User: 21-year-old male college student, Mechanical Engineering major, low-medium experience with serious games

Measurements/Log of critical incidents

• Time spent navigating Main Menu before clicking “start game”: 1:30. The participant actually clicked both the instructions and settings (and set/saved his physical characteristics).

Rating: not a usability problem

• participant wondered why the text boxes in the settings screen didn’t line up

Rating: cosmetic problem

• Time before participant realized he needed to absorb some food: 10 seconds. The participant started out clicking all food, but realized he needed to absorb some food too.

Rating: not a usability problem

• Time spent looking at the nutrient levels of a food before clicking: 0-2 seconds. Only when the total nutrient totals for the wave were unbalanced did the participant take the time to examine the nutrient levels of a particular food.

Rating: major usability problem

• Life lost on first wave: 0. Number of food that unintentionally passed through the stomach: 0. These two statistics point towards a game that is not challenging enough.

Rating: major usability problem

• Time spent reading mouse-rollover information about organs: 10 seconds. (It was only a brief skim)

Rating: minor usability problem

• Time to figure out how to submit a highscore: negligible

Rating: not a usability problem