PilotStudy-Group:Lucky Seven-Antony Setiawan

From CS 160 Fall 2008

Introduction

The system being evaluated is Defend Your Body, a serious game that aims to improve its audience’s eating habits through feedback provided during gameplay. The game is a strategy type game, and the main goal of gameplay is to defend a human body from taking in an excessive amount of food, while helping it consume a healthy diet. Extra features such as upgrading the body’s organs help familiarize the player with the functions of these organs, while enhancing gameplay by aiding in tackling the increased difficulty of each new wave. In addition, the settings feature allows the user to enter his or her age, height, weight, and level of physical activity; these stats are then integrated into gameplay such that the player is given a realistic view of what nutrient levels are required for his or her body type. Lastly, the highscore feature allows the users to compete with other players online.

The purpose of the experiment is to conduct the usability of our team's latest interactive prototype. Our group has made considerable changes and improvement to the previous version of the interactive prototype. Changes that we made are based on the feedbacks and suggestions that we got from the Heuristic Evaluation Prototypes. To further enchance our games, we use this experiment as a tool to get empirical feedbacks and suggestions from real users. And again, the measures that we use in this experiment are based on the three task.

Implementation and Improvements

Changes	Description
Setting screen	Remove the checkbox next to the 'mouse' text since the user could no longer choose between using the mouse and keyboard The mouse sensitivity bar from a continuous slider to an incremental one that varied from 1 to 5 Changed the interface of age/weight/height values from sliders to text boxes (with integrated limits) for ease of use reasons. "Physical Activity" dropdown menu is changed to “hours of physical activity per week”, as this made more sense Value in the settings menu is now reflected in the game. Physical characteristics entered and saved by the user will be applied to nutrient requireents for each wave.
Gameplay Screen	Create Delete and Absorb Mode and show the status on the left side of the screen User can toggle between modes by pressing "SHIFT" (Absorb Mode) and "CTRL" (Delete Mode) Delete mode will eliminate the food by reducing the health of the food (affected by attacks points) for each click Absorb mode will absorb the food in one click, the nutrients of the food is added to the health attributes for that wave. Added text representation on mouse roolover on the food, providing both text and graphical information to the player. The on food roolover information will stay for two seconds instead of disappear right away. Changed the value of fat/carb/protein from percent to numbers, make it easier to understand exactly how many grams of each nutrient is needed per day Implement increasing difficulty for each wave, make it more challenging and encourage them to play better Food's life is multiplied by a varied values depends on the level Food's speed is multiplied by a varied values depends on the level Fodd's amount that comes in to the body is increased by 10 for each level Changes the implementation of reducing lives: Once any of the daily required attributes is exceeded, all food that get absorbed would also reduce health.
Upgrade Screen	Implement score and upgrade score: score doesn't get decremented when upgrading organs. Adds repair health option which adds 1 life and the required value for next upgrade will be increased. Adjust the required value for upgrade with scoring system
HighScore	Display top 10 scores in descending order, make it more readable.

Methods

Participants

Age             : 21
Sex             : Male
Education level : College students - Senior
Major           : Computer Science
Games Experience: High
Serious game exp: Low

This is one of the person that we picked when we conducted our experiment using paper prototype for some reasons:

He fell under our target group.
He has experience with our low fidelity prototype.
He is expected to give more feedback compared with other person who plays my team's game for the first time.
Lastly, I promised him that I would let him play our game once the interactive prototype is made.

Even though the completion of the three task might be biased because of his experience in our game, feedbacks from him as a user who has played the game would adds to the variety of our team's collected data.

Apparatus

For this experiment, the participant is required to bring his own laptop. This way, once laptop can be used to play the game while the other laptop is used to gather all necessary data. For this experiment, internet is also being used since one of the task in this experiment is to submit score (which is now online). This experiment is done in the first floor of Moffit Library.

Tasks

Three tasks that is experimented are the same with the previous experiment, i.e.:

Submitting a high score (easy):
A player can submit a high score either by ending the game from the menu (situated in the gameplay and upgrade screens) or by dying (losing all health). When either case happens, a screen will pop up prompting the player to submit a high score. The actual process of submitting the score is done by typing a nickname or initials and clicking submit button. Highscores can be viewed from the main menu.

Upgrading organs (medium):
At the end of each wave, the gameplay screen will switch to the upgrade screen. This screen contains the pictures of three organs—heart, liver, and stomach—that can be upgraded by allocating points (acquired during gamepaly) to those organs. When the mouse rolls over the picture of each organ, text in the upper right of the screen will display information about the organ and its effects in the game. Upgrading the organ is categorized as a moderate task because the player can't just upgrade whichever organ that is shown on the screen; each organ has a different, positive effect on gameplay.

Playing the game (hard):
This task involves performing and demonstrating familiarity with the gameplay. This is done in the main gameplay screen, in which waves of foods will enter the stomach; the player must selectively allow a healthy combination of foods to pass through the stomach, so as to get the nutrient intake as close to the daily recommended values as possible. Once a food exits the stomach (through the bottom of the screen), it will be absorbed by the intestines and its calorie, fat, carbohydrates, and protein amounts will be added to the total nutrition for that wave. If too much or too little food passes through the stomach, the health bar below the heart will go down, and once it hits zero, the game has ended. Since each wave represents a day, the goal of each wave is to allow the desired daily values of the respective nutrients into the body. Additionally, the stomach, liver, and heart can be upgraded to aid in gameplay (upgrading these organs is our next task).

Procedure

Step by Step Procedure:

I set up an appointment with him to do this experiment and decided to do this experiment in the previous place:1st floor of Moffit Library.
We pulled out one table near the corner and set up the apparatus there.
I positioned my laptop adjacent to his laptop so that I can get a good view and he wouldn't be able to see my screen.
I tell him the purpose of this experiment and what do I expect from him in this experiment.
I asked him to say out loud whatever comes to his mind during the experiment
I tell him which part of the game is implemented using Wizard of Oz technique.
Send the game link to his laptop and open the game in Firefox (somehow the game does not run on his IE).
When the game is loaded, I present him a piece of paper containing three tasks and briefly explain what's the goal of each
I tell the participant what are the new features/rules of the game.
Gave him the control
Open up MS notepad to take note during the game
Let him finished the first wave. Upon entering the upgrade screen, I reminded him about "absorb mode" features
I stopped him on the third wave's upgrade screen to get additional middle-game feedback
Let him finish the game
Get end-game feedback
Say thanks, goodbye,etc

Test Measures

These are the measures that I used to rate my team's latest prototype.

Navigation time between each interface - I wanted to meausre the effectiveness of the participant in navigating throughout the interfaces.
Food decision Making time - I wanted to measure how well our interface in supporting the player in making decision of which food shoul be absorbed/deleted.
Gameplay time - I wanted to measure what is the gameplay time limit, i.e. until what point the player can survive the game
Life bar - To measure how easy/hard our game is.
Upgraded organs - Which organ is being upgraded and why
Task completion time - To see how well our interface supports the player to finish the given tasks(other than playing the game).
Confusion level - To see whether confusion arises and how high the frequency of the confusion.

Results

Incidents

Forgot to absorb food on the first wave
Misclick the organ to upgrade
Died on the fifth wave after attributes overlimit
After one game ends, have to close the browser to start another game (the participant decided not to play again)

Easy Task(submit the score)

Task completion time: 5 second (including filling the name)
Confusion:none

The participant seems to have no problem in submitting the score after he died and he said that the submit score interface is pretty intuitive and straightforward.

Medium Task(upgrading orgrans)

Task completion time -mm:ss- 03:32 , 00:01 , 00:30 , 00:20 , n/a
Confusion level - high, n/a, medium, high, n/a
Organs upgraded [heart][liver][stomach][restore] - [0,0,0,1] , [0,0,0,0] , [2,1,0,0] , [1,1,3,3] , n/a

The result of this experiment shows that there are various level of confusion and completion time for this task. As I asssumed the same between the participant and most other players, when they reach the upgrade screen, they tend to explore the whole selection to see what are the option that they have, hence resulting in high completion time. Tooltip that pop up for each organ does not clearly says what it contributes to the game, in addition to mismatched pictures of the organs with their expectation, creating high level of confusion .

Hard Task (playing the game)

Task completion time -mm:ss- 00:50 , 2:01 , 3:03, 3:50 , 3:30
Confusion level - low, medium, medium, high , very high
Life at each wave - 4, 5, 5, 2, 0
Decide which food to delete/absorb - <1-2 seconds

With some knowledge from the paper prototype, the participants clicked the food that he marked as bad. When he hover the food, status bars on the left changes, forcing him to pay attention to the increasing bar. The participant comments that even though he does not fully remmebers the stats for each food, when he sees huge increase on the bar, it mades him wants to delete that food. Using graphical object as bar doesn't take much time to process compared to number. Since our implementation increase the number of food by 10 for each wave, there would be around 60 food coming through and seems too much for the participant. He comments that the increasing difficulty is too much because he can't really thinks where there are too much food in the stomach. He also comments that there are no feedback to indicate whether he successfully clicked on the food or not.

Other Variables

Finding the right button to another interface - <1 second
Gameplay time - 33 minutes

Discussion

Derived from the result above, some consideration to be made:

Feedback control seems to be one of the most important key in the real time action game. As the participant mentioned during the experiment, there is nothing that can tells him whether he's successfully clicks the food or not. The solution that our team came up with a while ago was by either integrating sound effect or movie effect into the game. This way, whenever a player clicks on something, he/she can get better perspective of what's going on.
Players expect some direct changes in the gameplay whenever they upgrade an organ. Also, the current tooltip information does not give the expected information to the player in deciding which organ takes priority to be upgraded during the gameplay.
Most game attracted a lot of player because they're well decorated. Due to the game complexity, time constraint, and the manpower of our team, we haven't have much time to decorate/polish the interface. For "real" experiment, this essence also need to be considered.
Having bar chart that shows the effect of each food is useful and most players only refer to it when playing this game. However, if we're not very careful in putting the right real values for each food, this game might be misleading and educate players the wrong information. In the future, all the food specification, picture, and body status will be adjusted based on the real life nutrition facts and BMI.
I only have a chance to experiment with one person only. Generalizing based on one person is not suffice. Since this game can be accessed online and have it's own database, a simple feedback form at the end of the game would allow everyone who has played the game to give feedback and suggestion. This way, more things outside the box can be revealed and more bugs can be crushed.

Appendices

Demo Script

Say hello.
Positioning the participant's laptop and my laptop.
Briefing - state purpose.
Open the game (in participant's laptop if possible)
Handout three tasks and briefly tells what are expected.
Let the participant plays the game.
DON'T FORGET TO NOTE!
If level 5 is reached, stop the player
ask for feedback
let the participant continues the game
After the participant stopped playing, ask another feedback

Raw Data

(*)comment on the fly

Before Game

(*)"Still haven't change the main interface ya?"
Go to settings, does not change any parameter: only changes mouse sensitivity
back to the main screen and start the game right away

1st Wave

In delete mode deletes foods. Missed 1 (in purpose)
In upgrade screen went through all the picture, see the desc.
Restore health
Seems confused, hit nextwave right away.
I told him about absorb mode.

2nd Wave

Figured that absorb mode does not give much points
Again clicked most of the food. Miss none
Hit nextwave right away

3rd Wave

Food getting faster.
Start using the absorb mode but to delete food that almost passed through
missed none
Upgrade hearts and liver.

4th Wave

Lot's of food coming in
Don't have enough time to think
Start switching between absorb and delete
Barely made it
Exhaust all upgrade points, upgrading random organs

5th Wave

Start absorbing more food that almost passed through
overshoot the attributes
Died.
Submit score with no problem.

Comments

picture unrelated description shows no contribution to the game can accidentally clicked on upgrade heart overlapping upgradeable organ positioning button more cooler Fancier gameplay screen Need more feedback when clicking Sound would be great Too hard Intuitive bar.