PilotStudy-Group:一三三七-AlanMcCreary
From CS 160 Fall 2008
Contents |
Introduction (5 points)
The system under evaluation for this experiment is Fit Wars, a game created for those who enjoy playing video games and are looking for a way to make physical exercise (running) more enjoyable. In FitWars, a user plays a fighting video game while running, either against the computer or against another human; the player earns points for his efforts, and can apply these points to in-game item purchases or transfer points to other video games. We will ultimately implement the game on the iPhone platform, but for ease of testing, we have so far used a Flash implementation.
The purpose of this experiment is to obtain quantitative data and make qualitative observations to determine flaws in the game interface. Quantitative data will include factors such as time spent thinking and number of erroneous clicks on each screen. Other observations will include the tester's general sense of enjoyment during the game and any comments the tester makes during the test. With these data and observations, we hope to make the interface easier to use and make the game more entertaining.
Implementation and Improvements(15 points)
- During the interactive prototype presentation, it was pointed out that because the iPhone does not have a cursor, hovering elements cannot be implemented. For this reason, hover descriptions for menu items were removed.
- The "game settings" page was implemented. This page allows the user to toggle a set of options: game music, announcer's voice, game sounds, and "casual mode". The "casual mode" option allows the user to go through the game without any of the "chance" sections (in which a one-on-one battle pops up, and the player has to jump). In addition, the user can calibrate the iPhone to measure the player's physical movement more accurately.
- The "manage your stats" page was implemented. This page allows the user to view character information for the user: rankings, "strength" numbers, and items that the character owns.
Method (10 points)
Participant
The participant for this experiment is a 21-year-old business major at UC Berkeley. He enjoys playing games on the computer when he has free time (or when he doesn't feel like doing work), most often simple Internet Flash games. He occasionally goes running outside, but not consistently - he said that he thinks he should exercise more often. I found this person to be appropriate for this experiment because he falls in the target user group: he enjoys video games (although the points system might not apply to him, since he does not play large role playing games very often), and would like to exercise more often.
Apparatus
We ran this experiment on the participant's desktop computer (running Windows XP) at his apartment. To measure approximate times, I used a simple digital watch - I considered using a stopwatch, but constantly resetting and restarting the watch seemed impractical, since I was the only one conducting the experiment.
Tasks
During each of these tasks, I looked for signs of hesitation, and listened for any comments he made during the test.
Easy task: register a username and log in with it - This simple task requires the user to start with the "register" button, enter user information, and log in with the newly created username.
Medium task: go to message inbox, read a message, and send a message - For this task, the user must navigate through the correct screens to communicate with other users. In previous assignments, a slightly easier task was used (users were not asked to read any of the messages in the inbox).
Hard task: play a level against another human player - The user must enter another player's username and then go through a level, completing the required "jumps" and pressing the "running" keys.
Procedure
Before this experiment, I gave the participant a brief explanation of the experiment and its objectives, and gave him a consent form to review and sign. To get the tester slightly familiarized with the interface, I showed him important screens of the interface, careful not to show him how to go from one screen to another - I only showed him screen shots, since finding the correct buttons to click is the tester's job during the experiment. I then instructed the participant to try his best to go through each task without asking me for help.
With each of the three tasks, I began by explaining the task to the tester. I then asked him to begin the task. While he performed each task, I kept track of the amount of time (roughly, with a digital watch) that he spent on each of the major screens. In addition, I paid attention for any mistakes or signs of hesitation. At the start of each task, I had the tester start with the login screen. After the completion of a task, I asked for any additional comments from the tester.
Test Measures (5 points)
For each of the three tasks, I made two quantitative measurements: time spent on each screen, and number of errors (i.e., incorrect button clicks). I also kept track of any expressions of surprise, or any other positive or negative reaction to the interface. I did not use a stopwatch when measuring time, since the effort involved in stopping and resetting the stopwatch after each screen could prevent me from taking note of other observations. For each task, I added the number of time spent on all of the screens to calculate the total time spent on the task.
Results (10 points)
Easy Task
- Main login screen: 3 seconds, 0 errors
- Registration screen: 9 seconds, 1 error
- Main login screen: 2 seconds, 0 errors
Totals: 14 seconds, 0 errors
The tester had a slight delay when looking for the inbox on the main menu, since he began by looking at the white-colored buttons instead of the "new message" notification button. He expressed dislike for the loud, sharp sound during the click of some buttons.
Errors
The tester made a small error during registration: instead of clicking "go down" to go to the bottom half of the registration screen, he pressed the enter key, thinking it would take him to the next step. However, this should not be a problem when the interface is implemented on the iPhone, since there is no enter key on the iPhone.
Medium Task
- Main menu: 2 seconds, 0 errors
- Inbox: 2 seconds, 0 errors
- "Read message" screen: 6 seconds, 0 errors (most time was spent just reading the message)
- Inbox: 2 seconds, 0 errors
- "Write message" screen: 7 seconds, 0 errors (most time was spent typing)
Totals: 19 seconds, 0 errors
Hard Task
- Main menu: 1 second, 0 errors
- "Fight player" menu: 1 second, 0 errors
- Battle option menu: 5 seconds, 0 errors
- "Write player's name" screen: 3 seconds, 0 errors
- Game level: 43 seconds, 2 errors
Totals: 53 seconds, 2 errors
Errors
Both errors for the hard task occurred during the actual running portion of the game. When starting the level, the tester did not know how to move the character, since he had not read the tutorial. He first tried holding the right arrow key, and then tried a few other keys before figuring out how to move the character. He also did not know how to do the "jump" action, trying various keys and eventually losing the level because of this. Fortunately, both of these errors will not be a problem for the actual iPhone implementation, since the controls will be far more intuitive (moving the character will simply be done by running, and "jumping" will occur when the player jumps in real life).
Discussion (15 points)
During the difficult task, the tester made two significant mistakes during the gameplay. Although both of these errors would not have happened on the iPhone, the cause of the errors is important - the game did not show the user the (unintuitive) controls for moving the character. One important change in the interface may be to remind the user, before he begins a battle, to go through the game tutorial, just to ensure that the user is not confused in any way during the gameplay.
The tester had a slight delay when looking for the message inbox, since it is not obvious at first glance that the "new message" notification icon is a button that can be clicked. It may be a good idea to change the notification icon into an icon that is clearly meant to be clicked.
Another part of the game that may need change is the sound - in particular, a loud, high-pitched sound is played when certain buttons are pushed. The tester also commented that he would prefer slightly calmer music upon logging in - the current music sounds dramatic.
The timing data I gathered was not as useful to me as the qualitative observations, since a larger sample size of testers would be optimal.
Appendices (5 points)
Consent form:
