PilotStudy-Group:一三三七-SaliemThan
From CS 160 Fall 2008
Contents |
Introduction (5 points)
Introduce the system being evaluated (1 paragraph)
Fit Wars is a mobile application designed to help college aged users find it easier to stay in shape. It takes advantage of a novel popular piece of device in the age group: the iPhone combined with a college age pervasive activity (gaming) to achieve user fitness. The user can opt to play a given game course, but running against a computer or another actual player. The user can also transfer the points accrued to another particular game if wished, or message another player via the messaging aspect of the game. Fit Wars is an attempt to combine these two elements into a feasibly playable and enjoyable game. It has gone through three iterations so far, one via a paper prototype, and two via a Flash based prototype. Each successive prototype has been changed according to the results obtained from each successive testing. Most of the changes so far are fairly minor
State the purpose and rationale of the experiment (1 paragraph)
The purpose of the experiment is to determine how the application as it is currently designed can be improved. By measuring the time it takes to complete a task and by noting user suggestions for the application the design faults can be pinpointed an modified as needed. Some time and thought is also given to the initial assumptions and the overall design logic of the game. This way any particular pressing issues discovered at this point can be taken as a last iteration phase before the actual building of the actual application.
Implementation and Improvements (15 points)
Describe all the functionality you have implemented and/or improved since submitting your interactive prototype (1 page). <i>We will expect you to describe all the functionality your have implemented and/or improved since submitting your interactive prototype.
- When registering for the game, the user name, password email text boxes borders light up green when active
- Game settings section implemented
- calibrate walking
- options for various audio aspects of the game
- options for casual mode or not
- Manage Stats page added in
- stats, equipment, distance ranking compared with other users
- During game play - added in indicators as to what kind of game course it was and what the state status was (fighting, running etc.)
- During game play - added in a thick red border to the game progress indicator at the bottom
- Clicking on the avatar in the menu menu takes the user into the manage stats menu
Method (10 points)
Participant (who -- demographics -- and how were they selected) (1 paragraph)You should get the participants to sign an informed consent form and obtain other demographic information (e.g., age, sex, education level, major, experience with your type of tasks and application, etc.)
participant -- recent college graduate, male, plant biology major, familiar with smart phones including the iPhone and Blackberry, plays video games moderately with friends, including wii fit and xbox 360. interested in staying fit, plays tennis and runs regularly on a weekly basis. open to the idea of playing video type games combined with exercise. Apparatus (describe the equipment you used and where) (1 paragraph) laptop computer with access to the Internet and a browser. at a public cafe. watch for timing.
Tasks (1/2 page)
[you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed 1 easy task, 1 medium task, and 1 difficult task
- register and create a game account
- send a message to a friend
- calibrate and play the game
- looked to see how long it took to complete each task, looked to see where the user hesitated in between mouse clicks. Listened to user suggestions and thoughts for improvements in the interface and logic of the application
Procedure (1 page)
describe what you did and how You will give the participant a short demo of the system. Do not show them exactly how to perform your tasks. Just show how the system works in general and give an example of something specific that is different enough from your benchmark tasks. You should write-up a script of your demo and follow the same script with each participant.
I'm actually pretty hesitant of following a script even though that might lend way to more consistent accurate results. I think however that following a script might make the testing too formalized to the demise of the testing results: the testing might fall into the interviewer/interviewee mode and that is very undesirable as it might lead to the participant believing that there is something to be expected out of him or her. This might draw his/her attention away from actually using the game and he/she might focus instead to much on the idea that he/she is "testing" the application. Essentially I believe this adds an unmeasurable variable that one has to take into account. In retrospect this might just be the same as not following a script script. The unmeasurable variables resulting from this method might as well be the same as resulting from the highly scripted testing.
- introduced the basic idea of the game: iPhone application designed to improve user fitness
asked the user to register/create an account. watched listened and timed the user. talked with the user when various ideas and topics came up. did not tell how to do things but did confirm user questions as to what to do next.
- asked the user to send a message to a friend. discussed with the user the logic of sending a message on the iPhone application or whatever topic the user had in mind at this point
asked the user to calibrate and play the game. told user that to progress during game play, the right and left arrows had to be pushed simultaneously to simulate running and that the space bar had to be pressed to simulate taking an action when asked to "jump" or "kick" or such. noted to the participant that that was the end of the tasks and if he had any more questions or thoughts about the design. Thanked him for his time and help.
- designed to make the time with the participant as casual as possible allowing room for and encouraging lots of out-loud thinking. (for example, didn't instruct anything, just explained or discussed/conversed with an aim with the participant) late notice of the application testing allowed for little preponderance of the game application design beforehand (the participant working in the tech sector might ponder about it a little too much) and more natural reactions to the application design
Test Measures (5 points)
Describe what you measured and why (1/2 page) concentrate on process data. For example, you should instruct your participant to think aloud. You should make a log of critical incidents (both positive and negative events). For example, the user might make a mistake and you notice it or they might see something they like and say "cool". Set up a clock that only the observers can see (one or more of you should observe), and write down a log containing the time and what happened at that time when a critical incident occurred.measure some important dependent variables to get a feel for how it is done (i.e., task time, # of errors, etc.)
- Measured the amount of time it took for each task to be completed
- Also noted the times of specific parts of the tasks, and the specific parts themselves that took longest to complete. This is done with the assumption that the user wanted to get to the game play as soon as possible and that the amount of time it takes to do specific activities, like register for the game, ought to take the least amount of time possible.
- Took down user suggestions/remarks for the application logic and design that came up as the task was being undertaken. User suggestions taken and noted to improve the game application design and to be categorized under heuristic design principles
- Noted any mistakes made in each task. These are mistakes made on the part of the user or the application. Mistake is also meant to be understood as confusion or delay.
- Noted where the user was coming from before beginning the application testing and the time of the day. This variable might affect the cognitive process of the user. If the user is tired then it might take him a few more seconds longer to do a task. If the user is awake, less seconds. This should definitely be taken into account.
Results (10 points)
Results of the tests (1 page) You must report your results (values of dependent variables, summary statistics, and summaries of the process data)
Data Notes: - user came back from a weekend trip with a friend. was tired and drinking a lot of coffee. - took a long time to register. didn't know register was a button? not enough feedback for picking the avatar - expected to see information about the avatar to show up - took the longest time with this task. thought it would be a good idea to see information about each avatar as the avatar was moused over/clicked that was on display to be chosen (1.2 minutes, 0 mistakes)
- in messaging a friend, saw that there was a new message, but didn't see any indications of a new message in the inbox. thought that it might be simpler to just have or additionally have a button that takes him straight to the inbox. Also wondered if there could be a list of friends online to message. Wondered if it would be easier to just text a friend to ask them if wanted to play the online game instead of using the in game messaging system. Also wondered if he was suppose to know the user name of the friend beforehand, if he could just use their email address etc. Went back and forth to and from the main menu looking for the "new message". Wondered what the boxes with the 'x's were, when told what they were, thought that they should be trashcans instead (3.65 minutes, 1 mistakes)
- spent the least time playing the game. Game was pretty straightforward. Jokingly noted that the game was a lot like street fighter (3.13 minutes, 0 mistakes)
- calibrating the game - thought that was pretty straightforward. Mentioned that there was a iPhone application game where if you tilt the phone, it would effect the path of a ball that was traveling through a complicated maze, you had to calibrate for that game too. (2.3 minutes, 0 mistakes)
- user found the playing the actual game (running and using the iPhone) questionably fun and doable.
Discussion (15 points)
What you learned from the pilot run (1 page) what you might change for the "real" experiment? what you might change in your interface from these results alone? If you'd like, you may include results and assumptions from other group members' tests here as well.in the "Discussion" section you should draw some conclusions with respect to your interface prototype. You should also say how your system should change if those results hold with a larger user population. This should be the most important part of the write-up. We want to understand how you would fix your system as a result of what you observed.
Learned that a lot of improvements could still be made for the prototype. From the results of the first task, learned that perhaps the word register might work better if it looked like a button that is grayed out and then lit up red or some other bright color when an avatar is chosen. Perhaps also details of the avatar could be shown as an overlay with the background faded out when the avatar is clicked. The overlay could include a button for picking a different avatar, and a button for confirming the current avatar selection.
From the second task: The application can be changed to include an inbox button instead of a new messages button. perhaps the inbox button can also be grayed out when there aren't any messages, and a bright color with a new message icon indicator if there is. In the inbox, the delete icons should be turned into trashcan icons or red delete buttons. The message form could also be made to accept a user's telephone number, or email or user name. If the message is sent to a telephone number of an email, an additional message can be attached giving the message recipient instructions on how to install and begin playing the game. Also at the bottom of the message screen could be a screen listing other users familiar to the user who are currently online and currently playing the game.
From the last task, perhaps the option to change the type of course being played and the option to change the theme of the characters can be added in during game play. That would allow for a lot of flexibility so that the user doesn't have to feel stuck running the same course to completion.
Overall the participant in the study did find the idea of actually running and playing the game questionably fun and doable. Might have to rethink the game and perhaps make it purely audible and in sync with actual online/video games with actual online/other running players. Might have to rethink the actions of the game, Perhaps use GPS and maps to chart actual courses and use the type of courses and elevations run as goals to be achieved that would advance a user through a given game. Should perhaps do away with the transfer points aspect and perhaps set the preferences as to which game is to be played and which game the points ought to accrue to early on in the game, perhaps when first registering for the game.
Appendices (5 points)
Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
I. Demo Script
introduced the basic idea of the game: iPhone application designed to improve user fitness asked the user to register/create an account. watched listened and timed the user. talked with the user when various ideas and topics came up. did not tell how to do things but did confirm user questions as to what to do next.
asked the user to send a message to a friend. discussed with the user the logic of sending a message on the iPhone application or whatever topic the user had in mind at this point asked the user to calibrate and play the game. told user that to progress during game play, the right and left arrows had to be pushed simultaneously to simulate running and that the space bar had to be pressed to simulate taking an action when asked to "jump" or "kick" or such. noted to the participant that that was the end of the tasks and if he had any more questions or thoughts about the design. Thanked him for his time and help.
designed to make the time with the participant as casual as possible allowing room for and encouraging lots of out-loud thinking. (for example, didn't instruct anything, just explained or discussed/conversed with an aim with the participant) late notice of the application testing allowed for little preponderance of the game application design beforehand (the participant working in the tech sector might ponder about it a little too much) and more natural reactions to the application design
Raw data (i.e., entire merged critical incident logs)
II. Data Notes: - user came back from a weekend trip with a friend. was tired and drinking a lot of coffee. - took a long time to register. didn't know register was a button? not enough feedback for picking the avatar - expected to see information about the avatar to show up - took the longest time with this task. thought it would be a good idea to see information about each avatar as the avatar was moused over/clicked that was on display to be chosen (1.2 minutes, 0 mistakes) - in messaging a friend, saw that there was a new message, but didn't see any indications of a new message in the inbox. thought that it might be simpler to just have or additionally have a button that takes him straight to the inbox. Also wondered if there could be a list of friends online to message. Wondered if it would be easier to just text a friend to ask them if wanted to play the online game instead of using the in game messaging system. Also wondered if he was suppose to know the user name of the friend beforehand, if he could just use their email address etc. Went back and forth to and from the main menu looking for the "new message". Wondered what the boxes with the 'x's were, when told what they were, thought that they should be trashcans instead (3.65 minutes, 0 mistakes) - spent the least time playing the game. Game was pretty straightforward. Jokingly noted that the game was a lot like street fighter (3.13 minutes, 0 mistakes) - calibrating the game - thought that was pretty straightforward. Mentioned that there was a iPhone application game where if you tilt the phone, it would effect the path of a ball that was traveling through a complicated maze, you had to calibrate for that game too. (2.3 minutes, 0 mistakes) - user found the playing the actual game (running and using the iPhone) questionably fun and doable.
