PilotUsability-WendaZhao
From CS 160 Fall 2008
Contents |
Introduction (5 points)
The application is a serious game on the cell phone that encourage players to save gas usage. Each user will pick a character in the beginning, its mood will be reflected on players' driving behavior. And player will also be rewarded for cash if they met their driving goals (driving less than last week, carpool three times a week). Then player can use those cash to shop items for their character.
The modifications for the interactive prototype are based on the feedback from the participants and heuristic evaluations from the other group. The purpose of the experiment is to test the new interactive prototype. The experiment can help identify more usability problems in the current prototype.
Implementation and Improvements(15 points)
The entire application was moved from HTML to Flash for two reasons. First of all, Flash is a better fit for the application because Flash is more powerful and it can make a better looking interface. Second, a lot of the functions from our first prototype on HTML are hard coded due to the time pressure and the limitation of HTML. It will take more time to fix those than start our prototype from sketch in Flash. In general, A huge amount of changes were made since our first prototype. In big picture, the prototype is moved to Flash. All the functions and user input are now more responsive and give more feed backs to the users. None of the functions are hard coded anymore. One of the hard tasks carpooling were implemented. Each task was also modified. For set goal task, when users tried to input the numbers for their goal and hit confirm button, a confirmation text displaying "Are you sure to __ ? Yes No" was added. If they choose yes, the goal will be set and nothing happens if they choose "No". All the numbers to be set are from users this time, it is not hard coded to a fix number anymore. For the Reviewing data task, it only has a map view for our first prototype. a extra page to display the usage of the gas and other relevant stats was added. So the page has two tabs which are "Map" and "Stat" now. Users can choose either one. For "Shopping" and "My Items", they are not hard coded anymore also. Now users can actually buy the items and the money will be deduct to the amount accordingly and then users can equip items in the "My Items" page.
Method (10 points)
Participant's name is Stepan Grigoryan. He is a Computer Science major student from Berkeley. I met him in one of the computer science class last semester. I explained to him what I was doing for our project in CS 160 long time ago. He was very interested in our project and wanted to be participating our low-fi prototype evaluation. He also participate our low-fi prototype evaluation.
The experiment was conducted in one of the study rooms at Moffit Library. The equipments were a personal computer, a notebook, a pen and a watch.
The demo task was Setting Goals: Setting goals was selected as our demonstration task for two reasons. First, although we listed this as a moderately difficult task, this was largely based on the fact that the user would have difficulty setting both effective and realistic goals; the user interface required for it is actually rather simple. Secondly, setting driving goals and meeting them is central to the purpose of our serious game, so we hoped that by introducing this as our demonstration task it would help the user better understand the context in which other game elements were used
Tasks: Character Set-up The character set-up should be fairly intuitive, as it is the first menu that appears when the game is played for the first time or after the character dies or is reset. The user uses the arrow keys to navigate through the different types of characters he or she can choose from, then colors, and then uses the keypad to name his or her character. Each screen shows up sequentially after data has been inputted to the previous screen.
Collecting/Reviewing Data
Collecting data is a task to be performed shortly before the user begins to drive. It involves pressing [OK] from the main screen. This jumps to the normal home screen of the phone, with a small bar at the top showing, "Currently collecting data. Press [END]x2 to stop." This allows the data collection to run in the background, leaving the phone free for other functions. Following this, the user is shown the "Review Data" screen in the game. The level of interaction required for this task was deliberately set to be minimal, as users may often be in a rush during this phase of the game. We felt it important to test this aspect of the user interface first to ascertain whether it was as straightforward as the designers assumed, and secondly because it is a crucial, though easy, in-game task.
Character Customization
Items can be purchased from the store using "cash" acquired by minimizing acceleration. The store can be accessed from the main menu, where the amount of cash is visible in the upper left hand corner. The different types of items (hats, scarves, boots, etc.) can be accessed by scrolling through the menu using the up and down arrows, and the specific hat or scarf desired can be selected by scrolling left or right once a particular type of item has been selected.
These items are saved to the inventory. To dress the character, the interface is similar, but with only one level to the menu - hats are sorted linearly, one after the other, after which there are scarves, then boots. The user selects an accoutrement for his or her character by clicking [OK] once the cursor is over the desired item.
Carpooling
Carpooling consists of two phases. First, the user must determine with whom carpooling is ideal. The ideal procedure is to enter the carpool menu, select a route, and if the user chooses to do so, edit it. The routes uploaded by other users are then displayed in order of closest match, and the user (A) can select the name of the other user (B) of choice. This returns a new screen that allows the user to either call or request confirmation. Because the other user (B) has not yet been notified of user A’s intentions to carpool, the user should click “Call”. This results in a standard cell phone call.
Secondly, the user must be able to receive cash points associated with carpooling. To do so, he or she must go through the process of selecting a route described above, then the person with whom he or she would like to carpool with. Instead of clicking the “Call” button, however, the user must click “Request Carpool Confirmation”. The other user (B) will receive an automated voice message asking to confirm that they are indeed carpooling with user A by pressing 1. This helps prevent cheating and encourage carpooling through the Law of Reciprocity and the Law of Conformity, as both users must agree to carpool before any benefits can be reaped.
I was recording the time for participant to finish each task and number of errors he made for each task.
The procedure of the experiment was similar to our low-fi prototype experiment. I first introduced myself and read the script to the participant. And then I asked him if he had any questions. I started our prototype on the computer and showed him the demo task Setting Goals. Next I asked him to perform three of the tasks. I sit next to the participant record all the measurements while he was trying to complete the tasks. Once he is done with one task, I would tell him what is the next task. Once he completed all tasks, I asked him if he had any thoughts on our new prototype. And at last, I thanked him for participating the experiment.
Test Measures (5 points)
Time it took participant to finish each task and the number of errors he made for each task were measured. The same two variables were measured for our low-fi prototype. The time it takes participant to do each task and number of errors are very useful data because it can tell the user interface is not too good if it takes too long for them to do it or makes too many errors. The team can also compare the average time it takes users to finish each task and average errors they make for each task. This data can tell the dev team a lot about whether the new interface is improving or not. The process hesitancy and positive statement were also been recorded.
Results (10 points)
Dependent variable
(s = seconds, e = errors for the task, h=hesitancy, p=positive statement):
- Collecting Data: 15s/0e/0h/0p
- Reviewing Data: 76s/0e/1h/1p
- Carpooling: 200s/0e/3h/0p
- Customizing/Shopping: 140s/0e/2h/1p
Critical incident logs
- Collecting/Reviewing Data
- Not sure about what ideal speed is about
- Liked the stats page. Think it is very informative
Character Customization
- Not sure where to equip the items
- Think buying and equip page should be more consist
- Think that it is cute when equiped with hat
Carpooling
- Confused about how to choose the best route
- Did not understood what different color for different routes meant
- It is too complicated to carpool, should be easier
Discussion (15 points)
The results were showing that the new prototype was improved in terms of the dependent variables. Both the time it took the participant to finish each task and number of errors made dropped comparing to our first low-fi prototype for Collecting/Reviewing Data and Carpooling. The Character Customization took around the time. Part of the reason is that the participant participated our low-fi prototype experiment so he was familiar with the application. But also it is due to our changes to the prototype based on the feedback from the participants and heuristic evaluation. On the other hand, some of problems were also revealed after the experiment. First of all, on the review data page, the information on ideal speed and ideal acc are not too clear to the user. The way to fix this is to have a help mark on the top right hand corner of the text bubble. So when they are confused about the info displayed on the text bubble, they can just click on the question mark and get then an text box should pop up to explain what are the ideal speed and acc are about. The other big problem for them is that user is confused where to equip item after they bought the item. Our design require users to get to the menu and select on "My items" to equip. The other participants also had the similar problem with this. I think we design this way because it is constraint by the size of the cell phone screen so we do not have too much room for both buying and equipping. One solution we can try is that have a option saying "Buy and Equip" so they can buy and equip at the same time and immediate see it on the main screen.
Appendices (5 points)
Interviewee Stepan Grigoryan
Berkeley Undergraduate computer science major
