PilotUsability-KumarGarapaty

From CS 160 Fall 2008

Contents 1 Introduction 2 Implementation and Improvements 3 Method 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.3.1 Demonstration Task – Setting Goals 3.3.2 Easy Task - Collecting/Reviewing Data 3.3.3 Moderate Task – Character Customization 3.3.4 Difficult task – Carpooling 3.4 Procedure 4 Test Measures 5 Results 5.1 Task Measures for each Task 5.2 Collecting/Reviewing Data 5.3 Character Customization 5.4 Carpooling 6 Discussion (15 points) 7 Appendices (5 points)

Introduction

Tamagassi is a serious mobile game that is meant to lower the user's consumption of gasoline so as to save money as well as help improve the environment. We do this through two methods. One method is to test how much the user accelerates or brakes through an accelerometer. Excessive acceleration leads to a waste of gasoline, which could be reduced through this game. The second method is to allow for carpooling, so that the user can see whether his or her friends are taking a similar route at that time. If they carpool together, this can also help them save gasoline. The game aspect of the application is implemented through the creature that the individual takes care of. As the individual drives better and carpools more often, the creature develops over time.

The purpose of the experiment is to test our interactive prototype before it is tested with a broader range of people. The experiment will allow us to identify mistakes made in our prototype so that we can analyze them appropriately and implement the correct solution. The experiment will also allow us to figure out how much time each task will take.

Implementation and Improvements

The program was taken from an html based application to a flash based application so as to improve the overall appeal of the application through better graphics. This way our buttons, options, combo boxes look more professional.

The collecting data button, actually takes the user to a collecting data screen, where the user can then choose to stop collecting data. The user is forwarded to a reviewing data screen, once the user chooses to stop collecting the data.

For set goal, we added more feedback asking the user whether they were actually sure about that number. This way the user can type in a number and make no accidental mistakes in giving the wrong goals. We made it so that when the user enters in their goals, they can actually set the correct goals instead of hard coding it as we did in the earlier prototype. We still want to disable the ability to change goal after the user sets it for that week because goals are not meant to be instantaneously changed but something to be pursued.

For review data, we added a general statistics frame that gives the statistics for this week and in general. When clicking on the different spots that the user has driven badly is also done in one frame, so that the page does not reload and take up the user's time.

For shopping, we added other apparel than just hats so that the user has more options on what to buy. We also added a scroll bar and a selector so that this task can be performed with both touch screen and keypad interfaces.

For my items, more items were available for equipping and de-equipping than just the wizard hat, reducing the wizard of oz techniques used in our application. The equipping and de-equipping all happens in one frame, making the process of the task a lot faster and easier to perform.

Carpooling was actually implemented in our updated interactive prototype. It allows the user to select among saved routes. The user can then edit these routes, select a friend who is driving a similar route and then choose to carpool with that individual. Wizard of oz technique was used for displaying the maps for similar routes as well as the friends who had the same route as the user.

Overall, we reduced many wizard of oz techniques and improved the appearance of our application. However, other wizard of oz techniques can still be reduced and the prototype still needs many improvements. The user interface design can also be improved to be convenient for both touch and keypad devices.

Method

Participant

The participant is a nineteen year old sophomore, who is currently enrolled in the University of Berkeley: California as a physics major. He has played games of many types in the past so he is familiar with all formats that most games are designed in. He does use a car, but only occasionally. However, he does intend to use one regularly after graduation. The participant is highly interested in saving gasoline not just for personal financial reasons but also environmental reasons.

Apparatus

The participant used a laptop with a Macromedia Flash Player running. The screen of the interactive prototype is similar to the screen of a cellphone so as to make it look like the participant is actually testing on a cellphone screen. However, the participant still uses the keys and mouse of a computer to operate the application.

Tasks

The primary tasks associated with our game are as follows: (1) setting goals, (2) collecting data, (3) reviewing data, (4) demonstrating good driving behavior, (5) carpooling, and (6) customizing the character. Although we were only asked to test the user with three benchmark tasks, we felt that the diversity of features implemented in our pilot usability required more than three tasks to be tested.

Demonstration Task – Setting Goals

Setting goals was selected as our demonstration task for two reasons. First, although we listed this as a moderately difficult task, this was largely based on the fact that the user would have difficulty setting both effective and realistic goals; the user interface required for it is actually rather simple. Secondly, setting driving goals and meeting them is central to the purpose of our serious game, so we hoped that by introducing this as our demonstration task it would help the user better understand the context in which other game elements were used.

Easy Task - Collecting/Reviewing Data

Collecting data is a task to be performed shortly before the user begins to drive. It involves pressing [OK] from the main screen. This jumps to the normal home screen of the phone, with a small bar at the top showing, "Currently collecting data. Press [STOP] to stop collecting data." This allows the data collection to run in the background, leaving the phone free for other functions. Following this, the user is shown the "Review Data" screen in the game. The level of interaction required for this task was deliberately set to be minimal, as users may often be in a rush during this phase of the game. We felt it important to test this aspect of the user interface first to ascertain whether it was as straightforward as the designers assumed, and secondly because it is a crucial, though easy, in-game task.

Originally, reviewing data was considered a separate task. However, our user interface is designed such that there are two ways to review the data - the user can choose to review the data either following data collection, at the conclusion of their drive, or to return to the game (and the review data screen) at a later time at his or her leisure. Because ending data collection would jump to the review data screen, the interview seemed to run much more smoothly if we allowed the interviewee to examine the collected data immediately after it was collected, instead of asking them to exit the device and return to the review data screen at a separate time.

Moderate Task – Character Customization

Items can be purchased from the store using "cash" acquired by minimizing acceleration. The store can be accessed from the main menu, where the amount of cash is visible in the upper left hand corner. The different types of items (hats, scarves, boots, etc.) can be accessed by scrolling through the menu using the up and down arrows, and the specific hat or scarf desired can be selected by scrolling left or right once a particular type of item has been selected. These items are saved to the inventory. To dress the character, the interface is similar, but with only one level to the menu - hats are sorted linearly, one after the other, after which there are boots, then eye wear. The user selects an accoutrement for his or her character by clicking [OK] once the cursor is over the desired item.

Difficult task – Carpooling

Of the difficult tasks, carpooling was the obvious choice, since the driving component of the game involves virtually no user interface (except for the collecting data aspect described above). Carpooling consists of two phases. First, the user must determine with whom carpooling is ideal. The ideal procedure is to enter the carpool menu, select a route, and if the user chooses to do so, edit it. The routes uploaded by other users are then displayed in order of closest match, and the user (A) can select the name of the other user (B) of choice. This returns a new screen that allows the user to either call or request confirmation. Because the other user (B) has not yet been notified of user A’s intentions to carpool, the user should click “Call”. This results in a standard cell phone call. The user can also click on start collecting data from the screen once the user is actually carpooling with user (B).

Procedure

The procedure began with the interviewer expressing thanks for their willingness to participate, then giving the interviewee the context for the experiment. The user was then asked to sign a consent form, after which the goal of our serious game design was described in greater detail.

This was followed by the demonstration task, in which the facilitator instructed the user on an elementary task in the game, setting up driving goals for the week. This allowed the facilitator to provide a context in which the rest of the game tasks could be described.

Each task was introduced with a brief description of why a user would want to do it (in essence, describing the storyboard such that the interviewee would understand what he or she would be doing when he or she chose to do a particular task). At each screen, the interviewer would ask the user to pause and remind him or her to verbalize their thoughts and opinions about that particular screen. The interviewer refrained from asking questions about erroneous uses of the interface until after the interviewee had finished each task. In some cases, when certain features of our user interface were left unexplored, the interviewer would lead the user, asking them if they noticed anything in particular about the user interface or why certain features were labeled in a certain way.

Since the interviewer had no observers along with him, he had to take notes while he was interviewing, which could have interfered in the interview process. The interviewee, however, was not aware of this problem and seemed to perform normally through the process.

Test Measures

Done as a group

Results

Task Measures for each Task

(s = seconds, e = errors for the task, h=hesitancy, p=positive statement):

Collecting Data: 15s/0e/0h/0p

Reviewing Data: 65s/4e/3h/1p

Carpooling: 160s/4e/4h/0p

Customizing/Shopping: 210s/3e/2h/1p

Collecting/Reviewing Data

- Thought the ideal information on the map was related to the set goals task (3).

- No information about the start and stop of the drive even though it has the map of the route (2).

- Confused about what “All Time” meant in the Weekly Statistics page (2).

- Wanted to find information on how much gas they used but couldn't find it (3).

- Positive statement – enjoyed the pictures of the character, thought it enhanced the game aspect and made it more enjoyable.

Character Customization

- After equipping the character, it does not show up on the home screen that the character actually equipped it (3).

- Tried to click on the other apparel item without clicking on the scroll button (3).

- Wanted the shopping and my items screens to be more interconnected (4).

- Positive statement – liked the fact that he could access the different tasks from any part of the game through the pop up menu.

Carpooling

- Was confused about the click to add routes that were placed right after the route A and route B (3).

- Thought that the way the routes were organized was rather confusing (4).

- Did not understand the select friend to carpool with frame, where the user thought that all the similar routes would be shown in one frame. Later realized that each similar route was an option in the combo box (3).

- Did not understand the ranking system (3).

Discussion (15 points)

For reviewing data, the data given should be more related to the route that you are actually driving by stating what route it is. This allows the user to relate more to the actual events that happen across the route. The statistics given should be more clear so as to actually benefit the user and allow them to see how and in what areas they can improve. These features can be improved from the results alone, since they are rather easy to implement.

For character customization, after the character is equipped, it should become truly equipped and stored in a database that the character is currently equipped with that item. The design of the my items should be that multiple items can be equipped. These features can be improved from the results alone, since they are rather easy to implement.

For carpooling, the route organization currently violates many heuristics such as visibility of system status. The ranking system could be made more clear to the user, through other methods other than the use of a combo box. These features can be improved from the results alone, since they are rather easy to implement.

Throughout the pilot run, I learned that the game needs to be more fluid. Although, we built all the separate tasks, they are not interwoven correctly to make it a smoothly flowing and understandable game. For example, setting goals had no relation to the reviewing data, it should say that you have driven this much of your goal so far this week. Another example, is what is bought in the shop is removed from the shop and shows up in my items. Implementing features like these can allow the game to be more fluid and give more of an impact to the user. We should implement features such as these for the real experiment. A points system should also be added so that this game will become more long term, and the character can develop and level up as the user achieves their goals. This change is necessary to implement from the results alone because it is a central part of making an actual game fully functional.

Reducing wizard of oz techniques should be an important factor to consider because errors occurring due to the usage of these techniques can lead the interviewee to focus on those problems than actual UI problems. This should be done for the “real” experiment as well.

Appendices (5 points)

Demo script in group page

Raw Data:

Interviewee

Berkeley Undergraduate physics major

Has played many different types of games

Interested in many Japanese animes

Collecting/Reviewing Data:

1.Tried to find where it shows how much gas he used

2.Went to stats immediately

3.Spent some time trying to understand the statistics

4.Went to map again, and clicked on the numbers

5.Thought the ideal acceleration was related to the goals

Time: ~80 sec

Customization:

1.Went to Shopping to buy items

2.Confused about some of the images, did not look like actual boots

3.Liked the pop up so he could go to the my items instantaneously

4.Didn't have any problem buying the items

5.Equipped a hat onto the avatar, selected a bunch of other items, could only add one item at a time

6.Tried to click on pictures instead of scroll bar, which did not work

7.Went to Char Home and was disappointed that it did not show that the character were actually equipped

Time: ~160 sec

Carpooling:

1.Clicked on add a route from the combo box, confused about what that actually meant when it showed the new page

2.Went back and clicked route A, which actually had more form information

3.Understood what the form actually meant after that

4.Confused about why it doesn't show the routes of all the friends in one map, took some time to realize that the combo box allows the user to choose a particular friend to see their route and that it is ranked through the combo box.

Time: ~210 sec