PilotUsability-PaulIm

From CS 160 Fall 2008

Introduction

The system being evaluated is the serious game “Tamagassi”. Tamagassi is a convenient application designed for a user’s cellular phone. The purpose of the game is to try and help drivers decrease their gasoline usage on a weekly basis. Not only would this help decrease the amount of money they spend, but it would also cut gasoline emissions within our environment.

This user study was conducted in order to test a higher level of our prototype before we put it into the initial stages of production. It was a way to test the improvements we made to our original interactive prototype after the all of the feedback we had received through the heuristic evaluation. This stage tests a user’s perspective, as opposed to an expert’s; thus, we are following the testing cycle of user, expert, user, etc… We hope that this study showed us the final critical improvements that we need to make to our project.

Implementation and Improvements

Improvements

First of all, we converted our entire project from HTML/Javascript to Flash. Not only did this enable us to convert our project to a newer and better technology, but it also decreased the amount of work needed in order to make changes to our prototype. Overall, we found that Flash allowed us to create a much cleaner and more efficient interface.

Note: Because of our conversion to Flash, however, there may be some bugs here and there that we need to clean up before we deliver our final presentation.

There were a couple of changes we made to the actual functionality of our prototype as well. In the ‘Review Data’ section, we have now included a ‘Stats’ section that the user can check in order to compare their weekly data to their average, lifetime data. This was a result of some feedback indicating that there should be more data available in ‘Review Data’.

In the ‘My Items’ section, we added a couple of features that improved the affordance. The cyan selector box was added to make sure that the user knows what item he has currently selected. The ‘Press OK to equip’ dialog also helps the user understand exactly what he needs to do in order to equip a certain item.

The ‘Shopping’ section also now includes the cyan selector box in order to provide additional affordance to the user. There is also much more dialog allowing the user to see exactly where he is in the shopping process. There is dialog asking the user to confirm the purchase of an item, as well as information indicating whether or not the user has already bought the item.

Implementation

There were also a couple of additional functionalities that we implemented as well. The major addition is the implementation of the ‘Carpool’ functionality. The user now has the ability to locate, contact, and carpool with a friend who has the same route that he does. First the user selects his own route, then selects a friend with the most similar route, and then contacts the friend for confirmation. The process if fairly straightforward.

Also, the user can now begin collecting data from the main screen of the application. Once a user begins to collect data, the screen should convert back to the home screen of the user’s cell phone. At this point in time, however, the screen simply turns blank. Once the user is done driving and collecting data, the screen automatically shifts to the ‘Review Data’ screen, where he can review the route that he drove. We thought it would be obvious that a user would want to review the data for his route right after he finished driving it.

Method

Participant

The name of the participant that I selected is Aaron Shin. He is currently a fifth year, Integrated Biology student at UC Berkeley. He was selected because of a couple factors. Aaron owns his own car in Berkeley, and thus has the driving experience. He is also a somewhat passive gamer, which indicated to me that he would enjoy a not-so interactive game like Tamagassi. He is also not a computer science major, which helps me get the perspective of a non-computer science expert. Again, we wanted a typical user for this study, not an expert. With all the factors, I assumed that he would provide invaluable feedback for our project.

Apparatus

The experiment was conducted in my apartment. We used the Flash version of our game on my laptop. The conditions were ideal; there were no outside physical sensory distractions, and no timelines to adhere to. Other than the laptop, the equipment that was used was a notebook, a pen, and a clock. These pieces of equipment were enough to take down proper data.

Benchmark Tasks

The specific tasks that we had our users go through were collecting and reviewing data (easy task), character customization (medium task), and carpooling (hard task). I will now provide both the descriptions and procedures for each of these tasks:

Description of Tasks & Procedures

Collecting and Reviewing Data

In Collecting and Reviewing data, the user simply needs to press ok to begin collecting data, and then review the data afterwards. Reviewing the data consists of looking at both the map and the statistics screen. In the procedures, I explained to Aaron the general concepts of collecting and reviewing data, so that he would not surprised when the screen switched to the ‘Review Data’ screen after he was done collecting data. And thus, I asked him to pretend he was about to drive, and begin collecting some data. During this part, I looked for any errors or hesitancy in trying to figure out how to collect data. If he looked at any screen other than the home screen, it would have counted as an error. After the collecting data task, we came to the ‘Review Data’ screen. In this section, I asked Aaron to peruse the data and to think out loud while he was scanning the screen. While he was reviewing the data, I specifically looked for completeness. Did he look at every data point on the map? Did he realize that there was a separate statistics page? I noted if he ever lingered at any certain point. These were the main procedures in the Collecting and Reviewing Data task.

Character Customization

Character customization consists of buying items from the shop and then equipping them onto your character in the ‘My Items’ section. First off, I explained that to Aaron that he might run into practical issues, especially regarding the wizard of oz techniques used in this section. The procedure that I took here was fairly straight forward. I split the task up into two parts, the ‘Shopping’ part and the ‘My Items’ part. In the shopping section, I asked Aaron to buy an item for his character. If Aaron went to any screen other than the ‘Shopping’ screen, I counted it as an error. After he bought the item, I asked him to equip it onto his character. In this part, similar to the ‘Shopping’ section, I looked for any errors Aaron might make in trying to get to the ‘My Items’ screen. Once he got to the ‘My Items’ screen, I looked for any hesitancy that he showed in equipping items onto his character. Everything was fairly straightforward in this task.

Carpooling

The carpooling section is by far the hardest task out of the three. The user has to understand what it means to select and match different routes, and also how to contact their friend for confirmation. Thus, there were three parts to this procedure. In the first part, Aaron had to select the route that he wanted or create a new one if he wanted to take a new route. I asked him to select a default route out of convenience. The second part was that he needed to select the route of his friend that was the most similar to his own. And third, he needed to call his friend to confirm the carpool and begin collecting carpool data. I initially explained the concept of carpooling and the steps he needed to take in order to successfully carpool with someone else. I then asked him to complete each of these parts separately, in order to make the process more comprehensive. Again, like each of the previous tasks, I looked for hesitancy and errors. In the last section, I specifically looked for any signs of hesitancy to see if Aaron understood why the ‘Call’ button was there. These were the main procedures followed and issues looked for in the Carpooling task.

Test Measures

There were four main test measures that we looked at during the experiment:

1. The length of time for completion of the task.

2. The number of errors in each task.

3. The number of times the user showed noticeable hesitation.

4. The number of times the user reported something positive about the prototype.

The reasoning behind measuring the length of time for completion is obvious. The longer a user takes to complete a task, the worse our interface becomes. Out interface should be clear enough so that the user has little to no trouble in completing a certain task.

The number of errors per task test measure had similar reasoning behind it. The more errors that a user makes, the more unclear our interface is. Zero errors correspond to an almost-perfect interface.

Along the same lines, the number of times the user shows hesitation also indicates to us that something is confusing or unclear. Although a hesitation is not as bad as an error, we still want to minimize them in order to create a more fluid application.

On a positive note, we also recorded the good things that the user said about our prototype. The reasoning behind this was not only to see what we were doing well, but also to see if we could make similar positive adjustments to other sections of our interface. In other words, if a user complemented a certain feature in ‘Set Goals’, we could try and see if we could implement that same feature in our other sections.

Results

Data

The results varied across the different tasks:

Key:

Length of time for completion (s)
Number of errors (n)
Number of times the user showed noticeable hesitation (h)
Number of times the user reported something positive (p)

Collecting and Reviewing Data: 15s / 0e / 2h / 0p

Character Customization: 30s / 0e / 2h / 1p

Carpooling: 45s / 2e / 3h / 0p

Critical Incidents through each task

Collecting and Reviewing Data:

Not quite fully intuitive that collecting data happens at the home screen
Showed hesitation when asked how to look at weekly statistics

Character Customization:

Perused the ‘Shopping’ screen fairly slowly.
Showed hesitation when asked how to equip an item
Found that shopping and equipping items were cool add-on to the original Tomagotchi game

Carpooling:

Lots of hesitation throughout the process, but got through it okay.
Noted that there was no need for color coding the different routes if only one route shows up at a time
Did not realize why the ‘Call’ functionality appeared in the last section

Discussion

Experimental Changes

There are a couple things I would change for the real experiment. First of all, I would better explain each of our Wizard of Oz techniques. Even if we explain to the user that this is not our final product, they still expect a lot of fluidity in the application. Thus, the Wizard of Oz techniques simply caused Aaron too much hesitation and doubt as he went through the tasks. Another thing that I would change is that I would measure a couple more dependent variables. Our current data, although valuable, does not seem to capture enough information. For example, the time it takes to a user to complete a task as well as the number of hesitations a user commits are both fairly rough variables to measure. Because of the novelty of the application to the user, there will obviously be a slight drop in timed performance. It seemed as though Aaron was hesitating simply because he did not want to make a mistake and he wanted to double-check all of his actions. In this sense, I would probably add a couple more dependent variables to our current list. These may be things like a user confidence rating in response to certain tasks, or an annoyance rating as the user tries to fiddle with our application

Interface Changes

In terms of the interface itself, there are still a couple of changes that we need to make. From my study, there does not seem to be anything major outside of the Wizard of Oz techniques. There are a couple clarification issues with Carpooling, such as cleaning up the map routes as well as somehow getting the user to understand the ‘Call’ functionality a little better. This was expected, however, due to the fact that Carpooling is one of the more recently implemented functionalities of our application. In the ‘Shopping’ screen, there needs to be better feedback on which item you have currently selected to buy. And lastly, we may need to make the ‘Stats’ tab a little more visible in the ‘Review Data’ section. Most of these seem to be minor, cosmetic change to our interface. Also, on a positive note, Aaron mentioned that buying and equipping items was a good add-on to the original Tomagotchi game. Thus, perhaps there can be more we can do with our characters (e.g. feeding it). This however, will not be implemented in this prototype. These are the changes that my pilot experiment directed me towards making.

Appendices

Consent Form

Image:ConsentForm.doc

Demo Script

Script with Task Prompts

Raw Data