PilotUsabilityStudy-ShyamVijayakumar
From CS 160 Fall 2008
Contents |
Introduction
The system being evaluated is intended to represent the user interface of a game implemented for a touch screen cell phone. As the user being tested will see it, the user interface will be in the form of a flash application appearing on a laptop computer. The user interface is for a game that helps its users save gasoline usage. It takes advantage of the cell phone's capabilities such as the accelerometer and GPS tracking to keep track of the user's driving habits. These driving habits will be directly reflected upon the mood and health of a character that the user has already set up.
At this point, the user interface has gone through considerable testing (low-fidelity testing and heuristic evaluation) and my group has done its best to try to fix some of the problems that have already arisen. The purpose of this pilot usability study is to find more problems that need fixing or places for further improvement in the user interface of our game. Also, the solutions that we came up for problems in the past might not work. Thus, this study also helps us test the solutions we came up with. Finally, if were to test this improved interactive prototype with a larger pool of participants, this study might help us redesign our experiment.
Implementation and Improvements
Implementation
The biggest addition in our implementation is the user interface that enables the user to take part in the carpooling part of our game. This part of the interface allows the user to see who he/she can carpool with for routes that can be selected or added.
Next, we added the user interface for collecting data. This enables the user to start collecting data when about to go on a drive and stop collecting data when the user has arrived at the destination.
Improvements
A major improvement in the current version of our prototype is that it is implemented in FLASH instead of HTML. The older version did not have much detail in the way because HTML simply did not allow it. We couldn't make the buttons look clickable or images selected, for example, because it was difficult to do it with HTML. With FLASH, however, we were able to use buttons that looked clickable and add a selector box around images quite easily. Thus, level of detail in our user interface prototype improved over all.
In our user interface for setting goals, we added a confirmation question before accepting the goal from the user. This makes sure that the user really meant to set the goal instead of accidentally typing in an incorrent number.
In our user interface for shopping, we added a selector box for the different types of accessories that the user can by. For example, if the user has chosen to look at all the hats that can be bought, then the hats button will be selected by the selector box. We also prevented users from buying the same item twice by showing an error message. Finally, we added arrows above and below the list of categories of items to indicate to the user that he/she can scroll up or down to find more categories.
In our user interface for showing all the items that the user has bought, we also added a selector to indicate which item the user is currently looking at. Arrows are placed above and below the list of items to indicate that the user can scroll up or down to view more items. We also displayed the number of items that the user currently owns. Finally, for a selected item, we show a "Press OK to equip" or "Press OK to de-equip" so that the user immediately knows what to do when deciding to choose an item.
In our user interface for reviewing data, we added another page showing statistics for the current week, which include miles driven, average acceleration and average speed. This "Stats" page will be an option that the user can choose along with the "Map" option. The option of the user can again be seen by looking at where the selector box is.
Method
Participant
The participant is a 21 year-old male, who is an undergraduate student at U.C. Berkeley majoring in Computer Science. He was familiar with the concept of Tamagatchi, which helped in understanding the way our game works in general. He drives a car at Berkeley and is interested in saving his gasoline usage. The user was chosen at random from one of my classes.
Apparatus
The user interface appears on a laptop computer. Specifically, the user interface appears in a Flash application running on the laptop computer. The Flash application represents the cell phone that the user would have interacted with in real life. The user has to click on the various parts of the user interface as if he/she was touching the screen of the cell phone.
Tasks
Setting Goals (Demonstrated Task): Users need to set goals for themselves that help them save their gasoline usage. If users were not able to do this, there would no motivation in the context of the game for them to drive less or slower. It involves users setting a certain distance of driving and also a certain number of times a week that they must carpool.
Collecting/Reviewing Data: Users need to be able to collect and review data on their driving habits. It involves users choosing a certain option to start collecting data right before they start driving and choosing another option to stop collecting data once they have arrived at their destination. The cell phone’s built-in accelerometer and GPS tracking capabilities allow us to record data on their acceleration and the distance they drive. Once collecting data is done, users can choose locations from a map highlighted with streets that they accelerated too fast. The chosen locations will enable them to see statistics about exactly how fast they were driving and what the ideal statistics should be. This way, users can make adjustments when they arrive at that chosen location in real life. Collecting and reviewing data is relatively easy for users to do because it does not require the user to know that much about the controls of the user interface besides how the main menu works.
Character Customization: For this task, users obtain cash for meeting goals that they have set. With this cash, they can buy accessories for their characters such as hats and boots. Then the user views the list of items that he/she currently owns and can choose which items to equip or dequip the character. This is a medium difficulty task because it requires the user to know the basic controls of the user interface such as scrolling a list of items and going to multiple pages in our user interface.
Carpooling: Users can choose routes that have already traveled in or add new routes to see who among their contacts travels along the most similar routes. This allows the user to see how he/she can carpool with the easiest.
Procedure
The person being interviewed was given a consent form to sign before any interviewing was actually done. Once the interviewee has read and signed the consent form, I provided background information about the game, such as what the goal of it was and on what technology it is implemented in. Then, I went on to demonstrate the "set goals" task, while thinking aloud at the same time.
Once I finished demonstrating this task, I proceeded to describe what the first task that the user should complete is (but now how to do it). Then I asked the interviewee to complete the first task, encouraging him to think aloud while doing it. This was done for second and third task as well. Finally, I asked the interviewee about any overall impressions that he had and about any aspects of the user interface that still need some improvement.
Test Measures
1. Amount of time for completion of each task.
2. Number of mistakes that the user made while doing each task.
3. Number of times that the user hesitated while doing each task.
4. Number of times that the user said something positive about the prototype.
The amount of time for completion of each task shows us which tasks were harder to do than others. This is simply because harder tasks take longer to do and hence, we should focus on making those harder tasks easier through improvements in our user interface. The number of mistakes a user makes is a direct result of problems in our user interface. Looking at the origin of these mistakes can help us see these problems and fix them. The number of hesitations a user makes is a result of smaller problems in our user interface. Looking at the origin of these hesitations can help us see these smaller problems and fix them. The number of positive things that the user reported can help us see what the user liked and can help us add those positives somewhere else in the user interface.
Results
Test Measures
(s = seconds, e = errors for the task, h=hesitancy, p=positive statement):
- Collecting/Reviewing Data: 159s/3e/3h/0p
- Character Customization: 47s/0e/0h/0p
- Carpooling: 73s/1e/1h/0p
Critical Incidents
Collecting/Reviewing data
1. Kept clicking on the main menu to try to start collecting data. (Severity = 2)
2. Hesitated for a while about how to start collecting data.
3. Eventually went to different goals like carpooling and character customization. (Severity = 0)
Character Customization
No critical incidents where logged here. The user did everything perfectly.
Carpooling
1. Did not pick anybody from the ranked list to carpool with (Severity = 4)
2. Went to each person in the list and saw the routes on the map, but did not pick Alice, who has the most similar route. (Severity = 4)
Positive Comments
The interviewee did not have anything positive to say about the user interface as he was going through it. I think he was extremely focused on finding the negative parts of it that he failed to mention any positive things.
Discussion
From the pilot usability study that I conducted, I learned that I should directly prompt the user for any positive things that he/she has to say about the prototype at the end of the session if nothing positive has already been said. I believe the the interviewees are mostly preoccupied with finding negative points. Prompting the interviewees can force them to think back about what they liked about the user interface. Also, I think it might have been helpful to test the "set goals" task instead of collecting/reviewing data because I believe setting goals is easier to do in our user interface. Starting off the user with an easier task allows them to learn the different aspects of the user interface in an easier way.
From the results of the study, I learned that our group should probably put a "Collect Data" option in the main menu as well to be consistent. The interviewee thought that since many options were part of the main menu, the "collect data" option will be there as well. This makes sense and is something we should consider doing.
Also, we should make it as clear as possible that the list of contacts in the carpooling user interface is ranked based on the most similar route. We should also definitely not show "most similar" as an option in the drop down menu. Users have a tendency to chose this option and assume that the system will automatically choose the person with the most similar route.
Appendices
Materials
Consent Form
Demo Script
Hi, thank you very much for participating in this interview; my name is __________. This is a project for CS 160, User Interfaces, in which we are trying to design a serious game that will encourage people to drive in a manner that consumes less gas by linking their driving behavior to the well-being of a character whose image you can see on your cell phone whenever you enter the game.
I’d like you to begin by first signing this consent form, which in summary states that you agreed to do this of your own free will and that you weren’t associated with us in any way, so that we can attest that our interviewees were not biased.
Now, the way this interview will be set up is that you will be asked to utilize our user interface as if we aren’t here. You may communicate with me any concerns you have during the course of the interview, but I will be unable to provide you with any assistance. You should not feel in any way feel as if we have any performance expectations of you - our hope is that in running a pilot study, we will be able to assess what aspects of our interface prototype we will need to change before performing a wider study.
Next, I’d like to go through a demonstration task. (Show user how to set goals using the user interface).
Then, ask user to do each of the following tasks:
Task 1: Collecting and Reviewing Data:
Each time you drive you will want to set the game into the data collection mode so that it is able to keep track of how far you are driving and your rates of acceleration. How do you think you should do this?
Task 2: Character Customization:
In this game, good driving behavior is rewarded with cash points, which you can use to purchase items in the store and improve its mood and aesthetic appeal. How would you go about doing this?
Now that you have purchased an item, how would you put it on your character?
Task 3: Carpooling:
Our game has a feature that facilitates carpooling - this is one means of saving gas, and consequently, carpooling also increases the reward (in cash points) for your character. If you were to carpool, how would you go about achieving the points associated with it?
Raw Data
Collecting/Reviewing Data
1. Went to shopping, then reviewing data, then to carpooling screens.
2. Went back to character menu after not finding collect data button.
3. Hesitated and finally found the button at the top to start collecting data.
4. Pressed STOP button to stop collecting data and went to review data screen.
5. Did everything fine in review data.
Character Customization:
1. Went to shopping option from main menu.
2. Bought different hats, then went to boots section and bought different boots.
3. Went to My Items option from main menu.
4. Chose the items that he bought to equip and de-equip the character.
Carpooling:
1. Went to Carpooling option from main menu.
2. Chose route A and confirmed the route.
3. Saw that Most Similar was chosen in the drop down meny so went on to next page.
4. Went back to previous screen and looked at different people's routes on the map from the list of contacts.
5. But didn't choose any of the people from the list to carpool with, always went back to choosing Most Similar.
