PilotStudy-Group:!Xobile-GeoffreyLee

From CS 160 Fall 2008

Introduction (5 points)

Introduce the system being evaluated (1 paragraph)
State the purpose and rationale of the experiment (1 paragraph)

In this experiment, we are evaluating an educational game that is designed to help college students and young working professionals to start thinking about retirement planning. The goal of this game is for the user to get a feel for the interest rate and restriction-related mechanics of various retirement investment options without the user being conscious of actually learning. This game is meant to be fast-paced and is played by the player purchasing investment accounts in the game and transferring money across these accounts to maximize profits based on a changing target interest rate as set by the Fed.

The purpose of this experiment is to perform a simple pilot usability study using our interactive prototype. Results from the pilot will then be used to revise the experiment for a larger pool of participants. Ultimately, our goal is to test our interface on users and find out what works or doesn’t work well.

Implementation and Improvements (15 points)

Describe all the functionality you have implemented and/or improved since submitting your interactive prototype (1 page).

Based on feedback that we received from our interactive prototype, we added indicators for time, target interest rate as set by the Fed, and pop-up tool-tips to assist first-time users.

Time Indicator

One criticism we received from our interactive prototype was that the user could not relate the in-game income flow to the real-world because there was no indicator for the passage of time. We remedied this problem by defining a set game-length of six years and four months, which then triggers the end of the game. To show this, we added a countdown timer in the upper-right-hand corner of the screen which displays the number of years and months remaining. This serves two purposes: first, it allows the user to visualize how much money he/she is gaining from an account in one year, and second, this time limit forces the player to plan his/her investment accounts for a set future date.

Federal Reserve Interest Rate Indicator

We felt that we should add more educational content to our game, but we didn’t have enough time during the last assignment to do so. One of our planned features was to add some sort of financial indicator that the user would use to make his/her financial decisions. We felt that the target interest rate as set by the Fed would be a very broad indicator that affected all investment options and simple enough for the user to grasp. Thus, we added a number in the upper-left-hand corner of the screen labeled as the “Federal Reserve Interest Rate”, and it changes over time, requiring the user to change his/her investing habits.

Pop-up Tool-tips

One other criticism we received from our interactive prototype was that our game did not contain any help-related features. So in this iteration, we added pop-up tool-tips that trigger when the user hovers his/her mouse over certain game interface elements. The idea is that when a user performs a wrong action, the user would then proceed to look for help, and a location-based tool-tip would work well. We ended up adding four tool-tips, one for the account box indicating how to transfer money, one on the purchase bar indicating how to purchase an account, one on the money-transfer slider-bar indicating how to use that particular interface, and one on the Fed interest rate indicator explaining its purpose in the game.

Method (10 points)

Participant (who -- demographics -- and how were they selected) (1 paragraph)
Apparatus (describe the equipment you used and where) (1 paragraph)
Tasks (1/2 page) [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
Procedure (1 page) describe what you did and how

Participant

The test-user for this experiment is a 20-year-old male who is a third-year undergraduate business administration major. He plays video games approximately two to three times per week and considers himself moderately informed about personal finances, though he is unsure if that is accurate. This participant was randomly selected from campus and passed our criteria for someone who would potentially play this game.

Apparatus

We conducted this experiment on a laptop in a casual apartment room. This location was convenient, familiar, and comfortable for the test-user.

Tasks

[Easy] Purchasing an account - The first task in our game is to purchase an account. Without this, the game cannot be played. The test-user begins at the in-game screen and is told to purchase an investment account. He/she is expected to drag-and-drop an account from the bottom purchase bar onto the game area.

[Medium] Re-positioning an account - Immediately following the first task, the test-user is told to move an account to a different position on the screen. He/she is expected to drag an account somewhere across the screen using the header-bar on any account box.

[Hard] Transferring money across accounts - Finally, the user is told to transfer an arbitrary amount of money from one account to another. He/she is expected to examine the account box, notice the open-hand mouse-cursor when hovering over the box, and then drag-and-drop money into another account. This tasks should be the most difficult because it involves the largest number of steps.

Procedure

I first selected a participant randomly from a student group meeting who agreed to volunteer 30-minutes of his time for a pilot usability study. We then proceeded to my apartment, and during the walk, I explained that I was developing an educational Flash game for teaching young working professionals and college students how to manage their personal finances. After we sat down, I handed him the consent form, which he then read and signed. I then proceeded to collect information regarding his demographic and explain the ground rules for the experiment. Finally, he was instructed to perform each of the three tasks while I observed and wrote down notes. At the end of the three tasks, I asked him questions regarding areas in which he hesitated and asked him for any suggestions or other feedback.

The script is detailed in the appendix.

Test Measures (5 points)

Describe what you measured and why (1/2 page)

We measured the following statistics:

The time required to complete a task - Because this is meant to be a fast-paced game, a task should be quick to execute. In addition, if a user spends a lot of time during his/her first attempt at a task, then that indicates that the task has problems.
The number of positive events per task (such as facial and verbal cues) - Positive events are a good indicator of when the user feels that he/she is using the interface correctly and feels good about it.
The number of negative events per task (such as facial and verbal cues) - Negative events indicate that the user is frustrated with the particular task. This is bad because we do not want the user to be frustrated or feel that he/she is incompetent.
The number of times that the test-user hesitates on a task - Hesitation is a sign that the interface is not intuitive. This typically occurs when the user is unsure of the next step or after the user performs an action that leads to unexpected results. The interface should be intuitive and produce results that the user intended for, and hesitation indicates otherwise.

Results (10 points)

Results of the tests (1 page)

The pilot usability study was a success. The participant was able to complete all three tasks in a very short amount of time. The only tasks in which the test-user exhibited considerable hesitation and incorrect actions was the first task, which involved the test-user purchasing an investment account. The second task only took one second, and the third task took five seconds. Overall, only the first task showed problems.

After debriefing the test-user, I learned that he was uncomfortable with how the game started out really fast, and he felt that an in-game tutorial would've been helpful. He also did not like the clutter that the account boxes created on screen and suggested that the account boxes snap onto a grid. When asked about why he first clicked on the purchase bar, he agreed that the accounts on the purchase bar looked like buttons that should be clicked. He also noted that it was hard to select an exact amount of money with the slider-bar during money transfers, and that the tool-tips often covered up important information as he was trying to use the slider-bar. Given that, he suggested that the tips should be displayed in a separate window on the edge of the screen. Lastly, he suggested that we should add a history of actions so that he could keep track of what he had done in the game.

Discussion (15 points)

What you learned from the pilot run (1 page) what you might change for the "real" experiment? what you might change in your interface from these results alone? If you'd like, you may include results and assumptions from other group members' tests here as well.

I learned a lot from the pilot run both in terms of experimental procedure and interface improvements for our game.

In terms of experimental procedure, the participant definitely seemed to be more aware of problems in the game than if he were just playing it for personal enjoyment. This means that the participant isn't using the system in the exact same way that an ordinary user would use it outside of our lab conditions. However, everything else seemed to go fairly smoothly. We did encounter two program bugs during the experiment, one of which we were able to quickly correct, and the other which did not impede the experiment. Lastly, I felt that introducing my experiment to the test-user while walking to the lab location was a good way to build up rapport and explain our purpose without the participant listening uncomfortably. Overall, I would not change anything for the real experiment, except maybe to add video-recording equipment to help take notes.

In terms of our game interface, I learned that people will tend to incorrectly click on the purchase bar as opposed to dragging-and-dropping because the accounts look like buttons, which afford clicking. Secondly, I learned that tool-tips are not always the best way to display help information because this particular participant would've preferred seeing the help in a less-intrusive location on the screen. Finally, the participant felt flustered at the beginning of the game because numbers immediately began changing wildly and he did not know what to do. We can improve on this aspect by slowly ramping up the speed of the game as the user accomplishes certain key tasks.

Appendices (5 points)

Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
Raw data (i.e., entire merged critical incident logs)

Consent Form

Consent Form (Pilot Usability Study)

Demo Script

Introduction
1. Introduce yourself to help relax the subject.
2. Explain the game, detailing the overall premise and goal of it.
3. Explain the purpose of this study (e.g., to identify any usability issues that may necessitate a design change).
4. Show the participant the consent form, giving them time to read it. Re-iterate that he/she can quit at any time.
5. After the participant has signed the consent form, ask him/her for his/her demographic information:
  1. age
  2. sex
  3. education level and major(s) if appropriate
  4. frequency of playing games, and if he/she plays games, which types of games does he/she play?
  5. level of financial knowledge, how knowledgeable does he/she thinks he/she is?
  6. has he/she planned for retirement?
    1. if yes, how has he/she planned for retirement?
    2. if not, when does he/she plan to start planning?
6. Explain how the pilot study will be conducted.
  1. Note that he/she will be given three in-game tasks.
  2. Explain that you will be unable to provide help and the reasoning behind it.
  3. Explain how to "think aloud" and the reasoning behind it.
  4. To demonstrate how the game works, give a short demo of the system.
    1. During the demo, show tool tips to the participant.
Task 1 [Easy]: Buying (creating) an account: Ask the participant to buy an account.
Task 2 [Medium]: Rearranging the position of an account: Ask the participant to move an account to a different position.
Task 3 [Difficult]: Moving money from one account to another: Ask the participant to transfer $X from one account to another.
Outro
1. Ask the user which task he/she found the most difficult; which task was the easiest -- for both questions, ask the reasoning behind the participant's selection.
2. What would he/she recommend improving (interface-wise) and why?
3. Thank the user.

Raw Data

Demographics: 20 male, 3rd-year Haas, 2-3 gaming sessions per week, moderately comfortable with personal finances, does not plan to consider retirement until first job.

Time to completion:

Task 1 - 3 minutes
Task 2 - 1 second
Task 3 - 5 seconds

Number of positive events:

Task 1 - 1
Task 2 - 0
Task 3 - 0

Number of negative events:

Task 1 - 2
Task 2 - 0
Task 3 - 0

Number of hesitations:

Task 1 - 3
Task 2 - 0
Task 3 - 1

Event log:

Clicked on purchase bar
Hesitated
Read tool-tip for purchase bar
Attempted to drag-and-drop an account, but purchase bar changes
Hesitated
Attempted to drag-and-drop a red account
Successfully drag-and-dropped an account
Hesitated
Read tool-tip on slider-bar
Completed task 1
Dragged account across screen
Completed task 2
Hesitated
Read tool-tip on slider-bar
Drag-and-dropped money to another account
Completed task 3