PilotStudy-Group:!Xobile-PerryLee

From CS 160 Fall 2008

Contents 1 Introduction (5 points) 2 Implementation and Improvements (15 points) 2.1 A Federal Reserve Interest Rate Indicator 2.2 A Time Indicator 2.3 Tool Tips 3 Method (10 points) 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.3.1 Creating an Account [Easy] 3.3.2 Rearranging the Position of an Account [Medium] 3.3.3 Moving Money from One Account to Another [Hard] 3.4 Procedure 4 Test Measures (5 points) 5 Results (10 points) 5.1 Dependent Variables 5.2 Summary of the Process Data 5.2.1 Task 1 5.2.2 Task 2 5.2.3 Task 3 5.2.4 General Feedback 6 Discussion (15 points) 7 Appendices (5 points) 7.1 Prototype 7.2 Consent Form 7.3 Demo Script 7.4 Raw Data 7.4.1 Demographic Information 7.4.2 Task 1 7.4.3 Task 2 7.4.4 Task 3 7.4.5 General Feedback

Introduction (5 points)

• Introduce the system being evaluated (1 paragraph)
• State the purpose and rationale of the experiment (1 paragraph)

Fast Money is a fast-paced, short-length (5-15 minutes) game that raises players’ awareness of the different types of investment options that are appropriate for retirement. Players are challenged to maximize their money by (1) purchasing a number of investment accounts and by (2) transferring money from one account to the next. The problem at hand is a large number of Americans do not prepare or are unprepared for retirement. To combat this lack of readiness, Fast Money (1) raises players' awareness of the different types of investment options; (2) teaches players the larger picture rather than focusing on the specifics (e.g., what do you look for in a "good" stock); and (3) shows players the impact of the Federal Reserve interest rate on the various investment options they might consider.

The purpose and rationale of this pilot study is (1) to perform a simple usability test -- discover errors and areas of improvement -- with a single participant and (2) to use the results of this test to drive design changes in the prototype. In the real world, this kind of "pilot" study would be used to redesign our experiment before running the study with a larger pool of participants.

Implementation and Improvements (15 points)

• Describe all the functionality you have implemented and/or improved since submitting your interactive prototype (1 page).

Note: My group never received feedback from the heuristic evaluation.

Based off the feedback we received in previous checkpoints, we implemented three new features:

1. a Federal Reserve interest rate indicator
2. a time indicator
3. tool tips to show supplementary information (help)

A Federal Reserve Interest Rate Indicator

As the name suggests, the Federal Reserve interest rate indicator reflects the target interest rate set by the U.S. Federal Reserve. Lower rates stimulate the economy, whereas higher rates make U.S. dollar dominated assets more attractive.

In Fast Money, this rate affects all investment accounts (e.g., how fast or slow an account grows and its volatility). To maximize their money, players must keep an eye on this rate and strategically move their money accordingly.

A Time Indicator

As in the real world, investment accounts in Fast Money may be time sensitive and expire after a certain period of time. Using the time indicator, players can plan ahead and purchase accounts that maximize their money with the time remaining.

Tool Tips

In our earlier usability tests, we discovered that players are often unsure how to proceed due to a lack of visual cues. In this latest revision, we have implemented tool tips -- small hover boxes that appear with supplementary information whenever a player hovers the cursor over items -- to address this problem. Most items (e.g., transfer box) in our game are now linked to tool tips.

Method (10 points)

• Participant (who -- demographics -- and how were they selected) (1 paragraph)
• Apparatus (describe the equipment you used and where) (1 paragraph)
• Tasks (1/2 page) [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
• Procedure (1 page) describe what you did and how

Participant

The participant is a 22 year old male student attending UC Berkeley. He is double majoring in EECS and Materials Science. In addition to school, he works part-time as a systems administrator, averaging 19.5 hours of work a week. He is an avid gamer, playing games of all types, including Flash ones. He does not consider himself to be financially knowledgeable and has only briefly thought of retirement. He does not intend to plan for retirement any time soon. His highest priority is to graduate followed by finding a job.

This participant was chosen because he is part of the group that we are targeting our game for: working individuals who are unaware of and would like to learn more about the different forms of investments. He can be said to represent the stereotypical engineer, and moreover, to be representative of the young professional -- fresh (or at least almost) out of college with little to no financial background.

Apparatus

The study was conducted in an apartment with facilitator and participant sitting side by side. A normal MacBook running Mac OS X 10.5.5 and Firefox 3.0.3 was used to load the prototype and conduct the pilot study. Other than this laptop and a pad of paper to record notes, no other equipment was used.

Tasks

For our three representative tasks, we chose...

• creating an account [Easy]
• rearranging the position of an account [Medium]
• moving money from one account to another [Hard]

... because players' game play actions only consist of these tasks.

Creating an Account [Easy]

To create an account, players drag accounts from the purchase bar (at the bottom of the screen) to the main game window.

The accounts that are available for purchase vary with time and are randomly selected from a pool of type of accounts (e.g., a savings account, a mutual fund account).

Rearranging the Position of an Account [Medium]

Using the windows analogy, players rearrange accounts by drag-and-drop on the title bar. We use visual cues (the mouse is transformed to a directional pointer) to indicate that an account can be moved.

Rearranging accounts enables players to strategically position accounts:

• minimize the amount of time spent while transferring money
• group accounts with different volatility (or by other criteria)

Moving Money from One Account to Another [Hard]

Players move money from one account to another by drag-and-dropping the account balance (as indicated by a dollar amount) from one account to another. Upon dropping the account balance onto another account, two actions occur:

• the source account is "frozen" -- it no longer changes in relation to time
• a slider box indicating the amount of money to transfer is displayed

This slider box ranges from $0 (cancel the transfer) to the total balance of the source account. As a player moves his or her mouse from left to right, the amount of money to transfer is updated and displayed. A single-click on the slider box confirms the transfer.

To cancel a transfer, players can either:

• click outside of the transfer box
• select $0 to transfer
• press the escape key

In our game, players constantly move money from one account to the next in order to maximize their money.

Procedure

A similar approach to the low-fidelity prototype was used in conducting this study, with the more obvious exception that there were no differentiated roles (e.g., facilitator, greeter). There was only one person to facilitate, greet, observe, etc. -- me.

To help put the participant at ease, I first introduced myself, followed by an explanation of the game and of the purpose of the study. I then gave him the consent form and gave him several minutes to read it over and sign the form. During this time, I reiterated that he could quit at any time.

After he signed the form, I then asked for some demographic information (the specifics are listed in the Appendix). Following, I explained how the study would be conducted -- specifically noting that (1) he would be given three in-game tasks to perform; (2) I would be unable to provide help and the reasoning as to why; and (3) how to "think aloud" and the reasoning as to why I want him to think aloud.

To show him how the system works, I then gave a brief introduction to the different components in the game window, specifically drawing attention to the tool tips that provide supplementary information. At this time, I gave him instructions to perform the first task. Upon completion, I gave him instructions to perform the second task, and so on with the third task. He was only given instructions for the next task upon completion of the current one.

The three tasks asked of him were:

• Purchase an account.
• Rearrange the position of an account.
• Move money from one account to another.

While he performed each task, I kept track of how long it took him to complete each task, the number of errors, etc.

Following the completion of these tasks, I asked him the questions detailed in the Appendix (e.g., which task was the easiest? which task was the hardest?); and finally, thanked him for helping us improve our prototype. I noted that should he have any lingering questions, he should email us at the provided email address on the consent form.

Test Measures (5 points)

• Describe what you measured and why (1/2 page)

Four dependent variables were noted and measured during the pilot study:

1. The amount of time it took for a participant to finish a task
2. The number of logged "positive" events (e.g., user says "cool" upon seeing something they like) for a task
3. The number of logged "negative" events (errors) for a task
4. The number of times a participant hesitated because he/she was unsure how to proceed (per task)

If only a small amount of time is needed to complete a task, this result suggests that the user interface is intuitive and easy to use. If a lot of time is needed to complete a task, this result suggests that the user interface is unintuitive and requires a redesign to make it easier to use.

If a large number of "positive" events is recorded, this result suggests that the user interface is enjoyable to use. Note that nothing can be inferred if a small amount (or no) "positive" events are recorded -- users do not generally rave about how good an user interface is.

If a large number of "negative" events is recorded, this result suggests that the user interface needs to be redesigned and that there are likely serious heuristic violations present. Even if the number of "negative" events is low, one is enough to suggest that the user interface should be more closely examined and, if possible, redesigned to address the concern.

The number of times a participant hesitates helps give a measure as to how intuitive the user interface is -- the lower the number, the more intuitive; the higher the number, the less intuitive.

Results (10 points)

• Results of the tests (1 page)

Refer to the Appendix for the Raw Data.

Dependent Variables

Results

	Time to Finish Task	# Positive Events	# Negative Events	# Times Hesitated
Task 1	~1 Min.	1	2	5
Task 2	~5 Sec.	0	0	0
Task 3	< 10 Sec.	0	0	0

Summary of the Process Data

Task 1

Task 1 took the longest to accomplish because the participant had trouble understanding the instruction, "Purchase an account.", and the importance of the time indicator. Although the actual task of purchasing an account was "relatively easy" (in his own words) -- "actual task" referring to the drag and drop operation -- he noted that several elements seemed out of place while thinking aloud:

• "Why is the time counting down? Does the game end once it hits zero? Do I 'die'?" [Hesitation]
• "When you ask me to purchase an account, does that only apply to buttons that have the word 'account' in them?" [Hesitation]
  • He assumed that only accounts with the word "account" in them are accounts, and thus waited for an account with the word "account" in it.
• "I see buttons on the purchase bar, maybe if I click on an account it will be purchased." [Hesitation]
  • When a single-click didn't work, he used the tool tip to figure out he needs to drag and drop. [Error]
• "Why are some of the purchase buttons red?" [Hesitation]
  • At this point, he tried to purchase an account in red. When he noticed that the purchase did not complete, he attributed it to a bug rather than he doesn't have enough money. [Error]
• "Why are the buttons alternating? Fire sale?" [Hesitation]
  • He liked how they alternate and found it "intriguing", but he did not understand why the selection of accounts alternates. [Positive]

Task 2

This operation took the least amount of time. He noted the windows analogy and rearranged the accounts with no trouble.

Task 3

Surprisingly, this task did not take much longer than Task 2. He intuitively drag and dropped the money from one account to another, and used the tool tips to figure out how to confirm the transaction.

General Feedback

Note: I have rephrased his comments and left them written from his point of view.

• All of the tasks were relatively easy to accomplish, except the instruction "purchase an account" made Task 1 ambiguous.
• What does the countdown mean?
• There is no base line to compare how well you are doing -- how do I know if I am doing poorly or well?
  • Along the same lines, there are no consequences or terms of failure -- where is the incentive? How do I win?
  • Provide an objective list that is displayed -- give the player rewards or perks for a good performance.
• Provide negative feedback, specifically when you try to purchase an account you do not have enough money for -- that way there is no ambiguity as to why the operation failed. I thought it was a bug.

Discussion (15 points)

• What you learned from the pilot run (1 page) what you might change for the "real" experiment? what you might change in your interface from these results alone? If you'd like, you may include results and assumptions from other group members' tests here as well.

This pilot run showed that although we have addressed most of the ambiguity/unsureness associated with the in-game mechanics (e.g., how do I confirm a transfer?), we still have a long way to go in terms of the educational aspect. On a positive note, tool tips significantly improved the user experience. Many of the errors we saw in previous studies were due to a lack of visual cues -- using tool tips addressed these problems.

One in-game mechanic that still needs to be addressed, however, is error feedback. Although there are few "invalid" operations in our game, one that comes to mind is trying to purchase an account when you do not have enough funds. As this pilot study showed, when the operation failed, the participant attributed it to a bug rather than attributing it to the fact he does not have enough money. In our next revision, we need to provide error feedback, perhaps in the form of a message that says "You do not have enough money in your Checking Account to purchase account X."

On the educational front, we need to give players more guidance and explanation for their actions -- in a sense, positively reinforce their good decisions, and provide feedback when they make a "bad" decision. One comment that was raised is, "How do I know when I'm doing well?". In the current iteration of our interface, we do not provide any base line indicators that a player can use to determine how well they're doing. We need to address this in the next revision by including a base score they can compare against -- e.g., "I have $x more than the base score -- I'm doing well!"

Additionally, now that we have tool tips, we should put them to better use -- not only use them to provide supplementary information in regards to how to complete an operation, but to provide more metrics that a player may use to gauge where to move money or which accounts to buy. For example, it might be useful to tool tip an account's historical information and provide helpful information such as "When the Federal Reserve interest rate is low ~1%, this account's growth is accelerated at the expense of ...".

This pilot study also showed that we need to usability test the in-game Tutorial. A lot of the questions raised might be addressed if players were first walked through a Tutorial (e.g., why is this account red on the Purchase Bar). Not every situation lends itself to using tool tips -- if we put too much information in them, players might grow annoyed by them and ignore the helpful hints they provide.

In the next iteration, we also need to provide more explanation of the countdown timer -- most likely in the form of a label that indicates its purpose, a tool tip, and addressing it in the in-game Tutorial.

In terms of the actual experiment, rephrasing Task 1 as "Purchase an account from the Purchase Bar" might eliminate the ambiguity brought up by the participant.

Appendices (5 points)

• Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
• Raw data (i.e., entire merged critical incident logs)

Prototype

Prototype

Consent Form

Consent Form

Demo Script

1. Introduction
   1. Introduce yourself to help relax the subject.
   2. Explain the game, detailing the overall premise and goal of it.
   3. Explain the purpose of this study (e.g., to identify any usability issues that may necessitate a design change).
   4. Show the participant the consent form, giving them time to read it. Re-iterate that he/she can quit at any time.
   5. After the participant has signed the consent form, ask him/her for his/her demographic information:
      1. age
      2. sex
      3. education level and major(s) if appropriate
      4. frequency of playing games, and if he/she plays games, which types of games does he/she play?
      5. level of financial knowledge, how knowledgeable does he/she thinks he/she is?
      6. has he/she planned for retirement?
         1. if yes, how has he/she planned for retirement?
         2. if not, when does he/she plan to start planning?
   6. Explain how the pilot study will be conducted.
      1. Note that he/she will be given three in-game tasks.
      2. Explain that you will be unable to provide help and the reasoning behind it.
      3. Explain how to "think aloud" and the reasoning behind it.
      4. To demonstrate how the game works, give a short demo of the system.
         1. During the demo, show tool tips to the participant.
2. Task 1 [Easy]: Buying (creating) an account: Ask the participant to buy an account.
3. Task 2 [Medium]: Rearranging the position of an account: Ask the participant to move an account to a different position.
4. Task 3 [Difficult]: Moving money from one account to another: Ask the participant to transfer $X from one account to another.
5. Outro
   1. Ask the user which task he/she found the most difficult; which task was the easiest -- for both questions, ask the reasoning behind the participant's selection.
   2. What would he/she recommend improving (interface-wise) and why?
   3. Thank the user.

Raw Data

Demographic Information

• male
• 22 years old
• double major in EECS and Materials Science
• works part-time as a systems administrator; averages 19.5 hours of work/week
• was a gamer from an early age; plays all sorts of games -- Flash games included
• has vaguely thought of retirement -- only in the sense that he saves money and does not wildly spend it
• does not plan to formally plan his retirement anytime soon
• first priority is to graduate followed by finding a job

Task 1

Note: Each question was counted as one hesitation.

• "Why is the time counting down? Does the game end once it hits zero? Do I 'die'?"
• "When you ask me to purchase an account, does that only apply to buttons that have the word 'account' in them?"
  • He assumed that only accounts with the word "account" in them are accounts, and thus waited for an account with the word "account" in it.
• "I see buttons on the purchase bar, maybe if I click on an account it will be purchased."
  • When a single-click didn't work, he used the tool tip to figure out he needs to drag and drop.
• "Why are some of the purchase buttons red?"
  • At this point, he tried to purchase an account in red. When he noticed that the purchase did not complete, he attributed it to a bug rather than he doesn't have enough money.
• "Why are the buttons alternating? Fire sale?"
  • He liked how they alternate and found it "intriguing", but he did not understand why the selection of accounts alternates.
• Total Time: ~1 minute.

Task 2

• There was little to no hesitation; the operation took no more than 5 seconds.
• Noted that you cannot drag accounts off the screen, but that you can cover an account up with another.

Task 3

• This task did not take much longer than Task 2: 5-10 seconds
• The drag and drop operation was intuitive to the player
• Participant used the tool tip to confirm how to transfer the money

General Feedback

Note: I have rephrased his comments and left them written from his point of view.

• All of the tasks were relatively easy to accomplish, except the instruction "purchase an account" made Task 1 ambiguous.
• What does the countdown mean?
• There is no base line to compare how well you are doing -- how do I know if I am doing poorly or well?
  • Along the same lines, there are no consequences or terms of failure -- where is the incentive? How do I win?
  • Provide an objective list that is displayed -- give the player rewards or perks for a good performance.
• Provide negative feedback, specifically when you try to purchase an account you do not have enough money for -- that way there is no ambiguity as to why the operation failed. I thought it was a bug.