PilotStudy-Group:!Xobile-Mu-QingJing

From CS 160 Fall 2008

Contents 1 Introduction (5 points) 2 Implementation and Improvements (15 points) 2.1 Tool tips 2.2 Federal Reserve interest rate indicator 2.3 Time indicator 3 Method (10 points) 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.3.1 Creating An Account (Easy) 3.3.2 Moving/Rearranging Accounts On The Screen (Medium) 3.3.3 Transferring Money Between Accounts (Hard) 3.4 Procedure 4 Test Measures (5 points) 5 Results (10 points) 5.1 Summarized Results 5.2 Specific Results 5.2.1 Task 1 5.2.2 Task 2 5.2.3 Task 3 6 Discussion (15 points) 7 Appendices (5 points) 7.1 Prototype 7.2 Consent Form 7.3 Demo Script 7.4 Raw Data 7.4.1 Character Persona 7.4.2 Task 1 7.4.3 Task 2 7.4.4 Task 3 7.5 General Feedback

Introduction (5 points)

• Introduce the system being evaluated (1 paragraph)
• State the purpose and rationale of the experiment (1 paragraph)

Fast Money is a speed game that is designed to raise awareness for players regarding the different types of retirement/investment accounts that exist in the world today. Like most flash games, action is face-paced, the learning curve is very flat, and the game itself is fairly short in length. To goal for the player is to obtain as much money as possible before the end of the game. They can accomplish this by buying various types of investment accounts, as well as transferring money between them in order to best optimize the amount of gain. This game attempts to solve the problem of unawareness of many Americans towards the different types of investment accounts available to them. This games tries to address this by not only making people aware that these types of accounts exist, but also by portraying the general characteristics (risks, reward, and volatility) of each account. The game refrains from attempting to teach the player specifics in relation to market analysis because the huge scope of such topics require a much more complex and in-depth method of teaching (i.e. a college course). The game does, however, make note of the Federal Reserve interest rate in order to model account growth and volatility such that the indicator accurately maps to the real world.

The purpose and rationale of this experiment is to perform a simple usability test with a participant in order to flush out more errors or areas to improve on that can be addressed by design changes. The pilot test has a much lower cost at the expense of a much smaller test population; this can either allow for very large and egregious errors to be picked up, or to essentially act as an initial evaluation for the experiment itself.

Implementation and Improvements (15 points)

• Describe all the functionality you have implemented and/or improved since submitting your interactive prototype (1 page).

We never received any feedback from the heuristic evaluation, so all improvements were made based off of the impressions from our original user-testing and the Q/A section after our presentation.

Three new features were implemented:

1. Tool tips
2. Federal Reserve interest rate indicator
3. Time indicator

Tool tips

In order to address the problems of players being confused initially as to how different tasks or processes are performed, we decided to implement tool tips. Rather than using a tutorial or a lengthy documentation manual, we kept our design in line with one of the standards of online Flash games: intuitive and quick playability with minimal instructions or reading. We also used the tool tips to supply some more information to the player; when they hover over accounts or various parts of the game board, educational information pertaining to the object is displayed. In this way, we can provide a little bit more educational information to the player without sacrificing playability. An example of this can be seen on the screenshot at the end of this section.

Federal Reserve interest rate indicator

In order to provide the player with an additional tool to frame their decisions with, we decided to create an indicator for the Federal Reserve interest rate that maps to the volatility and growth of various accounts. An explanation is provided via the tool tips as to what this is and how it relates to various accounts. As players will see, for some accounts, a higher rate will mean stronger and more stabler growth, whereas for others it might mean very volatile growth patterns overall. The location of the indicator can be seen on the screenshot at the end of this section.

Time indicator

All of our previous test subjects had mentioned that they were confused as to how time is perceived/advanced in the game. Also, some of the accounts (specifically, the CD accounts) last only for a specific amount of time. We are able to address these concerns by actually displaying the time in the game and mapping it to all the accounts so that they all follow the same global time constraints.

Method (10 points)

• Participant (who -- demographics -- and how were they selected) (1 paragraph)
• Apparatus (describe the equipment you used and where) (1 paragraph)
• Tasks (1/2 page) [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
• Procedure (1 page) describe what you did and how

Participant

The participant for this pilot study is a 22-year-old Berkeley graduate with a degree in Environmental Economics and Policy. He is currently working full time at an internship at a San Francisco law firm, with the plan of applying for law school next year. He is a self-proclaimed gamer, but usually does not play online Flash games. He does not consider himself to be overly financially knowledgeable, claiming that the "Economics" portion of "Environmental Economics and Policy" is really a misnomer. He is not actively planning for retirement (financially), but does balance his money between a savings, checking, and money market account.

This participant was selected for the study because he is a good example of the target group that we are focusing on for the game: working, recent graduates who have an interest in investment accounts but not much current knowledge in that particular field. As a future professional, money (and the need to make smart investments) will undoubtedly be a large part of his life; as such, a strong base (which this game can hopefully start building) is a great necessity.

Apparatus

This study was performed on the participants desktop; specifically, a Firefox 3 running on a Windows XP machine with 1280x1028 resolution. A dedicated keyboard and wired optical mouse was used. Another laptop (MacBook OS X 10.5) was used in conjunction with TextEdit to record notes.

Tasks

We used the same three tasks that we used in the previous evaluations as those still remain the three main tasks that players must be able to perform efficiently and effectively. This included:

• Creating and account (of the financial variety, not a user account) (Easy)
• Moving/rearranging accounts on the screen (Medium)
• Transferring money between accounts (Hard)

Creating An Account (Easy)

To create an account, the player can either click and drag an account from the bottom purchase bar to the main game area above, or double-click to create an account instance that is placed at random in the main game area. The accounts available to purchase rotate randomly (though systematically) over time; this allows for more accounts to be used in a smaller area.

Moving/Rearranging Accounts On The Screen (Medium)

To move accounts around on the screen, the player has to click and drag on the "title bar" of the account box. We modeled this after the title bar that all windows have for almost every popular GUI in existence in order to maintain recognition; this window analogy is very intuitive. The ability to rearrange accounts allows the player place accounts commonly accessed near each other to reduce the amount of mouse movement needed. It also allows them to group similar accounts together in order to analyze in a more orderly and organized manner.

Transferring Money Between Accounts (Hard)

To transfer money from account A to account B, the player has to click and hold on the main account box area (not the "title bar") of account A, drag the mouse over to anywhere on account B, and then release. This freezes the source account (A) so that the player has time to work with all the funds in there. They are they presented with a slider bar, with which they use to determine how much money to transfer. To confirm, all they have to do is click the mouse button anywhere in the slider box. To cancel, they can click outside of the slider box, press the escape key, or simply confirm with a transfer amount of $0.

Since this is arguably the hardest task to perform, we utilized tool tips in order to inform players that they need to drag and drop from account A to account B in order to transfer money. We also added tool tips to the slider bar to explain how to confirm or cancel.

Procedure

I used a modified version of the procedures from the low-fidelity prototype to conduct this study. Rather than having specific roles, I essentially took on all the roles (except that of the subject, of course).

I introduced myself and explained how the game works and explained to him the purpose of the study. At the same time, I presented the subject with the consent form, allowing him to go over it. I then asked him for any questions, and helped clarify a few points to him. Once he signed the form, I proceeded to ask him for some information that would help me construct his character persona. I made sure to remind him to think aloud, and to point out any uncertainties or errors that he feels exist, no matter how trivial they are.

I then quickly demonstrated how the main aspects of the game works, and made sure to give notice to the importance of the tool tips. I then provided him with the instructions for tasks; one note is that I only provided him with the instructions for the next task upon successful completion of the task before. In order, the tasks performed were:

1. Create an account.
2. Move/rearrange the accounts on the screen.
3. Transfer money from one account to another.

As each task was performed, I recorded any errors that he made, how many hesitations he had, any incorrect movements he made, and how long it took.

After he successfully completed all of the tasks, I asked him some more questions regarding the game as a whole. Finally, I thanked him for taking time out to assist us in improving our game.

Test Measures (5 points)

• Describe what you measured and why (1/2 page)

Four dependent variables were measured during the pilot study:

1. The amount of time it took for a participant to finish a task
2. The number of logged "positive" events (e.g., user says "cool" upon seeing something they like) for a task
3. The number of logged "negative" events (errors) for a task
4. The number of times a participant hesitated because he/she was unsure how to proceed (per task)

For #1, we reasoned that more successful and intuitive designs usually take less time to learn (a flatter learning curve). With this in mind, we felt we could quantify and measure the intuitiveness of our design by how long it takes a new user to complete a task successfully.

For #2, we felt that the more "positive" events that the player exhibits, the more the player enjoys the game and has fun playing it. Even if not many positive responses are elicited, it does not necessarily mean that the design is bad; players usually only have responses if the gameplay is very bad.

With that in mind, for #3, we felt that if a number of "negative" responses were exhibited by the user, then there is a very high chance that our design is flawed in some way; there are likely many heuristic errors that we had made. Just one negative response would be enough to raise a flag that one or more of our processes have serious issues.

For #4, we felt that the number of hesitations can help us quantify and measure how intuitive the design is. This is similar to the first dependent variable, but is much more specific and subtle; hesitations don't necessarily correlate to the amount of time something takes.

Results (10 points)

• Results of the tests (1 page)

The raw data resides in the appendix.

Summarized Results

Results

	Time to Finish Task	# Positive Response	# Negative Response	# of Times Hesitated
Task 1	~30 sec.	0	2	2
Task 2	~5 Sec.	1	0	0
Task 3	~10 Sec.	0	0	0

Specific Results

Task 1

Task one went fairly smoothly, although there were two main points that resulted in negative responses and hesitations. The first point was not realizing that red accounts were unable to be purchased; the subject initially tried to purchase a red account, and then hesitated upon realizing that it was unsuccessful. Another hitch in the task came when he tried to single-click on the account; he only realized that he has to drag and drop once he saw the tool tips. In those two cases, the response "what the heck" and "um..." were elicited, respectively.

Task 2

Task two went very smoothly. The subject instantly made the connection between the "title bar" of the account and that of windowed screens.

Task 3

Task three was also completed very quickly. Upon hovering over the account, the tool tip instructions were provided, making the actual process itself very smooth and straightforward.

Discussion (15 points)

• What you learned from the pilot run (1 page) what you might change for the "real" experiment? what you might change in your interface from these results alone? If you'd like, you may include results and assumptions from other group members' tests here as well.

Although most of our usability issues have been resolved, we seem to still have some issues as far as making our game little bit more educational. The tool tips has definitely helped in providing a medium for which to convey information to the players with, but we can still stand to provide more substance and material with that affordance.

In order to address the issue with attempting to purchase in-eligible accounts, we definitely need some sort of error response. Possible solutions might be a pop-up box or some form of notification detailing the reason as to why the attempted action has failed.

Going back to the expanded usage of tooltips, we should definitely utilize it so that users can better understand the consequences of their actions, as well as offer more information (or even suggestions) in terms of financial responsibility. Smart decisions might be met with positive responses in the tooltips, and bad decisions might be met with negative ones. On that front, we can hope to improve the amount of information provided to the players by further analyzing how best to educate them. It would seem simple and straightforward to bombard users with terms, definitions, and other relevant information, but that may not be the best method with respect to the ability retain that information and maintaining the fun factor of the game. In contrast, we could also expand the gameplay, which would allow for a more non-direct implicit method of learning; that, however, would make the game more complicated and not fit in with the goal of creating a simple Flash game that is easy to pick up on.

It seems like the "real" experiment will need to take into account error detection, and how the logic of the game should be interrupted (or not at all) when errors are made. Furthermore, we definitely should implement some sort of tool tip functionality that explains how the Federal Reserve interest rate affects things; we can see if that new addition improves the educational factor of our game.

Appendices (5 points)

• Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
• Raw data (i.e., entire merged critical incident logs)

Prototype

Prototype

Consent Form

Consent Form

Demo Script

1. Introduction
   1. Introduce yourself to help relax the subject.
   2. Explain the game, detailing the overall premise and goal of it.
   3. Explain the purpose of this study (e.g., to identify any usability issues that may necessitate a design change).
   4. Show the participant the consent form, giving them time to read it. Re-iterate that he/she can quit at any time.
   5. After the participant has signed the consent form, ask him/her for his/her demographic information:
      1. age
      2. sex
      3. education level and major(s) if appropriate
      4. frequency of playing games, and if he/she plays games, which types of games does he/she play?
      5. level of financial knowledge, how knowledgeable does he/she thinks he/she is?
      6. has he/she planned for retirement?
         1. if yes, how has he/she planned for retirement?
         2. if not, when does he/she plan to start planning?
   6. Explain how the pilot study will be conducted.
      1. Note that he/she will be given three in-game tasks.
      2. Explain that you will be unable to provide help and the reasoning behind it.
      3. Explain how to "think aloud" and the reasoning behind it.
      4. To demonstrate how the game works, give a short demo of the system.
         1. During the demo, show tool tips to the participant.
2. Task 1 [Easy]: Buying (creating) an account: Ask the participant to buy an account.
3. Task 2 [Medium]: Rearranging the position of an account: Ask the participant to move an account to a different position.
4. Task 3 [Difficult]: Moving money from one account to another: Ask the participant to transfer $X from one account to another.
5. Outro
   1. Ask the user which task he/she found the most difficult; which task was the easiest -- for both questions, ask the reasoning behind the participant's selection.
   2. What would he/she recommend improving (interface-wise) and why?
   3. Thank the user.

Raw Data

Character Persona

• male
• 22 years old
• eep major, planning on law school
• working full-time at an internship in a sf law firm
• plays a lot of games, mostly console games
• usually refrains from short-length games (except minesweeper)
• has not really contemplated retirement; more focused on actually getting a job first
• currently lives permanently in berkeley; has lived mostly in the bay area; kansas prior to that
• conservative republican, views the economy with moderate importance
• currently has an mma, as well as savings and checking
• files own taxes

Task 1

• He noticed how the accounts were rotating. Although initially confused, he quickly realized that it was to fit more accounts in a limited space.
• To purchase an account, he tried clicking on one. This did nothing, resulting in the response "uh...". He then hovered the mouse cursor over an account and read the tooltip.
• He then tried to purchase a red account by clicking and dragging. That did nothing, eliciting the response "what the heck".
• Confused, he tried dragging a black account, which then worked. This snafu resulted in a hesitation and negative response.

Time: ~30 seconds.

Task 2

• Instantly recognized that the title bar is a parallel to the title bar on most window screens.
• No problems whatsoever.
• Positive response of "Nice!"

Time: Instant ~5 seconds.

Task 3

• He instantly saw the tool tip and thought it was very straightforward.

General Feedback

• He felt that all the commands and actions are very intuitive, especially with the help of the tool tips.
• He said that the overall fun factor of the game probably won't drop very much even if we throw in some more education factors.
• He did feel that even without the additional educational factor, it is currently an effective way to raise awareness for people.
• He would like to have some sort of graphing feature for account performance.
• Audio cues would be a nice touch.
• Overall, the gameplay is pretty refined, but there could be more added to the game later.
• The tooltips is definitely a necessity for this game because of the lack of a more conventional help manual.