PilotStudy-Group:!Xobile-ShaharyarMuzaffar

From CS 160 Fall 2008

Contents 1 Introduction (5 points) 1.1 System 1.2 Purpose 2 Implementation and Improvements (15 points) 2.1 Balloon Tool Tips 2.2 Countdown Timer 2.3 Federal Reserve Interest Rate Ticker 3 Method (10 points) 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.3.1 Creating an Account (easy) 3.3.2 Moving An Account (medium) 3.3.3 Transferring Money (hard) 3.4 Procedure 4 Test Measures (5 points) 5 Results (10 points) 5.1 Measured Variables 5.2 Task 1 5.3 Task 2 5.4 Task 3 5.5 Follow-up Interview 6 Discussion (15 points) 7 Appendices (5 points) 7.1 Script 7.2 Data

Introduction (5 points)

System

Our game is a flash game called 'Fast Money'. The purpose of the game is to educate users of the different types of investment options through a fun, fast-paced game. The game requires the player to manage a virtual account to gain as much money as possible within the limited time. They are able to purchase different types of accounts such as stocks, IRAs, and CDs each with their own characteristic of risk and return. These accounts are the main way the players learn. By showing them the trends of accounts, they will gain knowledge and experience of their investment options for their retirement.

Purpose

The purpose of the experiment is to test if our improvements from the first round of the interactive prototype increased the ease of use and intuitiveness of the game.

Implementation and Improvements (15 points)

Balloon Tool Tips

Our biggest improvement was the addition of balloon notification when the mouse is hovered over different elements of the game. This was added because during our first test, a common problem was that the users had no idea what to do for the given tasks. They first had to try several things out before realizing what actions needs to be done to complete the tasks. With the addition of these help balloons, users would be able to read what each element is capable of doing with doing trial and error testing.

Countdown Timer

We also added a count down timer on the upper right hand corner. This was added because users were unsure of how much time was left before retirement. With the timer, the users will know how much time they initially have and also how much time is left before the game is over.

Federal Reserve Interest Rate Ticker

We also added a Federal Reserve Interest Rate Ticker on the upper left hand corner. This was added because users had no idea how the money in their accounts were being affected. They felt like the money was changing randomly without any real cause. By adding this ticker, the users can predict how their account may increase/decrease in the near future and plan accordingly.

Method (10 points)

Participant

She is a 19 year old, 2nd year International Relations Major at UC Davis. She is a infrequent game player, only playing at most 3 hours a week of video games. She describes herself as not knowledgeable when it comes to financial terms or ideas. She has not given thought to her retirement options, and does not plan to do so until she is 40.

She was chosen, because she represent young students who have not given much thought to their retirement. Also, she would provide good feedback from those who are used to playing games.

Apparatus

She performed all three tasking in the dining room of her apartment. She used a 17" laptop with a touchpad mouse.

Tasks

Creating an Account (easy)

This task required the user to purchase a new account. In order to perform this task, the user must click and drag an account from the purchase, located at the bottom of the screen, to the game area. From there, they must slide a bar to indicate how much money they wanted to be put into the account.

Moving An Account (medium)

This task required the user to rearrange the accounts in the game area. To perform this task, the user needed to click and drag the header of an existing account to another area of the game screen.

Transferring Money (hard)

This task required users to move money from one account to another. The user needed to click on the money of one account and drag into the money of the other account. Then the user was to slide a bar to indicate how much money to be put into the account, just like purchasing an account.

Procedure

I first introduced myself to the user and them described how the experiment was going to work. I stated that I would first ask her to sign a consent form followed by a information gathering segment. After, she would proceed to attempt to complete 3 tasks on a game on the laptop in front of her. After she signed the form, I asked several preplanned question regarding her background. I then explained to her that while performing the tasks, she should speak her thoughts out loud so I can get a sense of how she thinks about completing the tasks. I also requested that she not think about properly stating her thoughts, but rather, say exactly what is going through her head. I then asked her to open the laptop in front of her, and then I proceeded to give a brief introduction to what the game was about and what the objective was of the game. After, I asked her to perform the three tasks, one at a time in increasing order. When she was done with the tasks, I thanked her for her time by giving her a Twix Candy Bar.

Test Measures (5 points)

Our group measured the four following things:

• The amount of time it took for a participant to finish a task. We measured this, because the more time it took to to complete a task, the more likely that a UI flaw exists in the game.
• The number of logged "positive" events for a task. This was an important variable to measure, because if this number was large, then we know that our design has a lot of good, interesting element.
• The number of logged "negative" events for a task. If we logged a large number for this variable, then it would tell us that our redesign is still confusing and would need more editing.
• The number of times a participant hesitated because he/she was unsure how to proceed. This number also tells us how confusing our game interface is to the user. If they hesitated a lot, then we know that the task given was not easy to perform through our interface, and so, the game would need to be updated again.

Results (10 points)

Measured Variables

	Time to Finish Task	# Positive Events	# Negative Events	# Times Hesitated
Task 1	~180 Sec.	0	3	4
Task 2	~ 5 Sec	0	0	0
Task 3	~ 15 Sec.	0	0	0

Task 1

After seeing for the first time, the user remarked, "the screen looks kinda of lame." She then proceeded to move the mouse around and read the balloon tool tips. During this time, one of the things she stated out loud was "what do i do?". She spent a great deal of time at this part, just reading almost everything. After a while, she proceeded to try to purchase an account the expected way, but ran into a problem by trying to drag an account for which she didn't have enough more. After a few attempts, she managed to drag a cheaper account onto the screen. During the post task question session, she stated that she was a bit confused about why the accounts she tried to drag at first did not work. She thought she had enough money, because the amount in her savings was enough to purchase the account. She was a lot a bit surprised when the slider bar came up. She stated that she expected to pay the amount listed to purchase an account, not drag and choose how much to spend.

Task 2

This task was performed quickly with no hesitation. The user simply clicked and dragged an account. I asked her how she figured out what to do so quickly, and she stated that she had read how to do this task while attempting to perform the last task.

Task 3

This task was also performed rather quickly. The user stated that this task easy to perform, because the tool tip was straight forward, and she had already seen the slider bar before.

Follow-up Interview

I asked her about how she felt about the overall game, she stated that it was okay, but is not that interesting. She said the the overall easiness to perform the tasks was moderate, meaning that the tasks were more easy than hard to perform. When asked about if anything felt unintuitive, she stated that the process of transferring money felt backwards. She stated that she expected to indicate how much money to move first by using the slider bar first, then which account to move to by dragging. Another improvement she stated is that the balloon tips should not show immediately, since they distract the user from what is going on. She also said an experienced user might find them annoying.

Discussion (15 points)

Overall, I believe our latest changes to our designed improvement the ease of use of our game. Although, from my results, it appears that our game till requires some time to getting used to, users will quickly learn how to perform tasks. I learned that giving the user some sort of in game guidance, such as the balloon tips, prove very useful for users, as they give at least some instruction rather than starting with nothing.

As for our interface, I believe most of our functional elements proved to be straight-forward and easy to use. We could possibly look into our slider bar, and see if we could improve it somehow, but any change to functional elements should be limited, because the existing ones proved to work well. However, it appears as if our aesthetics could be improve as suggested by my test user. From what I see, there somewhat seems to be a lot of things on screen, therefore, could make changes to improve the minimalist design.

For our real experiment, I believe we should take into account the time the user takes just to get used to our game. For example, my test user took a great deal for the first task, but hardly any time for the latter two. This was mostly because the user was getting used the game by reading the tool tips. This reading was not all for the first take, but she also gained knowledge for the subsequent two tasks. Therefore, the first task does include time that technically was spent on the other two tests, which somewhat weaken our test data.

Appendices (5 points)

Script

  1.  Introduction
        1. Introduce yourself to help relax the subject.
        2. Explain the game, detailing the overall premise and goal of it.
        3. Explain the purpose of this study (e.g., to identify any usability issues that may necessitate a design change).
        4. Show the participant the consent form, giving them time to read it. Re-iterate that he/she can quit at any time.
        5. After the participant has signed the consent form, ask him/her for his/her demographic information:
              1. age
              2. sex
              3. education level and major(s) if appropriate
              4. frequency of playing games, and if he/she plays games, which types of games does he/she play?
              5. level of financial knowledge, how knowledgeable does he/she thinks he/she is?
              6. has he/she planned for retirement?
                    1. if yes, how has he/she planned for retirement?
                    2. if not, when does he/she plan to start planning? 
        6. Explain how the pilot study will be conducted.
              1. Note that he/she will be given three in-game tasks.
              2. Explain that you will be unable to provide help and the reasoning behind it.
              3. Explain how to "think aloud" and the reasoning behind it.
              4. To demonstrate how the game works, give a short demo of the system.
                    1. During the demo, show tool tips to the participant. 
  2. Task 1 [Easy]: Buying (creating) an account: Ask the participant to buy an account.
  3. Task 2 [Medium]: Rearranging the position of an account: Ask the participant to move an account to a different position.
  4. Task 3 [Difficult]: Moving money from one account to another: Ask the participant to transfer $X from one account to another.
  5. Outro
        1. Ask the user which task he/she found the most difficult; which task was the easiest -- for both questions, ask the reasoning behind the participant's selection.
        2. What would he/she recommend improving (interface-wise) and why?
        3. Thank the user.

Data

• 19 years old
• Female
• 2nd year International Relations Major, UC Davis
• Approx. 3 hours per week. Tetris/ Guitar Hero/ TypeRacer
• Not financially knowledgeable
• No retirement plans, plans to start at 40 years old.
• Task 1: Move mouse curser around on screen. Read description of each item. Mouse to purchase bar, and dragged, one year CD account to blue area. Took around 3 minutes.
• Task 2: Clicked /hold. Drag. Unclick-drop.
• Task 3: Clicked on checking. Dragged to savings. Moving slider bar---transferring 50$.
• Followup: Easiest was moving an account. Tool tips make game look confusing.