PilotStudy-Group:!Xobile-BingWang
From CS 160 Fall 2008
Contents
|
Introduction (5 points)
- Introduce the system being evaluated (1 paragraph)
- State the purpose and rationale of the experiment (1 paragraph)
Fast Money is a flash based fast paced game that hope to raise the awareness of its players regarding different types of investments that exist in the real world. This game models many of the real world accounts, and players are asked to maxmize the their investment by putting money in the accounts that would generate the highest return yet have a reasonable risk. The problem that this game tries to solve is to simply let people know about all the investment options available out there. Many people put their money in checking and savings account which generate little or no interest and is often overshadowed by inflation. The learning aspect of the Fast Money involves raising players' awareness for different type of investments, have a "modeled" sense of the trade off between volatility and risk, and lastly the newest improvement which is the impact of Federal Reserve Targeted Rates on different investment accounts.
The purpose and rationale of this pilot study is to perform a simple usability test in order to discover more errors and areas to improve on before a larger scale usability test. From the cost perspective, this pilot study (with correct persona chosen as participant) can effectively model the larger pool of participants and thus fix large scale problems that might not be aware of by the designer yet is of concern by the targeted persona who is going to use the system.
Implementation and Improvements (15 points)
- Describe all the functionality you have implemented and/or improved since submitting your interactive prototype (1 page).
Note: My group never received feedback from the heuristic evaluation.
1. Federal Reserve interest rate indicator
The federal reserve indicator is one of the indicators that we hope to use to educate our users regarding different investment options. In the recent month, the Federal Reserve has been cutting its interest rate to combat the global economic slowdown. Such action would invoke some effects on different types of the accounts that we track. We hope that through the use of this indicator, the users become more knowledgeable regarding the changes in interest rates. The lowering of Federal Reserve Interest Rate spurs growth as businesses can lend more and take more risks to carry out their plans. On the other hand, the raise in Federal Reserve Interest Rate make U.S. dollar dominated assets more attractive. It would mean that the savings accounts as well as the Treasury Bills would generate more interest. We hope this indicator can tie the game together and offer some interesting learning aspects that relates the rates and different accounts.
2. A time indicator
We included a timer that not only indicate how long the game has elapsed, but also we hope to use that timer to indicate the expirations of various accounts. We have included that function in each of the accounts to take in a time. (i.e. 1 year CD will expire after 1 year is up). This global timer will take away any confusing how much time there is left in the game. We will also make each accounts possibly flash when the accounts are about to expire. Since this is not part of the three tasks that we are asking the users to do, it is still not implemented in the game.
3. Tool tips to show supplementary information (help)
The tooltip is to help users understand how to carry out each of the tasks. We also included the definition for the Federal Reserve Rate. We will also include the definition of each of the account in later improvements. The tool tip help the user know eactly what needs to be done during each of the step while they have their mouse hovered over an object. It is effective in terms of how it gives the user guidance during during the times that they need the help. My usability study participant also really liked the tool tip. He also said that he do not think he could have completed the task if tool tips were not present. I believe this addition is a great improvement.
Screen shot of these additions:
Method (10 points)
Participant (who -- demographics -- and how were they selected) (1 paragraph)
The participant is a senior at UC Berkeley student with a major in Molecular and Cellular Biology. He currently only has checking and savings account. He is not very knowledgeable regarding other types of accounts and have relatively no background in finance. He works in a lab on compus as a assistant. He does not have any plans for retirement at this point. He is pretty conservative in terms of personality. In the game playing aspect, he plays a sorts of games. He likes to play strategy games and RPG games more than flash games, but he is not totally out of touch with flash based games. He commented that some of the speed based flash games made his reflexes better.
He is chosen as a participant because he fits the criteria of our targeted audience. He fits with most of the aspect we try to target in the persona. Our game tries to target college grads and raise their awareness in terms of different investment options. His age and his lack of background in finance seems to be a good persona for our game. Another criteria for selection was the amount of games that the user plays. The participant plays a fair amount of game and even find speed games helping him with reflexes.
Apparatus (describe the equipment you used and where) (1 paragraph)
The study was carried out in the computing center in the Moffit Library. I used my Thinkpad to record notes while the participant carried out the task on the library computer. The machines in the computing center was running Windows XP. The machine was a desktop with regular keyboard and mouse. I also used my cell phone as a timer to see how long the participant took to complete each task. There were no other equipments around.
Tasks (1/2 page)
- [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
Creating An Account (Easy)
To create an account, the player can either click and drag an account from the bottom purchase bar to the main game area above. The scroll bar will then appear asking user how much they wish to put into the new account. The accounts available to purchase rotate randomly (though systematically) over time; this allows for more accounts to be used in a smaller area. When the player hovers the mouse over the purchase bar. The tool tip tells them what exactly needs to be done to purchase an account. In this task, I hope to observe how fast the user releases that the bottom bar is the purchase bar, and if the user will point his mouse there after he sees the title "Purchase Bar". I also hope to observe what the user does after he reads the tooltip. From our low-fi prototype, this seems to be the most difficult task to be carried out by the users. Most of them spent the most time on this task main because "Purchase Bar" title was not present and tooltip was not available. I hope to confirm that these two additions have facilitated the user interaction with the game.
Moving/Rearranging Accounts On The Screen (Medium)
To move accounts and rearrange them on the screen, the user has to click and drag the title bar of the screen of the account box. This is modeled after the popular windowing system where the user can click and drag windows around on the desktop. We followed the heuristic of recognition rather than recall style in hopes that the user can recognize the account box just like a regular window on the desktop. We give the user the ability to rearrange accounts as the user might want to put accounts that require frequent transfers closer to one another to minimize the time to move the mouse around. In this task, I try to ask the user if they recognize this system as something that they perhaps have seen before to confirm the recognition rather than recall assumption that we made. Most the users from the previous studies found this task to be a fairly simple one. I would also time the user to see if they can finish this task in less than 10 seconds. Lastly, I would confirm with the user to see if this interface to move around accounts is intuitive.
Transferring Money Between Accounts (Hard)
To transfer money from account A to account B, the player has to click and hold the money area of the account and drop it into the destination account. This action freezes the source account to let the player decide how much money to transfer. One the destination accounts, they are also presented with the slider bar to select the transfer amount. To confirm, all they have to do is click the mouse button anywhere in the slider box. To cancel, they can click outside of the slider box, press the escape key, or simply confirm with a transfer amount of $0. In this task, I want to see how the user go about start a transfer. With the introduction of the tooltip, I want to confirm that the user is able to quickly find out how to complete this task. I also want to time the user for this task and hope that they complete it under 20 seconds. As with other task, I would ask the user whether the interface is intuitive in terms of helping him complete the required task.
Procedure (1 page)
- describe what you did and how
I first welcomed the participant and introduced the system, the objectives that the game tries to acheive and the things that will be part of this study. I read out loud the items on the consent form and asked the participant to sign the consent form. I then asked then asked the participants to think out loud of the procedures that he is doing. I also told the participant to explore around the system for a bit if he is stuck to see if he can resolve the issue by himself. By not giving away any clues about the tasks that the participant is about to do, I showed off just the tooltips on the Federal Reserve Targeted Rates. There were no other tasks that can be used as a demonstration as all the functionalities of the game involve these three crucial tasks. I encouraged the participant to be active and inform me as soon as he is confused or is troubled in doing a task. Lastly, I asked the participant to make comments on interfaces that he finds unintuitive.
We then moved on to the three tasks:
- Purchase an account.
- Rearrange the position of an account.
- Move money from one account to another.
I read off the instruction from the script that we wrote. I tried to observe participants reactions whether it is confusion, complement, frustration, or error for each of the task. I tried to answer very limited questions during the study and encouraged the participant to use our newest addition of tooltips to resolve problems. I also used my phone as a timer to see how long it takes for him to complete each task. I recorded all the incidents by typing them up on my laptop. At the end of the study, I inquired the participants regarding the overall structure of the user interface and asked him for suggestions. I gave him another chance to ask me questions and thanked him for his involvement.
Test Measures (5 points)
- Describe what you measured and why (1/2 page)
As a group, we agreed to measure four dependent variables for the quantitive measures. To better understand their behaviors and improve on the user interface, I also asked the participants to comment on each positive and negative items as well as detailed aspect of confusion and errors while carrying out the task.
The following are the four dependent variables that I used as test measures during my study:
1.The amount of time it took for a participant to finish a task.
- If the participant used very little time to complete this task, it would mean that the interface design is fairly intuitive. If the participant uses a long time to figure out what needs to be done, it means that interface is not intuitive and requires a redesign. At this point, I would ask them details about the incident to better understand how the redesign might help the interface to become more intuitive.
2.The number of logged "positive" events (e.g., user says "cool" upon seeing something they like) for a task
- This would be an evidence that the user like the feature. It would be nice to ask more questions about why they like the feature and what aspect of it inspired them to regard the feature as something cool.
3.The number of logged "negative" events (errors) for a task
- This would be a sign that the user is not very content with the interface design. This could attribute to something that is not intuitive and causes frustration. Since only one user is present, it would help with the interface design if a detailed question is asked regarding the negative matters.
4.The number of times a participant hesitated because he/she was unsure how to proceed (per task)
- This is another negative effect that we have to take into account. With the addition of tooltips, we do not hope to see this even happening. A detailed question would be to ask whether the tooltip is unclear or if the participant simply doesn't know what to do next.
Results (10 points)
- Results of the tests (1 page)
Dependent Variables
| Time to Finish Task | # Positive Events | # Negative Events | # Times Hesitated | |
|---|---|---|---|---|
| Task 1 | ~43 Sec. | 1 | 2 | 4 |
| Task 2 | ~ 3 Sec | 1 | 0 | 1 |
| Task 3 | ~ 10 Sec. | 0 | 0 | 1 |
Summary of the Process Data
Task 1
With no surprise, this task took the longest to accomplish as demonstrated in previous prototypes. The tool tip and the "Purchase Bar" title seems to have alleviated some concerns, but the participant is still unsure regarding some aspects of the this task. This might attribute to the fact that this is the first task that the user performs and might require some time to just adjust to the system. I asked the participant to purchase another account afterwards, and it was almost instantaneous. Some of the comments that the participant made regarding this task is of the following:
Positives:
- The participant really liked the tooltip because it tells him exactly what to do mostly.
Errors:
- The participant transferred money into savings account and tried to purchase accounts. He noticed he could not complete the task, and then figured out that the money had to be in the checking account in order to purchase accounts. We might need to include this in the tool tip indicating the money should be in the checking account.
- He tried to purchase an account in red and was not successful. He did not realize accounts in red indicate that you have insufficient funds.
Hesitations:
- When the participant tried to make a purchase of an account, the account rotated into another set of 6. He dragged the account out and dragged it back to where it was (didn't release it into the main screen) and said, why does that it should not rotate otherwise he does not know exactly which account he is purchasing.
- I asked the user to purchase an account and he was hesitating as to which account he should purchase. He also asked what does these accounts do? What exactly is a Roth IRA?
- After he made the account, he started to transfer money. When slider was on the right side of the bar, the tool tip covers the amount that is being transferred. He was wondering how much he is putting into the new account.
- He was not sure why the timer says 5 years while there is 30 year CD's available to purchase.
Task 2
This operation took the least amount of time. He noted the windows analogy and rearranged the accounts with no trouble.
Positives:
- The participant noted that the mouse change which helped him complete the task
Hesitation:
- The participant complained that the account should not cover the timer and the interest rate nor the "Purchase Bar" Title.
Task 3
He found this task quite intuitive. He went right along to put the mouse over the account and the tooltip guided from there on.
Hesitation:
- This is related to one of the earlier hesitations that he had. After he transfer the money to the destination account. When slider was on the right side of the bar, the tool tip covers the amount that is being transferred. He was wondering how much he is putting into the new account.
General Feedback
- The participant thought that all the tasks are achievable if he goes over them just once.
- He commented that the initial task was confusing because of the layout of the interface and that it was the first time he saw the system.
- He commented that he would like to see the total amount of money that he has (like a game score to be displayed).
- He wanted all the accounts to be available to purchase at all times.
- He wants to have a account history to see how the accounts are doing.
- He wants some definition as to what each of the accounts mean.
- He played around with the system at the end and opened up around 10 accounts, he felt it was hard to manage all of them at once.
Discussion (15 points)
- What you learned from the pilot run (1 page) what you might change for the "real" experiment? what you might change in your interface from these results alone? If you'd like, you may include results and assumptions from other group members' tests here as well.
The pilot study demonstrated that we have a much improved user interface. I specifically
want to comment on the tooltip. I think it really helped in getting my participant in
finishing the task. As I read other team member's result, it also significantly correlate to
the fact that the addition of tooltip has significantly decreased the amount of time needed
on a certain. I would like to add more educational values and take into accounts of design flaws for the "real" experiment. I think one thing that it would be hard to do in the "real" experiment with many users would be the observer asking the participants specific questions regarding the negative comments. For the "real" experiment, we might want to give the users a survey after the study to see how they felt. Variables can include frustration, confusion etc. However, the downside would be that these variables are subjective and might be confounding depending on the personality of the individuals who we hope to study during the "real" experiment.
We still have to address some issues and many of them involves addition of the tooltip context. For example, my participant was not sure which account to deduct the money from in order to buy new accounts. We might hope to add that line in the purchase bar tool tips. I asked him why he thinks that way. His family background definitely played a big role there. His parents tell him to put everything into savings which automatically make him assume that savings account is the primary account. I think we cannot just assume things that we think is intuitive to us, it might not be intuitive to others. Another addition might be mentioning of the accounts in red are not purchaseable as well. I read through some of my group member's results and it seems that their participants also had confusions in similar areas. Perry's participant brought up the idea of negative feedback which I think should be implemented which gives users more feedbacks.
The task that took the longest was the first task. It is also the one with many negative events as well as hesitations. I have check other team members results and it seems they have similar findings. My participant was able to complete the same task almost instantaneously the second time around shows that we might need to have a tutorial of some sort that the first time users can refer to. Such addition would reduce the confusions that they have regarding the task. Another issue that my user noted is the tooltip is covering some important aspect of an game playing object such as the slider bar tooltip covering the amount of money being transferred. This would require improvement over better placement of the accounts. Currently, our tasks are all UI focused. We have not yet added much educational aspects to the game yet. I believe this should be one of the priorities on our list for the next phase iteration. We first have to tie in the Federal Reserve Targeted Rates into the accounts and make them show real values with respect to the indicator. Secondly, I believe there should be tooltips in aiding users in knowing the definition of the accounts. My participant was not sure which account to purchase. On one hand, it confirms that our user does not have an extensive knowledge about finance and this is the right game to get him to know about investment options. On the other hand however, it demonstrates that our game still lacks the educational value that we hope to achieve.
Another game-play value that Perry mentioned was to add a base score. My participant did not say that a base score is needed, but he hopes to see a global score or global amount to be displayed. My participant says that with each account going up and down so fast and having so many acccounts at the same time, it seems to make more sense if a global amount is added. Since the goal is to make the most of the money in the alloted time frame, it seems logical to have that score and that comparison with other player scores can add competitiveness as well as replay value to the game.
In the next iteration, we would also like to make the game playable. Instead of having individual tasks, the user should be able to play the game for the entire length allowed. This would give us even more knowledge of what needs to be improved on. Without only letting the user be able to carry out three tasks, it minimizes the scope as a whole.
Overall, I think the addition of tooltip is great and have recieved positive comments from the users. Our game play values and educational values still need to be greatly modified in the next few iterations.
Appendices (5 points)
- Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
- Raw data (i.e., entire merged critical incident logs)
Prototype
Consent Form
Demo Script
- Introduction
- Introduce yourself to help relax the subject.
- Explain the game, detailing the overall premise and goal of it.
- Explain the purpose of this study (e.g., to identify any usability issues that may necessitate a design change).
- Show the participant the consent form, giving them time to read it. Re-iterate that he/she can quit at any time.
- After the participant has signed the consent form, ask him/her for his/her demographic information:
- age
- sex
- education level and major(s) if appropriate
- frequency of playing games, and if he/she plays games, which types of games does he/she play?
- level of financial knowledge, how knowledgeable does he/she thinks he/she is?
- has he/she planned for retirement?
- if yes, how has he/she planned for retirement?
- if not, when does he/she plan to start planning?
- Explain how the pilot study will be conducted.
- Note that he/she will be given three in-game tasks.
- Explain that you will be unable to provide help and the reasoning behind it.
- Explain how to "think aloud" and the reasoning behind it.
- To demonstrate how the game works, give a short demo of the system.
- During the demo, show tool tips to the participant.
- Task 1 [Easy]: Buying (creating) an account: Ask the participant to buy an account.
- Task 2 [Medium]: Rearranging the position of an account: Ask the participant to move an account to a different position.
- Task 3 [Difficult]: Moving money from one account to another: Ask the participant to transfer $X from one account to another.
- Outro
- Ask the user which task he/she found the most difficult; which task was the easiest -- for both questions, ask the reasoning behind the participant's selection.
- What would he/she recommend improving (interface-wise) and why?
- Thank the user.
Raw Data
Demographic Information
- Male
- 21 years old
- Major in MCB
- Works in a biology lab on campus
- Plays game, mostly Strategy or RPG. plays flash games and think the speed games can help with reflexes.
- Does not plan to formally plan his retirement anytime soon
- Currently have checking and savings account. No financial background regarding other accounts.
- Parents tell him to put all money in savings account.
Task 1
With no surprise, this task took the longest to accomplish as demonstrated in previous prototypes. The tool tip and the "Purchase Bar" title seems to have alleviated some concerns, but the participant is still unsure regarding some aspects of the this task. This might attribute to the fact that this is the first task that the user performs and might require some time to just adjust to the system. I asked the participant to purchase another account afterwards, and it was almost instantaneous. Some of the comments that the participant made regarding this task is of the following:
Positives:
- The participant really liked the tooltip because it tells him exactly what to do mostly.
Errors:
- The participant transferred money into savings account and tried to purchase accounts. He noticed he could not complete the task, and then figured out that the money had to be in the checking account in order to purchase accounts. We might need to include this in the tool tip indicating the money should be in the checking account.
- He tried to purchase an account in red and was not successful. He did not realize accounts in red indicate that you have insufficient funds.
Hesitations:
- When the participant tried to make a purchase of an account, the account rotated into another set of 6. He dragged the account out and dragged it back to where it was (didn't release it into the main screen) and said, why does that it should not rotate otherwise he does not know exactly which account he is purchasing.
- I asked the user to purchase an account and he was hesitating as to which account he should purchase. He also asked what does these accounts do? What exactly is a Roth IRA?
- After he made the account, he started to transfer money. When slider was on the right side of the bar, the tool tip covers the amount that is being transferred. He was wondering how much he is putting into the new account.
- He was not sure why the timer says 5 years while there is 30 year CD's available to purchase.
Task 2
This operation took the least amount of time. He noted the windows analogy and rearranged the accounts with no trouble.
Positives:
- The participant noted that the mouse change which helped him complete the task
Hesitation:
- The participant complained that the account should not cover the timer and the interest rate nor the "Purchase Bar" Title.
Task 3
He found this task quite intuitive. He went right along to put the mouse over the account and the tooltip guided from there on.
Hesitation:
- This is related to one of the earlier hesitations that he had. After he transfer the money to the destination account. When slider was on the right side of the bar, the tool tip covers the amount that is being transferred. He was wondering how much he is putting into the new account.
General Feedback
- The participant thought that all the tasks are achievable if he goes over them just once.
- He commented that the initial task was confusing because of the layout of the interface and that it was the first time he saw the system.
- He commented that he would like to see the total amount of money that he has (like a game score to be displayed).
- He wanted all the accounts to be available to purchase at all times.
- He wants to have a account history to see how the accounts are doing.
- He wants some definition as to what each of the accounts mean.
- He played around with the system at the end and opened up around 10 accounts, he felt it was hard to manage all of them at once.
Very General Comments
- tool tip is neccessary to play the game
- tool tip persists
- purchase bar is bad, need to show all options at all times
- why do i have to use checking to buy these things? was transferring money to savings to make purchases?
- game is hard to manage when there are too many windows open.
- windows do not lock in
- better account grouping
- no total balance
- shouldn't make the account to be able to block the timer and fed interest rate and purchase bar
- cannot track history
- tool tip covers the amount needed to transfer.
- does not know the definition of the accounts and cannot learn them.
