PilotStudy-Group:Group4-TrinhVo

From CS 160 Fall 2008

Introduction

System

Programming4Lyfe, is a serious game that teaches people who are new to programming.
Our targeted user groups are:

Students in elementary through secondary schools: Programming fundamental classes are scarce in pre-secondary and secondary education while many students hope to started early into the field. Most importantly, the majority of them would rather play than read books.
Students in colleges: They want to equip themselves with the most basic skills in programming before taking off to the professional world with as little time as possible, but most of them do not major or minor in Computer Science to prepare themselves to work with software.
Researchers and professions in any industry: Programming skills are demanded for processing mass data from their research and product development. This user group typically does not have time to dedicate a fix block of day time in learning general programming concepts in academic setting.

First, learners may want to repeatedly practice in order to fully understand programming concepts.
Secondly, a serious game can be played at any time on their computers to have fun and learn at the same time. Such a game turns the tedious task of studying abstract concepts to an interesting game as entertainment while learning. Our targeted users are usually busy with work and family so that they would prefer something more entertaining than serious.

Purpose

The purpose of the experiment to evaluate how well the users would interact with our games. And we want our interface to be easily accessible to the users, so we want to minimize user's confusion as much as possible. We also want to minimize the time it takes a user to move the mouse cursor and click. Therefore, the game would be more enjoyable to users. In this experiment, we will put a user through a set of tasks designed to test how well our user interfaces are implemented.

Implementation and Improvements

Things we added to our games since the first interactive prototype:

In “What Goes There”
- Tutorial was added and revised twice after adding it.
- Pause, replay and help buttons and functions were added to the game play screen.
- Location of the cross mark is changed because it was unnoticeable by the users in our previous experiment.

What goes there's screenshots:

In “Datris”
- Tutorial was added and revised once after adding it.
- The buckets’ shapes was changed to match the shape of a bucket in the real life because we received feedback about that the original bucket’s shape might confuse players.
- Pause button will hide the gaming content so that the player will not be able to cheat.
- Revision of the help document because we received feedback about the help document was too lengthy from the heuristic evaluation.
- This new version also provides users with more feedback using a ding/buzz sound and a check mark/cross mark corresponding to the user's answer.
  - If the block was dropped into the right bucket type, a green check mark pops up, and the ding sound is played if the sound is turned on.
  - If the block was dropped into the wrong bucket type, a red cross mark pops up, and the buzz sound is played if the sound is turned on.

Datris's screenshots:

In “Bug-A-Boo”
- Game format changed for aesthetic design and excitement of playing.
- Tutorial was added and revised once after adding it.
- Sound added for the user's excitement and also for providing extra feedback.
  - If the sound is turned on, the ding sound is played if the buggy balloon is shot down, the text on top turns yellow and says: "BUG".
  - If the sound is turned on, the buzz sound is played if the bug-free balloon is shot down, the text on top turns green and says: "NO BUG".
- Pause button will hide the gaming content so that the player will not be able to cheat.
- Animation added if the jet crashes on a balloon.
- The jet will be repositioned at the origin if it crashes at a balloon. To provide user some amount of time to get ready after a crash, the jet is blinking and won't crash if hit any balloon during the blinking.

Bug-a-boo's screenshots:

Method

Participant

TN is a 22 year old Math major who took an introduction programming course. Even though he took that course, he still doesn't fully understand every materials and has forgotten a lot of them. He said he wanted to learn programming but he was too lazy to open his programming text to review what he had learned before, and after reading couple of pages, he got bored and gave up.
TN was randomly picked in front of Golden Bears Cafeteria when he was sitting and eating at noon.

Apparatus

The equipments consist of 1 laptop, which was for testing our games, a pencil, a piece of paper to take notes, and a digital watch for timing.
The experiment took place in Bechtel Library.
The game resolution is 1024x768.

Tasks

Easy - Datris
- Task Goal: Identify correct data type buckets and move falling data brick to the corresponding data type bucket to earn points.
- Player will be introduced to String, boolean, integer and double variables in the tutorials. The tutorials also instructs players to use the left and right arrow keys to move the falling block, and the down arrow key to drop the brick to the desired bucket. Then the player can start the game whenever he/she is ready by pressing "Ready to Play" button on the bottom of the tutorial screen.
- We also added instructions on how to operate the game on the bottom of the play area.
- If the data type block was dropped into a correct bucket, then the score increases by 5; a green check mark pops up, and dinging sound will be played if the sound is turned on. If the data type block was dropped into the wrong bucket then the user loses one heart; a red cross pops, and the a buzz sound will be played if the sound is turned on. Also, the user can make three mistakes in this game and on the fourth mistake the game is over.
- The block falls down faster as level goes up.
- This task would test the user's understanding of our user interface structures. I wanted to see if buttons are visible enough and user's perception of those buttons' functionality. I also observed the user's response to labels on buttons and instructions on the game screens. If these lead to any user's confusion or false responses, then I recorded them as UI usability problems. These were questions I asked during the test: "Can the user see if the text on top of the falling block? How intuitive are the buckets? What would you do when you need help? How do you open the Help and how often? How do you pause the game? What would you prefer, using keyboard or mouse to open/collapse Help, to pause/resume the game?"

Medium: "What Goes There"
- Task Goal: Based on the output of a program and the given programming codes, fill in the missing code.
- Before the game is started, the users will be instructed by tutorials. The users can skip the tutorials at anytime by clicking on the "Close Tutorial" button
- Each of the windows: the code window, output window, and desired output windows will contain labels and instructions on how each of the windows interact with each other and how they reflect the goal of the game. At first, tutorials will carry the user through a sample code fragment with which to fill in the blanks.
- At the beginning, only two blanks need to be filled for the given piece of code. As the user progresses to higher levels, the number of blanks and the difficulty of the presented code will increase to account for a wider variety of code syntax and practices.
- If so, the player will gain 20 points. Otherwise, the player can fix the code as long as there is time left. If the player completes the task prior to the expiration of the timer, the remaining time will be awarded to the player as "bonus points," which will encourage the user to play and practice a bit more.
- In this task, I wanted to see if the users knew which boxes he should fill in, and which boxes he should be looking at before filling in blanks. I also observed to see if the user knew that there's time constraints in this task, what would be the user's reaction when he typed something in the blanks, how many times the user looked for help, and what would he do if he needed helps. Since the structure of this game is little bit different from the Datris's, so I was looking for any confusions or strange reactions of the user.
- Note: This task is greatly simplified from asking the user to construct a running program from given functions in the low-fidelity prototype to choosing appropriate options for blanks in a piece of code now.

Difficult: "Bugaboo"
- Task Goal: In this game the user will have to visually "hunt" for the buggy lines of code.
- We made a lot of modifications to this game since the first interactive prototype. In this new version, the users no longer use the mouse device to shoot at blue circles with codes. They must use the arrow keys to move the jet plane around and press space to shoot at hot air balloons with faulty codes on top.
- Before the game is started, the user will be instructed by tutorials. The user can skip the tutorials at anytime by clicking on the "Ready to Play" button.
- If the user shoots at a buggy balloon, the text on top of that balloon says "BUG" and its color turns yellow, and a ding sound is played of the sound is turned on, and the user gets 5 points more. If the user shoots at a bug-free balloon, the text on top of the balloon says "NO BUG" and its color turns green, and a buzz sound is played, and the user looses one heart.
- The UI structure of this game is similar to the Datris, so I used the same questions to observe this game, but there were a couple questions came up during the test "Is it hard to move the jet around? Were the balloons moving too slow or too fast? Should the text on the top of a balloon be bigger?"

Procedure

My partners and I agree on the same script of demos & tasks, so I used the script to test on TN.
I followed the 10 steps as taught in lecture.

I introduced myself and my background.
I described to TN the purpose of this experiment is to test and evaluate our UI design. I also told him I wanted to measure task time and do some statistics on the results. I then gave him the consent form to sign. He glanced over it and signed without any questions.
I told him that it’s ok to quit at any time for whatever reason, and he didn’t have to explain why.
I talked about the equipment by showing him my laptop with the game’s main menu on the screen. I explained that I would use a watch to measure the time to complete a task and use a pen to take notes errors and any reactions of his during the tests.
I also explained to him how to think “aloud”. And the reason is that would help me know what’s going on how our UI structures affects him, and I’m supposed to write down his responses.
I explained that I couldn’t provide any help during the tests. The reason is the games’ UI should be self explained or giving clues to users to figure out what to do next. Otherwise our UI fails.
I described what each task’s goal is, and showed TN screenshots and orders of each task.
I asked him if he had any question before starting the test. He said “How long it took you to write the games?”
When the test is over. I concluded my observation by asking him if he had any other questions.
I went home and started to analyze the data I took during the tests.

Test Measure

Independent variable(s):

Device type: a mouse and a keyboard.

Dependent variable(s):

Time: Time was measure by using my digital watch. I think this data would be very helpful for finding out how quick a user can start or stop a game. Since one of the purposes of this experiment is minimizing the unnecessary time, I could use this data to see whether the games should be improved.
Error rate: I counted number of errors the user made during each task. This data is also very helpful for evaluating how well our game interfaces were implemented. Because the more number of errors, the more frustrated and confused the user will get, and this tells how bad our design is.
Number of button presses: This data is related to the time and error rate. For example, I wanted to see how many number of button presses to get a game started or to quit a game. The less is the better, so the time it takes to complete a task and the waste time will be minimized. Besides, the less number of button presses also help reduce the error rate.

Control variable(s):

Location/environment: the experiment took place in the Bechtel Library. We found a quiet area with a big table, so we could place our test equipments on it.

Random variable(s):

Age: 21
Gender: Male

Result

Quantitative

To start a game:

Task	Button Presses	Time Elapsed (Min:Sec)
Datris	3	0:42
What Goes There	3	1:10
Bug-a-Boo	3	1:00

To pause/resume a game using mouse:

Task	Button Presses	Time Elapsed (Min:Sec)
Datris	2: 1(pause) + 1(resume)	0:04
What Goes There	2: 1(pause) + 1(resume)	0:04
Bug-a-Boo	2: 1(pause) + 1(resume)	0:04

To pause/resume a game using keyboard:

Task	Button Presses	Errors	Time Elapsed (Min:Sec)
Datris	2: 1(pause) + 1(resume)	0	0:02
What Goes There	doesn't support	N/A	N/A
Bug-a-Boo	2: 1(pause) + 1(resume)	0	0:02

To get help and resume a game using mouse:

Task	Button Presses	Time Elapsed (Min:Sec)
Datris	3: 1(pause)+ 1(help)+ 1(resume)	0:09
What Goes There	4-5: 1(help) + 1-2(next or back) + 1(close help) + 1(resume)	0:12 - 0:14
Bug-a-Boo	3: 1(pause)+ 1(help)+ 1(resume)	0:09

To get help and resume a game using keyboard:

Task	Button Presses	Errors	Time Elapsed (Min:Sec)
Datris	3: 1(pause)+ 1(help)+ 1(resume)	0	0:03
What Goes There	doesn't support	N/A	N/A
Bug-a-Boo	3: 1(pause)+ 1(help)+ 1(resume)	0	0:03

To quit a game:

Task	Button Presses	Errors	Time Elapsed (Min:Sec)
Datris	1	0	0:03
What Goes There	1	0	0:03
Bug-a-Boo	1	1	0:42 due to a bug

Qualitative

Datris: TN didn't have any difficulty understanding the interface. Everything seemed clear to him. Since this game is similar to the famous Tetris. He could quickly grab the concept. His utterance was "Cool! I like the tutorial!" After closing the tutorial, it took him about 3 seconds to look around to make sure if he didn't miss anything(instructions or buttons) on the screen before he started. His next utterance was "Hey! The block is falling down so fast" when he reached level 8.

What Goes There: In this game, the UI structure is different from the Datris. After choosing this option on the Single Player Menu, its title screen showed up and took about 1 second and then tutorial showed up. He carefully read the tutorial, and this took about 1 minute before he closed the tutorial. The game started with boxes flashing. His reaction was "Cool!". It took him about 5 seconds again to look around to make sure the he didn't miss anything, then he started playing the game. His next reaction was "Cool! The output appears concurrently after I typed something!"

Bug-a-Boo: This game UI structure is similar to the Datris, so this time he didn't bother looking around when he got to the main game screen. However, it took him about 50 seconds to go over the tutorials. During the game, his reactions were "What? The plane has momentum?", "Wow! Cool explosion animation." The game was over after he scored 30 points. And he said "Dude, this game is hard." I asked him why the game was hard. He said that he didn't know the jet had momentum, and the balloons sometimes moved up and down and hit the jet. He was so surprised. I explained to him the reason the jet had momentum is to make the jet fly more realistic and look fun, and the reason that balloons sometimes move is they tried to avoid overlapping one another. He agreed and said it was a fun game. Then I asked him to quit the game, and this was where we found the bug in this game. The game screen was too big and it covered up the "Return to Main Menu" button. So there's no way he could go back to the main menu except pressing the Refresh button on the web browser.

Comments
- TN made a few suggestions to the games:
  - The game should have some kind of rewards to help users gain back hearts. It's so easy to lose the game. This might be frustrating.
  - Each game should have a back button to go back to the Single Player Menu instead only the "Return to Main Menu."
  - When the help box is opened, it shouldn't cover any part of the play area.
  - Make the UI structure of the "What goes There" more similar to both Datris and Bug-a-Boo, even though it was fine to him.
  - Both Datris and Bug-a-Boo should have borders around the game areas.
  - The gray background color was a little boring.
- Things TN thought were good:
  - He liked the concurrent output of the game: "What Goes There"
  - He liked the idea of applying the famous Tetris to Datris.
  - He liked keyboard supporting for pause/resume and help in both Datris and Bug-a-Boo.

Discussion

I wrote both Datris and Bug-a-Boo, so I understand my games pretty well and made a lot of assumptions of what a user should know about the games, but when I tested them on somebody, I realized that I had made so many foolish assumptions. For example, after the first revised version of Bug-a-Boo came out, it didn't have any tutorials and explanation what a buggy balloon was. Users would think a buggy balloon was the one that had many insects flying a round it, and they thought they were supposed to shoot at them, but when game started they didn't see any bugs flying around and they were looking for them. So my partner(Karen Huang) decided to make tutorials for every game to minimize this kind of confusion.

I think it would be much better to have another partner to work with while conducting the experiment. It was very hard to collect all data and guide the participant. Sometimes my mind was wandering around and distracted by thoughts of how to improve the games. It also would be better to have more participants to compare my data, because each participant has different programming background and experienced of game types or applications. Therefore I don't totally trust my quantitative data. Also, I wish I wasn't so busy with CS162, so I could have more time to fix silly bugs in both Datris and Bug-a-boo right after my partners' experiments.

One thing I noticed about our game is it doesn't really fit for those who never have any programming experiences. It actually doesn't teach a newbie how to program, but instead it's just a game to help him/her practice and develop more reflexes on what he/she had learned before.

During the tests of both Datris and Bug-a-Boo, I realized after a game won or game over, it took a user 2 button presses to get the game started. First, the user has to click on the "Replay" button, and then move the mouse cursor way to the middle of the play area to click the "Start" button to be able to play the game. Some participants then had to move the mouse cursor away from the play area and accidentally clicked somewhere outside of the focus area; Therefore they couldn't use the arrow keys to control, and I thought it was a bug.

My suggestions are:
- Instead of having a "Start" button after the user clicking the "Replay", I should have a countdown message for 4 seconds(eg. 3, 2, 1 and Go) to give the user some amount of time to get ready for the new game.
- The help box should be placed away from the play area.
- There should be borders around the game play area to make it more distinguishable from the background screen.
- The Bug-a-Boo resolution should be smaller so it wouldn't cover up the "Return to Main Menu" button
- In the Bug-a-Boo game, when a buggy balloon is shot down, it should explain why or what the bug is.
- Each game should have a "Back" button to return to the Single Player Menu
- In the Single Player Menu, we shouldn't split up to more levels, since each game has its own level increment.
- The "What Goes There" should have more questions.

Appendices

Materials

Raw Data

Task times:

Intro and Questions: Start(12:35 pm), End(12:42pm)
Task1 - Datris: Start(12:45pm), End(12:51pm)
Task2 - What goes there: Start(12:57pm), End(01:10pm)
Task3 - Bug-a-Boo: Start(01:12:pm), End(01:18pm)
Conclusion and Questions: Start(01:19pm), End(01:20pm)

Number of errors:

Task1 - Datris: 0
Task2 - What goes there: 0
Task3 - Bug-a-Boo: 1 - resolution bug

Number of button presses:

To start a game:
- Task1 - Datris: 3 : 0'42
- Task2 - What goes there: 3 : 1'12
- Task3 - Bug-a-Boo: 3 : 1'00

To pause/resume using the mouse:
- Task1 - Datris: 2 : 0'04
- Task2 - What goes there: 2 : 0'04
- Task3 - Bug-a-Boo: 2 : 0'04

To pause/resume using the keyboard:
- Task1 - Datris: 2 : 0'02
- Task2 - What goes there: N/A
- Task3 - Bug-a-Boo: 2 : 0'02

To help/resume using the mouse:
- Task1 - Datris: 3 : 0'09
- Task2 - What goes there: 4-5 : 0'12 - 0'14
- Task3 - Bug-a-Boo: 3 : 0'09

To help/resume using the keyboard:
- Task1 - Datris: 3 : 0'03
- Task2 - What goes there: N/A
- Task3 - Bug-a-Boo: 3 : 0'03

To quit a game:
- Task1 - Datris: 1 : 0'03
- Task2 - What goes there: 1 : 0'03
- Task3 - Bug-a-Boo: 1 : 0'03