PilotStudy-Group:Group4-KarenTran

From CS 160 Fall 2008

INTRODUCTION

System

Programming4Lyfe is a serious game that aims to make learning the concepts of beginning programming easy and fun, by abstracting away all the hard-to-grasp-at-the-beginning concepts and developing a fun yet intuitive interface conducive to understand the crucial elementary building blocks of Computer Science.

The primary problem my cs160 group finds is that the programming learning curve is fairly steep for those who are just beginning to learn Computer Science. It is hard for beginners and children to learn concepts without going through the pain of learning syntax and compilation. The game hopes to serve as entry-point to people who would like to get a taste of what programming is. Therefore, the range of users targeted for this game is very broad. Anyone who is interested in Computer Science, in programming specifically, should pick up a copy of Programming4Lyfe.

Purpose and Rationale of the Experiment

The purpose of the experiment is to observe and examine the interaction of users with Programming4Lyfe. For each atomic, essential concept of programming, we devised and structured a fun game, with the rationale of teaching it to the users without making them feel they are under strict academic environment. In our game model, we chose three representative concepts of data types, syntax style, and debugging. We want to games to be as easily comprehensible and user-friendly to the users as possible. Therefore, in this experiment, we would like to measure the performance of our game system by evaluating how it is understood and enjoyed by a targeted user.

IMPLEMENTATIONS AND IMPROVEMENTS

Things we added to our games since the first interactive prototype:

In “What Goes There”
- Tutorial was added and revised twice after adding it.
- Pause, replay and help buttons and functions were added to the game play screen.
- Location of the cross mark is changed because it was unnoticeable by the users in our previous experiment.

In “Datris”
- Tutorial was added and revised once after adding it.
- The buckets’ shapes was changed to match the shape of a bucket in the real life because we received feedback about that the original bucket’s shape might confuse players.
- Pause button will hide the gaming content so that the player will not be able to cheat.
- Revision of the help document because we received feedback about the help document was too lengthy from the heuristic evaluation.
- This new version also provides users with more feedback using a ding/buzz sound and a check mark/cross mark corresponding to the user's answer.
  - If the block was dropped into the right bucket type, a green check mark pops up, and the ding sound is played if the sound is turned on.
  - If the block was dropped into the wrong bucket type, a red cross mark pops up, and the buzz sound is played if the sound is turned on.

In “Bug-A-Boo”
- Game format changed for aesthetic design and excitement of playing.
- Tutorial was added and revised once after adding it.
- Sound added for the user's excitement and also for providing extra feedback.
  - If the sound is turned on, the ding sound is played if the buggy balloon is shot down, the text on top turns yellow and says: "BUG".
  - If the sound is turned on, the buzz sound is played if the bug-free balloon is shot down, the text on top turns green and says: "NO BUG".
- Pause button will hide the gaming content so that the player will not be able to cheat.
- Animation added if the jet crashes on a balloon.
- The jet will be repositioned at the origin if it crashes at a balloon. To provide user some amount of time to get ready after a crash, the jet is blinking and won't crash if hit any balloon during the blinking.

NOTE: The screenshots for these changes are shown in the storyboards of each task in the "TASKS" section below.

METHOD

Participant

T. is a first-year undergrad student at UC Berkeley. Although undeclared at this point, he is working towards being admitted into the Economics program. He loves playing computer games in his spare time and has an impressive knowledge of virtually all video games in the market. It was because of this hobby that he developed a keen interest in Computer Science and hopes to program his own game some day. He expressed interest in being an evaluator of Programming4Lyfe when he first heard of it and volunteered to participate in the usability pilot study.

Apparatus

The user was given a gateway laptop with only the window for the games open.
A notepad and a pencil were used to record the observation of the user as he tested out the games.
A stopwatch was used to make time measurements.

Tasks

Easy Task: Datris

Objective: The game aims to familiarize the users with the concept of data types. Datris introduces some of the most common data types used across programming languages: String, int (integer), double (decimal), and Boolean (true/false).

Goal: Each falling brick falls into one of the four data types introduced above. As each brick appears, the user’s task is to use the keyboard to direct the brick into the correct data type bucket.

How to Play:
- Once the Datris Task is picked, the User Interface for the game is displayed.
- The game first starts out with a tutorial (of three pages): the first page stating the goal of the game, and giving examples for each data type; the second page explaining the interface of the game, and the uses of the screen buttons; and the third page describing the mechanics of the game, of how the game is played. At any point, there are “back” and “next” buttons at the bottom of the screen to navigate through the tutorial pages.
- Once the user is done with the tutorial, the “Ready to Play” button at the bottom of the screen is used to start the game.
- Player can use left and right arrow keys on the keyboard to move the falling brick, which is representative of a data type.
- The task is to sort the brick into the appropriate data-type bucket at the end of the screen. There are five buckets: String, int , double, true and false. The description of each bucket is self-explanatory. If a falling brick contains a String, the brick should be directed to fall into the String bucket, etc.
- For each brick sorted correctly, a green checkmark appears, followed by ‘ding’ sound, indicating that the correctness of the sorting. On the other hand, a red cross is displayed instead, coupled by a “buzz” sound to indicate the incorrectness of the sorting.
- For each brick sorted correctly, 5 points are earned. On the right hand side, the current score of the game is shown at the top corner in the score panel. For each ten points earned, the level is increased and the rate at which a brick is falling also increases. For each wrongly sorted brick, a Health Point (HP) is lost; the user can lose at most 4 HPs before the game is over. Once the user reaches 100 points, the game is won.

1. The initial Datris screen displayed when DATRIS TASK is selected.	2. First Datris Tutorial Page.	3. Second Datris Tutorial Page.	4. Third Datris Tutorial Page.
5. When a brick is sorted in the correct bucket, a green checkmark appears, coupled with a 'ding' sound.	6. When a brick is sorted in the incorrect bucket, a red cross appears, coupled with a 'buzz' sound.	7. The game is won when the player earns 100 points.	8. The game is lost when the player loses all four Health Points.

Medium Task: What Goes There?

Objective: The objective of “What goes here?” task is to familiarize the users with the coding style and syntax of the Java programming language.
Goal: The user is given a partially complete snippet of code and the task is to fill in the missing pieces to make the code an output that matches the desired output.
How to Play:
- Once the “What Goes There?” Task is picked, the User Interface for the game is displayed.
- The game first starts out with a tutorial (of three slides), stating the goal of the game and the description of the interface of the game and how it is played in three easy steps.
- The first problem is presented with only two missing pieces of code, with the desired output displayed in the “What We Want” window. The user can try many inputs to fill in the blanks as he wants in the time allowed.
- If the task is finished correctly before time is up, the player earns 20 points and moves on the next level. The remaining time is converted into bonus points. The point of this strategy is to encourage the players to practice more and be able to finish the task as fast as he can to earn maximum points available.
- As the player moves higher and higher up the levels, the complexity of the code snippets increases as well as the number of the missing pieces.

1. The initial What Goes There? screen.	2. First What Goes There? Tutorial Page.	3. Second What Goes There? Tutorial Page.	4. Third What Goes There? Tutorial Page.
5. When tutorial is complete, player is asked if s/he is ready to play.	6. If all the blanks are filled in correctly, a green checkmark appears at the Output window.	7. If the blanks have not been filled in correctly, a red cross appears at the Output window.	8. If the player could not fill in the blanks correctly within the time limit, the game is lost.
9. If all the blanks have been filled out correctly before timer expires, the game is won with the remaining time converted into bonus points.

Difficult Task: Bugaboo

Objective: The objective of this game is to teach users to identify bugs by just looking at the line of code.
Goal: The goal of the game is to hunt and shot down buggy hot air balloons.
How to Play:
- Once the Bugaboo Task is picked, the User Interface for the game is displayed.
- The game first starts out with a very simple tutorial stating the goal of the game and of the mechanics of the game.
- When Bugaboo is started, the task is to maneuver the jet with the arrow keys so that it would not crash into any hot-air balloon and to identify the buggy balloons and shoot them down. The jet can fire at a buggy balloon by pressing the spacebar key.
- The balloons will move randomly around within the bounds of the playing screen. Therefore, the jet should be careful of when the balloons are passing by each other because it might intent to shoot a buggy balloon but it the bullet might end up shooting the non-buggy balloon instead.
- For each buggy balloon shot down, a ding sound is played with a message “BUG” appears, and five points are earned. For each non-buggy balloon shot down, a buzz sound is played with a message “NO BUG!” appears, and a Health Point (HP) is lost. The player has a maximum of 4 HPs before the game is over. The game is won when the player reaches 100 points.

1. The tutorial for the Bugaboo game is displayed when the game is selected.	2. Once the tutorial is complete, the Start page for the game is shown.	3. If the jet fires a buggy hot-air balloon, the "BUG!" message appears and a 'ding' sound follows.	4. If the jet incorrectly fires a non-buggy balloon, the "NO BUG!" message appears and a 'buzz' sound follows.
5. After firing 4 non-buggy balloons, the game is lost.

Procedures

The experiment is conducted in a quiet environment of a Cory classroom.
I first introduced myself, giving a brief overview of my background, and a broad overview of cs160.
The user is introduced to the purposes of the game and of the experiment. I stated clearly that I wanted to evaluate the game being devised and developed by my group and any confusion of any kind is solely the responsibility of the game’s failure to convey its purposes and not the user’s fault. The user is also told that he is free to leave at any point in the game for any reason.
I also told him that throughout the experiment, I will be observing his interaction with the game. I will record some statistics to evaluate the performance of the games, and the results will only be used by my group to improve the game’s implementation.
The user is then given a consent form, which he glanced over and signed without any question.
The user is then given the laptop with the games already loaded at the main screen.
I explained to the user that the game should be very self-explanatory and that I will not be providing any help because I would like to evaluate how well the user interfaces of the games and the games themselves are understood and any “manual” outside help would contaminate the results.
Having said that, I also asked the user to think aloud because it would help me a lot in my observation of how well the user interface is implemented.
I showed him screenshots of each of the game tasks and briefly explained to him the objectives and the goals of each. He asked how each game is played, to which I answered that there is a tutorial for each game and the mechanics of the games should be very clear from those tutorials.
The user is ready to start the experiment and I took notes of how long it took him to complete each task, of how many practices he tried until he won a task, and of any comments that he thought aloud.
When the test is over, I asked him if he has any questions and if there are any suggestions/comments he would like to make.
The user is then shown the notes I’ve taken during the experiment.
I thanked him for his time and we went home.
Finally, I started my analysis of the experiment.

TEST MEASURES

One of the resounding important principles of programming is “practice makes perfect.” Therefore, one of the measurements that I decided to make is the number of times a game task is tried until the user wins the game.
Since there is a limit to how many errors a user can make before a game is over, it is pointless to measure the number of errors the user makes. But what is a useful measurement to make is the rate at which a task is learned and how quickly the game is won.
I also measured how long the user spent time going over the tutorial before starting a task.
I also measured how many times the user paused the game due to confusion and clicked on the “Help” button to go over the tutorial again.

Datris Task Measures

Description	Measure	Game Result
Time the user spent on tutorial:	19 sec	N/A
Number of times tried before the game is won:	5	N/A
Duration of first round played:	21 sec.	Lost
Duration of second round played:	1 min 28 sec.	Lost
Duration of third round played:	2 min 20 sec	Lost
Duration of fourth round played:	2 min 01 sec	Won
Number of times the user used “Help”:	2	N/A

What Goes There? Task Measures

Description	Measure
Time the user spent on tutorial:	27 sec
Number of times user tried before conquering the first code snippet:	2
The number of inputs the user made before entering the correct inputs:	6
The time user spent on the first code snippet:	3 min and 45 sec.
Number of times user tried before conquering the second code snippet:	1
The number of inputs the user made before entering the correct inputs:	5
The time user spent on the second code snippet:	2 min and 3 sec.
Number of times the user used “Help”:	1

Bugaboo Task Measures

Description	Measure	Game Result
Time the user spent on tutorial:	10 sec	N/A
Number of times tried before the game is won:	8	N/A
Duration of first round played:	18 sec.	Lost
Duration of second round played:	37 sec.	Lost
Duration of third round played:	39 sec.	Lost
Duration of fourth round played:	45 sec.	Replay
Duration of fifth round played:	1 min 20 sec.	Lost
Duration of sixth round played:	1 min 49 sec.	Lost
Duration of seventh round played:	2 min 40 sec.	Lost
Duration of eighth round played:	3 min 57 sec.	Won
Number of times the user used “Help”:	2	N/A

RESULTS

From the experiment, it is clear that our group predicted the difficulty level of each task correctly.
- The time spent on the tutorial of each task before the task is attempted is a good indication of that.
According to the statistics, the number of times Datris is retried before the user won the game is more than that of the “What Goes There?”
- However, this is because in our current implementation, we have only two obstacles to the “What Goes There?” Task.
- The more difficult and higher levels of the task have not been implemented; whereas in Datris, the game is more or less complete with all functionalities realized and implemented.
- Datris has 10 levels of difficulty, with each higher level more difficult than the previous one; while What Goes There? has only 1 level.
The time that it takes the user to understand the objective and the goal of Datris is less than that of “What Goes There?”
- In Datris, once the user grasped the differences between the data types, the concept is learned and the game is merely a measurement of how quickly the user can identify the data type.
- However, in What Goes There? Task, the user took a while to try to make sense of the code snippets, especially, never having seen one before.
- The user first tried to guess blindly the values that should go into each blank and learned by trial-and-error of what to fill in by comparing their Output to that of the “What We Want” window.
- Having conquered one code snippet, the user has to, in a sense, “start over from scratch” with the next code snippet by taking time to decipher the new code presented and try guessing again.
What is interesting to note here is that the number of times the user used the “help” functionality only once for the more medium task whereas he used it twice for the easy task.
- think this is because while the mechanics for both is easy enough to understand, Datris introduces 4 different data types and it is easier to get them mixed up but the “Help” tutorial offers the definition and clear examples of each, whereas the tutorial for “What Goes There?” does not offer the explanation of each code snippet presented. Therefore, the user does not find it helpful to go back to the tutorial for help when he is stuck here.
Bugaboo Task takes the most number of retries to win the game, which is as expected because it is the most difficult task implemented in Programming4Lyfe serious game.
- This task assumes the user to have gained some experience and knowledge about programming from the previous two tasks.
- It also requires the full attention and focus of the user to quickly isolate the buggy code from the non-buggy.
- To accomplish this task, the user is assumed to be familiar with the code syntax already and have a good understanding of the data types concept.
- From the statistics gathered, we see a clear progression of the user’s improvement.
- After each retry, the user seemed to have a better understand of what to do, and of what to expect.
- From this experiment, it is notable that the user was not discouraged after each “game over.”
- On the contrary, he seemed to be more determined to win the next time. He decided to “replay” at the fourth round because the (buggy) balloons he meant to shoot were blocked out by (nonbuggy) balloons and the wrong balloons got fired instead.

DISCUSSION

What you learned from the pilot run what you might change for the "real" experiment?
What you might change in your interface from these results alone?

From the pilot run, I gathered great conclusion about our implementation of the Programming4Lyfe game. I think that the Datris game is successful. It was quick to introduce the concept of data types to the user and it allows the users to learn the differences between the data types in a very engaging approach. The addition of the ‘ding’ and ‘buzz’ sounds as well as the green and the red checkmarks certainly serve as excellent encouragement in the sense that they provide “real-time” update of how well the player is doing. The user interface of the game is improved in this game because of those additions. In the previous implementation, sound was not a part of the game, therefore the player relied on the red or green flashing of the bucket, indicating of the incorrectly or correctly, respectively, sorting of the bricks. This old approach distracted the user from focusing on the next falling brick right away. As far as interface is concerned, I think not much more improvement could be made for this task. Perhaps, minor cosmetics changes could be applied such as the shapes of the buckets and the bricks could be more “fun” and more distinguishable from each other to provide yet another level of visually pleasing experience. In this experiment, we didn’t want to drag the test on for long, therefore, we rewarded 5 points for each correctly sorted brick. In the “real” experiment, we would make the game longer by rewarding only 1 point. And we would also add even more complicated expressions for the falling bricks as the level increases.

I think this experiment also proves that the user interface for the “What Goes There?” Task is very promising. The user only went to seek for “Help” tutorial once throughout the whole task. For this task, the ultimate intent of our group for this task is to expose the users to the coding style and syntax of the Java programming language; and we wanted to do this in a way that would not intimidate the users by showing too many technical details. I think the revised interface for the task accomplished just that. This task has undergone a few major interface changes from making the users build a function from scratch to allowing them some built-in functions presented in a graphical interface using drop-down menus, and now to simplifying the whole system and just asking them to fill in the most crucial pieces of the code. The fundamental goal of Programming4Lyfe game itself is to make it as easy on the users as possible so that they do not feel learning Computer Science is a pain. And the progression of the development of this task is representative of that. In this user interface, as the user fills in each blank, he gets “feedback” immediately because the output window displays the result of the code in real time. As a result, by comparing this result side-by-side the expected result in the “What We Want” window right next to it, any mistake is instantly recognized and fixed. In this way, the task achieves the goal of being as interactive as possible. One more note to add to the possible effects of this Task is that the bonus points earned as a result of left-over time proves to be an effective incentive during the experiment. One of the aims of this game is to encourage the players to practice as much as possible, because that is the only way to improve one’s programming skill. The more the players practice, the faster each code snippet is completed, and the more points the players will get.

However, from the pilot study, I also find some room for changes that could be done to make “What Goes There?” game to be even better. The cosmetics of the way the code snippets are presented should be improved. Some of the blanks lie on top of, and therefore “hide”, some the code itself. This confused the user and made it harder for him as he tried to make sense of the code. Moreover, we could also add in more levels to the game to make it more exciting. This is under construction. In the “real” experiment, the addition of extra levels would certainly entice the users to the game even more. Lastly, an introduction to each programming concept should be added to the tutorial. For example, if level 1 introduces the “for loop,” the tutorial should give a brief overview of what a “for loop” is. This will help the users a lot when they get stuck staring at the code and help them make educated guesses as opposed to blind ones.

Much, much improvement has been done to our last Task, Bugaboo. And from this pilot study, I think our latest product of this task is significant. This task has also undergone major interface changes from the original confusing graphical interface of nested boxes, to a Nintendo-like shooting plates game, to now a jet firing buggy hot-air balloons game. Gathering from the statistics of the pilot run, this game proves to be very challenging as it takes many replays before the player actually wins the game. And I think the level of difficulty of this task should stay the same to reflect the challenges of real-life debugging in a programmer’s life. Debugging is an acquired skill that demands lots of practice, and visual sharpness. And this version of Bugaboo, in my opinion, certainly fits the bill. From observing the user and his progression of how well he performed this task, I concluded that practice did make him better. He was sharper to detect a buggy balloon in later rounds of his retries. And we see a clear improvement of his feat through the time it took him to finish each round. A possible enhancement that could be added to the interface is enlarge the playing field (screen) in the beginning levels to give the users more time to think.

Again as in the Datris game, we didn’t want to drag the test on for too long, therefore, we rewarded 5 points for each buggy balloon shot. In the “real” experiment, we would make the game longer by rewarding only 1 point. And we would also add even more complicated expressions for the balloons and possibly even more balloons as the level increases.

APPENDICES

PilotStudy - DemoScript

Informed Consent Form

Raw Data Collected during the Pilot Study