PilotStudy-Group:group4-WittonChou

From CS 160 Fall 2008

Contents 1 Introduction 1.1 System 1.2 Purpose and Rationale 2 Implementation and Improvements 3 Method 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.4 Procedures 4 Test Measures 5 Results 5.1 Qualitative Task Results 5.2 Quantitative Task Results 6 Discussion 7 Appendices 7.1 Task Data 7.2 Script & Consent Form

Introduction

System

Programming4Lyfe is a serious game designed to assist users in learning to program. The target audience ranges from secondary school through college students and up who may be interested in programming. We understand that programming can be overwhelming, which discourages many people from learning it. While this user group encompasses a vast range of users, we expect that the primary users will fall between the ages of 18 and 25 during the users' college career. The users will use Programming4Lyfe as a tool to practice their programming skills while having fun. The goal is to present the user with common programming tasks, increasing their awareness and understanding of various programming functions and bugs.

Purpose and Rationale

While conducting our usability study, we want to evaluate several aspects of the game:

• We want the interface to be intuitive. The user should not have to worry about how to play the game as much as how to solve the problems presented to them through the games available in it.
• We want the game to effectively improve the user's understanding of programming concepts.
• We want the users to enjoy playing the game. Although there may be signs of frustration, we want to stay within the zone of proximal development and elicit the desire to successfully complete the tasks.

Users will be asked to perform several tasks within the game environment, providing us feedback on the aforementioned design aspects.

Implementation and Improvements

The game is organized with a typical menu screen leading to submenus. Currently only the single player function is implemented. There are three stages available in the first level. Since our first interactive prototype, we have made various changes to the three stages which were in dire need of improvement.

Datris

• Tutorial screens were added.
• The bucket graphics were adjusted to better reflect the shape of an actual bucket to improve visual recognition.
• The Pause button now properly hides game content to avoid cheating, where the users could pause the game to allow more time to think.
• Help content was revised to be more concise.
• Audio feedback and visual cues have been added to inform the user the correctness of their decisions.

What Goes There

• A short welcome screen was added prior to the automatic appearance of the newly added tutorial screens.
• Pause, Help, and New Game buttons and their respective functionalities were implemented.
  • Pausing will cause the contents of each of the three windows to disappear, timer to stop, and a "Paused" message to appear in the main "Code" window.
  • Help will bring up the tutorial windows seen prior to the start of gameplay and automatically pause the game as well.
  • New game will reset the game with a fresh slate.

Bug-A-Boo

• Bug-A-Boo received a major facelift and concept redesign and has gone through several iterations
• The task is similar in that buggy codes are shot. However, the targets are now moving balloons and the 'shots' now come from a moving fighter jet controlled by the player.
• Visual and audio feedback are given when the user shoots the right balloons or if the user crashes the jet into a balloon and loses a life.
• The goal remains the same in having to shoot enough buggy lines of code to achieve a score of 100. However the play style is very different with the new modifications.

Credit to Trinh for collecting and merging screenshots of each game

Method

Participant

I will refer to the participant as KM throughout the paper to maintain user confidentiality.

• Age: 21

• Sex: Female

• Education Level: College

• Major: Political Science

• Previous programming experience: a little bit of html.

• Interest in programming: A little bit.

• Comfort with computers: Very comfortable.

• Pastimes: Watching TV shows, playing computer games (mainly MMORPGs, and recently solitaire).

The user was selected based on several criteria: She falls within the typical age range, has little to no experience in programming, is from a field of study we have not examined yet, is familiar with general computer usage, and enjoys and plays a variety of computer games.

Apparatus

• Location: Soda Hall

• Equipment: Dell Inspiron 1420 Notebook Computer, 1440x900px 14.1" LCD monitor, Generic Wireless Optical mouse.

• Software: Mozilla Firefox 3.0.4, Adobe Flash Player 9, Windows Vista.

• Timing device: iPhone stopwatch.

Tasks

Datris

Users are asked to sort various fragments of data into different data type categories. Following the Tetris style game design, blocks with data written above it will fall down. Users will have to align the object with the correct data type. When the user correctly sorts the object into the right bucket, they will gain 5 points. Once the user reaches 100 points, she wins. However, the blocks will drop faster and faster as the user correctly sorts the data objects. We are looking to see if the user's recognition of data types and expressions over the course of this stage.

What Goes There?

Users are presented with a fill in the blank type interface and a section of real java code. It is the user's goal to fill in these missing spaces with the correct arguments so that the output of the code matches the desired output. The user will be able to see how her inputs in the text boxes change the output of the code. Once the user successfully fills in the correct arguments from the two available fill in the blank scenarios, the user wins. A 3 minute timer is also imposed and the user loses if she fails to complete the two scenarios prior to the experiration of the timer. Points are earned with each correct scenario with a time bonus dependent on the remaining time.

Bug-A-Boo

Users are given the task to shoot down hot air balloons with buggy code written above each moving balloon. Users will control the fighter jet that can shoot at the balloons. They will have to avoid colliding with balloons and evaluate the validity of lines of code snippets. Each balloon with buggy code shot down awards the player points with the goal of achieving 100 points. Here, repetition is key in helping the player recognize buggy lines of code in an interactive, fun, and challenging manner.

Procedures

My group used a common demo script which is linked in the Appendices on the wiki page and attached as the last page of the printed document.

• I first introduced myself and the purpose of this usability study and how the results of this study will be used.
• I clearly described how her identity will remain anonymous and had her sign the consent form
• Then I proceeded to demo the game as outlined in the demo script.
• Before having her use the game herself, I reminded her that I will not be giving her hints on how to use the interface or complete the tasks after this point.
• I also reminded her that it is very helpful to me and my group if she talked about what she's thinking and even slightly exaggerate any frustrations with the interface or game content.
• I told her that I would be timing and taking notes while she performed all of the tasks but to relax that the timer is just for recording/reference purposes.

• I then turn over all controls to her and ask her to navigate to and play each of the tasks one after another beginning with Datris, followed by What Goes There?, and lastly Bug-A-Boo.

• Following the completion of the tasks, I asked for any questions she had, any frustrations with the interface, what she liked and disliked about the games, as well as any improvements that would help her enjoy the game and learn/benefit more from playing the game with respect to having fun while learning/practicing programming.
• A final thanks and we parted ways.

Test Measures

Dependent Variables Measured:

• Time elapsed per Stage: The time the entire stage (Datris, What Goes There?, Bug-A-Boo) took in its entirety. This variable is affected by how long tutorials took to read and how long it took to perform the task successfully including failed attempts and re-plays.

• Time Elapsed per Successful Attempt: This is the elapsed time for a successful attempt. This variable is affected by the difficulty of the task.

• Times Played: This is a count of how many times the user played the game. It is affected by navigation errors, failed attempts, and how much she liked the game.

• Interface Errors Made: This variable is a measure of how many times the user mistakenly navigates through the user interface and incorrectly performs a requested task.

• Game Accuracy Errors Made: This variable is a measure of how many times the user makes a mistake in completing the task objective. For Datris, this is how many times the user sorts a data type into the wrong bucket. In What Goes There?, this is how many times the user inputs the wrong argument into a text box. In Bug-A-Boo this is how many times the user shoots a balloon that does not have a buggy fragment of code written above it.

Random Variables:

• Age: 21
• Sex: Female
• Education Level: College (4 year university)

Results

Qualitative Task Results

Datris: KM had little to no trouble using the interface. It seems that her experience with many different types of games and her careful reading of the tutorial pages yielded good knowledge of the task's workings and understanding of hotkey usage. As a natural gamer, she immediately exploited the use of the pause hotkey to buy time to get the answers correct. It was, however, evident that the speed of the falling blocks became unexpectedly fast when she reached level 8.

What Goes There? The tutorial pages allowed her to understand the concept of the game. However, KM was expecting the pictures of the tutorial to be interactive and tried to click on the pictures with text boxes but realized they were not interactive. She was also scared to click on the "Close Tutorial" button as her intuition told her that this would close this stage of the game. However, once the game began, she very quickly input the correct answers, only making one mistake. She played the game a second time, realizing that the questions do not change. KM completed both tasks in about 10 seconds this second time around. The "Not Yet" button the play again prompt was not intuitive until she tried clicking it.

Bug-A-Boo: This task proved to be very difficult and was not completed successfully until the fifth attempt. She understood the concept of bugs but made many mistakes each of the first few attempts. On the fourth attempt, KM would repeatedly shoot the balloons she knew to be buggy and avoided shooting those she either wasn't sure about or knew were not buggy. Finding which lines had bugs was a hard task. Avoiding balloons was also quite difficult. The combined effect of the two made this stage very difficult. However, she kept wanting to play again and beat the level. While she showed some frustration, she was always determined to complete the task. After she successfully completed Bug-A-Boo, she was very excited.

Quantitative Task Results

Below is a summary of the quantitative results of the three tasks the user was asked to perform:

Task	Time Elapsed Per Stage - Includes Reading Tutorial (mm:ss)	Time Elapsed Per Successful Attempt - Actual Play Time (mm:ss)	Times Played	Usability Errors	Game Accuracy Errors
Easy Task: Datris	02:10	01:40	1	0	1
Medium Task: What Goes There?	02:00	00:20	2	1	1
Difficult Task: Bug-A-Boo	07:10	02:30	5	0	20
Total	12:20	Not Applicable	8	1	22

Discussion

Experiment Overview

In general, the tasks presented to the user in this pilot test were quite successful. For the most part, the interface was quite intuitive. Despite a few confusing button labels, there was no severe usability flaw apparent from the interaction of my participant. Despite limited programming experience and being intimidated by the concept of programming, she clearly enjoyed the game as a whole. There were some tasks, such as What Goes There?, which she completed surprisingly quickly, and other tasks where she struggled for several minutes, as in Bug-A-Boo. She exhibited great determination to complete the Bug-A-Boo task successfully despite some signs of frustration after four failed attempts. However, it was apparent that, with each failed attempt, she was beginning to understand what constitutes valid syntax and data types/assignments. Through this study, we have proved to address our program as a fun game that helps users understand programming less painfully.

Breakdown

Datris:

• Repetitive pausing allows the user to "cheat"
  • Fix: restrict the usage of pause to a certain number of pauses per attempt or set a cooldown timer on the pause feature.

What Goes There:

• Here, KM seemed confused about the tutorial being a static image. Currently the tutorial occupies the entire screen, which implied that this screen is the interface she will be performing the task in.
  • Fix: Reduce the size of the tutorial screen to show that the working interface is underneath the tutorial window.

• The buttons on the tutorial do not lead the user back to the Single Player Menu if the user is not interested in playing this stage.
  • Fix: Add a button that will "Return to Single Player Menu"

• The "Close Tutorial" button can mislead the user into thinking that the button will close the game when they want to start playing the game.
  • Fix: Change text to "Start Game"

• The prompt that asks the user if they want to play has a "Not Yet" option. This label is not intuitive that the button will return the user to the Single Player Menu screen.
  • Fix: Change the button text to "Return to Single Player Menu". Add a button that will close the prompt but remain in the What Goes There screen

• KM completed the two scenarios very quickly and wanted to play the game again only to find that there are no new questions to answer.
  • Fix: Add more questions of progressing difficulty to take most users 2 minutes or more to complete everything

Bug-A-Boo:

• Initially, KM was frustrated with how the balloons would shift up and down. This is actually due to the collision detection algorithm used on the balloons which can happen if balloons are in close proximity with each other.
  • Fix: Scale down the size of the balloons. Make the algorithm detect which vertical alignment to create the balloon to avoid vertical shifts

• The feedback of incorrect hits is merely a correct/incorrect type response and makes it harder to understand where the player's mistake is.
  • Fix: Have the feedback hint at where the error is in buggy code by highlighting or somehow draw attention to where the source of the bug may lie.

• Being able to shoot the same two pieces of buggy code to win the game avoids the purpose of helping the player understand different types of bugs as opposed to one single bug getting drilled into their memory.
  • Fix: Bugs that have been shot will no longer reappear in a future balloon spawn. Rather than have the goal be to achieve a certain number of points, there should be a count of how many buggy balloons remain and the player has to shoot every buggy balloon to win.

Overall Design

Some of the button designs are still inconsistent and primitive. A button set should be designed and used throughout the game. The button set would reflect mouseovers, button presses, invalid buttons (those that are there but cannot be clicked due to the state of the game), and a set of names that clearly reflect the actions the button performs on click.

Tutorial designs are inconsistent as well. They should be fixed to maintain consistency throughout the different games in how each game is presented. This means making the tutorials cover the same screen area so the user understands that what they are seeing is a tutorial and not the interface they will be interacting with. The photoshop work in certain tutorials are inconsistent in design with each other and sometimes with the interface of their respective games.

Although I don't think this issue came up through our pilot user studies, some people use the keyboard's Tab button to navigate through the interface elements. It may be a good idea to explicitly set the tabbing hierarchy so power users can use their Tab button to quickly jump to the relevant input box or button. If possible, I would like to make buttons like "next" on tutorials to be preselected as in the Windows wizards so that a simple press of the spacebar or enter key will yield a button click on the preselected button.

Targetted Users

It also became apparent that our design can help those who have a little bit of programming experience very well. However, it still has problems addressing an audience with absolutely no programming experience. While the tutorials show how to complete the tasks, it provides little information on how a for loop works or what System.out.println means. Though users get a sense of the behaviors of these functions through trial and error, we don't do a good job explicitly teaching the users valid arguments and merely leave them to infer what the arguments are. I think an animated/interactive lesson can be very helpful if we were to address programmers with absolutely no programming experience and are looking to learn programming rather than to just practice what they have learned from another resource.

Conclusion

There are still many more improvements to be made, both visually (eye candy) and for usability. For the most part, I think we have developed an excellent foundation and have created tasks that address a variety of concerns that will help new programmers learn more efficiently. Although there may not be enough time for us to make all of these fixes and improvements, there are still a lot of details that we overlooked in some of our iterations that have been drawn out through our pilot usability studies.

Appendices

Task Data

Menu Navigation

• Main menu -> single player menu -> datris: no problems navigating to tasks

Datris (0:10-2:20)

• No problems with tutorial

• Immediately used 'P' hotkey to pause game expecting to buy time to think about where to place the data blocks.

• Resumed and paused again - got mental image of block text, determined the answer - resumed again and immediately aligned block to correct bucket

• User did this a few times for the first few blocks.

• Once user was accustomed to evaluating, no longer used pause to "cheat" until the blocks became a lot faster when at 80 points.

• User only made one mistake on a "<=" comparison and quickly acknowledged her mistake.

• User won on first try.

• User did not use help button.

What Goes There? (2:40-4:40)

• Spent about 20 sec on each tutorial page.

• Was expecting an interactive tutorial with dynamic text boxes when the pictures displayed screenshots

• The "close tutorial" button was not intuitive and was confused where the actual game was.

• 3:50 - started game and immediately began playing.

• 3:51 - tried pausing but couldn't remember enough to "cheat" this time

• 3:57 - first question completed: immediately input the right arguments in both text boxes (was this lucky?)

• 4:20 - second question completed: took a little more time to understand what to input for the condition.
  • Initially put 6 expecting 6 lines to print out in the output.
  • Successful input on second guess of 12 (another valid input would be 11)
• 4:30 Played again expecting different questions.

• 4:40 Finished both tasks (same inputs on both tasks)

• Was not sure what "Not Yet" would do but eventually just tried it and realized it took her to submenu.

• Did not use help, once again.

Bug-A-Boo (5:00-12:10)

• Was confused to the difference between a == and = when playing the game.

• Most of first few shots were incorrect.

• Confused at first but seemed to vaguely pick up the meanings.

• Failed first 4 times.

• Had difficulty avoiding balloons (esp when the balloon collision avoiding script caused balloons to bounce back and forth)

• Hard problems coupled with avoiding balloons proved to be quite hard.

• Saw really big improvement after 2nd failure.

• Got to score of 95 on 4th try.

• Really wanted to beat the level so kept trying over and over again.

• Would often repeatedly go for the balloons with the lines that she was sure to be buggy to rack up points.

• Did not use help.

Script & Consent Form

• Demo Script

• Consent Form PDF