Heuristic Evaluation of Prototype-Group:Orquesta

From CS 160 Fall 2008

Contents 1 Problem 2 Violations Found 2.1 H2-2: Match system and real world 2.2 H2-3: User control and freedom 2.3 H2-4: Consistency and Standards 2.4 H2-5: Error Prevention 2.5 H2-6: Recognition Rather Than Recall 2.6 H2-7: Flexibility and efficiency of use 2.7 H2-8: Aesthetic and minimalist design 2.8 H2-9: Help users recognize, diagnose and recover from errors 2.9 H2-10: Help and documentation 2.10 Other (Bugs, Missing Features) 3 Summary of violations 3.1 Overview 3.2 Statistics 4 Recommendations

Problem

We performed a heuristic evaluation on Programming4Lyfe, a game that teaches people introductory programming through syntax evaluation, data type matching, and bug recognition.

Violations Found

H2-2: Match system and real world

1. [severity 3] The game focuses on a few aspects of programming without explaining the context behind them. For example, the "What Goes There?" game expects players to fill out a for-loop without explaining what all the code does. Even if this game were used to supplement a class, most people learning these skills would require a little refresher.
2. [severity 1] In "What Goes There?" the left box continues to read "Current output in this window" even after the game starts, until the user actually has somewhat correct code filled in. This may be confusing to a new user who thinks the output should change when it has not yet.
3. [severity 1] There isn't a "Run" button in the "What Goes There?", instead, the code is evaluated in real-time. This may be confusing to novice users whose code input is not sufficient to make the game actually generate any output. It is a little surprising to see a victory screen pop up without pressing a button, and conversely, users may look for a button to press when they see their code isn't outputting anything.

H2-3: User control and freedom

1. [severity 4] On the "What goes there?" screen, the "Ready for Fun?" dialog box restricts user flexibility, and executes an unclear action. Clicking "Not Yet" takes the user back to the Single Player menu, which is not immediately clear (and is also the only place in the entire game where clicking a button performs that action). For example, the user may think the game is about to start, and so will click "Not Yet" because they want a little more time to read the game instructions. This dialog box comes up far too fast for a brand-new user to the game, and they may be confused by its function. On the other hand, an experienced player might want to start the level immediately, without waiting for the box to prompt him or her to begin play.

H2-4: Consistency and Standards

1. [severity 2] On the two "Output" and "What we Want" boxes at the bottom of the "What goes there" screen, the phrase "Current output in this window" appears in both. This offers the user a conflicting description of the function of each box.
2. [severity 1] For new players, it can be confusing to navigate within a mini-game in single player mode due to differences between them. The different color schemes and layouts in each game make the "Return to Main Menu" look irrelevant in later screens. Also, it can be difficult to find the help and start buttons on a new game. Perhaps their positioning should be standardized among all games.
3. [severity 1] In Datris, the data types at the bottom of the screen appear to be buttons.
4. [severity 0] In Datris and Bugaboo, it's not entirely clear what HP means, since the term is typically used in some form of a combat game.
5. [severity 0] Victory dialogs should have borders to differentiate them from the background.

H2-5: Error Prevention

1. [severity 2] The target dots in Bugaboo can be difficult to click even when the user quickly recognizes a bug, due in part to the lag inherent in Flash and also because the targets simply disappear before the user has time to reach them with the mouse pointer. Fitts' Law would suggest that making the target area larger would reduce the severity of this problem; perhaps a larger hit area would be appropriate.

H2-6: Recognition Rather Than Recall

1. [severity 0] It may be hard to remember which game is which when you're trying to select one. In the game selection screen, each game should have a short (less than a sentence) description underneath each. Skimming a description can trigger memory.

H2-7: Flexibility and efficiency of use

1. [severity 1] The "Return to Main Menu" button is always available, but there's no way to simply go back to the mode screen (single player games menu screen) if the user just selected the wrong game.

H2-8: Aesthetic and minimalist design

1. [severity 0] The game screen resolution is too high: the overall game does not fit standard display size of 1024x768 pixels. This makes it difficult to use for players with older computers or inexpensive monitors. We suspect this may also contribute to lag issues in Flash (in particular with Bugaboo).

H2-9: Help users recognize, diagnose and recover from errors

1. [severity 4] Overall, these games don't teach per se, they just grade the player. Specifically, the user is never told why their answer was wrong. Novice / inexperienced users are thus prone to making the same mistakes over and over, until they figure it out themselves.
2. [severity 2] In the Bugaboo mode, it is unclear whether the red button means pressed or "fault," especially since the "round" can expire just as you click a button. Following that, a green tick (to signal "correct") over a red (just pressed) button is confusing.
3. [severity 1] In "What goes there?" it is not clear what the red "X" in the bottom of the screen between the two "Output" and "What we Want" boxes does.

H2-10: Help and documentation

1. [severity 3] In Bugaboo, the help menu is verbose and may be overwhelming to some users at first. It has small text and is not suitable for skimming or referring to.
2. [severity 1] In Bugaboo, the game doesn't pause when you open the help menu. It appears that the help menu is supposed to be kept open for reference while the game is active, but the games require too much attention to read large amounts of text while playing.
3. [severity 0] The help menu in Bugaboo has the same target cursor when you are simply looking at a text menu, which can be a bit confusing.

Other (Bugs, Missing Features)

Unrated because we felt these are bugs rather than design problems.

1. In Bugaboo, the labels for each target can appear on top of each other, making it difficult for players to read or click on them.
2. In Datris, an extra block appears at the top when you succeed.
3. Multiplayer mode and Settings are currently unimplemented. We look forward to seeing these in the future.

Summary of violations

Overview

The games contained in this prototype are fun and function well, but they can be hard to learn and navigate. "What goes there?" is laid out differently than, and has the opposite problems of, the other two games. Where Datris gives an easy-to-understand start button, "What goes there?" has a cryptic dialog. Whereas "What goes there?" has terse, but highly visible documentation, Datris and Bugaboo have verbose and somewhat hard-to-access documentation. Additionally, we found the lessons incomplete; the user may need a little more guidance as to the learning process, and more feedback as to what they've done wrong. There are also some scattered technical errors and improvements that we saw, but overall it's a fun set of minigames that just need clearer introductions and standardized UI's.

Statistics

Number of violations per heuristic (in parentheses following heuristic description):

• H2-1:Visibility of system status (0)
• H2-2:Match system and real world (3)
• H2-3:User control and freedom (1)
• H2-4:Consistency and standards (5)
• H2-5:Error prevention (1)
• H2-6:Recognition rather than recall (1)
• H2-7:Flexibility and efficiency of use (1)
• H2-8:Aesthetic and minimalist design (1)
• H2-9:Help users recognize, diagnose and recover from errors (3)
• H2-10:Help and documentation (3)
• Miscellaneous bugs (3)

Number of violations per severity category:

• Severity 0 (5)
• Severity 1 (7)
• Severity 2 (3)
• Severity 3 (2)
• Severity 4 (2)

Total number of heuristics violations in the entire interface [not including bugs] (19)

Recommendations

1. Give a short description of each game on the game selection screen
2. Display the instructions / background for each lesson to the player when the game opens, and include a start button underneath these instructions that the user can click when they're ready to begin the game
3. Try to use less text to describe these lessons; don't overwhelm new players
4. Give the help menu a bigger font and organize it with bold headings (in Bugaboo, Datris)
5. Give the user feedback when he or she makes an error
6. Pause the game when the player opens the help menu
7. Have a "Return to Single Player Menu" button instead of "Return to Main Menu" in the games
8. Have similar Start, Restart, and Return to Menu concepts in each game
9. Put controls like Help and Restart in the same place as Return to Menu, standardize their positions consistently for all games
10. Get rid of the "Ready for Fun?" dialog, and replace with a "Start", "Go", or "Play" button to give users flexibility over when the game starts (in "What Goes There?")
11. In "What Goes There?" explain how the code is being automatically updated, and what's happening in each box at the bottom of the screen (by making the labels more clear and distinct for each box).
12. Make the Datris data type targets not look like buttons
13. Use a different term than "HP" in Datris and Bugaboo, or clarify what "HP" means for non-gamer-type users
14. In Bugaboo, use a larger hit area (to make it easier to click the buttons)
15. In Bugaboo, give better feedback as to whether the player got the correct answer by choosing a new color scheme for "hits" or flashing confirmation text somewhere near the cursor
16. In Bugaboo, the cursor over the help menu should not be a target symbol (like in the regular game), but should morph to a regular cursor or perhaps a text selection one (to eliminate confusion with actual gameplay).
17. Explain what the Red X means in "What Goes There?", or label it for first-time players.
18. Scale down the overall display to at most 1024x768 pixel resolution