PilotStudy-Group:Orquesta-StuartBottom
From CS 160 Fall 2008
Contents |
Introduction (5 points)
Orquesta is a serious game that is intended to teach basic DJ'ing skills and theory to players of all ages. Crucial tasks that successful DJ's must perfect include learning to follow a beat and making musically-correct "transitions" between tracks in a "beatmix." These tasks are both taught in the first level of Orquesta's Campaign Mode. Another crucial aspect of DJ'ing, practice and experimentation, is encouraged and supported in Orquesta's Practice Mode. The advantage of using a serious game to teach these skills is that it can be provided in a simple interface online for free to anyone in the world, in contrast to real DJ hardware which is often heavy, expensive, and difficult to use.
Since Orquesta is already in a fairly mature prototype form, this experiment is being conducted to evaluate its usability, and determine how well players can complete several key tasks within the game. This is done by observing the user as they complete the required tasks, getting their feedback on the ease or difficulty with which they can manipulate the UI, and discussing possible improvements or strengths of the interface after the formal task evaluation is over. The overarching objective of this procedure is to gain concrete understanding of areas the game needs improvement in, so that improvements in these areas can be implemented in the next iteration of the design cycle.
Implementation and Improvements (15 points)
- Made several visual enhancements, including
- improved color scheme
- added a snazzy new logo
- Made buttons consistent across the UI, removed sometimes confusingly different button types
- Interface clutter cleaned up by removing the seldom-used
- pause button
- quit button, which only worked if the user was playing in a standalone Flash window
- save and import functionalities, whose functions were unclear to users
- Improved user feedback and error recovery in several areas, including
- Added visual feedback in Campaign Mode that shows the player when a transition is valid
- Improved error recovery for the user when duplicate tracks are selected in either Campaign or Practice mode
- Improved user feedback on the post-Campaign Mode statistics screen by color-coding success or failure and telling they player what they can do with what they've just accomplished
- Simplified the menu hierarchy and placed "Return to Main / Previous Menu" buttons on each screen so it is far more clear where the player is within the menu tree at any point
- Clarified ambiguities in the pre-Campaign Mode instructions screen - told players that they can only advance to the next level if they complete ALL of the level objectives
- Added several new features, including
- A level selection screen so it is clear users can only go to the first level the first time they play Orquesta
- The ability to go back to previous steps (those already viewed) in the tutorial, in case the user missed or forgot something
- Power-up functionality - gives 2x points for three successful transitions in a row in Campaign Mode
- Enabling players to select the volume of a track by simply clicking where on the slider axis you want the volume to be, so the slider bar snaps to that location instantly instead of requiring the user to drag it to the right location
Method (10 points)
Participant
We first interviewed Jeff, a 15-year-old male freshman in high school, for our group's contextual inquiry. He was eager to return and see what we'd come up with based on the ideas he gave us when we first started this project. Jeff is an avid gamer and enjoys playing many "hardcore" recreational PC games, primarily first-person shooters, driving simulators, and classic arcade games. His favorites include Halo, Halo 2, True Combat Elite, Battlefield 2, Call of Duty 4, Test Drive Unlimited, Gears of War, Asteroids, Pac-Man, and chess. He happens to be a big fan of "instant" games found on websites (e.g. Flash games that don't require the player to install any software). He also has experience with educational/serious games, which he played as a child. These games include Math Blaster, Midnight Rescue, Oregon Trail, and certain titles in the Jump Start series.
Jeff has never had experience with DJ'ing or beatmixing, but he has had several years of music theory training and was already familiar with the premise of Orquesta. Thus, we feel he is well-qualified to evaluate a "serious game" such as Orquesta from a nonprofessional, non-domain-expert perspective. As with the contextual inquiry, we chose Jeff because he represents a younger demographic of players who would potentially have great interest in our game. Additionally, the fact that he is not focused on becoming a DJ (he is interested in Orquesta more from a random skill-building / fun-factor / recreational aspect) allows us to consider the appeal of our game to a broader group of users than only those who are seriously setting out to learn DJ'ing techniques.
Apparatus
The test was run on the participant's laptop computer, which was also equipped with a touchscreen. We chose this hardware instead of the interviewer's computer because it provided a unique opportunity to view (for the first time) how Orquesta would be used by a player on a touchscreen hardware interface. The interviewer sat at a table to the participant's left (a position that allowed him a full view of the participant's interactions with the game) and took notes on his computer.
Tasks
Complete the tutorial (Easy)
This task merely requires the participant to complete Orquesta's built-in tutorial, in order to familiarize himself/herself with the application. The participant had to step through each stage of the tutorial, which demonstrates and describes the functionality of each GUI component in Orquesta. We looked to make sure the user didn't get stuck on too many sections in the tutorial, and that they found the explanations / demonstrations clear and easy to follow.
Obtain the power-up in Campaign Mode (Medium)
This task requires the user to complete at least three transitions (that is, turning one instrument off and another instrument on) in a row in Orquesta's "Campaign" mode. These transitions must be performed "on beat" in order to be counted. The power-up grants the player 2x the normal score they would get, and is only activated if three successful transitions are performed in a row (with no off-beat transitions or mistakes, denoted by "broken vinyls"). We evaluated how difficult this task was for the user to perform, and how many tries it took them to complete it (denoted by the number of unsuccessful transitions before the power-up was obtained).
Successfully complete Level 1 in Campaign Mode (Hard)
The last task requires the user to successfully complete Level 1 in Orquesta's Campaign Mode. This is done when the player successfully performs 8 "on beat" transitions without running out of vinyls (a vinyl is broken every time the user performs an unsuccessful, off-beat transition). We evaluated how long it took the player to reach the 8 required transitions, and how many tries it took them (each time they needed to restart the level).
Procedure
The participant was introduced to Orquesta and presented with our reasons for conducting this usability study (see script in Appendix below). Following this, they were given a quick demo / run-through of the Practice Mode interface in Orquesta. Once they were ready to begin the test, they were given the first task. Once they completed each task, they were given the others in sequence.
The evaluator took notes throughout the usability test, watching for critical incidents and recording these, along with all comments made by the participant, in a detailed log (see Appendix). After the participant had completed all the required tasks, the evaluator conducted a "debrief" with them to determine how they felt about various aspects of the interface. This debrief was held in an open, unstructured format, with the evaluator and participant trading ideas and thoughts back and forth as they discussed different parts of Orquesta.
When evaluating this particular participant, the evaluator chose to focus more on obtaining insights into the participant's thought processes and likes / dislikes about the interface, than on the minutiae of the participant's performance (e.g. when they missed a transition or not). This is because the participant in question was already highly familiar with the premise of Orquesta and its objectives, and their time could be better spent in giving highly insightful feedback on the ideas behind the game itself.
Test Measures (5 points)
In this evaluation, we used the following metrics to gauge the effectiveness of the interface in enabling the participant to complete the required tasks:
- Time required to complete each task
- Number of retries or mistakes
- Relative ease or difficulty in completing each task
- Positive expressions of approval
- Negative expressions of disgust, confusion, or questions about why the interface is taking a certain action
- Instances where the user uncovered a bug or inconsistency in the interface
- User comments and suggestions during the debrief
Results (10 points)
The participant in this study was able to complete the required tasks very rapidly, with few mistakes and only two retries, which were both required in the Medium task (obtaining the power-up). The completion of all tasks was achieved in a grand total time of 2 minutes 50 seconds, with no prior practice on the part of the user. While the participant was already familiar with the premise of Orquesta and an avid gamer, he was not able to complete the tasks flawlessly.
The participant uncovered several inconsistencies / bugs during the course of the evaluation, all in the Orquesta Tutorial. The first bug was that clicking the knob for one of the tracks returned the tutorial sequence to an arbitrary position, which was highly confusing for both the participant and the evaluator. The second bug was that only one beat-feedback light in tutorial mode lights up at a time, which does not reflect the lights' behavior in the actual Campaign Mode. Finally, the user noticed that any attempt in Tutorial Mode to turn tracks on or off outside of the prescribed sequence had no effect, which also does not reflect the actual behavior of the mixing board in Campaign Mode.
The participant noticed two areas where Orquesta's interface did not provide adequate feedback or flexibility to complete certain tasks. The first area is in the Campaign Mode level selection menu: while the user commented that they liked being able to select which level to go to, it was not immediately clear how the game would keep track of the levels a player had completed since there is no user identification or sign-in process at the beginning. The second area is that the game only records a successful transition when a track is turned ON in Campaign Mode, never when it is turned off. This is what a transition is, by definition, but neither the Tutorial nor the Level 1 Instructions screen communicated this clearly.
The post-experiment debrief with the participant was extremely helpful and enlightening. The participant suggested a myriad of improvements to the Orquesta interface; his comments are detailed in the Appendix. A few of the major suggestions included:
- enabling better control over transitions; adding the ability to make multiple tracks transition simultaneously
- improving the affordances for visual beat feedback
- improving user feedback when a track is on vs. when it is off
- boosting the richness of the game by having an actual song to follow the beat with
- having cool animations and noises happen when the user takes certain actions, which would make the interface more fun and game-like
Overall, the participant gave a very positive review of the game. While he suggested many improvements, he was pleased with the overall look and feel of the game, and enjoyed the fun aspects of its gameplay. This is demonstrated by the fact that he went immediately to Practice Mode after the formal task evaluation was over - not because he was prompted to by the evaluator (Practice Mode was outside of the list of tasks), but because he simply wanted to try it out - he was having so much fun with the game. This was also because the participant was disappointed that there were no additional levels available in Campaign Mode at the time of the test.
Discussion (15 points)
Given another chance to conduct the experiment, this evaluator would definitely try to gain access to video cameras to increase the accuracy with which data is collected. It can be difficult to hold a conversation with the participant while simultaneously and furiously scribbling notes. Use of video cameras would alleviate this problem and allow the evaluator to focus on the user-computer interactions rather than recording information for later analysis.
The results from this experiment are in some cases very unique, and in other cases duplicate exactly what other users have said in usability testing by other group members and during previous iterative evaluations. The results are not unique where the participant has asked for improved visual feedback, which has been a weakness of Orquesta from the beginning. The changes to implement in this area would be individual lights that come on when each track is on, and a single, synchronized transition light that follows the beat of the music. Also like other users, this participant desired better control over transitions, and suggested the natural affordance of a spinning vinyl record on a turntable that can be manipulated and scratched to control transitions, just as DJ's do on real audio hardware. This is also a feature well-worth considering for inclusion in the next iteration of Orquesta.
More unique, yet still compelling, conclusions can be drawn from this evaluation. It is clear that the hidden bugs in the tutorial uncovered by this user will need to be fixed immediately in the next iteration. The ambiguity over level selection (and perhaps even a second level) should also be considered as fixes to be made. More important, though, is the necessity of clarifying the nature of a transition (as described by this user). Transitions can only occur when a track is turned on, and this must be clear to the player if they are to advance beyond the first level of the game. Increasing the "fun factor" of our game is also important, too (being too "app-like" instead of "game-like" has been a problem for Orquesta from the beginning). This can be done by following the user's suggestions on including cool animations and sounds for specific player actions.
The participant's suggestions about the ability to make multiple transitions simultaneously with the use of keymaps or special GUI affordances cannot be ignored either, and deserve serious consideration. Finally, the participant's suggestion to make the game more invigorating by including actual songs to follow the beat with (instead of just tracks alone) may also prove to hold merit with other users in future tests. In conclusion, the participant's positive review of the application is encouraging, but the bugs he discovered and the suggestions he made reveal there is still much usability work to be done on Orquesta. In the next iteration, our group will need to fix bugs, prioritize the most essential improvements, and implement them with natural affordances and playful, fun UI elements.
Appendices (5 points)
Materials
The participant introduction script is below, followed by the task statements, which were introduced at appropriate times as the participant progressed through the test.
Participant Intro Script
"You are going to help us test a new application, Orquesta, which is intended to be a fun game to teach basic DJ'ing skills such as mixing tracks and following a beat.
The objective of this study is to test the game, not to test you. That is, we're evaluating the game's layout and controls because we want to make it as easy as possible for anyone to use. This means that YOU get to test this game for us and tell us what you like and don't like - we're especially interested in hearing your thoughts as you play the game. Go ahead and think aloud as you play the game - this will greatly help us identify where the game needs some work. Also, if you like anything; find something fun, surprising, or entertaining; or are confused at anytime, we'd definitely like to hear about that, too.
You'll be given three tasks to complete. When you're ready, we'll get started: I'll give you a short demo of the game, and then present you with your first task. I'll be taking notes as you go through the tasks so I can talk over the results with my teammates later."
Task Instruction Statements
"Your first task is to start Orquesta and complete the tutorial."
"Your second task is to successfully obtain the power-up in Campaign Mode."
"Your third task is to successfully complete Level 1 in Campaign Mode."
Raw Data / Critical Incident Log
(Units in m:ss)
0:00 - Started tutorial task
0:10 - Participant states he likes the fact that objectives are displayed right in the tutorial window
0:15 - Participant begins playing tracks in the tutorial, and decided he liked the sound; glanced at the interviewer, grinned, and turned up the bass on his speakers
0:31 - Participant found a bug in tutorial: when the "Bells" track knob is clicked, the tutorial jumps back to an arbitrary previous step with no explanation
0:40 - Particpant notices another tutorial bug: only one light (for Drums) blinks with the beat, the rest of the tracks have lights turned off
0:54 - Participant tried to turn off an instrument while in tutorial mode and found he wasn't able to; commented that this was unexpected
1:06 - Participant given second task (obtain power-up in Campaign Mode)
1:15 - Participant comments that he likes being able to select the level to jump to directly after switching to Campaign Mode, says that another way to do this is to have a password displayed on completion of a level so the player can jump directly to it next time he/she plays
1:31 - Participant comments that he now notices you only get a successful transition when you turn ON a track
1:55 - Participant acheived power-up, completes second task, is told that the last task is to complete Level 1
2:50 - Participant completes Level 1, reviews statistics screen
2:59 - Participant comments "I'm itching to press the next level button, man!" and is informed that the second level is unfortunately not implemented yet
3:05 - Participant goes straight to Practice Mode to continue playing with the interface (he was not asked to do this, he did it because he wanted to try it out)
3:30 - Interviewer begins debrief with the participant, and discusses the user's likes and dislikes about the interface. User comments include:
- You need to have an animation for the power-up, like a vinyl spinning across the screen with stars coming out of it
- It would be nice to have a beatboxing feature so you can record your own beats and add them in
- The interface should have an animation that actually has a record playing a song in the background that you match the beats to, then you should be able to squeak / scratch the record ... but this shouldn't be in Level 1, in an upper level instead.
- User should be able to transition between songs instead of just tracks and simple beats
- Comments that the problem with Orquesta right now is that you're just making up stuff in the transitions between tracks. Instead, if you have something for inspiration like a prerecorded song, you can make it better by throwing in your own transitions
- User wants to be able to upload his own songs
- User comments that transitioning between two songs is better because its richer, and you're not making stuff up, but building upon the work of others
- Suggests that the game should have a little 5-second theme song play when you go to the credits because simply viewing them is boring.
- User should be able to set the error margin in Campaign Mode (that is, the tolerance that determines whether a transition is counted or not)
- User did not know that the lights on the tracks were synced with the beat until it was explained to him. Said it didn't make sense that each track blinked since they're all on the same beat anyway, so the interface should only need one light.
- It's hard to tell whether or not a track is on. You should have a light ... it's hard to see the little tick mark on the knob pointing to on or off. User wants to be able to look across at a glance and tell what's on and what's not.
- The game should have a pause button in every mode, or the ability to hit P to pause
- User likes the mouseover sheen on the buttons in the mixer.
- User pointed out that the game needs a toggle switch interface with a single button to enable transitions between multiple tracks simultanously, which will enhance the level of creativity and expression available to the player. Also commented that user is not able to change the volume for multiple tracks at the same time - suggested making keybindings to control volume and mute controls for each track.
