PilotStudy-Group:Orquesta-GregNagel

From CS 160 Fall 2008

Contents 1 Introduction 2 Implementation 3 Method 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.4 Procedure 4 Test Measures 5 Results 6 Discussion 7 Appendicies 7.1 Script 7.2 Raw data

Introduction

Orquesta is a game designed to help you learn how to be a DJ. What Orquesta tries to teach you is the basics of beatmixing: attempting to synchronize the beat and make transitions between different parts of the song. To do so, it introduces you to a mixing board and asks you to make transitions, giving points for transitions so that the user is more inclined to get more practice.

The purpose of the experiment is to further test the Orquesta system. Having gone through a lo-fi usability study and heuristic evaluation by peers, Orquesta has seen many changes from the initial paper prototype, but we have not yet done a study with actual users. This test will help us figure out what usability problems a normal user might have.

Implementation

We have made many changes from our last update, using the feedback from the heuristic evaluation and graders' comments, as well as issues that we noticed while taking a closer look at the game ourselves.

1. Consistent Layout: We got rid of different button styles and attempted to standardize the positions in which buttons were placed. In addition, the artwork and color scheme is much more attractive and less distracting. I find it much easier to navigate personally, and hopefully the usability studies will validate this opinion.
2. Improved Tutorial: We added back buttons and made the tutorial explain things a bit better. The color scheme was altered here as well.
3. Technical Improvements: We implemented scoring and timing, which makes for a much more realistic test. The game is now feature-complete, except that it only includes level 1.
4. Level Selection Screen: We added a screen where the user can select which level to use. Only level 1 is implemented, but it gives the user an idea of how he or she would go directly to a specific level.
5. Timing Indication: We added visual feedback for when the beat occurs. One limitation of flash we ran into is that the music does not synchronize with real time; however, we were able to get it close with some tinkering.
6. DJ Power: We added a power-up feature where the DJ gains double points for several good transitions in a row. Right now, it uses a simple, but highly visible textual notification.
7. Removal of Confusing Features: We took out the import, saving and pause features because users found them confusing and they relied on Wizard-of-Oz techniques. We now rely on no Wizard-of-Oz techniques.
8. Failure Notification: One feature we intended to add was an audible notification when a failed transition occurs. However, it caused some inexplicable bugs and was left out of the final version.

Method

Participant

This section will describe the participant in the test. The participant is female and in her late 20s, going to medical school. She likes electronic music and describes herself as an excellent dancer. She does not play many games or DJ herself. She uses a Macintosh and is as competent with computers as anyone in our generation. She was selected because she is a friend of a friend.

Apparatus

The game was played on my own computer, an older Windows machine with ergonomic two-button mouse. The test was conducted at my house, where that computer is located.

Tasks

1. Complete the Tutorial (Easy): The user must find and successfully navigate the tutorial to completion. I was focusing mostly on usability problems within the tutorial that would stop the user from successfully completing it or fully understanding it at this stage.
2. Obtain DJ Power in Campaign Mode (Medium): The third task is to obtain DJ power in Campaign Mode by making three successful transitions in a row. The main things I looked for at this stage are misunderstandings from tutorial, how the user attempted to obtain DJ Power, and navigation problems which the user had.
3. Beat Level 1 in Campaign Mode (Hard): The user must enter campaign mode and complete 8 transitions within 1:30, without failing 10 transitions. The main things I looked for at this stage are misunderstandings about the objectives and deeper insight into what she thought was going on.

Procedure

We introduced ourselves and I explained the purpose of the experiment. She signed a consent form and I did an initial interview about demographic information.

Then, I introduced her to Orquesta, explaining what it was supposed to teach and the project thus far. I explained the process of the study, how she was supposed to think aloud and how I was not allowed to help her.

We then went to the computer and I gave her her first task while I sat and took notes. She went through the task until completion without help. After completion, I had her navigate to the main screen. The other two tasks proceeded the same way.

Then I conducted a closing interview, getting her thoughts on the system and more insight on the problems she had.

Test Measures

1. I kept a log of critical incidents as suggested.
2. I kept track of the time it took to get to the appropriate screen and how long it took to complete the task from there.
3. I kept track of her score, as well as the number of successful and failed transitions she made at each stage to get some data on how long it takes to figure out the game.

Results

What surprised me about this candidate was the way she rushed to completion of each task. Unlike previous interviewees, she focused solely on the task at hand.

First Task: She was very quick to complete the tutorial (in about a minute), and I think she only skimmed it. She was noticeably having trouble with my mouse (an unconventional design).

Second Task: She skipped the Campaign Mode instruction screen the first time, and could not figure out how to make transitions. She was audibly frustrated when she failed the first time because the time ran out, as she had figured how to make transitions out just before then.After that, she restarted the level, but did not understand why the transition button wouldn't work sometimes (she was failing transitions). Then, she went back and read the campaign instructions, started the game and got into power mode. She later explained that she was having trouble synchronizing to the beat at that time.

Third Task: She quickly entered Campaign Mode and started making transitions. However, she misclicked, right-clicking and hitting "play" by accident. This broke the application. I had to intervene and start it over.

After that, she was able to quickly complete the first level after a few missed transitions.

Discussion

For a "real" experiment, I'd like to have a better testing environment. My mouse is a bit unconventional. Maybe I should have had her use her laptop. Other than that, she seemed to get into the game from the start, but was frustrated several times when she did not understand what to do (and that I would not give her help). However, she was able to go back and solve those problems on her own.

I learned a lot from this run, mainly about how the game needs some contextual help while playing. She tried to click on buttons with my mouse but missed them slightly. Perhaps a larger hit area for some of the buttons would be wise.

She also did not understand that the transition buttons were actually buttons. They afford turning, so she tried to drag them, and due to my mouse, she had though she'd done that.

She also tried to use the volume sliders to turn the samples on. Perhaps we should disable the volume sliders or make them more obviously ineffective somehow while a specific sample is off. The volume sliders aren't specifically used in level 1 and might be removed.

She tried to click the vinyls on the bottom. Maybe we should make them look less round. The could be appear to be buttons. Maybe level 1 could have a few contextual hints. She was often lost in what to do and a less thoughtful user might be completely stuck rather than figuring out to go back to the help screens.

Appendicies

Script

• Introduction
  • Pleasantries
  • Purpose
  • Consent
  • Initial Demographic Interview
• Explanation of Experiment
  • Class Project Details
  • Purpose of Orquesta
  • Her Job
  • My Job
• Test
  • Set up on computer
  • Task 1
  • Task 2
  • Task 3
• Closing Interview
  • Thoughts
  • Explanation of critical incidents

Raw data

• 7:31
  • Quickly found tutorial
  • Could not find next button right away
  • Tried turning on the second instrument
  • Hovered over quit before seeing the next button
• 7:32
  • Followed the arrow with mouse
  • Tried to click the button in the window
  • Hovered over restart before clicking
  • Finished tutorial
• 7:33
  • Started Task 2
  • Rapidly found campaign mode, skipping introduction
  • Tried to click the vinyls (strikes)
  • Played with volume (was trying to turn on with volume)
  • Explained she was trying to turn the button rather than click
• 7:34
  • Figured out button was button
  • Failure (time up; 200; 2 transitions 1 strike)
  • Frustrated at failure screen
• 7:35
  • Restarted level
  • Didn't understand turning off wasn't a transition
  • Made 3 transitions with 5 strikes
  • Did not understand why didn't achieve power mode
• 7:36
  • Quit level (300; 3 transitions 5 strikes)
  • Went through tutorial again
• 7:38
  • "Oh" (understood strikes)
• 7:39
  • Several errors until completion (800; 7 transitions 5 strikes)
  • Quit
• 7:41
  • Quickly entered campaign mode
  • Made several transitions
  • Exposed bug which required intervention
• 7:43
  • Restarted game
  • Went to campaign mode
  • Missed button
• 7:44
  • Completed 8 transitions (1000; 8 transitions 6 strikes)
  • Quit level