PilotStudy-Group:ExampleGroup-JimmyNguyen

From CS 160 Fall 2008

Contents 1 Introduction 2 Implementation and Improvements 3 Method 3.1 Participants 3.2 Apparatus and Environment 3.3 Tasks 3.4 Procedure 4 Test Measures 5 Results 6 Discussion 7 Appendices 7.1 Materials/Scripts 7.2 Raw Data

Introduction

System Being Evaluated: The system that is being evaluated is our serious game, called Orquesta, that teaches people the basics on how to DJ. To elaborate, learning to become a DJ could be a long, expensive process, as all their equipment can range from hundreds to thousands of dollars. The purpose of this game is to introduce users to the DJ-ing aspects of the musical world. More than just an application, tracks are required to be played on beat to be considered a good, smooth transition. People are capable of selecting and playing up to six different tracks, and even capable of adjusting volume levels to perfect the sound levels of each track.

Purpose of Experiment: The purpose and rationale of this experiment is to get another perspective on yet another usability test. The purpose of this assignment is to leave the user with a usable prototype and they will freely go about to try and achieve three different tasks, ranging in difficulty from easy to hard. The purpose of the "pilot" usability is to allow the user to guide freely through the prototype so that the design team can revamp and reconsider any implementations before running the prototype with a larger pool of participants. Without telling the user what to do, they must rely on the game's interface to complete the three tasks at hand.

Implementation and Improvements

Ever since our interactive prototype, we have changed a lot of different aspects, in addition to implementing more and more features. Being a pretty ambitious group, I think our semester project has a lot of room for lots of different implementation and improvements because this is a multimedia based project. All of our recent modifications can be grouped as:

Graphical Interface: All the buttons, graphics, fonts, and logos were made to be consistent. The background is consistently gray, closely resembling a metal DJ mixer that a DJ would use. There is lots of white space on the menu screens to have a simple, yet sleek and sophisticated look. The Orquesta picture and title are consistent across all screens. Buttons have all been modified to look similar to each other, and moved around so that the focus is on the main buttons and the "Main Menu" buttons are in the corner. The color scheme is now gray, white, black, and variances of blues and purples.

Game Screen: The game's graphics are now all pretty much set. The button definitions are all now more explicit. Volume meter was fixed so that once you click on part of the volume bar, the slider will slide there and adjust volume as well. The timer and timer bars were successfully implemented as well. The most important new implementation was the game itself. On every beat, there is a light behind each instrument (in campaign mode) that will turn on and off as an indicator to the user if it is a good time to transition or not. If so, the song will play, if not, a vinyl will break and a sound will play, indicating a bad transition. The timers were all completed to complete this implementation, thus completing the game aspects. If three transitions are successfully achieved in a row, then you will go in "super"/"overdrive" mode, in which you will score more for combos.

Overall Game Work-flow: Scores are now dynamically computed when you finish campaign mode, thus you will see your stats when completed, signaling if the user has either succeeded or failed in completing a level. Overall, all the menu buttons have been fixed to allow a user to be able to move back and forward pages easily, including tutorial mode. The main menu is more minimalistic and each page allows the user to go smoothly from one page to another, thus being able to complete tasks easier.

Method

Participants

The participants for this application primarily target those who have some sort of interest in music. More specifically, it is a game to reach out to the aspiring DJ's. People who play Guitar Hero and Rock Band will have an easier time learning and getting acquainted with a real guitar or drum set than any aspiring DJ would. We selected this subset of people because we felt it would be best and flexible to allow users of general musical interests, while also appealing to people who have little or no exposure to DJ-ing. Our flexibility also allows people of different musical levels as well.

Apparatus and Environment

We did this experiment at the Unit 2 Underground All Purpose Room. There were not too many people in the big room, so it was perfect. It wasn't an environment in which it would be awkward if we were too loud or if other people were too loud, so it was somewhat ideal. We sat on the couches in the corner and I placed my laptop (and USB mouse) in front of the user on the smaller coffee table. I typed up the 3 tasks and put it on one side of the window, and the actual game on the other side. I took notes off to the side on a notebook.

Tasks

Our three tasks were originally:

1. Complete Tutorial Mode
2. Successfully complete two transitions in Campaign Mode
3. Saving a composition

After some more exposure, our implementation and the limitations have hindered us in accurate and useful file system interactions with our game. To adapt to this, we have revised and elaborated our tasks to:

1. Attain the Power-Up in Campaign mode: To do this, the player must successfully complete 3 consecutive transitions in a row. This allows the user to earn double the points for each transition completed while "powered up". This is analogous to power up modes in many other games.
2. Beat Level 1: To do this, the player must complete 8 transitions total. The obstacles that they must over come are that every transition must be on beat (which is cued by the blinking blue light), they cannot break 10 vinyls (which occurs when a bad transition was attempted off beat) and they must complete the objective before time runs out (level 1 has a minute and 30 second time limit).
3. Complete the tutorial: To complete the tutorial, the player must simply navigate to the tutorial section and read through each frame of the tutorial to gain a better understand of the game and the interface. While they do this, they get to see and hear all the cool things about the game.

Procedure

To complete the whole study, what I did was to take my user to the Unit 2 All Purpose Room hoping for a good working environment to do the experiment. Luckily, the room only had a couple of people so they were not a distraction to use as we were not a distraction to them.

To start, we sat in the corner and I set up my laptop, my mouse and my notes. I put the three tasks on part of the screen so that the user could read it and loaded the .swf file in Adobe Flash Player for the actual game.

From here, I read my brief notes on scripts. While greeting the user, I was clear on the fact that if he felt uncomfortable at any time, it would be OK to let me know and/or leave. Following the rest of my scripts, I introduced our system in context and asked any questions he may have asked me. After explaining the processes of the pilot usability study and the tasks that that he must accomplish without me, I gave a brief 5 second demo of the game just to get a small feel for how to start.

After answering any remaining questions, I tried to tell him to be critical and to think out loud, and thus we began the process. I tried to take notes on the users overall reaction, thoughts, and possible nitpicky ideas as much as I could. During the first task, the user was impressed with the intricate tutorial mode. However, as he went to the campaign mode to complete the other tasks, he was confused about the blinking light and when to actually time the entrance of other tracks. I had to explain it to him momentarily, but afterwards, he got the general idea rather quickly, being a huge fan of Rock Band.

At the conclusion of the experiment, I went over the whole game and tried to get him to exploit as many holes in the game and interface as possible. Overall, he was more impressed than anything else, but he also gave some good suggestions here and there. I thanked him for his time and thus we completed our pilot usability study. The time of the test took approximately 30 minutes.

Test Measures

What I measured was with what ease did the user have to use to complete each of the three tasks as mentioned before. I counted up the number of frames it would take to complete each task. For the tutorial, it was an exception because the user may have been testing the "Previous" button, but other than that, all tasks were completed within the approximate estimation of frames/time.

The reason for this is because if a user needs more frames than required to complete a single task, then it may be the case that something in the game is either unclear or the directions were not clear enough. In either case, my user was successful in completing these tasks with relative ease.

I also wanted to count how many tries it would take the user before he could get a feel for transitioning songs in campaign mode. It isn't 100% clear, on how or when to transition beats, as it also takes me a couple tries before I can get a streak going. The reason for measuring this is to get an idea for if the user understands when to transition, and also if it is too easy/difficult to get a feel for the game, thus being able to complete a 3 transition streak.

Results

The results of the Pilot Usability Study were very constructive and informative. They will help us as a group revise our game and interface further for the next iteration of the project. For privacy purposes, I will refer to the user as Alan.

During the Test: Overall, completing the three tasks was pretty easy and smooth. He was not exactly sure why the game mode was referred to as "Campaign Mode", because he wasn't sure if it applied to a game like this. The menu was simple enough overall to be able to go in the right direction to achieve and complete the tasks. While doing the tutorial mode, Alan found it a little bit difficult to differentiate what was on beat and off beat and why a transition had to occur at these specific moments. He was also wondering if there was any cross fading functionality, but the idea behind that is that these are individual instrument tracks, not like full on club songs that need cross fading.

Measurements: These are the results of what I "measured" for the experiment.

	Minimum/Approximate	Actual
Number of Frames to Complete Tutorial	12	14
Tries To Get Streak	~4/5	8
Ratio of Objectives Completed to Failed	8/10	26/6

To elaborate, the first row may be slightly inaccurate because to complete the tutorial, they may have been experimenting or my have skipped over a section too quickly, thus going back and forth, requiring the two extra frames. The task was still completed with relative ease and no stress.

As expected, it takes a couple tries to get a feel for the system and timing the beat. After awhile, it was just a simple key press on every single beat to turn the mechanism on and off. Anyone with a sense of rhythm will be able to do this. The flickering lights may have been confusing at first, but the opacity tweens may help later.

Finally, the ratio of objectives to allowed failures is 8 transitions needed before 10 failures to pass the level. After awhile, Alan got used to the system and just started experimenting with other features. The level was completed really quickly and he wanted to see progress to level 2.

Criticism Ideas Afterwards: Alan brought up some good points and suggested ideas that we have heard in the past, like turn tables, tempo changer, more tracks, overall, more visuals to represent a DJ mixer. Again, to elaborate on the group's decision, there had to be a compromise between different musical and DJ levels that things had to be simplified for game and implementation purposes. All songs have already been set to one tempo manually, so it lets the songs sound a lot better looped overall, especially as time goes later and later.

Overall Impression: Overall, Alan was very impressed with the whole game all together. He liked the graphics and logo, and also the DJ implementation. He was wondering if individual sounds, like a cowbell or record scratch, could be easily implemented. Just to keep everything simple and consistent, I had to explain to him the difference between instrument sounds, instrument layers/tracks, and full on songs. Our game consists primarily focused on instrument layers and tracks. Sounds are too arbitrary to implement while songs are already there. Alan liked how all the songs sounded layered together and was very surprised Actionscript was capable of this.

Discussion

What I learned from the Pilot Usability Study was that the fact that the users have to manually go through everything to complete the tasks, the communication is all done in the implementation of the game itself. The workflow of the game screens has to be pretty much perfected at this point. All actions that the user wants to take needs to be either explicit or very implicit. An attractive logo, game and interface will almost immediately draw the user's attention.

What I might change for the "real" experiment would be that I would want to test things on a real website/server just to see how the application performs online. I know multimedia might be a little bit heavy, so it might or might not have latency issues.

From the results of the experiment, we may want to change tutorial mode to include the flashing lights as well. This part was the roughest part of the whole game implementation itself. It may even be a good idea to have the flashing light fade in and out just so the beat is accented with 100% opacity of the light. The light may need to change color to be more interesting. Visual effects like an equalizer or a turntable would also be really cool, but that is getting more and more aesthetic.

From a group perspective, I feel that there is a split in terms of half the people wanting to be a bit more ambitious, and the other being more explicit in terms of implementation. As a whole, I feel that we have made the best choices in terms of implementation and the whole game is coming together as a whole. Everyone is finally happy with the graphics, the implementation, and the game itself. In the future, I assume we are going to make the flashing lights clearer, as well explain a little bit more, either via a background or in tutorial mode, what "on beat" transitions are. That is the biggest issue I believe right now. There may be other bugs, but it seems to be mostly there.

Appendices

Materials/Scripts

I gave my user my computer to use. Our game was developed in Flash and Actionscript 3.0, so I just used Adobe Flash Player 9 to load up our Orquesta.swf file. I also gave the user a mouse because the game is based on clicking, so I thought a mouse would be easier than my laptop touch pad.

In terms of scripts, this is what I told the user:

Greeting: "Hello. Thank you for taking the time out to help me complete my Pilot Usability Study for CS160. At any time, if you feel uncomfortable with something, please let me know.'

Introduction: "This is our game, Orquesta. It is a DJ-ing game, with similarities to that of other musical games such as Guitar Hero, Rock Band, etc. DJ-ing is a difficult hobby to get into, especially when the equipment and software is so expensive and hard to obtain. As a serious game, Orquesta will allow you to get an introductory feel to what it would be like to be a DJ. Our 'Pilot Usability Study' is basically going to be completed by you. You will do whatever you want with this game to do your best efforts to complete three given tasks that I will give to you later."

Starting: "So if you don't have any questions, then we can begin. If you do not mind, I will be taking notes as you do this. The three tasks are going to be here: <put in a document on the side of the computer screen> while the game is right here: <short click demo>. Feel free to explore our game as much as you want, but try not to ask me questions pertaining to the game or tasks."

Conclusion: Thank you for your time and I really appreciate the work and input you have given to me and my group. Please let me know anything else that you would like to see implemented, revised, etc.

Raw Data

Task: Complete the tutorial

• The beats/lights should flash in tutorial mode
• Red arrows in tutorial mode are a little "weird"..
  • could be different/better color
  • more consistent and accurate.
• Should the restart button do anything?
• There should be a "finish tutorial" button
• User did this quickly

Task: Attain the Power-Up in Campaign mode Task: Beat Level 1 (both tasks obtained simultaneously)

• Selecting multiple tracks makes an error
  • Should be pop up screen
  • Should not bring you back to level select screen.
• Time and score should either be bolded or more obvious
• Game/scoring broke after awhile
• Score sheet should have ALL transitions because points don't add up
• Transitions after completing objective should be 3x

Miscellaneous

• Practice mode should have toggle for the flashing lights
• More visual animations, feedback
• More pop up/confirmation screens
• Why is turning off a track not count against the vinyls?