PilotStudy-Group:BuTtErFlY-MohammedAli

From CS 160 Fall 2008

Contents 1 Introduction 2 Implementation and Improvements 3 Method 3.1 Participant 3.2 Apparatus 3.3 Tasks 3.4 Procedure 4 Test Measures 5 Results 5.1 Game Incidents 5.2 Variables/Statistics 6 Discussion 7 Appendices

Introduction

The system that we are evaluating is our math RPG prototype. This is not a complete version of the game, however, it has enough functionality to span a lot of the features of the complete intended version of the game. Our math RPG prototype now cleverly named "Quadratic Curve Sin of the Buffalo", is a role playing game intended to provide a recreational way of gaining basic math skills.

We will be conducting a pilot usability study on our prototype. We will be mainly conducting a qualitative study on our prototype. Since we do not have enough time to test our prototype with many subjects, we will not be conducting a full blown quantitative study. We will however include a little quantitative analysis along with measures, summary statistics, and variables. This study will provide for us a fast, inexpensive way of figuring out what fundamental things we have to change, find and prioritize problems, and roughly evaluate the usability of our prototype.

Implementation and Improvements

As per our observations and the heuristic evaluation provided by Group 4, we have implemented the following changes:

• Tutorial - added a tutorial level that helps the player learn the game
• Game Title and Title Screen - added a title and title screen
• Puzzle Backtracking - fixed the puzzle so even after you solve it to cross the bridge you can go back and forth
• Recognizable Signs - signs are added for the dungeons
• Emphasis on Math - balanced the game so physical attacks do very little; the focus is on math
• Random Events - randomly generated monsters now appear in dungeons
• Meditation Status - added status windows that tells the player if the meditate succeeded and how much SP was restored
• Meditation Change - changed to restore a certain percentage of the player's SP
• Monster Status - now we can use a scan skill to see the status of an enemy.
• Escapability - now users can use escape to get out of a battle.

Method

Participant

The user I selected was a 6th grader with sharp math skills and a low level of RPG experience. One of our biggest concerns is how this game will played by those users who do not have too much RPG experience. We hope that the tutorial and general user interface affordances will help them through the game. These test will help us figure that out.

Apparatus

I used a Vista laptop in my own living room. (It should be noted that there was a lot of background noise when the user was playing the game.

Tasks

• Easy
  • The easy task is moving around in the world and interaction with objects. Moving can be done with directional arrows and interaction with the 'enter' and scan buttons.

• Medium
  • The medium task is to solve our current puzzle which involves the fibbonacci sequence. Before traversing through a bridge over fire, the user will be presented with a sequence of numbers. The user is to enter in thier answer with the directional pad.

• Hard
  • The hard task consists of battle with simple and hard level monsters. Battles in our Math RPG includes simple attacks, using special skills, solving math problems in order to meditate for additional skill points and using items.

Procedure

I started out by explaining the purpose of the test, namely, that we are trying to improve our prototype and that thier input would indeed assist us in improving the game. We then proceeded with the evaluation. I took out a stop watch to record the times of the things that I wanted to record, and a notepad to record the number of errors made during the evaluation. I then proceeded to explain to the user the basic controls of the game: The back button, the enter button, and the navigation buttons. I then generally described the tasks that I wanted her to complete. After that, I started up the game for her and proceeded with the evaluation of the prototype.

Test Measures

I wanted to measure a few things, namely:

• Time it took to solve the puzzle
• Time it took to defeat an easy level enemy
• Time it took to defeat a moderate level enemy
• Time it took to defeat the boss
• The number of times the back button was used
• The number of errors the user made and how they made it.

Results

Game Incidents

• When the game started, she expressed her concerns that she usually doesn't know what to do with games that she is not used to. I assured her that there is a tutorial that will help you.
• As she entered the world map, she completely passed up the entrance into the tutorial and traversed the entire map all the way up to the end. I let her do it. She asked, "what do I do now?". I pointed to the tutorial portal all the way towards the bottom of the screen and she yelled, "You let me go all the way up there?!".
• When she got to the tutorial portal, it took her a moment to realize that she had to push 'enter' in order to enter into the tutorial.
• She read the dialog boxes carefully at first, but scanned through the ones towards the end. I think this is the reason why she had trouble during the rest of the game.
• Her first interaction with the monster went rather smoothly. She used the simple attack option to finish off the easy monster.
• During her battle with the fox (the tutorial boss), the first thing that she did was use the meditate function. When I asked her later why she did that, she said, "When I meditate and solve a math question, I get stronger to do an attack". It took her a while to realize that it only increases her skill points(SP).
• Throughout the game, she kept on using the meditate function even when she didn't need it. I asked her during the game shy she did so frequently, and she responded with the fact that it increased her points. I pointed towards the SP and then she realized that she had maxed out on the amount of SP she could increase on.
• The math question also went smoothly for her, although sometimes she surprisingly got the wrong answer.
• In one of the math question, she accidentally chose the wrong answer.
• One way that I knew she didn't readily understand the difference between skills, meditate, and attack was that when she would use the meditate function, she would immediately use the attack function. I think this was so, because she didn't understand the power of skill and how meditate is used to restore skill points. Another attributing factor to this was that she probably didn't pay attention to the tutorial as much as she needed to.
• During the puzzle, she tried to enter in her number through the number pad.
• Her first input for the puzzle was 11. When she got that wrong, she tried to enter in another number but said, "Oh, I thought this was for two numbers!" She then got the correct answer, 13, and was very happy for getting that.
• Another point that I noticed, is that she found some jokes of the game funny. She didn't try to eat the mushroom though, so she didn't get a chance to enjoy that joke.
• During the boss, she wondered whether he would ever die. She thought she was doing something wrong. In my opinion, I don't think she would have wondered that if she had seen a status bar for the monster.
• After the subtraction dungeon, she went on to the multiplication level. I then explained to her that those levels weren't available yet.

Variables/Statistics

• Monster Defeat Times:
  • Ghost (Tutorial) - 15 seconds
  • Fox (Tutorial) - 45 seconds
  • Buffalo (Addition Dungeon) - 50 seconds
  • Goblin (Subtraction Dungeon) - 30 seconds
  • Boss (Subtraction Dungeon) - 85 seconds

• Total Monsters Fought: 5 Monsters
• Total Monster Defeat Time: 225 seconds
• Average Monster Defeat Time: 45 seconds

• Puzzle Completion Time:
  • First Pass - 65 seconds

• Number of Times 'Back' Button Used: 4

• Errors Made:
  • Misdirection in map - 1
  • Math Errors - 2
  • Incorrect Usage of Meditate - 3

• Total # of Errors: 6

Discussion

One of the first things I would make note on is my users lack of understanding the different battle functions of the game. Although my user knew what attack was and how to use it, she had a tough time understanding what meditate does and the effectiveness and use of skills. I noticed that she scanned the messages in the tutorial but still didn't understand the power of skills and also didn't understand that the purpose of meditate was to increase skill points so more skills can be used. The first thing I would change is to make our good and clear tutorial into a clearer tutorial so that our users understand our functions well. I think most game players already understand the purpose of attach and the specialty of magic and skills. However, when they are faced with another element of battle technique, as they are with our meditate technique, there is a good chance that thier understanding of battle technique might be lower. As a result, there is a chance that their confidence in using battle technique can be lower now as well.

Overall, I realized it didn't take her long to defeat monsters, even though she still had not mastered strategy for the game. This was obvious because of the way she switched between attack and skills when she had enough skill points to use skill. The average monster defeat time for her was 45 seconds. She had a total battle time of 225 seconds. This might not be enough to keep the player engaged and interested in the game. I would suggest an increase in the number of monsters and the average time it takes to defeat each one. The bosses of course, should be considerably harder than the normal monsters so that the users can find incentive to fight a larger number of normal monsters to level up. When a user is confident about their level, then they will fight the boss. The more monsters they beat, the more math question they would have to answer.

I would also try to make it more clear that the dungeon entrances are in fact dungeon entrances because my user had some difficulty in knowing what to do on the world map. Once she was helped in finding the entrance, the other entrances were similar so she had no trouble in finding those. Through the entire evaluation, she did not use one sign tip. Maybe a clearer marking and sign post will help here.

One little detail that I will also add, is to add the ability to use the number pad to enter in numbers as well as the input control we already have (using the up and down arrows).

Before any real usability tests are done, I would enhance and build on our prototype to include more monsters, items, and a better tutorial. This would allow us to collect more statistics and see where our users are spending the most time. We would also be able identify more common errors that our users make if we have more monsters to fight and more dungeon to traverse.

Overall, I think this prototype and game is turning out to be a success. I really believe it to be a wonderful balance of fun and education to keep any kid engaged in the game while their parents and teachers are confident that they are getting smarter. Yes, while playing a game!

Appendices

• Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
• Raw data (i.e., entire merged critical incident logs)

Informed Consent Form

Demo Script/Task Instructions