PilotStudy-Group:BuTtErFlY-JordanBerk
From CS 160 Fall 2008
Contents |
Introduction
The system we are evaluating is our interface for our Math RPG game, in which uses a player math to accomplish tasks (mainly achieve victory over monsters in battles), as in a standard RPG. Our purpose is to get young elementary school kids learning and practicing math in a fun yet educational environment.
This pilot usability study is being conducted in order to evaluate the features and gameplay of our interactive prototype for our Math RPG. Numerous interface changes have take place since the low fidelity prototype evaluations, and gathering qualitative data on the usability of the features of the interface will provide valuable data for refining and perfecting the final prototype.
Implementation and Improvements
As per our observations and the heuristic evaluation provided by Group 4, we have implemented the following changes:
- A scan function which displays enemy HP upon request.
- Made dungeon signs more visible and obvious.
- Added the ability to go back over the bridge in the dungeon map.
- Made physical attacks weaker to emphasize magic/math usage.
- Added random event monsters to allow more free roaming map navigation yet maintain a good amount of battles.
- Added a title to the start page.
- Made all battles escapeable.
- Added a window to inform the user upon answering a math question incorrectly.
- Added a tutorial section at the beginning of the game.
Method
Participant
Not having any access to any people in our target user age range, my participant for this study is a 4th year political science major at UC Berkeley. Although he is obviously much more advanced at simple math than our intended user group, he is very inexperienced with RPGs, and because our target user group includes RPG novices, he can represent them appropriately. Being able to answer the math questions will be easier for him, but this should not effect the testing of the critical functions and tasks (none of the tasks below are math-knowledge dependent).
Apparatus
The test was conducted in a quiet room in my apartment. My notebook computer was used as the platform that our game was run on. The user had access to a mouse as well as the built in trackpad, although he quickly figured out that neither is used in the game. As the participant played the game, I monitored his status with a notepad and stopwatch.
Tasks
Easy - Navigation
Our interface follows the norm for the Role Playing Game genre. The easy task of moving around and interacting with your world is achieved by using the arrows (or the d-pad, if you use a joystick) and there is one button for “confirm” and another for “back/cancel.” Interacting with elements in the world is as simple as walking up to them and pressing the confirm button.
Medium - Puzzles
When a puzzle sequence starts, you don't know what the input method is going to be. Maybe you will walk around and press buttons in a specific order, maybe you will input a large number one digit at a time. Either way, the player will have to interpret the puzzle as it is presented, do some mathematical analysis on it and then somehow give the answer back to the game. In the prototype, our puzzle is very simple, but additional puzzles will be more challenging and/or have alternate input/solution methods.
Hard - Combat
In addition to a standard RPG combat system, our combat system throws a twist at the player by making them solve math problems in order to replenish their SP (spell points). This task is harder because of the problems and because the strategy of knowing when to use a turn to 'meditate' as opposed to using a weaker non-magic attack.
Procedure
I first explained to the participant the nature of the experiment and what exactly would/wouldn't be expected of him. I made sure he understood that there was no such thing as a mistake, and then had him sign the consent form.
I then sat the participant down in front of my computer with the game loaded up. I instructed him to choose "New Game," and from there, thanks to the in-game tutorial, very little instruction was needed to prepare the participant for the experiment.
The first task I asked him to perform, as described above, was to walk around the map and enter the Subtraction dungeon.
Following this, I allowed him to explore the subtraction dungeon. Because of random pop-up enemies, the hard task of combat came up before the medium puzzle task. I was expecting this, however, because we recently added this feature to the prototype.
Following this and any other pop-up battles that occurred, the participant made it to the bridge and I instructed him to complete the medium task, which is solving the puzzle.
The participant was then instructed to battle the Ogre boss, which is a much more difficult battle than the pop-up ones, and requires more strategy and basically requires the user to use math magic.
Test Measures
- Number of times user tried to use magic without enough SP
- Time between running out of SP and using meditate to get more
- Time to get into the subtraction dungeon
- Whether or not user looked at the signs to figure out which dungeon was which
- Time to defeat pop-up enemy
- Time to defeat boss enemy
- Time from puzzle entry to puzzle completion
- Number of times math magic used vs. number of times standard attack used
Results (by task)
Easy - Navigation
The user took a total of 20.57 seconds to get into the subtraction dungeon after having completed the tutorial and upon having been instructed to enter the subtraction dungeon. The user used the signs to find the correct dungeon.
Time to get into the subtraction dungeon: 20.57
Whether or not user looked at the signs to figure out which dungeon was which: Yes
Medium - Puzzle
The user took a total of 37.6 seconds to complete the puzzle. It should be noted that the user, being a college student, recognized the Fibonacci sequence very quickly and thus probably completed the puzzle quicker than someone who isn't familiar with the sequence would have.
Also, the user went back (to the left side of the screen) once in the puzzle map, and then went back in upon realized that his action had sent him back to the original map.
Time from puzzle entry to puzzle completion: 37. 6
Hard - Combat
The user started out a bit confused about how to do combat. The user beat the first enemy in 54.6 seconds, using only standard attacks. The user discovered magic in the second one, and was able to beat the pop-up enemy in only 19.5 seconds (two magic attacks). The third and final pop-up battle was won in 59.7 seconds after using meditate (see next paragraph) and one powerful magic attack. Overall, the pop-up battles took an average of 44.6 seconds.
Upon running out SP, the user went immediately back to use magic. Although the option was faded out, the user clicked enter 3 times trying to access the attack. It took the user a few more seconds to find the escape button to get back to the attack options screen. The user then tried a standard attack, and upon seeing that addition magic was still not accessible, went to meditate, and correctly answered the question. The entire process took 47.53 seconds.
The boss battle took 1 minute, 22 seconds, about double the time of a normal pop-up battle. The user first attempted a standard attack, and upon realizing that it had no effect, continued with magic attacks for the rest of the fight, replenishing SP with Meditate when necessary.
Number of times user tried to use magic without enough SP: 3
Time between running out of SP and using meditate to get more the first time: 47.53
Time to defeat pop-up enemy: 54.6, 19.5, 59.7
Time to defeat boss enemy: 1:22.28
Number of times math magic used: 7
Number of times standard attack used: 10
Discussion
Overall, the user seemed to be fairly comfortable working within the interface of the game. His comments afterwards only strengthened this viewpoint. Although he certainly was not immediately familiar with some of the controls, such as pressing enter to jump to a dungeon or using Escape to back out of a menu, he was able to grasp these concepts fairly quickly and didn't have a recorded error following the initial learning.
The tutorial seems to have played a major role in the user's understanding of the interface. Especially for RPG novices like my participant, the tutorial is a crucial way to teach the basics of combat, and especially to explain the game-specific features like Meditate. Even to an experienced RPG player, it is not openly clear what the meditate ability does, or how/when to use it, so by explaining its use in the tutorial, the learning curve of combat strategy is made much less steep. That said, it does seem like the tutorial is missing a few key elements. Things that are not explained and may not be obvious are how to use an item and the various functions of the escape key. Also, the user is required to figure out how to get into the tutorial by pressing enter on the portal. I think a good change would be to make the tutorial automatically loaded at the beginning of the game, with an option to skip it for people replaying the game.
The first task was very easy for the user, especially because the tutorial covered it. My participant made good use of the signs that give helpful tips by going up to each one. This was a change we put in since the first stage of the prototype, and it seems as if the change was effective.
Getting through puzzle was also a smooth process for my user. Although he temporarily left the puzzle interface by exiting through the left, he figured out how to get to the puzzle question, and didn't hesitate on using the number input system.
The battles were the most interesting. It seems that there wasn't a clear enough explanation of the "Esc" button, which is kind of all purpose for representing both "Back"(in a battle) and "Menu" (in a map). An explanation of this button's features would be very helpful in the tutorial. The user tried many times to use magic without enough SP. After a few tries of clicking and even performing a standard attack, he finally realized that Meditate is the only way to replenish SP (items will accomplish this as well in the full version of the game).
Against the boss, the user quickly figured out the optimal strategy to fight the ogre, especially after realizing that standard Attack did 0 damage. He quickly switched to a Math magic/meditate method of attack and made short work of the boss.
Overall, the user seemed to enjoy the game, and hardly made any indications verbal or otherwise of frustration or confusion. I'll take this as a good sign that we're on the right track, although we certainly need to expand the game play and balance certain gameplay elements.
Appendices
Incident Log
- Liked the title (got the math reference although our target user audience probably wouldn't)
- Went up to the tutorial portal expecting entry but figured out enter needed to be pressed (didn't seem to like that he had to press enter based on facial expression)
- Tutorial smooth, completed tutorial tasks no problem
- Used the signs to find subtraction dungeon (didn't go into addition dungeon)
- Walked around subtraction dungeon a little, found the sign that explains what to do in it
- Encountered first pop-up battle, little surprised but understood that battle was about to occur despite never playing an RPG before
- Used only standard attacks first battle...maybe forgot about more powerful magic or maybe just wanted to test effectiveness of standard attacks
- Left puzzle to the left, little frustrated that screen didn't scroll, I think, but went right back into puzzle screen
- Used up/down arrows instinctively for number input for puzzle
- Second pop-up battle: remembered/used magic, didn't run out of SP, beat enemy in two quick hits
- Third pop-up battle: tried to use magic, out of SP, seemed frustrated that the magic was inaccessible, tried standard attack, magic still not accessible, remembered and used meditate, killed enemy in one magic hit
- Boss battle: tried standard attack once, saw had 0 effect, then switched to a magic/meditate attack plan (tutorial works!)
- Reacted positively to the exploding new level
