PilotStudy-Group:BuTtErFlY-MikeKendall
From CS 160 Fall 2008
Contents |
Introduction
The system that we are evaluating is our math RPG prototype. This is not a complete version of the game, however, it has enough functionality to span a lot of the features of the complete intended version of the game. Our math RPG prototype now cleverly named "Quadratic Curve Sin of the Buffalo", is a role playing game intended to provide a recreational way of gaining basic math skills.
We will be conducting a pilot usability study on our prototype. We will be mainly conducting a qualitative study on our prototype. Since we do not have enough time to test our prototype with many subjects, we will not be conducting a full blown quantitative study. We will however include a little quantitative analysis along with measures, summary statistics, and variables. This study will provide for us a fast, inexpensive way of figuring out what fundamental things we have to change, find and prioritize problems, and roughly evaluate the usability of our prototype.
Implementation and Improvements
As per our observations and the heuristic evaluation provided by Group 4, we have implemented the following changes:
| Updates | Description |
|---|---|
| Recognizable Signs | Signs are the prototypical way for a game to communicate with its player. In our previous iteration, we did use signs but they weren't regular enough to be recognized as a standard. This time our use of signs is more formulaic and will hopefully be easier to use. |
| Puzzle Backtracking | When the player enters the puzzle sequence, was previously going through a one way tube with a boss at the end of it. Once on the other side of the bridge, there was no way out other than through that boss. This IS a legitimate strategy for constructing dungeons, but wasn't what we intended so we fixed this by adding a path to get back from the right side of the bridge. |
| Escapability | Again, we had a problem of style. In most RPGs, there is no running away from boss monsters. Since this game is meant to challenge your math skills before challenging your strategic/role playing skills, we decided to let the player run from boss battles. |
| Emphasis on Math | The player's physical damage is now reduced so that more emphasis will be placed on using skills to attack. As you will see, my subject didn't see the benefit of this change and never really understood the use of meditate as a part of combat. |
| Random Event Monsters | In dungeon maps, players will now encounter enemies without first seeing them on the map. This serves for giving the player more enemies to level up against, but also allows for more practice of math skills as they are needed. |
| Game Graphic and Title | The game now has an (in)appropriate graphic and title screen. |
| Meditation Feedback | In our old prototype, the only feedback that a player received when they used meditate was a number on your character indicating the number of SP that he gained. Although this does follow RPG standards, it wasn't enough to make the mechanics of the game obvious to players. To fix this, we added a message after the meditate skill is used to tell the player what happened. |
| Increased Meditation Effect | In tandem with making the player's normal attack weaker, we had to give him more skill-firepower to fuel his math cannon. Yeah. This forces the player to work within our combat system, which would be pretty confusing without... |
| Tutorials | There is now a tutorial that starts the game and explains the basics of combat and the skill system. This seems to be enough for most players, but as you'll later see, our system is not approachable enough for extreme non-gamers. |
Method
Participant
The user I used for this my pilot usability study was the same one that I used for the Task Analysis assignment earlier. She is a non-technical 4th grader and, as described before, she covers the non-mathy, non-gamer crowd... This is the kind of child who might be forced to play an educational game at school or home.
Apparatus
I used a desktop in my room to run the game. I ran it full screen so there there would be no distractions.
Tasks
| Level | Description |
|---|---|
| Easy | Our interface follows the norm for the Role Playing Game genre. The easy task that we are tracking is simple interaction with the world. This is achieved by walking up to objects and pressing the "Enter" button on them. |
| Medium | Our medium task is solving a puzzle. This is a harder task because the player has to solve something in an abstracted world with only four directions and two buttons to work with. In our game, the puzzle that we have implemented involves inputting a two digit number. This is done by using left/right to change between the 10's and 1's digit and up/down to change the value in that digit. |
| Hard | Our game's combat system requires that the player have a balance between using healing skills, attacking skills and meditating. The goal is for the player to realize that using their character's normal attack is less effective than doing these larger attacks, which require math. Our hard task is defeating a boss monster, the hardest type of monster in RPG games. He will have more hit points and attack for more than a normal enemy. |
Procedure
I first explained this game as an educational one. With our newly added tutorial, I simply explained that the "Enter" key was "Ok" or "Yes" and that the "Escape" key was the equivalent of "Cancel" or "No." I noted that the arrows were for movement and selection and that no other keys would be needed. I was keeping close notes but as you'll see, the test didn't go quite as planned.
Test Measures
I wanted to measure a few things, namely:
- Time it took to solve the puzzle
- Time it took to defeat an easy level enemy
- Time it took to defeat a boss
- The number of errors the user made and how they made it.
Results
Game Incidents
- At first, she didn't know where to go. She wandered around the first screen and I had to talk to her. I asked her what she saw that she could interact with and she said "Plants." I said that that wasn't what she should be looking for and then she realized that I meant the portal. She then said to me "But that's where I came from!" and entered the tutorial.
- When using meditate for the first time, my subject got confused and had to pause for a couple of seconds before realizing that she had to hit "Enter" again.
- She used meditate twice more before returning to attacks.
- She understood using the "Enter" button to interact with her environment almost instinctually.
- She didn't use skills when attacking. Instead she used normal attacks through most of our time together. At one point i stepped in and used "Heal" for her because she was going to die. The second time that she found herself in this predicament, which was against the boss to the addition dungeon, I did not step in and let her die. It was clear to me that our system was too complicated to understand this quickly.
Variables/Statistics
- Monster Defeat Times:
- Ghost (Tutorial) - 20 seconds
- Boss (Addition Dungeon) - died
- Boss (Subtraction Dungeon) - n/a
- Puzzle Completion Time:
- First Pass - n/a
- Errors Made:
- Misdirection in map - 1
- Game Flow Stall - 1
- Misunderstanding of Combat - 2 (and then some)
- Total # of Errors: 4
Discussion
From our first iteration to this one, we decided to shape the game so that the player would be forced to use meditate more. We thought that if we made meditate much more powerful, and gave the player more useful spells, then they wouldn't use the normal attack function.
It seems that we pushed this difficulty too far since non-RPG gamers don't seem to understand the balance between meditating, SP, HP and attacking. When I played through this prototype, the game seemed quite balanced, though I did worry that we were pushing the level of complication too much.
My subject complained when a random encounter happened. I later answered that we had warned her that this would happen, but the problem isn't that she forgot about random encounters... Random battles against easy enemies are used to get stronger and in our game are the easy/free way to heal yourself. That battle could have been used for meditating and healing so that by the end of it she was ready to take on a boss. Maybe it is the case that RPGs will always be the hardest games for casual gamers to get into.
On top of problems with respect to making the game approachable, there are problems with the maps/tilesets that we need to work on. It seems that most of our tests with this prototype led the player to be confused about where to go from the start. The signs, although a good idea, don't pop out enough as elements to be interacted with. Enemies and portals, however, seem to be very obvious. After the first portal is used, all of the remaining portals are obvious.
On the bright side, the game mechanics do work. Although our numbers and our growth curves are a bit unforgiving, the game itself makes a lot of sense to RPG players. The flow of the game is quite good.
As a side note, I asked my subject if she would be willing to wait for me to play through to a puzzle so she could solve it and she wasn't interested. I feel like I have collected some of the most useful data, even though one of the main tasks wasn't attempted at all.
Appendices
- Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
- Raw data (i.e., entire merged critical incident logs)
