PilotStudy-Group:BuTtErFlY-HaoLuo
From CS 160 Fall 2008
Contents |
Introduction
This Pilot Usability Study is to test out our educational math game after the heuristic evaluation of our prototype. We have made a lot of changes to streamline the experience and make it easier for the user to play. Our new prototype has three complete worlds including a new world which serves as a tutorial that helps the player familiarize himself with the game.
The purpose of the experiment is to see what design problems we can catch to further improve our game after it has been evaluated by another group already. Have our changes made the game easier to play and a more enjoyable and educational experience? What more changes do we need? We are re-evaluating this because iterative design is needed to produce a good product.
Implementation
- Tutorial - added a tutorial level that helps the player learn the game
- Game Title and Title Screen - added a title and title screen
- Puzzle Backtracking - fixed the puzzle so even after you solve it to cross the bridge you can go back and forth
- Recognizable Signs - signs are added for the dungeons
- Emphasis on Math - balanced the game so physical attacks do very little; the focus is on math
- Random Events - randomly generated monsters now appear in dungeons
- Meditation Status - added status windows that tells the player if the meditate succeeded and how much SP was restored
- Meditation Change - changed to restore a certain percentage of the player's SP
Method
Participant - my roommate. Demographic: college student. He was selected because I had no one else younger to interview. He can do the math problems faster but many design flaws would stump him as it would stump our target audience, since he does not play much RPGs.
Apparatus: I used his computer using notepad to take down notes as he was playing and telling me what he was doing.
Tasks: There are three tasks. Easy, medium, and hard.
- Easy - The player should be able to navigate around the world map, enter dungeons, and exit dungeons back to world map. He should be able to use the arrow keys to figure out where to move, notice how to encounter monsters on the way, and find his way to the boss and the exit portal.
- Medium - The player should be able to solve a puzzle. He should be able to navigate to the puzzle, and give the correct input.
- Hard - The player should be able to beat a boss in battle. This is difficult because the boss is designed to be hard to take down and deal a good amount of damage. If the player is not careful, he could easily die and have to start over. The fact that when the player is low on SP he needs two turns to meditate and then heal makes the task harder. A good amount of solving math problems is needed.
Procedure: I told him to talk out loud about what he's doing in the game. Starting off, I said absolutely nothing about the game. He was able to navigate the main screen fine and enters the World Map. Then, he walked around and completely missed the portal. I still said nothing, wanting to see if a college student (someone who should perform this easier than a middle-schooler) can figure out how to enter a dungeon. After a while he notices the portal but has no idea how to enter it, and is stuck. At this point I had to tell him to hit the enter key.
After that he proceeded smoothly through the rest of the prototype. I had to give occasional hints like using an item, in order to help him not die because then he would have to start over. For a user, this is not a big problem because if the player hits game over he will simply restart and learn from his mistake, and learn through trial and error.
For this process I tried to talk as little as possible in order to see what was intuitive and not intuitive. Having a college student go through it was helpful because I know for sure that whatever problems he runs into is serious and needs to be fixed.
Test Measures
For the Pilot Study I wanted to measure how well my roommate performed the tasks, whether or not he gets stuck in the game, and how fast he completes the tasks. He should be able to complete the tasks much faster than a middle-schooler, so any task which takes him a while to figure out needs to be looked at. I paid special attention to time because anytime that my roommate struggles means that task is not intuitive by nature, whereas the tasks in which he doesn't struggle, but a 5th grader might struggle with (which another project group member might discover), probably means that the task is simply difficult in nature and not necessarily a design flaw. This is why it's useful to have someone outside our demographic test the game. Is the problem too difficult for our target group? Or is the design inherently flawed and thus needs to be scraped together?
Results
Initially, my roommate got through the menu screen fine but had trouble once he entered the world map. He walked around because (I figure) he's the exploring type. This took a long time and therefore I concluded that it's a rather serious problem. The problem is most of the map is empty because new dungeons have not been unlocked. This is something that perhaps we need to address. Then, he notices the portal into the first level but does not know how to enter it. This is a major problem because it's supposed to be a simple task yet he got stuck, and could not do anything until I told him what to do. After he enters the dungeon, the first monster teaches you how to use the attack command. However, he tried every command possible, disappointed at the lack of skills and confused at what meditate does (the tutorial hasn't explained it yet). After that he encountered another monster, which explained to him how to use skills and meditate, and he had no problem after that, although he expressed annoyance at having his skill miss occasionally. This is a minor problem which we will try to fix. Finding the exit was not a problem, and he liked having a new dungeon pop up after he beat the tutorial dungeon. Battling did not take him a long time to figure out, it went very smoothly. He proceeded to go through the dungeon with little problem, and complimented the "Addam" monster. However, he was a bit annoyed at how hard it was to beat the boss. It takes a long time, but it's designed this way to give the player more practice. For the puzzle, he had no problems with the puzzle itself, but we put in a mushroom as a lighthearted joke for students to not do drugs. He didn't get the joke and expressed confusion at why it was there or what it has to do with the game. However, he was able to get the puzzle easily and cross the bridge. He did not even bother to go back, which was one of our fixes, so it backed up my intuition that it's a minor fix. While fighting the boss in the third world, my roommate uttered "melee attack is useless." This is something we wanted the player to notice from the start since we want the player to use magic which needs math. This shows that we need to make the attack weaker and make it more apparent in the start.
After we finished he tried to go into the next dungeon which we haven't implemented yet and was disappointed that it's just an unfinished area. However, he still walked around and fought a few monsters even with no purpose which suggests that our game can be addictive. I felt the same way when I was play-testing so this is a good sign.
Summary:
- Easy task (navigating the world): Surprisingly took the longest time, mostly because he could not figure out how to enter a dungeon which is a big problem. Also because he wanted to explore the empty world map, which is a minor problem. Other than that, no problems navigating the world map as well as the dungeon map. Finding the boss, finding monsters, and finding the exit came naturally and required no effort.
- Medium task (solving a puzzle): No problems at all except something not relevant to the task left him a bit confused. The puzzle itself took no time which means the interface is easy to understand, it's just the difficulty of the puzzle itself that might give the players trouble.
- Hard task (beating a boss): No problems mainly because he already battled a few monsters through the tutorial, except it took a long as expected since the task is hard.
Discussion
Even though my roommate is 21 years old and far older than our target audience, he nevertheless ran into problems which tells us we still have a lot of work to do. There's a problem in that even though we included a tutorial, we do not teach the player how to access it. Rather ironic. Another group member's tester also ran into the same problem. I have since fixed it so that the player gets a basic message at the start of the game that teaches him how to navigate the world map. This also teaches the player to access the menu, which I myself did not know how to access until after my roommate discovered it while playing the game. I have also streamlined the tutorial a bit to make the text a bit more concise and easy to read, because I noticed my roommate simply skipped some texts. I also tweaked some minor stats to balance the game better and now attacks are very weak even in the start so the player is encouraged to use magic. One of my concerns was with how the leveling up was not very noticible, but my roommate apparently noticed it and told me, so perhaps the problem isn't as big as I thought it was.
My roommate apparently liked to explore, so this has me thinking that we should make the maps less linear, and add more roaming monsters on-screen like "Addam" and also add some items to encourage the player to explore the map, because apparently players like exploring the map instead of going from A to B.
He also liked the monster "Addam" because of its large graphic. This confirmed my thought that large monsters are more appealing to players. As such, I will try to make the boss monsters the larger characters. This is one of those design ideas that seems very simple and childish but is in fact very relevant to gaining the interest of kids. Also, he seemed intrigued by Addam's story, which further makes me think that we need to have a solid story in order to keep the player more engaged. Although, I noticed that he kept playing the game afterward just to try to beat more monsters and level up. This showed that leveling up by itself can be fun and addictive. In fact this is the conclusion I came to in the first day of class which is why I wanted to work on such a project. I think a good strategy is to have a good story and incorporate it into major events in the game like the bosses, but have longer dungeons in order to give the player more opportunities to battle normal monsters and gain levels because it is enjoyable.
Some of the problems he ran into were also problems that the other project group members found with their Pilot Study. These are areas that we need to look into as design problems that need to be fixed or scrapped and remade from scratch. This includes not knowing how to enter a dungeon or using attacks when it is highly discouraged. Other problems that the other project group members found which my roommate did not have problems with are probably difficulty problems, and these are not design problems but require some minor tweaking and balancing.
Appendices
Incident Log
- walks around, passes tutorial
- walks to the end
- (I'm stuck)
- (doesn't know to press enter)
- tries to use kill, sees no skill
- tries meditate
- (what does sp stand for?)
- tries item
- (i want to fight more)
- (impatient)
- (pissed at missing)
- (likes the spells)
- realizes he has no sp and meditates
- uses meditate too
- uses attack and spell
- wrong meditate, doesn't know how to use escape
- (can't heal your enemy?)
- keeps attacking, too strong
- notices some questions keep repeating
- how do i know how much my max hp is?
- noticed the level up
- didn't notice the item
- ate the mushroom
- kept exploring in the bridge before trying to cross it
- doesn't heal unless he's almost dead
- surprised by the variable amount of magic
- notices melee sucks
