PilotStudy-Group:BuTtErFlY-GaryWu

From CS 160 Fall 2008

Introduction

System

This system is a short demo of an interactive and educational role-playing game designed to teach elementary and middle school kids the fundamentals of mathematics and prepare them for the learning of higher-level math, such as algebra and pre-calculus, in their future studies. Using gaming to our advantage, we designed this game to allow players to enjoy the gameplay, yet retain valuable educational information while doing so.

Purpose

The rationale behind this experiment is to do a medium-fidelity test on real users. However, the demo used in these tests will incorporate changes made after the heuristic evaluation of our game as well as the feedback given during our presentation. Ultimately, we want to be able to evaluate the user's satisfaction and performance in our game. Stemming from the low-fidelity prototype interviews, we will continue to monitor critical, interesting, and unusual incidents that occur. This information will help us mold the game to a more usable state.

Implementation and Improvements

Updates	Description
Monster Status	The player is now able to determine how much life the monster has left when battling. This is done via a new skill called "Scan". This is a magic spell that provides the player with information about the monster during a battle. This addresses the visibility of the system status.
Recognizable Signs	Signs and tips are now all consistent and match with the real world. A graphic of a signpost is used to convey that information is available to the player when touched. Because they are consistent, signs now stand out in dungeon maps, as well as world maps. This matches the system to the real world as well as addressing some aesthetic issue.
Puzzle Backtracking	When the player enters the puzzle sequence, he/she has the option of returning to the dungeon map without having to solve the puzzle. However, the player will eventually have to solve the puzzle in order to cross the bridge. Additionally, after correctly solving the puzzle, the player is now able to go back across the bridge to the previous side. This addresses the issue of user control of freedom.
Escapability	Before, BOSS monsters were inescapable. Added a fix where now, all monsters are escapable in a battle. This addresses the issue of user control and freedom as well.
Emphasis on Math	The player's physical damage is now reduced so that more emphasis will be placed on using magic attacks to successfully advance through the levels. By guiding the user towards using magic attacks, the player will have to meditate more, which in-turn allows the player to practice and learn math.
Random Event Monsters	In dungeon maps, players will now encounter random event monsters that appear without warning. This is used to provide challenge to the player to keep the player interested and focused.
Game Graphic and Title	The game now has an appropriate graphic and title screen.
Meditation Status	After a player uses the skill "Meditate", a status window appears after the answer has been selected to alert the player whether he/she chose the correct answer and if any SP had been restored. This addresses the visibility of the system's status.
Increased Meditation Effect	Along the lines of decreasing the physical effect of a players attack, the Meditation skill effect is now increased to replenish more of a player's SP. This was done to balance the reduction of the player's physical attack.
Tutorials	At the start of the game, the player is introduced to the system and it's gameplay with a tutorial that explains the game. This addresses the help and documentation issue of the game.

Method

Participant

The user selected for this interview was a 5th grader who did not regularly play RPG games (low-level) but had an intermediate-level of math skills. This user was selected to represent the players who have little to none RPG game playing experience.

Apparatus

The equipment that was used to run the demo was a laptop running Windows XP with a version of our latest interactive demo. Information and notes were taken with pen and paper. The experiment was conducted in the living room, inside the home of the user.

Tasks

Level	Description
Easy	Our interface follows the norm for the Role Playing Game genre. The easy task of moving around and interacting with your world is achieved by using the arrows (or the d-pad, if you use a joystick) and there is one button for “confirm” and another for “back/cancel.” Interacting with elements in the world is as simple as walking up to them and pressing the confirm button.
Medium	When a puzzle sequence starts, you don't know what the input method is going to be. Maybe you will walk around and press buttons in a specific order, maybe you will input a large number one digit at a time. Either way, the player will have to interpret the puzzle as it is presented (which should be obvious), do some mathematical analysis on it and then somehow give the answer back to the game. In our prototype, our puzzle is very simple, but the method for inputing the numbers comes from standard console input. It's counter-intuitive to make the user use arrows and a confirm button to input a number when they're already using a keyboard, but if they wanted to play this on a gamepad or other input device, this method will still work.
Hard	Our combat system throws a twist at the player by making them solve math problems in order to replenish their SP (spell points). This task is harder both because of the problems and because the method of input is slightly different. This is also the same method that is used when you are choosing a skill to use when attacking an enemy, except that in this case when you make the wrong choice, your turn ends and you get attacked without doing anything useful.

Procedure

I first explained to the user the rationale behind running this interview. I explained to him the focus of the class and how this interview would help in revising the current prototype for our RPG game. Because the user is of the age demographic we are focusing our product to, I explained to him how his input would be invaluable information. After he had understood what I said, I asked if he wanted to proceed with the testing. I then provided the consent form which was basically the printed version of what I had just explained to him. After he had signed the form, we proceeded to testing the interactive demo.

After loading up the demo, I proceeded to explain what the controls were: keyboard arrows, enter key, and esc key. I showed him how to walk around the map and how to enter dungeons before going into the demo script. No questions were asked during this process. After performing the demo script, I asked if he had any questions. The questions he asked were answered and I proceeded to restart the game.

From here, I told him about the goal of the game and how not all parts of the game were implemented. This was where I introduced the three tasks that we were benchmarking with our dependent variables. After explaining the tasks, and making sure that the user understood, I watched and took note of interesting, unusual, or critical incidents that occurred during the game. I also kept track of the time and error rate of performing these tasks. I would only answer questions if the user was completely stuck or had given up.

After the test had been completed, I asked the user to rate his overall satisfaction with the game, as it is one of our dependent variables, and asked if had another other input he would like to share about the game. All this was recorded and taken account. Once again, I thanked the user for his help and reiterated the fact that the information gathered was invaluable.

Test Measures

Variable	Reason
Task Time	This was chosen to measure the efficiency and speed of task completion. A consistent low time could be attributed to either a UI issue or knowledge-based mistake. Whatever the case, this data can be compared to and contrasted with other participants to rate the effectiveness of the layout and design.
Error Rate	Consistently high error rates can be easily distinguishable and be the basis of re-design.
User Satisfaction	This is what really matters at the end of the interviews. No matter how great a design or layout we come up with, if the users do not like using it, there's no point of continuing to develop an interface that won't be used.

Measured the time it took to defeat a simple monster
Measured the error rate during a battle with a simple monster
Measured user satisfaction with battle system
Measured the time it took to defeat a boss monster
Measured the error rate during a battle with a boss monster
Measured the time it took to understand the navigation system
Measured the error rate during navigation
Measured user satisfaction with navigation system
Measured the time it took to solve the puzzle
Measured the error rate puzzle solving
Measured user satisfaction with puzzle system

Results

Easy Task

Variable	Result
Task Time	27 seconds to figure out controls and map
Error Rate	Low. Wasn't sure how to enter and leave dungeons in the beginning. Thought you only had to walk up to the portal but later, quickly, figured out that you had to press the Enter key. Automatically knew how to read the sign posts when navigating.
User Satisfaction	Liked the simple navigation system. Reminded him of previous games.

Medium Task

Variable	Result
Task Time	8 minutes 13 seconds to solve the puzzle
Error Rate	Medium. Instinctively knew how to use the input system, which was good. However, couldn't figure out the sequence until he went online and searched for math puzzle help (critical incident). Was guessing and checking for a while until he found the pattern online and figured the puzzle out from there.
User Satisfaction	Thought input system was fine. However, thought the puzzle was a little hard. But he mentioned he felt cool knowing a math pattern.

Hard Task

Variable	Result
Task Time	30-50 seconds for simple monsters, 4 min 43 seconds for the Boss monster
Error Rate	Low. User instinctively knew how to attack and use magic spells. Mentioned that the tutorial helped during combat. Spent most of the time experimenting with the attacks, skills, and meditate. Learned how to properly use meditate after three tries.
User Satisfaction	Liked the battle system. Reminded him of Pokemon (one of the few games he actually played). Thought that battling was really fun and didn't mind doing the math.

Summary

The tutorial played a major role in explaining the system to the user. Also, game recognition helped in navigating through the interface (i.e. Pokemon style battle system helped my user fight efficiently because he had played Pokemon before). Signs and tips were definitely looked for and read. Navigation and input usage was easily and quickly picked up.

The puzzle seemed a bit hard for my user. I suspect that my user was a bit young to understand the pattern and may not be the case for older users. Most of the time spent in solving the puzzle for my user was in the actual research of the pattern.

Battling seemed to be the favorite part of the game and most exciting. Also, by making magic attacks a more pivotal part of defeating boss monsters (our forcing function), more math was being used during battles than our previous prototype. Also, the user liked to experiment with all the options in the battle menus before figuring out the best way to defeat the boss (magic + meditate combo). This was where most of the time was spent.

The status menu was barely used because it was not well documented and was overall confusing. Because of this items were not used. However, I could not effectively test this during battle (using items during battle is an option) because the test ended after the defeating the boss monster.

Overall, the user enjoyed the game and even wanted to continue to play the game after the test was over (definitely, a good sign). Many of the mistakes the user made could be remedied by a better tutorial setup.

Discussion

After conducting the pilot study, I can see the huge importance of the tutorial section. Without it, the learning curve of our system for inexperienced RPG players would be quite high. I was surprised by how quickly my participant picked up on the combat system of our game. Upon questioning, he attributed his success to the tutorials and to the association he had with past games he had played (more specifically, Pokemon). I would continue to develop and create a more completed tutorial section that spans our whole system in future iterations. An example of something that could be covered would be the Item and Status menu.

I was also surprised at how helpful the sign posts and tips were during the game. My participant actively looked for them and read every one of them before continuing with the game. If anything, more helpful hints could be added in the future to help guide the novice user.

The puzzle input system seemed to work out great. Because we drew from past games, the user was not confused on how to input the answer. However, I would definitely change the difficulty of the puzzle depending on the user. We might have to adjust the puzzles to fit with the personas we have for our game. I was actually excited to see that my user went online to research the pattern and try to find the answer out that way. He discovered the Fibonacci sequence online and understood what it was because he wanted to solve the puzzle and continue on with our game. I thought that was a significant sign. An edition I would also make is an option to escape/cancel out of a puzzle. Right now, once the user enters a puzzle situation, he/she is stuck until it is solved.

As expected, the battle system seemed to be the most substantial part and draw of our game. The user thoroughly enjoyed battling and picked up the system very quickly. This was due in-part to the tutorial system as well as recognition. My user learned the optimal strategy to defeat boss monsters (magic + meditate combo), which forces the user to do more math in order to continue on with the game. I thought that our addition of the Meditate status messages really helped the user know whether something happened when using that skill. The item portion of the battle system was not tested and this was due to lack of documentation and items. In future iterations, I would like to incorporate a more significant role of items in battling to see how that would work. Overall, I think we got the core functionality of our battle system intact.

Overall, I thought our demo was a success. This was shown in my user's want to continue playing the game. Furthermore, some documentation on how to save the game could be added in future iterations as my participant wanted to save his progress. I feel that the core functionality of our system is there and most of the edits from this point on will address design/aesthetic issues and balance of fun/education.

Appendices

Informed Consent Form

Demo Script/Task Instructions

Incident Log

Approved of tutorial
Thought the music was cool
Didn't know you had to press the Enter key to enter dungeons; thought you just walked up to it (figured it out later, though)
Automatically recognized sign posts and read every one of them
Pressed Esc on the world map and got to the status menu, but closed it right away; when asked, he said he didn't know what it did
Thought the visible monsters were cool and scary looking; initially avoided them at first
Said "Whoa" when a random event monster engaged him
Experimented with entire battle menu; but he looked like he knew what he was doing; when asked, he said he just wanted to see what he could do and what everything did
He mentioned that the tutorial helped in using the system
Pressed Enter when engaging a visible monster; he didn't need to as monsters are activated by player touch, but he didn't realize
Thought magic attacks and animation were really cool; was excited when battling
Entered the puzzle map, but then went back and read the sign; then went back to the puzzle map
Got to the puzzle and wanted to escape after reading, but couldn't
Knew how to use the input system, but couldn't figure out the pattern and was randomly guessing
Asked if he could use the internet (I said yes)
Most of the time taken to complete puzzle task was in researching
User found the pattern online and put in the correct number
Mentioned that he felt smart knowing this pattern
Engaged Boss monster, but immediately escaped
After walking around for a bit, re-engaged Boss
Initially used physical attacks, but then after seeing the effects of the magic attacks, he relied on them
Used Meditate correctly right off the bat; when asked, he said that he remembered from the tutorial
Asked if he could use pencil and paper (I said yes); also used his fingers sometimes to count
Mentioned that he felt really powerful killing the Boss monster
Didn't use the item after receiving it; when asked, said he didn't know how
After telling him that the test was over, asked if he could save (I said yes), but he didn't know how to; also, mentioned that he wanted to play more (because a dungeon was just unlocked)