Low Fidelity Prototype-Group:BuTtErFlY
From CS 160 Fall 2008
Contents |
Team
| Name | Contribution |
|---|---|
| Gary Wu | Paper prototype of battle sequence, played cpu during interviews, method section |
| Mike Kendall | Observer/notetaker during interviews, results and discussion sections |
| Mohammed Ali | Facilitator during interviews, results and discussion section |
| Hao Luo | Paper prototype of world/dungeon map and puzzle, played cpu |
| Jordan Berk | Greeter/notetaker during interviews; introduction/mission statement and appendix sections |
Introduction
The system we are evaluating is our interface for our Math RPG game, in which uses a player math to accomplish tasks (mainly achieve victory over monsters in battles), as in a standard RPG. Our purpose is to get young elementary school kids learning and practicing math in a fun yet educational environment. For this stage of the usability testing, we have created a lo-fidelity prototype and run simulations with various users. After a short demo, we had them perform three tasks. Meanwhile, we observed and learned from their comments and reactions, and will use this to improve our system.
Mission Statement
With our Math RPG, we hope to get kids to learn, practice, and use math in a fun yet educational environment. Optimally, the game will be able to get kids who have little interest in math to practice and improve their knowledge.
Prototype
World Map
Dungeon Map
Battle Screen
Magic Menu
Item Menu
You Attack
Enemy Attack
Puzzle
Meditate
Method
Participants
We decided to select participants that matched our pre-determined criteria. We wanted a good representation of players that would have a varying degree of experience playing role-playing games and aptitude for math. The following table outlines the skill level of users tested in our interviews:
| Role-Playing Game Level | Math Skill Level |
|---|---|
| Superuser | Moderate |
| Moderate | High |
| Unfamiliar | Low |
We felt that these participants represented a broad range of our spectrum. There would be the super-users, the moderate users, and the inexperienced users.
Environment
The environment used in our testing was a quiet and spacious room with a table in the center of the room. On one side would be the "computer." The computer was played by two of us during the tests--the two would switch playing the CPU depending on the task being done. On the other side would be the participant. Next to the participant stood the facilitator who explained the process to the participant and answered any questions. Next to the facilitator stood the greeter. To the side of the table, the observer stood quietly watching with a clipboard for notes.
Tasks
We chose three tasks to have the participant accomplish. The tasks varied in difficulty from easy, moderate, and to hard.
Easy Task: Navigating the World Map
Navigating the world map should be an easy task. The basic principle of moving in a game should be almost second nature to any player. Thus, an intuitive design that draws on previous games and familiar real world examples (i.e. a compass, or grids on a real map) would make sense here. In this task, we asked the user to walk to different places in our world. This would be done by clicking on icons on the screen. Once the player moves to the selected icon on the world, the user would then enter the "dungeon" of choice. This would bring the player to a dungeon map that behaves similarly to the world map. Once again, the task of navigating would be proposed to the player. Navigation on the dungeon map works the same way as the world map with the exception of items, monsters, and puzzles blocking the players path. These objects would then lead the player to our next task.
Moderate Task: Battling a Monster
Once the player encounters a monster, the battle screen appears. The menu layout of the screen draws heavily from the Pokemon role-playing game. The use of website icons to draw familiarity with drop down menus were used. The heavy use of status messages during the battle sequence were used to help the player while battling. The battle sequence is turn-based. This means that the player and monster trade turns so only one player is performing an action at a certain time. There are four basic actions in battling: attack, use a magic attack, meditate (replenishes magic power), and use an item. In our system, the player would have to answer math questions whenever meditate is used. Meditate can be used multiple times in a turn to replenish magic points for use in magic attacks. Magic attacks are much more effective than regular attacks and each attacks takes a turn to use. Items can be used however the player wants but also takes a turn. This was deemed a moderate task as the player has to orient him/herself with the menus, figure out the meaning of certain actions, and strategize the best way of defeating the monster.
Hard Task: Solving a Puzzle
Another roadblock that the player will run into while navigating a dungeon map would be puzzles. These were deemed a hard task as the interface for solving puzzles are arbitrary and are sometimes the puzzles themselves. A puzzle can be frustratingly hard to solve if the interface used to solve the puzzle is very complicated and cryptic. The puzzle used in our testing was a simple graphical representation of the Fibonacci sequence.
Procedure
Before we started, we made sure to do dry runs of our prototype as a group to knock out any major kinks and get a general feel on how the system should behave. Once this was satisfactory to us, we froze the paper prototype and went ahead with testing. We based our testing on the reading "Prototyping for Tiny Fingers." Therefore, we had a greeter, a facilitator, an observer, and two computers. As described in the environment section, two people in our team played the computer. This depended on the task being done. One person acted as the CPU when navigation and puzzle solving was being performed. The other person played the CPU when the task of battling a monster was being performed.
When the participant entered the room, the greeter welcomed the participant and put the participant at ease. We wanted to make the participant feel comfortable and relaxed. The greeter gave a quick introductory speech on what was going to happen. This included mentioning the cs160 class and curriculum and a general overview of why and what the interview was all about.
After the greeter had finished, the facilitator was introduced. The facilitator then explained the actual concept of our project. This included what serious games were, what focus our game was going to take (math-based rpg), what paper prototyping is, and how the interview was going to be conducted. The facilitator introduced the rest of the team and the parts they were going to be playing. Once finished, the greeter asked if the participant understood everything that was going on and asked the participant to sign the consent form.
Once the consent form was signed, the facilitator told the participant that they were going to run through a demo. Using the demo script, a mock test was ran in front of the participant. During the time, the facilitator prompted the participant to make sure that everything was clear. If something wasn't the facilitator would explain.
When the demo was complete, the facilitator explained the three tasks to the user in detail. After each explanation, the computer would come the life and the facilitator would ask the participant to accomplish the task. Any questions the participant asked, the facilitator would field. Also, during the process, the facilitator would also ask the participant the reason why he/she chose to use that attack, or why a certain icon or button was clicked. Everything here was noted by the observer. Questions asked by the participant and answers from the facilitator was duly noted by the observer as well. The computers would switch off, depending on the task that was being currently performed by the participant.
After the tasks were complete, the facilitator then asked the participant if he/she had any comments or final questions regarding the tests. The facilitator also had a couple of set questions to ask the participant, such as "What did you think of the battle layout", "Did the map make sense?", and "Did the status messages help?" Questions and answers in this portion of the interview was also noted by the observer.
Once all the interviews were complete, we met as a group and categorized the feedback we got from the participants. We discussed, as a group, the information obtained from these interviews and used them to improve our interface and assessed the severity of problematic layouts that the participants pointed out.
Test Measures
The main goal of our tests was the pinpoint the problematic and cryptic layouts of our game to users of varying levels. Thus, we felt that participants who did not regularly play games had more substantial feedback that was relevant to our goal. We wanted to focus on how the user was interacting with our system and if menu layouts and on-screen icons made sense. This would definitely help us on our iterations of layouts.
The big-picture detail or process data of our test was to actually see if the user would enjoy playing this game knowing that the concept of math-learning was involved. Basically, if given two choices (one being doing problem sets from worksheets and the other being this game), which one would students prefer to do in a class?
The bottom-line data of our test was just familiarity of menu layouts, time it takes fight a monster, repetition of math problems solved, and confusing interface designs. One of the most important detail here was the repetition of math problems solved. This, being the main selling point of our product, had to be solid. As said before, practice makes perfect and the more math problems solved while retaining a certain "fun factor" of the game was a substantial piece of information we were looking for during our tests.
Results
It was notable that the super user started each task by clicking on everything that seemed clickable before actually making a decision. Other than that there was nothing too notable about the world map task. He didn't like that he couldn't move across the whole map freely, but in the exit interview he ended up agreeing with us about our design decision so it's hard to say. The moderate and unfamiliar users effectively did the same thing and explored the menus and maps before taking further actions.
The super user generally had no problems playing the game at first. This was the case until he came upon a button that took action without letting him go back.. This was the attack button at first, and he had no chance to try the meditate button. After the first attack was made, he went on with the battle as though the meditate button didn't exist, until he noticed that there were no mp potions. At this point the super user got confused. He asked the computer why there weren't any mp potions and received a blank stare. He started experimenting again and found the meditate function.
The moderate user had a issue with not knowing what the meditate button did. After clicking it and successfully solving the math problem, he got a message of "Tried to restore MP, however MP is already full!". He was pretty upset because he lost a turn for no reason and expressed his views that he should get a choice to restore HP or MP.
The unfamiliar user got confused at first with some of the status messages. For example, she was confused about who Hedgehog was (The name of the monster she was fighting).
The super user noticed the elements (type) button and changed his character's element. This changed the type of questions asked when meditating and also made his attacks super effective. He didn't seem pleased that changing elements took a turn, but that's a question of battle mechanics. We weren't pleased that it took so long for him to find the elements button, but that's getting into discussion. As the battle went on, the user realized the power of magic attacks and didn't use normal attacks at all. This is the point of the game, since in order to use magic attacks, you have to spend mp which are only refilled by solving math problems.
The moderate and unfamiliar users were not so quick in figuring out strategy as the super had done, however, they both won their sample battles.
In general, all the users were confused about the type of math questions and the relationship to the dungeon because of the fact that they solved other types of math problems in the addition dungeon other than addition. The moderate and unfamiliar users harldy used the meditate button, however, once they ran out of MP, they eventually found the meditate button to restore it. All the users also had an issue with the difference between meditate in battle mode and dungeon model which they explained at the end.
A simple piece of feedback that was given by the unfamiliar user was why the status message said "okay" when she solved the problem correctly as she was confused of whether or not she was successful in doing so. She said that instead of using "okay" we should have said "correct" or something of that nature.
The moderate user also seemed annoyed when he had to solve a problem for using the items such as the "exponent bomb". He believed that the item should have been "free".
The moderate and unfamiliar users had some trouble in figuring out how to input the answers for the puzzle. They had trouble figuring out how to input their answers and struggled to find a sort of "enter" button to submit. However, once they figured out the general flow, it was fine. The puzzle task for the super user went relatively smoothly, with a handful of incorrect clicks at the beginning as well. The moderate user also felt as if he should add feedback to the puzzle when he solved it at the end. He expressed that he didn't like the fact that the input is at the bottom when he tried to add it to the circles themselves.
Discussion
After the interview was over, the super user stuck around and made comments that there wasn't enough feedback, especially in the puzzle section. Although he did understand the puzzle quickly, he thought the interface could have been more intuitive. In order to make the layout more intuitive and less confusing, we decided to change they layout so that input is more natural. Instead of having an input box on the bottom, we will have the input entered directly into the box with immediate feedback if the correct answer is entered by having a green border around it or a red border if the answer is not correct yet. We also noticed that there was no cancel button in order to get out of a puzzle so we added it to allow users who have difficulty to cancel if they wanted to.
The group discussed what happened in the super user interview and decided that the structure of the game was somewhat flawed. The whole “element” part of the game is counterintuitive but maybe this is only a product of someone who has already played rpgs. There isn't much of a reason for someone to click on a box labeled “element.” Not only that, if the user were to change his type to addition, he could refill his mp much easier than with algebra, in turn circumventing the point of learning math.
The element system has been removed and replaced by meditate questions that depend on what dungeon you're in. This way, the player is forced to do the hardest level of math available as he progresses through the game. Attack will still be a weak attack and magic will not have any element, instead just getting stronger with larger mp costs (very simple, very old school – we believe this will work since the target audience is pretty young).
In order to avoid forcing a tutorial down the player's throat (or in order to make the game playable even if the don't pay attention), we decided that it would be a good idea to mention meditate during the feedback message for not having enough mana to cast a spell. Also, there should be an explanatory window between clicking on meditate and actually meditating, explaining what will happen.
Because it seemed counterintuitive for the user to have to get attacked repeatedly and go through status windows while meditating over and over, we also concluded that the meditating task should have some kind of chain effect built in so that the player can keep answering questions until they have full mp, if they wish. In order to balance this out, we would have to make enemies stronger, of course.
Appendix
Critical Incidents (see Results/Discussion section for elaboration):
Task 1 (moving around world map/dungeon):
- Lots of exploration of what is click-able
- Confusion over what meditate does in dungeon mode
Task 2 (battle):
- Confusion with the Meditate button
- Status message confusion
- Strategy confusion (about using magic)
Task 3 (puzzle):
- Required a lot of clicking around to find correct buttons
- Lack of feedback expressed
- No cancel button
