PilotStudy-Group:Team Pint
From CS 160 Fall 2011
Contents |
Team Members
- Wrote Up Design Doc + Conducted Interviews
- Wrote up Design Doc + Conducted Interviews
- Wrote Up Design Doc + Conducted Interviews
- Wrote Up Design Doc
Introduction
Physdom is a mobile game for learning physics in a fun and engaging manner. With the current iteration of our game, we performed a simple pilot usability test which gave us results that we will incorporate into our design. Rather than focusing on the physics and mathematical aspect of the game, this study centers on the interface of the application, seeking to determine whether screen layouts and controls are intuitive to the user in performing tasks. With a necessity for user input and the lack of time to perform a large-scale study, the experiment entailed a few observational studies of users being asked to perform tasks with the current system. Included is also a qualitative analysis of the users' experiences and performances.
Implementation and Improvements
- The "Tutorials" section has been updated with more complete documentation.
- Added backgrounds to menus to avoid the default solid black background.
- "Lives" feature, the player is given 3 chances to shoot the ball into the basket.
- In the "Projectiles" game, we give the player either the velocity or angle of the shot, and then they must calculate the other variable to score.
- Rearranged button arrangement to give a friendlier user interface.
Method
Participants
- Participant #1
- Is a 20-year old EECS major. He has significant physics experience and has strong opinions on how physics should be taught, and is also very opinionated about user interfaces, especially concerning games. He was chosen as a sort of critical voice who would point out problems with the interface and game concept overall.
- Participant #2
- Is a 28-year old software engineer. He has been working for a small startup for the past 3 years. I chose to show our application to him since he has experience and knowledge in the development of mobile applications. So as an expert, he was able to give me tips and suggestions that were directly relevant to our game design. He was also easy to talk to and was able to explain things on a coding level regarding our application. This helped in discovering and squashing bugs.
- Participant #3
- Is a 21-year old physics major. He has taken both AP Physics classes in high school, as well as an introductory Physics course at UC Berkeley, followed by a few higher level Physics classes. His expertise in introductory physics was valuable in determining the correct approach in teaching such concepts. This participant was chosen for his high level knowledge in physics so that he would be able to give us more insight regarding how our game should be structured in order to most effectively teach players.
Apparatus
- The experiments were conducted using no more than the android tablet on which the application was loaded, a pen and paper for notes and scripts, and a stopwatch for keeping time. Subjects were also provided chairs to sit on for the duration of the study, with pens, paper, and calculators given to them on request. The test environment and apparatus were kept simple for the sake of avoiding unnecessary elements, but care was taken to keep the environment stable and to avoid mitigating elements such as noise or bad lighting.
Tasks
- Task #1: Adjust Settings (EASY)
- The first task was to navigate to the settings screen and to select a preferred measurement system. This task is extremely simple, and measurements were limited to checking whether the user could navigate easily and whether the settings and their controls were intuitive.
- Task #2: Look at Tutorial (MODERATE)
- The second task was to navigate to a tutorial on projectiles and to explore the tutorial page. This task contained the meat of the observations, including time to find and navigate through the game mode menu and any errors encountered while doing so, as well as reaction to the layout of the tutorial screen.
- Task #3: Play A Level (HARD)
- The third task was to navigate to and explore the first level on projectiles. This task was important to checking over the actual game screen interfaces, and important observations included time the user took to understand and utilize the various controls, whether or not they made errors or had problems actuating the controls, and whether they had difficulty understanding the premise of the level.
- Additionally, all tasks included observation of counts of misnavigations, significant pauses, and instances of verbalized confusion or complaint, as well as recording of all 'critical incidents' that were observed and that might inform design changes.
Procedure
- As with the low-fidelity prototype tests, subjects were greeted and introduced to our game and study concept. They were then shown the prototype on the Vizio Tablet and were then given a brief demo of the touch-screen controls. At that point the proctor read through a script which instructed the subjects on the tasks that they must complete. Then, after waiting for each task to be completed the participant was given instructions regarding the next task. Subjects were also given some verbal descriptions of the intent for features not yet fully implemented, and how some of our on-screen elements in the current iteration are placeholders and serve a larger purpose that is yet to be implemented. In addition, the participants were encouraged to voice their thoughts and feelings on the implemented features that they used to get through each of the tasks. As this step was taking place, any and all experimental data was observed and recorded closely.
Test Measures
- Time taken to reach the task page. This is a simple measure of convenience, to ensure that the user does not need to actuate overly-many controls to access desired parts of the app
- Number of misnavigations per task and what these missteps entailed. This helped us to identify places where the structure of the application is not intuitive.
- Number of significant pauses and verbalized instances of confusion or complaints. Again, this helps us to gauge the intuitiveness of features, by noting when a given feature takes a long time to be understood by users. Furthermore, verbal instances of confusion or complaint point to features that, while potentially intuitive, may not provide the most desirable user experience
- Critical incidents. While not quantifiable, this was the most important measured data. Looking for places where users had unusual ease or difficulty, and those where they provided significant positive or negative verbal response, provides pointers to what parts of the app are currently well-designed vs. what parts currently pose recognizable and conscious difficulty for users. Redesign of the app after this experiment will focus on those traits seen in critical incidents.
Results and Discussion
Results
- Participant #1 had difficulties with the application, and voiced complaints about its structure. In particular, he suggested that he did not think physics was something that could be effectively taught in a 'friendly' manner, but that rather the application should be aimed at convenience while providing the academic content (equations and problems) in a rigorous manner. As such his major complaints included the incoherent colour schemes and inclusion of 'splash' images, as well as what he described as a 'cluttered' and 'busy' layout. His suggestions included restructuring the application to include a smaller scope, or perhaps to organize different physics topics more hierarchically within the teaching and playing sections rather than in broad top-level categories. He also complained about text buttons, suggesting that large pictorial buttons would benefit the application more than text buttons, and that controls might be placed on the diagrams themselves, with explanatory popups for some elements on mouseover. Finally, his assertion was that the colour palette should be made more consistent, but rather than 'brighter' or 'friendlier' that a more 'classy' or 'minimal' scheme might be more effective.
- Participant #2, as a software engineer, thought that this application had the basic form of what a game app should be like. But he did suggest making some changes, such as when a button is pressed, the user needs to receive some sort of response which indicates to the user that a button has been touched. Perhaps a sound response, or a graphical change when a button is pressed would be sufficient. He also suggested that the "back" button be relocated to the top left part of the screen to best adhere to common touch-interface design practices. As a game, he suggested that buttons and such be converted to graphics, so that the application conveys that it is a game. But it is also worth noting that he liked our ideas to fill the physics engine placeholder with videos. Finally, he suggested a background score in accompaniment with the game.
- Participant #3 generally liked the application, but had a few complaints. His complaints were mainly about cosmetic features of our design. He suggested that we keep backgrounds consistent across all of our screens. He also suggested making our background stylistically more simple. He suggested that we spend more time on our level design. He believes that although for the first level our game is of suitable complexity further levels might need to engage the player more. Also the levels could use clearer sliders with units and other necessary information being more clear to the user. He understood our lack of animation due to complexity issues, but he thinks that it will be necessary if we hope to truly engage the player. On a more structural note he thinks that we should have only one access point to the settings menu. He also believes that we should have a section containing ALL of the tutorials and not just having each section containing only the pertinent tutorials.
Future Changes
The three interviews we conducted revealed several weaknesses in our design. These weaknesses will make us focus on two broad categories: cosmetic changes and structural changes.
Cosmetic Changes
- We need to focus on keeping backgrounds consistent. It came up multiple times that the subjects were not happy with having a main menu background that differed from the other selection screens. Also we need to make these backgrounds simple enough so that they can be used as the background for game modes and tutorials.
- It was noted that there was some difficulty in operating some UI elements and that it took significant time to understand some of these elements. To combat this we want to use more distinct buttons and are considering making the buttons larger / spaced further apart to make it easier for the user to interact with the UI elements.
- The general layout of buttons could be a little bit more interesting. We will try moving buttons around / changing their appearance to make the interface more appealing.
- We will also try out different looks for the interface. Some subjects have liked a simple, but cartoony layout while others would prefer a more professional looking application.
- We are considering including music and sound effects in our game, but we want to make sure that if we include this that the user has the option of turning the sound off.
Structural Changes
- We need to either only allow one access point for the settings menu or include a note that no matter where you access the settings menu from it will affect all game modes.
- One subject suggested that we make a more broad tutorial section that includes ALL of the tutorials. This would allow users to more easily access a tutorial if they were only interested in that feature and didn't want to play the game at that moment.
- We will try to make the tutorials more interactive and also break it apart into different sections that the user can swipe through rather than scroll through. We believe that breaking apart each tutorial into sections will make it easier for the user to learn the material.
- Although users had some issue with figuring out the different game modes we probably won't change this because ultimately it allows the application to be expanded much more easily.
