PilotStudy-Group:Phi-tus-JuanPadilla
From CS 160 Fall 2008
Contents |
Introduction
Word soup is designed to provide users who are trying to learn or master a foreign language with an enjoyable and interactive environment that is both educational and fun. To accomplish this, Word soup incorporates elements like word association and collaboration from word games such as Pictionary® and Taboo® into the system and extends those ideas into learning a language. Because it has been shown that these elements are essential to acquire even a basic knowledge of a language, our system aims to exploit these characteristics of learning while using the proven methods of delivering an enjoyable game from the above mentioned games.
The goal of this experiment is to see how users will respond to the system’s design and to incorporate the results of the test into design changes of the system’s prototype. Specifically, I will be looking at variables such as time it takes to complete the specified tasks and the number of errors that the user makes while trying to compete these tasks. Once the data is collected, it will be used to further refine and optimize the system’s design.
Implementation and Improvements
With the help of the valuable information we received from the other team’s heuristic evaluation of our prototype, our team has continued implementing and improving features in our prototype. Specifically, we have made improvements on some of the major heuristic violations our prototype contained such as, limited user control and freedom in many of the user screens and a lack of consistency and standards in some of our screens. We also tried to address many of the small violations as well, noting that many small violations can sometimes be worse than one or two major violations.
Two of the major heuristic violations our prototype contained were limited user control and freedom and a lack of consistency and standards. In many of our screens the user was simply trapped once they entered it, meaning they had no way of turning back if they perhaps got there by mistake or just changed their mind. Fortunately, this was easy to resolve by going through each screen and adding a back or cancel button that allowed the user to escape if and when they chose to do so. Another major violation we addressed is the fact that our screens were inconsistent in our button placement. We had buttons containing the same functionality in different areas on different screens and in the case of the chat box, we used different layouts on various screens and had buttons that performed functions that had nothing to do with the chat box aligned with the buttons that performed chat box specific functions. To resolve these issues, we went through the system and placed all the buttons containing similar functionality in like places throughout the prototype. Additionally, we set a common layout for the chat box in all of the screens and grouped only those buttons that had relevance to the chat next to it and placed those that had no relevance elsewhere on the screen. The fixes we implemented have provided more user freedom and control and a more consistent interface.
In addition, to the two major heuristic violations described above we addressed several other issues. • In the drawer screen, there was no feedback as to which tool the user had selected and was currently using. To address this, we have added a border around each tool that is highlighted when the tool is selected. • Some of the text was too small so we changed the font to one that is larger and easier to read in those instances. • Provided better navigation throughout the screens by implementing the functionality that did not previously exist although the button indicated that it should. For example, the link In the main screen to "create new account" is now clickable allowing the user to use that feature directly. Also, the join game button now takes the user to the waiting room to wait until they can join in the game.
Method
The participant I used for the study is Joy Johnson. Joy is a college graduate who holds an associates degree in Interior Design from the Colorado Institute of Art. I chose her as my participant for two reasons. First, as a design graduate, she has experience with functional design and space planning. Second, she is planning a trip to Rome and is looking for a quick and easy way to brush up on her Italian. The combination of Joy’s background in design and her desire to brush up on a foreign language made her a natural fit for the task.
Joy and I met at a local Starbucks for the test. This location was chosen because it is offers a similar environment, as far as noise level, interruptions, etc., that a typical user will use during actual game play. Joy performed the three tasks on my laptop computer, which we set up at a table with the prototype of the game running inside of a web browser. As Joy performed the tasks, I took written notes of her actions and used a stopwatch to time her movements.
Rate the drawings (easy task):
The user is simply presented with a screen showing all the drawings that have been submitted from the last round with a set of an unfilled stars used to perform the rating under each drawing. Therefore, the user must simply hover the input device over the stars and once the amount of the stars is “filled”, the user then clicks to lock in the result. The user repeats this process over the number of drawings then selects the submit button for the results to be tallied. Create a new game (moderate task): Game creation was chosen as our moderate task because it requires the user to interact with elements not used in other parts of the game; namely, the information form that is presented to users to choose the basic parameters of the game such as language and difficulty level. This form is reached by selecting the “create a new game” button form the main screen. Once here, the user chooses the game’s parameters, which are mainly guided through a set of drop down menus for easy setup. Once the user is satisfied with their choices they can click the “create” button and are then taken to the waiting room where they await other player to join their newly created game.
Draw a picture (difficult task):
The screen we used is very simple and familiar to that of most drawing programs such as Photoshop but not nearly as robust. The task simply requires the user to select the mostly familiar tools in the left hand toolbar such as the shape or pencil tool and use them to draw a picture. The task of selecting the proper tools to create the desired effects was the crucial step for this task. Once the image is drawn, the user can either wait until the timer runs out or submit the picture early for possible bonus points.
Procedure
I met Joy at a local Starbucks coffee shop for reasons mentioned above. I bought her a cup of coffee and we set up the laptop at an empty table. While setting up the laptop I explained to her what the purpose of the experiment was, what I was going to be looking for during the experiment and how the data was going to used afterward. Also, I wanted to make it clear that any mistakes she encountered were not her fault but ours and that she was able to stop the test at any time, for any reason. Once the introduction was completed I asked her if she had any questions, which at this point she did not.
To begin the experiment, I explained to her the rules of how the game is played then went through logging in and getting her to an area of the game to which she could begin the experiment. Once here, I explained the first task to her, as described below, and allowed her to proceed with the task. Once finished, she continued with the remaining tasks until all were completed.
While the tasks were performed, I kept notes using a stop watch to record the time it took her to not only complete each task, but to measure the time it took her to perform the steps within each task. These incremental measures were done so that upon determining that a task took too long, it would help us to determine which aspects of the task were creating the bottlenecks and could therefore be improved. Along with the time measurements, I also kept watch of each time she made an error, or used help for assistance.
Once the test was completed, I answered her questions about the overall design and she gave me some additional feedback on the overall game design. As an established designer, she had some helpful tips on button placement and font readability that I hope we can incorporate into future versions of our prototype. We finished our coffee and left Starbucks after about an hour.
Test Measures
As mentioned above, I not only recorded the total amount of time it took to perform each task using a stop watch but also measured the time it took Joy to perform the steps within each task. It makes sense that by recording the breakdown of each task it would allow us to easier go back and see qualitatively where the most amount of time during each task was spent and could be potentially improved. Another important measurement was the number of errors/pauses Joy made during the task. I felt that this was especially important for our game because it is a timed gamed and a user does not have much time to spend correcting their errors and reading help screens or trying to figure out what they need to do. Also, it is an important measurement because if a user is frustrated because they can’t figure out how to use the interface, they will simply not use the product and look for alternatives. While there are probably other good measures to have looked for during the test, for the reasons just listed, time measurements of the tasks and “sub-tasks” and the number of errors/pauses encountered during the tasks, were viewed as the most crucial.
Results
Over all, the testing went well. Joy was able to perform each task within a reasonable amount of time and seemed to be comfortable in moving through the tasks. She had only a slight bit of hesitation in some sections but it seemed to be due more to personal thoughts than due to the interface itself; explained in more detail below. Most of her difficulties came in one area, however, the first task, but it is hard to determine if this was due to a lack of expectation of how she should perform the tasks, which might have been due to a lack of instructions or because she actually had difficulties with the interface; this is questionable because she simply flew through the other tasks, which we deemed more difficult, with little effort and full understanding of what she had accomplished. Additionally, Joy never consulted the help menu, even though it was available to her on all screens.
The first task Joy performed was to create a new game. She completed this task in a respectable time of 26.8 seconds. The time breakdown for each “sub-task” is listed below. Even though we have listed this as our easy task, this task seemed to be the most difficult for her and as explained above, it is hard to determine why, so comparing data with my group will be essential here. One interesting thing to note when considering time on this task was that she hesitated to think of a name for the game. Initially, I thought she was confused but when I questioned her after the task she explained that she was simply thinking of something creative. Another problem she encountered was that she added a password even though she wasn’t sure what it was for. When I asked why she had entered in a password and showed here the help menu, which explains that it is to be used with a private game, she said she was confused and felt she was required to add a password. Even though she was confused she said she saw the help but just didn’t think of using it.
