PilotStudy-Group:Group Phi-tus-Frank Yang
From CS 160 Fall 2008
Contents |
Introduction
When learning a new language, students are forced to completely change the way they formulate a word or sentence. Translating a sentence is like deciphering a code. It has been understood that one key way of improving a student’s conversational skills in a foreign language is simply through practice. However, in a classroom environment, students’ are limited to practicing with only their peers during class or organized practice sessions that are hard to schedule. Furthermore, the act of practicing vocabulary words in a classroom is typically monotonous without a casual environment. The system we created aims to take advantage of the accessibility of the internet and open up possibilities for students to easily practice their learned language. Furthermore, by creating a “serious game” the students are provided with a casual environment that encourages the student to do well to win the game.
The purpose of this experiment is to observe the effects of the design changes that were implemented to the system’s prototype. Specifically, the experiment will test the effect of the user’s ability to complete the tasks, which will be measured by the time elapsed as well as users’ reactions and comments.
Implementation and Improvements
When considering the design of this system, the ease of the process of completing the task at hand is crucial. After reviewing the comments by the heuristic evaluating group, there were some issues we felt were crucial to the gameplay. These issues are of the utmost importance because the player must feel comfortable completing the task at hand due to a time limit. Any hindrance that the player may feel with the interface will deter the player from allowing his or her own skill to benefit the gameplay.
One important heuristic issue we felt we needed to address was the player feedback heuristic. Any issues with players not receiving ample feedback will only result in confusion, which will result in time wasted trying to maneuver through the interface. Before, the drawer’s screen had a palette to work with and ample tools to draw the picture. However, it was brought to our attention that the drawer’s screen lacked any feedback as to what tool was selected. We fixed the problem by keeping the selected button highlighted to provide the player with information as to which tools and settings are selected. We also included an indicator for both the drawer and the dictator that indicated when the player’s partner was speaking or drawing.
For this phase of the project, the aesthetic level of the design was still kept to a minimum. The purpose of this experiment was to test the functionality and flow of the completion of a task. The purpose was not to attract the player and keep the player visually engaged. Some minor adjustments were made, but not many were changed.
There were a few issues regarding the player’s freedom and control that we also fixed. The severity of these problems were not as large as the before mentioned feedback problems because the pace of the game is relatively one-directional. The lack of a back arrow in some menus would be undoubtedly preferred to none at all, and we updated the system to assess the issue.
We updated a few other basic functions that were not functional in our initial prototype. For example, the line tool was just a planned tool to implement in the initial prototype. For this phase, the line tool has been implemented as well as the eraser tool.
Based on the experiment conducted, a few other proposed changes were mentioned, and will be mentioned in sections below.
Method
The intended demographic for this game is actually for anyone that is currently learning a second language. Since this demographic is quite large, I decided to simply look for a student that was currently enrolled in a foreign language course. Ideally, the participant will have some sort of limited conversational ability in the foreign language. The oral ability of the participant cannot be too high or else feedback may not be accurate, for there is much less appeal and incentive to play the game as a learning tool. I found Derek, a junior majoring in computer science who is currently enrolled in Chinese 10BX at UC Berkeley.
Apparatus
There were very few things that were needed to conduct the experiment. The only equipment that was needed was a pad of paper for to take notes and record observations and the laptop for the participant to interact with the prototype. In order for proper time to be recorded, a stopwatch was also discreetly used, as to not create any pressure for the participant. For the sake of convenience, we decided on conducting the experiment in the labs of Soda Hall.
Tasks
We felt that our tasks remained accurate in outlining the goals of our system, so we felt that the tasks needed no revision.
The tasks outlined below have been taken from previous stages of the design of our system.
Rate the drawings (easy task): We felt that rating a drawing was a relatively easy task. The user is simply presented with a screen showing all of the drawings that have been submitted from the last round with a set of an unfilled stars used to perform the rating under each drawing. The rating process is similar to that of rating a product on Amazon.com, Apple.com and various other familiar websites. As such, the user must simply hover the input device over the stars and once the amount of the stars is “filled”, the user then clicks to lock in the result. The user repeats this process over the number of drawings then selects the submit button for the results to be tallied. During this task, time elapsed for the participant to complete the task will be recorded, as well as movement of the cursor on the screen. Extremely random cursor movement will be disregarded, but any movement that implies exploration will be observed and recorded.
Create a new game (moderate task):Game creation was chosen as our moderate task because it requires the user to interact with elements not used in other parts of the game; namely, the information form that is presented to users in order to choose the basic parameters of the game such as language and difficulty level. This form is reached by selecting the “create a new game” button form the main screen. Once here, the user chooses the game’s parameters, which are mainly guided through a set of drop down menus for easy setup. Once the user is satisfied with their choices they can click the “create” button and are then taken to the waiting room where they await other player to join their newly created game. As with the first task, the time elapsed for the participant to complete the task will be recorded. While the time elapsed between each element of the menu will not be officially recorded, the time taken for each individual element will be noted.
Draw a picture (difficult task):Drawing a picture is one of the fundamental tasks of the game so we felt it very desirable to make sure that we got this right. The screen we used is very simple and familiar to that of most drawing programs such as Photoshop but not nearly as robust. The task simply requires the user to select the, mostly familiar, tools in the left hand toolbar such as the shape or pencil tool and use them to draw a picture. The task of selecting the proper tools to create the desired effects was the crucial step for this task. Once the image is drawn, the user can either wait until the timer runs out or submit the picture early for possible bonus points. This task requires a slightly different assessment than the tasks above. Because the round only lasts one minute, the elapsed time will always be one minute. Instead, the actions of the participant must be carefully observed and noted. While time to complete the task will not be an issue, the time spent on learning the interface will be of much use. It is difficult to correctly assess when exactly a participant is drawing or simply just learning the interface, so the observations will be based on reasonable judgment. Furthermore, the time spent using each tool was also recorded
Procedure
In order to keep the informality to keep the participant in a relaxed position, no official script was recited to the participant. Instead, a paper with bullet points was used to ensure that main points were explained and nothing important was omitted. First, the participant was given an explanation of what the system was and how the game worked. Allusions to the existent games Taboo® and Pictionary® were made as to help familiarize the participant with the game without even having seen it. After the initial description, the participant was shown the main screen of the game and various other screens of the game. This quick demo of the game was very quick as to have the participant learn as little as possible from the demo to have an authentic new experience completing the tasks. In order to avoid having the participant learn about the system through the demo, the tasks were presented to the participant after the demo had been shown. The initial presentation of the tasks was a simple overview of each task. After briefly mentioning each task, the participant was then informed of the first task in full detail. Observations and notes were recorded during the execution of the task. Upon completing the task, the participant was allowed time for his own comments on the system. This was then repeated for the second and third tasks. After the completion of the three tasks, time was allowed for the participant to give his comments for the system as a whole, as well as any recommendations he had for future improvement.
Test Measures
For task 1, the time taken for the completion of the task was measured. The number of clicks per picture was also noted. Since the pictures up for voting are provided using the “Wizard of Oz” technique, it was made apparent which drawings were good and which were poorly drawn. This allows us to use the time as a more accurate data in analyzing how effective the interface was. Because this was simply a pilot usability experiment, it was not desired to have the participant spend time on deciding what ratings to provide to the pictures, but only to spend time on learning how to give the ratings.
For task 2, the total time elapsed was again recorded. The total time elapsed would be reflective of how easy the interface was to use. There is very little contemplation involved, so extensive time spent would be due to poor design.
For task 3, as mentioned before, total time elapsed is set. Because of this, it was more reasonable to see how often each tool was used or selected.
Results
Time for task 1: 13 seconds
- Time elapsed to rate picture 1: 9 seconds
- Time elapsed to rate picture 2: 2 seconds
- Time elapsed to rate picture 3: 2 seconds
Observations: The participant immediately hovered over the stars. He read through each of the ratings and lightly laughed. He then preceded to rate the pictures pretty quickly.
Time for task 2: 22 seconds
- Game name: 13 seconds
- Language: 2 seconds
- Difficulty: 2 second
- Num Players: 4 seconds
- Private: 1 second
- Password: 0 seconds
Observations: Very little hesitation in all parts except game name and number of players. Hesitation of game name only due to the participant’s want to think of a non-generic name.
Time for task 3: 60 seconds
- Pen: 38 seconds
- Eraser: 14 seconds
- Line: 0 seconds
- Change color: 2 seconds
- Change width: 2 seconds
- Clear screen: 4 seconds
Observation: As the participant started to draw, the participant used the clear screen often. The participant would try to correct mistakes with the eraser around the middle of the minute and near the end used it much less. The colors were changed as the participant felt was needed. The width of the pen was changed once or twice near the beginning but remained the same width for the majority of the minute.
Discussion
Task 1 Discussion: Judging by the results, it can be seen that the majority of the time was spent reading each of the captions under the ratings. Just by observations alone, it seemed that there were no problems in completing this task. From this, given that other participants for the experiment experienced the same results, it can be concluded that the task
Task 2 Discussion: From the observations and results, we can see that the only extended amount of time was spent at the creation of the game name. Because the participant wanted to create an interesting game name, we cannot attribute the extended time to a poor interface. Furthermore, it is expected for a field that is a simple textbox to take longer to complete. The only other slightly longer time was for the number of players field to be completed. The slight hesitation was matched with a slight furrowing of the brow, indicating a slight bit of confusion. The participant tried clicking the arrows and was surprised that the numbers decremented and incremented by 2. He settled on creating a room of 6, but only after he tried decrementing past 4 and incrementing past 10. It seems that even though it wouldn’t make much sense to play the game with only 2 people, because the arrows provide the option to do it, players may try to do so only to hit an invisible wall. It should be made clear what the minimum number of players are as well as the maximum number. Also, since the range is only 4, 6, 8, and 10, the need for the arrows to increment and decrement may be unnecessary. With so few choices to choose from, by changing it to a dropdown menu, the player can see all the choices at once and know exactly what the limits are without trying to increment and decrement past the bounds.
Task 3 Discussion: From the results, it can be seen immediately that majority of the time spent drawing was with the pen tool. However, the next highest time was with the eraser. Since this is a drawing game on a computer, the player is forced to use a mouse to draw. Drawing a picture with a mouse is no easy task, and mistakes are consistently made, as seen with the participant in this experiment. However, seeing how the clear drawing button was used so often in the beginning, it can be deduced that the further along in a drawing, the less likely it is for a drawer to feel the need to erase a part of the drawing. Regardless, neither the eraser nor the clear all buttons should be taken out, as both are needed. However, given the short timeframe, providing the user with an undo and redo button may be useful, but also unnecessary. There is not enough information just from one participant for any conclusions to be reached. It was also seen that the color changes were minimal. The participant definitely felt the need to use the color palette, but it should be noted that color changes were minimal. Again, due to the time constraint, the participant felt that there was not enough time to draw too many details, nor did he feel like he had the ability to draw all the details. The color palette has only a limited number of colors, which is great for the task at hand. Most surprisingly, though, was the lack of any use of the line tool. Although the tool is very popular in most simple paint programs, it was completely untouched by the participant. This is a reasonable situation, however, as many objects do not have perfectly straight lines. It may even be reasonable for the tool to be taken out. However, again, the data from a single participant is not reason enough for a tool to be taken out completely. Even so, the time taken to switch from the pen tool to a line tool to draw a single line is longer than the time in which it requires to draw a passably straight line with the pen tool. Consideration of removing the line tool altogether should be noted.
Appendices
Script Outline
- system introduction
- problem and proposed solution
- brief demo
- brief overview of tasks
- explain task 1
- allow participant to complete task 1
- comments, answer questions, brief discussion
- repeat for task 2
- repeat for task 3
- final questions and comments. Longer discussion about any task
- conclusions and thanks.
Screenshot of drawing by participant
