PilotStudy-Group:Group Phi-tus-Haosi Chen
From CS 160 Fall 2008
Contents |
Introduction
The serious game our team have designed is call Word Soup. This is a game that designed to help players to improve their secondary language skills, and this is kind of a mix up with the Pictionary®/Taboo® types of games. Our research has shown that the vast majority of the language-learning process involves verbal communication in the language in question. Our goal in this design is to bring this "best" aspect of learning into an environment where players will try very hard to both speak and comprehend the language. The purpose of the pilot usability study is looking feedback about the usability by average users and possibly to find any confusion from the user interface. Besides that, we would like to make sure from the pilot usability study that we have improved and fixed all the confusions or errors from the Heuristic Evaluation.
Implementation and Improvements
Following are the functionality improvement that our team has made since the interactive prototype.
- In the drawer screen, there is no feed back when choosing drawing tools so that I can not see which tool I choose. Solution: We added a bold boarder on the selected drawing tool.
- The mission statement on the first screen is too small. Solution: We changed the font size to be bigger.
- The user doesn't have the freedom to navigate back to the previous screen. Solution: Added back button for player to get back to the previous screen.
- The button in "the waiting room screen" showing start violates consistency. Because, the buttons on the left side of the start button is used for chatting. Users may be confused by different sort of button aligning in a straight line. Solution: We have changed the layout and alignment for the buttons.
- In the main screen, the link "create new account" is not click-able/linked. Solution: Implemented the feature, made it click-able.
Method
Participant
Karena (24 years old), Berkeley Alumni, is a female international student that come from HongKong. The reason that I selected her as the participant is because that she had a lot of experiences for taking different language classes as well as learning foreign languages herself.
Apparatus
The study was done at Karena’s apartment and having the participant sitting on a comfortable sofa. I have used my laptop computer for the purpose of demonstration of the prototype in Flash. I used a paper pad for notes and my digital camera for recording the experiment. I also use my cellphone as the timer to keep track of the duration of the user performing different tasks.
Tasks
Rate the drawings (easy task): The user is simply presented with a screen showing all of the drawings that have been submitted from the last round with a set of unfilled stars used to perform the rating under each drawing. Therefore, the user must simply move the input device over the stars and once the amount of the stars is “filled”, the user then clicks to lock in the result. The user repeats this process over the number of drawings then selects the submit button for the results to be tallied.
Create a new game (moderate task): Game creation was chosen as our moderate task because it requires the user to interact with elements not used in other parts of the game; namely, the information form that is presented to users in order to choose the basic parameters of the game such as language and difficulty level. This form is reached by selecting the “create a new game” button form the main screen. Once here, the user chooses the game’s parameters, which are mainly guided through a set of drop down menus for easy setup. Once the user is satisfied with their choices they can click the “create” button and are then taken to the waiting room where they await other player to join their newly created game.
Draw a picture (difficult task): The screen we used is very simple and familiar to that of most drawing programs such as Photoshop but not nearly as robust. The task simply requires the user to select the mostly familiar tools in the left hand toolbar such as the shape or pencil tool and use them to draw a picture. The task of selecting the proper tools to create the desired effects was the crucial step for this task. Once the image is drawn, the user can either wait until the timer runs out or
Procedure
- Greeting the participant, and thanking him in advance for taking the time to participate this pilot usability study.
- Go over the idea of the pilot usability study and made sure that the participant understands the consent form and he is voluntarily doing the study and she could quit at any time. Then I asked the participant to sign the consent form.
- Start with giving the introduction of the study with overall idea of our serious game “Word Soup” to the participant and the major functionalities it offers.
- Setup our Improved Interactive prototype using Flash Player
- Gave a quick demo of the serious game with explanation to all the actions.
- Ask the participant if he has any questions regarding the interface then handed participant a list of tasks to perform.
- Record the test measures while the participant performed first task, and included any error, comment, or critical incidents on the notepad.
- Repeat step 6 and 7 for all the tasks.
- After participant finished the last task, I asked him for comments and suggestions regarding our serious game.
- Thank the participant again for taking part in our pilot usability study.
Test Measures
Track the time for participant to perform each task - This measurement can tell us how fast each task can be done and indirectly reflected the usability of the user interface since we don't want the user to get stuck in performing the task, and want to shorten the time for performing each task.
Track the errors that appear during each task - This measurement indicated the flaws of our interface design and those errors should be a high priority issue to be fixed in later builds.
Track the confusions that happen to the user during each task - These confusions indicated that the user interface does not make sense to common average users and we should learn from the user's logic to improve the interface design. Plus this measurement indicator of how easy the user interface was designed.
Track the participant's reaction during each task - By keep tracking this, we could understand how well our interface design appeals to users and find possible flaws according to how the participant react.
Track the conversations with the participant about the program - we can gather the participant's suggestions, comments, and other feedbacks by tracking this.
Results
Rate the drawing (easy):
Task completion time: 47 sec
Number of errors: 0
Number of confusions: 0
Participant's mood: normal
Comment: After the user was given the task and the screen of the rating system, she easily figured out how to rate the drawings. She told me that our rating system is really similar to the rating system from other popular website.
Create a new game (moderate):
Task completion time: 2 min
Number of errors: 1
Number of confusions: 0
Participant's mood: normal
Comment: The user started from the home screen of our serious game, after clicking the “Create a new game” button, she was able to fill out the game creation form without any help from me. One error she found is that when she try to create a private game, and she accidentally forgot to put in the password, but the system didn't catch that, instead it let her create a private game.
Draw a picture (hard):
Task completion time: 1 min
Number of errors: 0
Number of confusions: 1
Participant's mood: normal
Comment: After presented the task and the drawing screen, the user were able to draw with the tool we provided on the drawing screen without any help from me. One of the confusions is that she wasn't sure how to close the help pop-up after she finishing reading the help. It took her a while to figure out the way to close the help pop-up is to click on the help button again. Then she suggest that maybe we should have a exit on the help pop-up at the top right corner.
Discussion
From this pilot run, I learned that it is really important to make sure that the prototype being tested is absolutely free of bugs. Because some bugs in the prototype will cause the program to crash, and wasted the participant's time. The interruption cause by the bugs will frustrated the participant and make them lose their patient. Also, from the results of pilot usability study, I learned that users could easily and quickly figure out how to use the user interface by recognizing the design pattern which is similar to products they have used. While Karena was performing the easy task of rate the drawings, she found out how to do so from his past experience with some existing rating system from some popular website, such as Amazon, Youtube, etc. Besides that, I learned that we provided users with little feedback of user actions, during the easy task, the user confused when he was not able to decrease the number of the players to 2 because 4 is the minimum number of players required. It will be very helpful if we could provide some visual feedback of the user action that informs the user why his or her action didn’t change anything of our serious game. From this pilot usability study alone, I would like to change the game creation form to have extra text that indicates that the maximum number of players is 10, the minimum number of players is 4 and an even number of players is required. I would also like to add a form of feedback if the user action is not logically to our user interface. Besides, I would like to standardize the help menu that would contain close button at the top right corner so the users will no longer be confused by our current method of closing the help menu by clicking the help button again.
Appendices
Consent Form - I have the signed copy in my possession
