PilotStudy- Group:Group Phi-tus-KevinFriedheim
From CS 160 Fall 2008
Kevin Friedheim
Contents |
Introduction
During the contextual inquiry process of our design, we discovered that learning a language tends to be more of a chore than it is an adventurous and fun process. The system that our group has constructed is a game that turns the focus of learning a new language away from its "boring" aspects and into a fun and challenging process. The unique method we used to accomplish this is to use the game designs that are found in the popular games Taboo® and Pictionary®.
Also learned from the Contextual Inquiry phase of our design was that learning a new language is best achieved by practice -- and with language, the best form of practice is verbal communication. This gives the user a chance to forcibly recall foreign words and phrases in order to complete the tasks of either dictating or drawing - which are both very important aspects of our game. Dictating (or in this case, describing a picture to the drawer) allows the player to practice speaking, pronunciation, and word recollection and usage. As a drawer, a player will get the chance to use interpretation skills in order to both decipher and understand words and phrases and demonstrate their understanding by drawing a picture that is being painted by the dictator with whom this particular drawer is paired.
Implementation and Improvements
According to our heuristic evaluating group - we had many areas that were in dire need of improvement. Some of these areas (which I will go into more detail to follow) included: providing user feedback in multiple areas of the game, many aesthetic properties that were not optimal, functionality that seemed important to game play, many user control and freedom errors, and finally some conceptual model conflicts
There were two areas of our design that, in order to correct the user feedback heuristic violation, needed to be implemented: first was in the drawer's screen where the user is confronted with a vast number of options/settings (colors, sizes, tools) and our first implementation offered no feedback as to the current settings. This was easily repaired by adding a highlight feature to indicate the tools and settings that are currently selected. The second area that required attention in regard to user feedback heuristic violation was some form of an indication that, from the drawer’s perspective, input from the dictator is being received – likewise, a feedback indicator for the dictator should be present to indicate actions (drawing) is performed by the drawer. Both these features are in the works of being implemented.
Some of the aesthetic properties that our first implementation lacked were the size, style, and layout of our overall design. We’ve made some big changes to make the design more pleasing to the eye. However, I must say that by not making a flashy (more “hi-fidelity” like) design, the interactive prototype was able to better server its purpose since the observer(s) of the heuristic evaluation were able to concentrate on more complicated problems rather than ones like the color of a flashy design is as it should be.
Some added functionality that pertains to game play involved added features with the painting program. We added the line tool to work properly along with the pen and eraser tools. In later versions, more and more functionality will be added and existing ones, improved upon.
Our initial design did lack in user control (back buttons, cancel buttons, etc…) which we added in appropriate places throughout our design.
When a game was created in our initial design implementation, the user would be added to the existing players in the game, and remaining possible open slots remained opened and were simply labeled as such. The group that evaluated us found this to go against their conceptual model of how this should work. Their suggestion was to dynamically add players to the game and do away with the “open” representative slots or to simply rename them as “still open” or create a more clear representation of their state and purpose. I didn’t agree with their assessment and to test it, I decided to ask my volunteer what she thought. I’ll elaborate on the results of this inquiry later in this deliverable.
Method
Participant
I wanted to find someone who fit the description of having had already learned a second (or third, forth,…) language and has been through the painstaking learning process. This way they can offer more insight about how the game should function and stands to enjoy the game more than someone who’s language skill level is not as advanced (this applies to games like Taboo® and Pictionary® as well - a better drawer/guesser will have a more enjoyable time than someone who isn’t so great at drawing/guessing). Other demographics were unimportant in consideration – which made the selection process easier since I needed only find someone that matched the before-mentioned qualification.
The person I found was Jenny – a 4th year pre-med Biology undergraduate student. Her first language is English, and Russian is a language she learned as a sophomore.
Apparatus
The only equipment that was needed was a prompt to read from, and a laptop on which Jenny could test out our improved design. We decided to meet up at her convenience at the Free Speech Movement café near Moffit Undergraduate Library. This way the test could be kept casual and she would feel like less of a lab rat.
Tasks
Since we decided that our tasks from the previous Low-Fidelity prototype have not changed, I’ll use the summary we made there, here. Also, I should note that no drastic changes have been made and so the descriptions below are as they should (and hopefully with our modifications, should be easier to carry out).
After carefully considering the list of tasks available to the users, we chose the three tasks we felt were the most representative to our game that also covered the major components of the user interface. The three tasks we chose were creating a game as the task, rating the drawings as the task and drawing a picture as the task.
Rate the drawings (easy task):
We felt that rating a drawing was a relatively easy task. The user is simply presented with a screen showing all of the drawings that have been submitted from the last round with a set of an unfilled stars used to perform the rating under each drawing. The rating process is similar to that of rating a product on Amazon.com, Apple.com and various other familiar websites. As such, the user must simply hover the input device over the stars and once the amount of the stars is “filled”, the user then clicks to lock in the result. The user repeats this process over the number of drawings then selects the submit button for the results to be tallied.
Create a new game (moderate task):
Game creation was chosen as our moderate task because it requires the user to interact with elements not used in other parts of the game; namely, the information form that is presented to users in order to choose the basic parameters of the game such as language and difficulty level. This form is reached by selecting the “create a new game” button form the main screen. Once here, the user chooses the game’s parameters, which are mainly guided through a set of drop down menus for easy setup. Once the user is satisfied with their choices they can click the “create” button and are then taken to the waiting room where they await other player to join their newly created game.
Draw a picture (difficult task):
Drawing a picture is one of the fundamental tasks of the game so we felt it very desirable to make sure that we got this right. The screen we used is very simple and familiar to that of most drawing programs such as Photoshop but not nearly as robust. The task simply requires the user to select the, mostly familiar, tools in the left hand toolbar such as the shape or pencil tool and use them to draw a picture. The task of selecting the proper tools to create the desired effects was the crucial step for this task. Once the image is drawn, the user can either wait until the timer runs out or submit the picture early for possible bonus points.
When each of the three tasks were carried out, I looked for the following:
Rate the drawings (easy task): Although we did not notice any difficulty in our low-fidelity prototype user tests for this particular task, I felt it necessary to watch for how the user interacted with the stars and if it was easily understood as to what each star selection represented. As predicted, Jenny had no trouble at all selecting the stars that corresponding to the predefined pictures that we have in our implementation.
Create a new game (moderate task): Again, I watched how Jenny dealt with the login screen (I told her to assume that she has a username and password already since the register feature is not implemented). I also was sure to take notice as to how long it took her to get from point a (the login screen) to her destination (the game creation screen). One thing that we upgraded this time around was the button placement on the screen – I wanted to make sure that they are in their ideal positions and that they are easily recognizable as to their function.
Draw a picture (difficult task): Here is the task that seemed to create the most confusion in previous user tests and so I was very careful to watch for a few things: one which was the reaction time of the user in her ability to recognize what to do when confronted with the screen; especially after having seen the countdown timer at the top of the screen. I certainly don’t’ want the user to freak out thinking that time is running out and I don’t know what to do. In order to facilitate (wizard of oz) the dictator, I acted as the dictator.
Another point of concern was the buttons that related to the tools for the painting and whether or not Jenny would have any trouble with them.
Procedure
After determining what kind of user I wanted to find, I thought of who I could use and decided on the best candidate – Jenny came to mind first. Since I’m good friends with her, I did not have much trouble getting her to agree to the test, so at that point we decided on a place and time (I wasn’t ready then and there since I had a little preparation to do beforehand. Also, I had planned on voice recording her, and I was a little surprised that she was not willing to allow me. But I did respect her request and told her that I would not be using any recording equipment to respect her wishes.
Next step was do my preparation which consisted of drafting a prompt that would help me run the user test smoothly and without interruption. When I was not as prepared during the last user test that I orchestrated, after asking a question and following up on it, I was lost as to where I needed to take the user test next. I was sure not to make the same mistake twice. See the appendices for my prompt.
Now I was ready for the actual user test, so we met at Free Speech Movement and found ourselves ready to begin. I began by asking her to review the consent form that I provided for her, and made sure to ensure her that she would not be recorded. I went over the purpose of the user test and the importance of her involvement. I asked if there were any questions that she had up front. And finally I assured her that she could leave or stop the test at any point for any reason (she found this funny).
I went over the three tasks very briefly and gave her a quick demonstration of a simple task that was not one of the three that she would be performing. This sample task was to join a pre-existing user game. This was very simple to do and did not give enough help to her so that she would not have to think about how to carry out her tasks.
I then explained the first task as follows: “You will be undertaking the role of the dictator – and the particular task will be to rate the other players’ (drawers’) drawings. You’ll be given an answer key picture from which to compare the other drawings with and to assist you in your judgment. “ I set up the computer so that the screen was that particular screen so that she did not have to navigate through to that particular screen.
After the first task I asked if there were any points that she felt gave her trouble.
Then I explained the second task: “This second task will require you to create a game by any means you are able. The game can be public or private, its up to you. Once the game has been created, you are done with the task.”
Again I asked for any difficulties she may have experienced.
The third task: “For the final task, which is the most fun of the three tasks, will be to draw a picture given a verbal description. I will be acting as the ‘dictator’ for this part of the user test. So when you think you’ve drawn what I’ve described, submit your picture”
After the three tasks were completed, I wrapped up by asking if she had any further questions, or any comments. I asked for any improvements she might suggest (anything from color of the user interface to the functionality of the game). I told her again that her data will be kept confidential, thanked her for her time, and dismissed her.
Test Measures
For this Pilot Usability Study, I decided to measure the time taken to complete the three tasks. In order to accomplish this, all I needed was a stop watch, and to *not* let Jenny become aware that I was timing her. Of course I didn’t want her to have to go through all the tasks a second time just so that I could time her, so I decided that I would mark the time prior to beginning a task, and stop the time upon completion. Since there isn’t much involved in documenting the time, doing so did not hinder me from properly analyzing her performance.
As for why I chose to test the time for completion – I think that its important to quickly get through the administrative steps so that a player can quickly get to what really counts, the fun stuff – actually playing the game. Also, when playing the game, the player should not spend a lot of time wondering what to do or how to do it. S/he should be able to go right to the tools needed to accomplish the task at hand.
One thing to note is that the third task is actually drawing a picture given input and for this task there is a timer associated with it. The timer length is 00:01:00 in length. However, since the instructions that I gave were to press submit when complete, this disregards the timer and so I will subtract the total time minus one minute to assess how much more time she needed as compared to how much time a player in a real game would have.
Results
Time for task 1: 15 seconds It appears that Jenny had no trouble with this task and she completed it in a timely manner.
Time for task 2: 22 seconds Again, she didn’t appear to have any difficulty with this task whatsoever. In fact, I timed myself on this one and she ended up performing the task more quickly than myself.
Time for task 3: 1:31 seconds – 1:00 = 31 seconds OVER I was a little surprised by this outcome actually. I didn’t expect that she would even go over the time limit as it was clearly posted above in red. I wondered what she was thinking when she saw the timer hit 0. A quick note about the timer: currently it is not implemented as it should be (i.e. when it hits zero, the screen automatically turns to the next screen and disallows continued drawing. I used this fact to my advantage to see how long an “easy” drawing given a description, would take. Apparently 1:00 is not long enough. Perhaps we might end up changing this to 2:00. That might be more reasonable. I did notice that Jenny was having some difficulty with the size of the buttons on the drawing screen (for example, the size of the color options could be a little larger as there is a lot of page “real estate” that I can use to expand the size of the color pallet).
Discussion
I learned a lot from this piolet run and I’m certainly glad that I did it prior to running a “real” experiment because there are quite a few changes that still need to be made. Those changes really came apparent as a result of this test.
Certainly I would want to find a more logical placement for the next, back, and cancel buttons. I think that they need to be a little larger, and more obvious on the screen so that the eyes go right to them instead of searching the entire screen for them. Also they probably could use more consistancy from screen to screen.
Another change that will be made is the overall functionality. As we have it currently, many of the links are broken and many of the tools don’t work even though you can select them. We will attempt to get the game working but the backend stuff still has quite a bit of work before we make large advancments.
A concern that Jenny had was that, while waiting in the created game screen, why there were other slots open but not filled. She said that she would not put them there at all and dynamically update the players as they enter or leave the created or joined game (she didn’t quite say it like that, but that was her concern and advice). I agree with her one hundred percent and I do plan on making this change if I’m able to.
From the last review of our design, it was said that the dictator’s “fun factor’ was a bit low and almost non-exisitant. One resolution that our team had for this was to implement a “hint” system for the dictator that he can use to spend points for hints on what to say (both english and forign version). This way, in case the dictator takes one look at the picture that he is supposed to discribe, he doesn’t just say nothing. We were playing with other ideas as well – another was to allow “special gameplay buttons” that can be used only once per game. One can be “say it in english” button. When used, the button remains inactive for the remainder of the game (note: there are several rounds in a game for which the button would remian inactive). These proposed inhancments not only solve the problem of making the game more fun for the dictator, but add to the “serious” aspect of the game (which is to learn a new language).
Also I noted that Jenny did not like the overall look and feel of the game. “It doesn’t seem playful enough.” This is something we will certainly look into before the real “tests.”
Appendices
Pilot Usability Demo Script: greet sign consent form explain purpose of test explain her role explain tasks explain task 1 perform explain task 2 perform explain task 3 perform ask any follow up questions explain purpose and use of data collected thank and dismiss
Below is the link to our groups Lo-Fi appendices and at the bottom is the consent form that I used.
