Project Title: ETA – Electronic Travel Assistance for Group 4.
Online Doc.: http://inst.eecs.berkeley.edu/~swlee/assignment4.html
Electronic Travel Assistance (ETA) is a program designed to help travelers quickly access powerful features such as accommodations, dining, entertainment, maps, and other information, while at the same time, help to create a custom itinerary for any desired location. The goal of the project is to improve the overall experience of budget travelers by providing up-to-date information not found in tour books and the ability to manage those resources at hand. The program runs on Java Chai on Microsoft’s PocketPC handhelds.
We are now at the stage of evaluating the high-fidelity prototype that we have produced during our last assignment. The purpose of this study is twofold: first, to perform a usability test on the modifications to our interface since our low-fidelity experiment, and second, to use this gather information to enhance our current working model of the program.
Our Participants were the following:
First participant was 20 yr., Mechanical Engineering major, 3rd year at Berkeley.
Experience: No prior experience.
Second participant was 25 yr., IEOR major, graduate student.
Experience: Some experience with Sony Clie.
Third participant was 24 yr., Business major, 4th yr.
From: United States
Experience: Has used a Jornada before.
The equipment that we used in the study consisted of: a HP Jornada, JVC Digital Camcorder, paper, and pencil. We performed the study in a quite room in the International House, which offered us a relatively distraction-free environment. Jun greeted the subjects and introduced them to the study by giving them the consent and information forms. Bikas introduced them to the system by showing them a short demo of ETA. Fahd made observations, and Sean filmed the entire study on video tape.
Our Tasks were the following:
1. The first task assigned to the participants is to set up their profile. The goal of this task is to identify not only how easy the profile section is to navigate, but also to evaluate if the questions asked reflect characteristics of the user that will be relevant and helpful when making recommendations. We insisted on avoiding asking questions that sound restrictive, and tried rather to get input that helps customize the recommendations without limiting the options of the user.
2. The second task is to find an accommodation, the user navigates through the accommodation section and uses the suggestions to locate a convenient accommodation. This section will be used frequently, so we were concerned about the ease of use and the completeness of the format for displaying information about the place. It includes price, address, and a review as well as options such as mapping the accommodation. We also were looking at how flexible and convenient the navigation between the suggestions, the details of a specific accommodation such as it map, and then back to the listing of accommodations.
3. The third task consisted of mapping a location. We think it is one of the most important features that will be used on a daily basis. We looked at how easy it was for users to input addresses using the keyboard interface of the Jornada. We also tried to replicate the standard way of displaying maps with zooming buttons, to be consistent with the standards the users might be used to.
Our Procedures were the following:
We followed a standard procedure with all participants. First, one member of the group was assigned the task of greeting the user and helping him/her go through the release forms. A second member of the group explained the general dynamics of the user interface and the purpose of our tool as a travel assistant. The introduction was meant to give the participant a general feel for the interface without revealing the way tasks are accomplished. The user was also instructed on the procedure used during testing, especially the absence of assistance with tasks and the importance of speaking loudly as the task execution progresses.
A third member of the group provided the tasks for the participant; he only interfered when a task was finished and to provide the next one. Throughout the testing process, a fourth member videotaped the complete interaction of the user with our interface; this allowed us to review the performance of participants with greater details and without loss of information after all the participants were done. The taping of the three tests allowed us to save valuable input from the users in the form of critical input. Once participants finished the third task, they were thanked for their help. We also insisted on getting more input from the participants by asking about their general impressions about the feel and look of the interface. We also asked for suggestions to improve on the interface and its features.
We measured the following:
· Notes on bottom-line data:
o Testers took most of their time filling in the textboxes using Jornada’s on-screen keyboard. Also, it took them longer to complete a task if they were more verbose about what they were thinking. So, the time to complete a task might not correctly interpret the how much thinking they needed to put in to perform the task, and how difficult it is to use the particular feature.
o Tester 1 had never used Palm or Windows CE devices. Tester2 had some experience with Palm devices (Sony Clie) but not with the Windows CE devices, and tester 3 used to own HP Jornada.
o # of Errors included any minor button click that was not necessary as well as failing to recognize/read/fill-in parts that was pre-filled due to the reason below.
o During the pilot tests, we used the same Jornada and when users clicked on some pages, the screens were scrolled down already, thus confusing users to recognize the information on the upper part. In real life situation, it is not very common to have two different users using one Jornada.
o Severity rating and implementation difficulty on the critical incidents range from 0 to 4.
Exiting the Application:
Integration with Jornada’s Keyboard:
Better On-Screen Instruction:
The results of the hi-fi pilot test revealed a remarkable improvement over our initial lo-fi prototype. Perhaps the most important was a considerable improvement navigating through the interface. In our lo-fi prototype, navigation was done through a pull down menu, with many of the sections hidden in submenus. This created a startling problem for users during our lo-fi testing. The menus were redone in the hi-fi prototype to make it more clear how to reach a section, and a new main page was added to the interface with icons that provide clear links to each of the sections of the interface. The result, as seen in the pilot test, was that all of the participants were able to clearly find the appropriate section of the interface to complete the necessary task. Furthermore, a ‘home’ icon was placed on each page of the interface, thus allowing users to quickly return to the main screen. The results indicating the success of this design were evident in that none of the users need assistance beginning a task, and that all of the tasks were completed successfully on the first attempt, with one exception (discussed later).
Another positive result from the pilot test was the combining of the mapping feature of the interface with the mapping features of the listing sections. Users can pull up a map of a particular listing (hotel, point of interest, restaurant …), as well as a get a map given an address. The success of having a single map section (with links from two places) was evident in the results with participant 1 completing task 3 by using what he remembered from task two, that he was able to link to the map section from the accommodations section.
Our most significant design flaws were found in the profile section, with the most severe being that participant 1 exited from the application by mistake. The most likely reason explaining the results for the first task are that the users were not able to clearly see what they were doing. This was due to scroll bars and the Jornada keyboard hiding areas of the screen the user needed to see when typing. The most simple fix, first of all, is to move text entry fields higher up on the screen so that when the Jornada keyboard is activated, the text fields are not covered by the keyboard. Second of all, eliminating scrolling where ever it is at all possible will eliminate the problems the users had because they could not see all the on screen information, such as page titles or ‘next’ and ‘previous’ buttons. Participant 1 exited the program because he was not sure if he was in the right section, even though he clicked the correct icon from the main page. This was because the page was partly scrolled down covering the title of the page. Other participants did not fill in all of the information asked of them because they did not see to scroll down.
Finally, our pilot testing continued to reveal minor problems as a result of unclear, or incomplete directions or information. For example, none of the participants knew that they could choose multiple selections in the profile task for check boxes. Refining the on screen instructions and text was something that we spent a lot of time on from our lo-fi prototype to our hi-fi prototype. However, continued refinements are necessary to provide dialog that is both clear and concise.
To conclude, the pilot testing revealed a dramatic improvement between our lo-fi and our hi-fi testing. Many of the major interface decisions proved to provide a much more usable interface. However, the numerous minor flaws still yield an interface needs to be refined for our next iteration of design and testing.