Prepared for the:
Public Affairs Branch
Canada Revenue Agency
FINAL REPORT
June 2007
POR# 046-07
Contract #4655888873
Prepared by:
Phoenix Strategic Perspectives
Le rapport complet en français sera fourni sur demande.
To request a full copy of this report, please contact Library and Archives Canada at:
613-996-5115 or 1-866-578-7777 or www.collectionscanada.ca
Media Enquiries:
Media Relations
Canada Revenue Agency
4th Floor 555 MacKenzie Avenue
Ottawa ON K1A 0L5
media.relations@cra-arc.gc.ca
Executive Summary
Background
The Canada Revenue Agency (CRA) commissioned Phoenix SPI to conduct qualitative research in the form of one-on-one interviews to assess two proposed new designs for the CRA Web site home page. A new CRA corporate look chosen earlier in 2007 is ready to be applied to the Web site. Concurrently, the Agency wanted to begin to apply the first of several changes mandated by the Treasury Board's Common Look and Feel (CLF) 2.0, namely the removal of the top institutional (red) navigation bar and re-location of its contents on the home page. As the home page is updated with the new corporate look and the removal of the red institutional navigation bar, it was also seen to be an opportune time to make some improvements to the labelling and organization of information, to make the site easier to use and navigate.
The main purpose of this research was to obtain feedback from CRA Web site users about the mock-ups for the CRA home page. More specifically, the research objectives included evaluating participant impressions of the home page mock-ups, obtaining suggestions for improvement with respect to all aspects of the design of the home page, and obtaining feedback concerning the new CRA corporate look.
Target Audiences
The target audiences for this research included individuals and business. For the purposes of this research, ‘individuals' included wage-earners, seniors, and students (aged 18-24). The business audience included self-employed individuals, representatives of small and medium-sized businesses (less than 100 employees), and third-party service providers (e.g. businesses involved in accounting, tax preparation, and payroll services).
Methodology
A total of 20 interviews were conducted, with 10 in each of Toronto (English) and Montreal (French). The interviews were conducted May 22-24, 2007.
This research was qualitative in nature, not quantitative. As such, the results provide an indication of participants' views about the issues explored, but cannot be generalized to the full population of any of the audiences included in this research.
Note on Design Concepts:
For research and reporting purposes only, the design concepts tested in this research were given names or working titles. The names given to the design concepts were not revealed to/used with participants. Instead, they were referred to simply as ‘Concept A' and ‘Concept B' and their order of presentation varied in each interview. The names were chosen to reflect the main design feature of each concept. The two design concepts were designated as follows:
- Columns: Designated as such because of the focus on two columns featured in the middle of the page (i.e. ‘Individuals' and ‘Business') with a list of popular links under each.
- Portal: Designated as such because of the portal style designed to lead the user into the site to choose from a more complete menu of information at the next level. The focus in middle of the page is on key on-line services and ‘Information about' popular topics.
Key Findings
The Columns version was the clear favourite between the two home page designs presented to participants and was preferred by a considerable margin over the Portal version. In explaining their preference for this version, participants routinely pointed to their belief that it would make it easier for them to quickly locate information they typically need when visiting the CRA site. The prominence of the individuals and/or businesses categories (with the associated columns often identified as containing key links), and access to comprehensive information directly from the home page were seen as facilitating this. Some said they preferred the Columns version because of its layout/display in general and the colours used.
While most preferred the Columns version, participants did perceive weaknesses. Perceived weaknesses identified most often included too much information on the home page, a lack of structure/organization in the left-hand menu of the home page, and the lack of indication that the ‘Individuals' and “Business' headings are also active links that lead to additional topics of relevance (i.e. over and above the items listed in the respective columns).
Despite the preference for the Columns version, nearly everyone felt that they would be able to navigate the Portal version and find information they needed, though some added that it might take them some time to familiarize themselves with the home page.
There was widespread agreement that the relocation of links from the top institutional (red) navigation bar was well done and virtual unanimity that visitors will be able to easily find these links in their new locations. That said, some suggested that the links be highlighted or bolded to make them even more visible.
Suggestions for Improving Columns Version
Participants offered a variety of suggestions for improving the Columns version. Two of them were routinely offered for improving the home page mock-up. They included reorganising the left-hand menu of the home page and matching it to the look of right-hand menu. Specific suggestions for reorganising the left-hand menu included adding headings and/or distinguishing more clearly between groupings of links, including drop-down menus, and placing links in alphabetical order in the French version. The suggestion to match the left-hand menu to the look of right-hand menu was based on the widespread impression that the menu on the right-hand side of both versions (but especially the Columns version) is better than the menu on the left-hand side. Some frequently-made suggestions for improvement involved incorporating elements or features from the Portal version of the home page, including adding the ‘What's new' link to the top items on left-hand side, including red bullets in the ‘In focus' section, and adding an ‘Information for' heading to the left-hand menu of the home page. Moreover, many participants liked the prominence accorded to the online services link in the Portal version and felt this feature should be incorporated, if possible, into the Columns version. Other frequently-made suggestions included adding all possible links under the ‘Individuals' and ‘Business' headings or making it evident that this list is not comprehensive by adding a new link – ‘More…' – at the bottom of each list, and making it evident that the headings ‘Individuals' and ‘Business' are also links.
Participant Exercises
As an exercise to test the user-friendliness of the home page mock-ups, participants were asked to try to find various types of information on each version. The mock-ups had limited functionality, allowing participants to review the home page as well as a few second-level pages. The exercises took this limited functionality into account by focussing primarily on the starting point of people's searches for the needed information. On the whole, most participants had little difficulty locating the information they were asked to find, though some tasks were more complicated than others. All participants were able to find at least some of the information they were asked to find and many were able to find everything. Moreover, in some cases where participants did have difficulty locating the information, it was because they simply had not noticed it when they arrived at the right location. Business representatives, especially third-party service providers, were most likely to be able to locate information, while seniors had the most difficulty.
Participants tended to have more success finding information on the Columns version than on the Portal version. Having said that, the success rate was not substantially higher and participants who found information on one version of the home page were usually able to find it on the other. The participant exercise revealed that most participants regarded the labels ‘Individuals' and ‘Business' on top of their respective columns in the Columns version as headings and not live links on which they could click. This was the factor most likely to affect the ability to find information on the Columns version of the home page.
Conclusions and Implications
The main purpose of this research was to obtain feedback from CRA Web site users about proposed new designs for the CRA Web site home page. The research findings provide the CRA with definite direction regarding the most effective home page design (and how it could be improved). The Columns design emerged as the clear winner and was the preferred option among all audiences, with the exception of students. Moreover, the preference for this version was a categorical, positive choice, not a reluctant one.
The main strength of the Columns design was the emphasis given to topics/links for individuals and businesses, which allows relatively quick and easy location of links commonly used when visiting the CRA site. Note that the most frequently-identified weakness of the Portal version was the perceived lack of prominence/visibility of these two categories (and associated topics). Having said that, some participants said that the ‘Individuals' and ‘Business' links in the Portal version have one advantage over the ones in the Columns version – that it is evident they are links. Indeed, one of the main perceived weaknesses of the Columns version is that there is no indication that the headings ‘Individuals' and ‘Business' are also links. This emerged through the participant exercises.
While there was most likely an element of familiarity at work in participants' preference for the Columns version (the design more closely resembles the current CRA home page), the choice also reveals a preference in terms of how participants want to be able to search for information on the CRA site. The advantage of presenting participants with two very different home page designs was that it offered them distinct alternatives in terms of navigation. In choosing the Columns version, participants revealed that they prefer to have access to more comprehensive information at the surface level (i.e. on the home page), even at the price of a more cluttered or loaded home page.
Participants offered numerous suggestions for improving the Columns version, all of which were concrete and precise, and most of which involve relatively small changes or adjustments. Particular attention should be paid to suggestions for making it evident that the headings ‘Individuals' and ‘Business' are also links, for increasing the prominence of online services, and for better organizing the left-hand menu.
