Evaluation of the Canadian Archival Information Network (CAIN)

Library and Archives Canada

Final Report

Prepared by the Audit and Evaluation Division
Library and Archives Canada
March 2004

Table of Contents

• 1.0 Introduction

• 2.0 Evaluation Findings

• 3.0 Conclusions

• 4.0 Management Response

1.0 Introduction

1.1 Background

As stated in early documents ¹ the stated vision was "to enhance the knowledge and appreciation of our past by providing access to the nation’s archival record via the information highway." CAIN was described as "a distributed, searchable network of networks which will link Canadians on the internet with every Canadian archives." In its key characteristics it would be flexible, bilingual, user-friendly and inclusive. Its main strategic framework elements were:

1. Accessing Canadian archival content
2. Developing standards
3. Training the builders and users
4. Funding the Canadian Archival Information Network
5. Communication, consultation, coordination and cooperation

With regard to implementation it was anticipated that it would take place in three phases considering the uneven development of information networks and institutional databases among the various archives themselves (including provincial and territorial archives) who would be the key partners.² These were:

1. primary descriptions of all Canadian archival collections  -  by the end of the year 2000;
2. detailed finding aids and descriptions of each collection; and,
3. finally, availability of significant items including photographs, documentary art and maps.³

The target audience (i.e. communities of interests or end users) identified in the planning stages included the following:⁴

• Educators, researchers and students
• Archivists
• Canadian public
• Cultural industries
• Genealogists
• Government employees
• Communities, and
• Corporate entities

In the early period efforts focused mainly on promoting the concept of CAIN and in securing a source of funding. Early estimates of the cost of CAIN were in the neighbourhood of $16 Million over a four year period and it was decided to approach the federal government for half of the funding required.⁵ After preliminary consideration of various options, the department of Canadian Heritage seemed to be the logical choice and presentations began as early as December 1998.⁶

Work was also done in developing terms and conditions for the program. These were approved by CCA in March 1999 and identified five funding streams:⁷

1. Preparation of Rules for Archival Description (RAD) compliant fonds and lower level descriptions;
2. Support for CAIN technical infrastructure;
3. Select scanning and digitizing;
4. CAIN related training and education; and,
5. Program management.

Thereafter efforts were made to develop the technical infrastructure to support CAIN. Progress in this area took a major step forward in January 2000 when the National Archives offered the CCA needed infrastructure in the areas of a web server, search engine and internet connection.

Although efforts had been made by the CCA to secure private sector sponsorship for the initiative, these were not successful. During the development period a significant effort was made by archives across Canada to make CAIN a reality. As well, the CCA spent over $200,000 of its own funds to get CAIN up and running.⁸

In February 2000 the federal budget allocated $20 Million for 2000-01 and $30 Million in the following year for the purpose of increasing Canadian cultural content on-line.⁹ Funding for the CCA to work on CAIN was finally secured from Canadian Heritage in October 2000 in the amount of $700,000 for the remainder of the 2000-01 fiscal year (a total of 20 projects were mounted). In the interim work continued on a web site prototype and the site itself was launched in October 2001. Funding for 2001-02 of $2.3 Million supported a total of 134 projects although there were delays in getting this funding approved and adjustments were required for some of the projects. Funding of $1.7 Million for 2002-03 was less than anticipated and required more adjustments including a contribution from CCA from its own regular funding to cushion the reduction.

Over the course of the three years in which funding was received from Canadian Heritage, the guidelines under which the funding was provided have been increasingly focused on making digital content available on-line. Canadian Heritage has become increasingly unwilling to fund projects involving other activities (infrastructure, descriptions, training, etc.). While the vision for CAIN included digital content, it was seen as a later stage in the work and not the major emphasis particularly in the first few years. Although there have been adjustments in the mix of projects proposed to meet the requirement for increasing digital content, the results have been that some projects have had to be adjusted, others have had to be dropped entirely, and funding levels have been lower than expected since CAIN is solely dependent on CCOP funding.

1.2 Objective of the Evaluation

From the outset funding for CAIN as provided by Canadian Heritage through Canadian Culture Online (CCOP) and its predecessor has not been a good fit due to different objectives. An evaluation of CAIN at this time would demonstrate whether or not the current approach is working and what lessons can be learned to date. On October 31, 2003 the Audit and Evaluation Committee of the Library and Archives Canada (LAC) approved the Terms and Conditions for the evaluation as presented in Appendix 1.

1.3 Methodology

The evaluation examined the program as it existed on September 30, 2003 and was directed by the Audit and Evaluation division of LAC with the assistance of Consulting and Audit Canada. An Advisory Committee assisted in the development of the methodology that was used in the evaluation, assisted the evaluation team with documentation, facilitated contact with program participants, and reviewed the draft report (see Appendix 2 for the Terms of Reference for the Advisory Committee).

The evaluation itself used a number of methods including a review of documentation at LAC and at the Canadian Council of Archives. Interviews were conducted with staff at the Library and Archives, Canadian Council of Archives and National/Provincial/Territorial Archives Councils. Phone interviews were also conducted with a representative sample of program recipients across the program components and categories of recipients. An online client survey was also conducted using several listserves for which it was believed that the listserve members were possible users of CAIN (see Appendices 3A through 3D for more details).

Five case studies balanced across program components and categories of recipients were also completed. These case studies involved a file review as well as interviews with program recipients.

2.0 Evaluation Findings¹⁰

2.1 Achievement of Program objectives

2.1.1 Original targets and objectives of CAIN

The objectives of CAIN were not well aligned with the funding program CCOP.

The evaluation found that the original vision of CAIN included both online descriptions and digitized content as part of a sequence. The national catalogue of online descriptions was to be the first priority and once significant progress had been made and the technical infrastructure/network had been built, selected content was to be digitized and made available online.

All informants agreed that the primary objective of CAIN was to create a national online catalogue of descriptions of the holdings of Canadian archives. In terms of actual targets it appears that the goal was to have all fonds level descriptions for archives in Canada (up to 800 institutions) online and RAD compliant over a five year period.

While efforts were made to secure funding for this vision of CAIN, the only funding source that materialized was the Canadian Culture Online Program  -  CCOP. Over time CCOP placed greater emphasis on digitization and virtual exhibits which caused the CAIN projects to do likewise. The archival community has perceived this as a change in the direction of CAIN requiring it to move away from its original vision.

2.1.2 Extent to which CAIN has achieved its objectives

Some progress has been made in achieving the goal of a national online catalogue.

In terms of the objective of a national catalogue of descriptions it appears that CAIN is about 20% complete after 3 years of funding. The main reason it appears is that to date only about 250 out of the 800 archives have participated (those who have participated are about 50% complete but only 37% of this group felt that all descriptions would be completed in the next two years).

Although the funding devoted to the preparation of descriptions declined over the three year period, it still represented 44% of total expenditures. An additional 24% was spent on technical infrastructure and 12% on training and management. Only 20% was spent on digitization and virtual exhibits (see Appendix 5 for details). In total, CAIN received and used about 45% of the funding it had originally requested ($8.0 Million) towards the development of the national online catalogue and its supporting infrastructure.

Informants believed that significant progress was made in achieving a network and modest progress was made with regard to raising the profile of archives through an improved presence on the web. Those who participated in the program felt that the funding achieved its purpose (60-90% agreement) and that CCOP funding for CAIN was important in achieving program objectives (70-100%).

The use of the Archives Canada/CAIN website is not at a level where it could be said to be engaging Canadians but still appears to be primarily used by the archival community. It is not apparent that CAIN has improved access for Canadians. One of the possible reasons for this result is that CAIN has not been actively promoted to potential users. It should also be noted that traffic on CAIN is not reflective of the entire network of which CAIN is only one part. Data and time limitations did not allow for the gathering of statistics on provincial networks.

2.2 Program results and impacts

2.2.1 Impact of CAIN on the archival community

Positive benefits to the archival community have been apparent.

Informants believed that the impact of CAIN on the archival community had been positive particularly in the areas of standardized descriptions and improved communications among archives. Case study respondents also felt that some improvements in physical management of documents and space had occurred.

With regard to increases in requests from the public, participants responded that the greatest increases have been in online visits and e-mails which is reflective of their greater online presence which in turn was assisted by CAIN.

Although 44% of participants indicated that CAIN had improved their capacity to leverage funds, the funding profile of the institutions in the sample showed that their sources of funding are not any more diverse than before.

2.2.2 Satisfaction of clients with CAIN products

Desires of the public appear to be different from those of the archival community.

Participants, case study respondents and key informants from within the archival community all believed that clients were satisfied with CAIN products but admitted that few mechanisms were in place to assess client feedback in any systematic fashion.

The survey of the Archives Canada/CAIN website conducted as part of the evaluation was very revealing with regard to this finding. Firstly it would appear that the vast majority of those who visit the website are those who are already intensive users of archival resources (Appendix 4B). Those with less previous experience with archival resources were less enthusiastic about the website and its contents and when asked about future priorities were more likely to give a higher priority to digitization and genealogical resource. It would appear that there is a difference between the wants and desires of the archival community (i.e., the traditional users) and some segments of the public they would like to serve (i.e., the non-traditional users). Having a catalogue of online descriptions is a useful research tool and enables archivists to help both traditional and non-traditional users locate what they are looking for. Knowing where something is located (i.e., having a catalogue of online descriptions) might be the goal of archivists and those who are familiar with and sympathetic to the profession but it is only one of a number of priorities for the public.

2.2.3 Satisfaction of participants with program impacts

Recipients are satisfied with program outcomes.

Recipients were generally satisfied with the impacts of the program and found that funding was adequate to achieve its goals for the projects that were undertaken. Some 70% indicated that they would apply for CCOP funding again in the future.

2.3 Lessons Learned

2.3.1 What worked/didn’t work with design and implementation of CAIN

Views of what worked or did not work were seen only from the perspective of the needs of the archival community.

It was generally agreed by informants that the concepts underlying CAIN were sound (national network, standard descriptions, putting descriptions and content online) and that there had been positive impacts for those who had participated.

The issues most often cited with regard to what did not work were the adjudication process and what was perceived as CCOP’s changing objectives, changing program criteria and unstable funding from year to year, and untimely funding within each year. It was also felt that promotion efforts had been unsuccessful and public interest had not yet materialized.

Some within the archival community saw digitization of content and virtual exhibits as not being part of fundamental archival work. This led to the view that the community should have passed on the CCOP funding and waited for another source of funds. Others, however, pointed to what had been accomplished by CAIN and felt that the effort had been worthwhile.

2.3.2 What lessons from CAIN could be applied to similar initiatives

Agreement and understanding on program objectives by the funding authority, the program administrator and participants is crucial.

In terms of the experience of CAIN and CCOP it was apparent that misunderstandings occurred on both sides. Some informants felt that CCA was not effective in explaining to PCH what archives are all about and why a national catalogue of descriptions was an important first step towards an online presence for the archival community. They also felt that Canadian Heritage did not require CCOP criteria to be met from the outset thereby giving the archival community the impression that its vision had been accepted.

In terms of lessons learned from the CAIN/CCOP experience, it was clear that more careful consultations between the CCA and Canadian Heritage about the objectives and expected results of the two initiatives was needed to see if there in fact was a fit before deciding to go ahead. It may, in fact, have been wise to do a pilot or series of pilots to test the concept first. Once the program was up and running there should have been consultations with the archival community as changes were needed.

3.0 Conclusions

1. Data collected during the evaluation suggests that CAIN is not on track to complete its goal of a comprehensive catalogue of archival descriptions within the budget or time frames originally estimated. Taking this experience into account it is concluded that the archival community needs to do a better job of establishing targets, milestones and performance indicators when seeking funding in the future.
2. The results of the evaluation show that CAIN and CCOP were not well aligned but this does not mean that the relationship could not have worked better to achieve the goals of both initiatives. The critical factor may well have been the inability of CCA to position the national online catalogue as a necessary part of the infrastructure needed to support an online presence.
3. The evaluation shows that the non-traditional user is more interested in direct access to source documents and virtual exhibits via the web than in access to descriptions of those documents. The archival community should consult with non-traditional users to determine what would serve their needs. Part of this exercise should involve making use of research into how the public searches for and uses information of this type and how this is changing over time. Efforts should also be made to put in place proper mechanisms to collect ongoing client feedback and use this feedback to continually improve the quality of products and services offered to the public.
4. The archival community as represented by the CCA viewed CCOP funding as an opportunity to develop a comprehensive online catalogue of Canadian archival holdings (the primary objective of CAIN) rather than an opportunity to make archival content widely available to a non-traditional audience (the primary objective of CCOP). Changes in technology have presented to the archival community an unprecedented opportunity to make archival descriptions and documents widely available to the public. Future considerations should take into account the broader agenda of the Government of Canada in providing citizencentred access to Canadians and how archives can support this goal.

4.0 Management Response

1. In Bill C-8 which establishes LAC, section 7(f) states that one of the objects of the institution is "to support the development of the library and archival communities." Section 8 also states that the Librarian and Archivist may do anything that is conducive to the attainment of the objects of the LAC, including (f) "enter into agreements with other libraries, archives or institutions in and outside Canada"; and (i) "provide professional, technical and financial support to those involved in the preservation and promotion of the documentary heritage and in providing access to it."
2. LAC agrees with the conclusions of the report. In the future, initiatives which assist in making archival collections known to a large segment of the Canadian population will be a primary objective of LAC. LAC will concentrate on supporting programs and activities that are aligned with this primary objective.
3. Many of the conclusions in the report are addressed to the Canadian Council of Archives and/or the archival community. We intend to raise these issues with our key partners. As well, LAC, in cooperation with the Canadian Council on Archives (CCA) and the National, Provincial, and Territorial Archivists (NPTAC) has established a committee to look at the future of CAIN, now Archives Canada. CAIN was conceptualized in 1997 and its objectives reflected the priorities of the period. These have since been superseded by technological developments that have improved access tools and have made the digitization of materials possible and relatively affordable. LAC will ensure that the committee receives a copy of this evaluation so that they may take its findings into consideration.
4. In response to the conclusion that CAIN and CCOP were not well aligned, LAC and Canadian Heritage (PCH) have agreed on a new approach to administer the digitization program for the archival community funded by the Canadian Content Online Program (CCOP). The CCA and NPTAC have agreed to work closely with LAC in administering this program. The program focuses on the digitization, showcasing and promotion of nationally significant archival collections. Although CCOP funding now focuses on digitization of content, there are close links with Archives Canada and close cooperation is necessary. A LAC staff member has now assumed the coordination of this project.
5. In response to conclusion 4, which highlights the opportunities provided by technology, LAC also intends to share its recent technological developments, for instance the creation of an XML repository, with the archival community and will work closely with it to continue the development of a Canadian archival system.
6. LAC and the archival community need to encourage greater discussion with users. LAC is currently in the process of developing tools and mechanisms to gather client feedback and incorporate this feedback into developing better products and services. We are committed to sharing these tools with the CCA and the archival community.

Footnotes

1. Raising CAIN: Building Canada’s Archival Information Network, November, 1997.

2. Raising CAIN: Exploring the Collective Memory of our Nation, October, 1998.

3. The timelines shown were based on work beginning immediately (1998). No further description of the work to be undertaken was provided at that time, however, the CCA was in the process of establishing the infrastructure needed to make CAIN a reality such as a CAIN Steering Committee and a Technical Sub-committee.

4. Canadian Archival Information Network, Business Case Framework, May 1999.

5. CCA Annual Report 1998-99, p. 19.

6. Ibid., p. 19.

7. Ibid., p. 20.

8. This included $71,111 in 1998-99, $115,521 in 1999-2000 and approximately $45,000 in 2000- 01 as noted in the Annual Reports for the respective years.

9. Budget 2000: Making Canada’s Economy More Innovative, Department of Finance, February 2000, p. 13.

10. A more complete version of the findings is presented in Appendix 4.