Foreign Affairs, Trade and Development Canada

international.gc.ca

Common menu bar

• Français
• Home
• Contact Us
• Help
• Search
• canada.gc.ca

Breadcrumb

1. Home
2. >
3. About the Department
4. >
5. Accountability

Summative Evaluation of the International Science and Technology Partnerships Program (ISTPP)

(May 2010)

(PDF Version, 1.53 MB) ^*

• Abbreviations, Acronyms and Symbols
• Acknowledgements
• Executive Summary
  • 1.0 Introduction
  • 2.0 Evaluation Scope & Objectives
  • 3.0 Key Considerations
  • 4.0 Evaluation Complexity & Strategic Linkages
  • 5.0 Evaluation Approach & Methodology
  • 6.0 Limitations to Methodology
  • 7.0 Management of the Evaluation
  • 8.0 Evaluation Findings
  • 9.0 Conclusions of the Evaluation
  • 10.0 Recommendations
  • 11.0 Management Response and Action Plan
  • 12.0 ISTPP Programming Activities

Abbreviations, Acronyms and Symbols

Acknowledgements

The evaluation team would like to extend its appreciation to many individuals and firms who agreed to participate in interviews and surveys for this evaluation. The participants included: the Heads of Canadian Diplomatic Missions, Canadian Trade Commissioners, Provincial representatives in Canada and abroad, ISTPCanada and CIIRDF representatives, Canadian participants and companies involved with ISTPP, non–Canadian participants and firms involved in ISTPP related activities as well as Government and non–governmental representatives of Israel, China, India and Brazil. The team would also like to acknowledge the advice given by the members of the Evaluation Advisory Committee on the approach, findings and recommendations of this evaluation.

Executive Summary

The Government of Canada in Budget 2005 announced $20 Million over five years to support new joint science and technology initiatives in four priority countries: Israel, China, India and Brazil. This new initiative, the International Science and Technology Partnership Program (ISTPP), was built upon the successful approach of its predecessor the Canada–Israel Research Development Foundation (CIIRDF). ISTPP now includes CIIRDF in its programming. The ISTPP is the subject of this summative evaluation, in year five of its funding cycle, to fulfill the Government of Canada requirement to assess its relevance and performance of grants and contribution programs in consideration for program renewal.

ISTPP was examined closely in this summative evaluation conducted between January and March 2010. The evaluation methodology employed mixed–methods to collect qualitative and quantitative data. The evaluation approach was comprehensive in that it was based on an almost 100% response rate. The evaluation included interview and/or survey data from ISTPP stakeholders in federal departments and agencies, in provinces engaged on international S&T, Canadian recipients, the ISTPP delivery agent in Canada and in each partner country (Israel, China, India and Brazil), Canadian Trade Commissioners, diplomatic representatives and officials from partner countries as well as Canadian and non–Canadian industries. In addition, given the absence of ISTPP data, the evaluation collected primary data from three cohorts to conduct longitudinal assessments of the nature and extent of ISTPP relevance and performance on commercialization and workforce impacts – considered to be key outcome indicators.

The evaluation faced two important limitations. First, it was not mandated to assess the impact or effectiveness of the S&T Treaty negotiation on ISTPP objectives and secondly, the ability to fully assess the performance of ISTPP was premature. This was because ISTPP, as delivered through its third–party delivery agent, started operations mid–term in its five year program cycle, in 2007 and only in 2009–10, did it become fully operational in all four partner countries. Although the timing of this summative evaluation may have been considered still early in ISTPP's development, the use of auxiliary data sources and the collection of primary empirical data have enhanced the evaluation's ability to measure an influence of ISTPP's success despite these limitations.

ISTPP, designed to contribute to efforts on international trade, investment and innovation, through the Government of Canada's Global Commerce Strategy, was found relevant in engaging partner countries in S&T innovation by supporting Canadian industry and universities towards international commercialization. ISTPP financial support is important to Canadian industry and universities in their efforts to commercialize their joint R & D collaborations but the amount of funding available was not sufficiently robust to create a critical mass in furthering Canada's S & T cooperation relationship with partner countries. Commercial success, based on a purposive sample, was considerably less than expected for Canadian–recipients. However, there is a need to further examine what constitutes success beyond a monetary sense for a program of this nature. Factors which may be mitigating the effect of ISTPP for Canadian recipients included but were not limited to shifts in market demand, protecting IP rights and overcoming barriers to market access.

The absence of sufficiently robust ISTPP accountability at a strategic level was found to contribute to management issues at the operational level. ISTPP programming was delayed in its implementation both in Canada as well as in partner countries which led to deferred program disbursements. Despite programming delays, ISTPP, through its arms–length delivery agent in Canada, was able during a 3 year implementation period to successfully leverage non–federal funds at a ratio of 3:1. The Government of Canada, however, would need to work closely with the arms–length delivery organizations in order to lead bi–lateral relationship building and partnering missions in support of S&T Treaties with partner countries.

Four recommendations have been derived from the evaluation findings. They are as follows:

1.0 Introduction

• 1.1 Background and Context
• 1.2 Programme Objectives
• 1.3 Governance
• 1.4 Delivery Organizations
• 1.5 Program Resources
• 1.6 Significant Findings of Past Program Evaluations

The Evaluation Division (ZIE) at the Department of Foreign Affairs (DFAIT) is housed within the Office of the Inspector General (ZID) and is mandated by Treasury Board of Canada Secretariat (TBS) through its new Evaluation Policy (effective 1 April, 2009) to conduct evaluations of all Direct Program Spending of the Department (including Grants & Contributions). The Evaluation Division presents to the Departmental Evaluation Committee (DEC) each quarter. The DEC is chaired by the Deputy Ministers and Associate Deputy Minister of DFAIT.

The summative evaluation of the International Science and Technology Partnerships Program (ISTPP) is being conducted to meet the TBS's requirement under the "Transfer Payments Policy" and for the "Renewal of Treasury Board Terms and Conditions for Grant and Contribution Programs." This evaluation is in accordance with the TBS decision for a summative evaluation to be conducted in 2009–10, the final year of the ISTPP, and it is identified in the Department's five–year evaluation plan. This evaluation is supported by the Department's Global Business Opportunities Bureau (BBD) and Innovation Science and Technology (BBT) Division, both of which are key target audiences along with the Canadian delivery organization, the four partner country delivery organizations as well as business and research communities of interest.

1.1 Background and Context

In its February 2005 Budget the Government of Canada had allocated $20M CDN over five years to support new international science and technology (S&T) initiatives such as those contained in the Joint Declarations with India and China. DFAIT was given the mandate to negotiate science and technology agreements with China, India and Brazil, renew the agreement with Israel, and to undertake or continue S&T cooperation activities with these countries. The ISTPP was to become the dedicated instrument to promote industrially–oriented international partnerships under bilateral science and technology cooperation agreements with those priority countries identified in Budget 2005.

ISTPP builds on the success of the Canada–Israel Industrial Research and Development Foundation (CIIRDF) Program to support bilateral S&T agreements. The aim of the program is to promote collaborative international research with other selected countries. This five year, $20M CDN program, is focussed on S&T relationships with four selected countries: Israel; China; India; and, Brazil. Canada signed bilateral agreements with each of these selected countries to develop and facilitate cooperative activities in S&T for peaceful purposes in fields of common interest and on the basis of equality and mutual benefits.

To be more specific, the purpose of S&T initiatives is as follows:

• To promote collaborative research between Canadian and foreign scientists and technologists in a variety of areas, such as the use of cleaner and more efficient forms of energy, biotechnology, information and communication technology;
• To foster and support bilateral research projects between Canada and these selected partner countries that have been proposed by companies, universities/colleges and other private sector R&D institutes; and,
• To stimulate bilateral S&T networking and matchmaking activities to further new partnerships and accelerate the commercialization of R&D.

In 2005, the Government of Canada approved its grants and contributions allotment by $1M CDN per year for 2006–07 to 2009–10 to the Canada–Israel Industrial Research and Development Foundation (CIIRDF). The funds however were frozen until the approval of the ISTPP Terms and Conditions and the approval of a Results–Based Management and Accountability Framework and Risk–Based Audit Framework.⁽¹⁾ These were approved in 2006.

In 2006, the Government of Canada approved to increase its grants and contributions allotment for a combined total of $13.5M CDN to support the China ($6.75M CDN), India ($5.25M CDN) and Brazil ($1.5M CDN) portions of the ISTPP program. It also exempted the ISTPP from the sections of the TBS Policy on Transfer Payments to allow the Department to make contributions on an annual basis to the delivery organisation in advance of need. The contribution agreement dates were signed as follows: Israel (CIIRDF) in March 2006; China in January 2007; India in March 2007; and, Brazil in March 2009.

1.2 Programme Objectives

In its broadest sense, globalization has provided the opportunity for Canada to establish its presence abroad on many fronts, including international commerce, where it continues to participate in world markets on technology and investment. The Government of Canada has therefore implemented a Global Commerce Strategy that includes industry and scientific engagement on international S&T to complement and supplement other efforts aimed to increase the international competitiveness of Canada and Canadian business.

The Department, through its international capacity, supports this engagement strategy. It manages and delivers commerce services and advice to Canadian businesses in an effort to enhance Canada's competitiveness. Despite substantial government and industry investments in S&T in Canada on either up–front science research or end–product commercialization, there appears to be less investment and capacity to support many demonstration studies on the market worthiness of pre–export ready S&T. ISTPP, an important mechanism, is therefore strategically situated to help reduce the gap between innovative science research and commercialization for Canada. Its focus is on seeking beneficial international bi–lateral collaboration which will develop Canada's capacity on science, technology and research in its business environment.

In support of the Department's objectives on international services for Canadians, ISTPP⁽²⁾ specifically focuses on:

• Encouraging domestic competitiveness through the transfer of technology and knowledge resulting from international S&T partnerships;
• Fostering international S&T partnerships and collaborative research in all sectors, with an emphasis on industry–industry partnerships and university–industry partnerships;
• Accelerating the commercialization of R&D that would benefit Canada through international partnerships with a focus on small and medium–sized enterprises;
• Accessing international technologies for Canadian enterprises;
• Promoting Canadian R&D capacity and Canada as a destination for foreign technology–based investments;
• Encouraging the mobility of researchers and promoting Canada as a career destination for foreign researchers and highly qualified personnel; and,
• Strengthening overall bilateral S&T relations.

The ISTPP expected results are as follows:

• Greater visibility by the Canadian S&T community into science and technology developments around the world;
• Increased collaboration between companies, universities/colleges and other private sector research and development institutes in Canada;
• The development of internationally competitive technology consortia in areas targeted by Canada and the partner countries (global networks);
• Increased investment in technology areas targeted by Canada and the partner countries;
• Improved international competitiveness and economic position for companies, universities/colleges and other private sector research and development institutes in Canada by creating wealth through improved commercialization; and
• Expansion of Canada's innovative workforce through the increase and/or retention in the number of highly qualified personnel in Canada.

1.3 Governance

The overall responsibility and accountability for the ISTPP rests with the Minister of International Trade. The Minister of International Trade is responsible for the program, including its development, implementation and administration. The Minister of Industry, while not responsible for the program, is consulted and plays a key role in the development and implementation of the program. Responsibility for these functions at an operational level has been delegated to the ISTPP Steering Committee⁽³⁾ and the ISTPP Secretariat. The senior–level Steering Committee is jointly appointed and co–chaired by Foreign Affairs and International Trade Canada and Industry Canada. It is supported by the secretariat.

The responsibilities of the Steering Committee include:

• Providing advice on the allocation of ISTPP resources to the identified program delivery organizations for China, India, Brazil and Israel;
• With the approval of Cabinet and based on Government priorities, modifying country priorities, as necessary, as the program evolves or is enhanced;
• Reviewing the annual activity conducted by each program delivery organization;
• Reviewing the evaluation of the chosen delivery mechanisms to be conducted on a regular basis (at least once prior to the consideration of renewal of funding);
• Providing advice to Ministers on changes or additions to program priorities; and
• Providing an annual progress report to the Minister of International Trade.

The responsibilities of the Secretariat are:

• Providing support to the Steering Committee;
• Assisting in the negotiation of the bilateral S&T Agreements with the partner countries;
• Assisting in the negotiations of how the program is delivered under each bilateral S&T agreement;
• Negotiating and managing the financial agreements between Canada and the delivery organizations;
• Drafting Treasury Board submissions and any follow–up action required;
• Collecting reports and audits from the delivery organizations; and
• Supporting program evaluations and audits.

1.4 Delivery Organizations

The Canada Israel Industrial Research & Development Foundation (CIIRDF) is the delivery organization for Canada's S&T relationship with Israel. CIIRDF is an arms–length bilateral organization incorporated in Canada. The Foundation is governed by a six member Board of Directors with half of its members selected by each country. The Board is responsible for all major funding decisions. CIIRDF, with its Program Office located in Canada, is managed by a Canadian President and a Chairman of the Board who is appointed by the Israeli government. CIIRDF received a $5M contribution for 2005–2010 sourced from DFAIT's International Science and Technology Partnerships Program (ISTPP). The Government of Israel matched this contribution amount. The contribution agreement was signed in March 2006.

CIIRDF supports participation in bilateral science and technology networking activities and organizes and delivers a number of "match–making" events designed to bring together Canadian and Israeli companies, universities/colleges and other research and development institutes to explore partnership opportunities. The partnering activities supported may include, but are not limited to, technical and scientific seminars, conferences, workshops and other events designed to foster partnerships with the selected countries. In addition to networking and matchmaking activities to foster new partnerships, CIIRDF fosters and supports bilateral research projects between Canadian and partner country companies, universities/colleges and other research and development institutes (the ultimate recipient) by funding joint research projects and activities.

As stated in the Contribution Agreement, the responsibilities of CIIRDF for program delivery are as follows:

• To promote and enhance all forms of private sector industrial cooperation in accordance with the legislation of Canada and Israel, as well as their respective economic and development policies and priorities by encouraging, supporting and facilitating:
  • The maintenance of a database on Canadian and Israeli companies' capabilities as prospective research partners;
  • The exchange of information on technologies and know–how, licensing arrangements and industrial consultancy;
  • Contacts between respective science and technology communities;
  • Exchanges of views on the formulation and application of science and technology policies;
  • The matchmaking between Canadian and Israeli businesses in their effort to establish R&D joint ventures; and
  • The transfer of technology through research programmes in order to promote the application, adaptation and improvement of existing and new technological products and processes.

ISTPCanada is a non–governmental organization formed through a partnership between CIIRDF and Precarn Incorporated. It serves as a policy implementation instrument and was created in response to the Government of Canada's call in 2006 for an ISTPP delivery organization for Canada's S&T relationships with China, India and Brazil. It was selected through a competitive tendering process to provide funding and other services in Canada to program proponents. The Government of Canada announced in 2007 that ISTPCanada was named as the successful candidate as the ISTPP delivery organization. Therefore, ISTPP delivery through a third– party agent, ISTPCanada, has been in effect since 2007 which represent three years of the five year program cycle supported by the TB submission.

The ISTPCanada Board of Directors is comprised of sectoral–specific sub–committees that are responsible for the delivery of each bilateral program other than Israel (i.e., one for Canada–India, one for Canada–China and, one for Canada–Brazil). This includes oversight for the delivery of the program, major funding decisions for the bilateral partnering activities. The ISTPCanada office manages the program activities while Precarn Incorporated is subcontracted to provide a range of corporate services, the most important of which are those involving project information and financial management. ISTPCanada received three (3) separate contributions sourced from DFAIT's International Science and Technology Partnerships Program (ISTPP) to engage in activities that primarily support the areas of cooperation as set out in the Agreements on Scientific and Technological Cooperation with each of the partner countries: China ($5.25M), India ($6.75M) and Brazil (1.4M⁽⁴⁾). The contribution agreements for China, India and Brazil were signed in January 2007, March 2007 and March 2009 respectively.

As stated in the three respective Contribution Agreements, the responsibilities of ISTPCanada are to:

• Identify and coordinate matchmaking activities;
• Market the program among companies, universities/colleges and other private sector research and development institutes who may become potential ultimate recipients;
• Accept and assess applications for the funding of bilateral research projects and provide funding to successful applicants;
• Provide ongoing monitoring of the status of the funded projects. This includes collecting and reviewing project reports from ultimate recipients including technical reports, financial reports and commercialization reports and the ongoing interaction with the ultimate recipients;
• Collect repayments from ultimate recipients in cases of overpayment and, if so stipulated in the bilateral arrangement with the partner country, in the event of commercial success;
• Conduct performance and risk monitoring activities and ensure that sufficient records and documentation related to these activities are maintained to support operational and management information requirements as well as departmental due diligence requirements. This includes:
  • Implementing and maintaining a system for tracking partnering and matchmaking activities (i.e., a database);
  • Collecting and analysing the performance information on an ongoing basis and reporting these to the Steering Committee/Secretariat in its annual Activity Report;
  • Monitoring risk in support of the ISTPP risk based monitoring activities;
  • Fully and promptly cooperating with any and all Program evaluations and audits;
  • Conducting audits of the ultimate recipients and reporting the findings of these audits to the ISTPP Steering Committee;
  • Preparing a succession plan to outline how activities can or will be conducted should there be significant changes in the delivery organization's structure or personnel;
  • Forming a Board of Directors to oversee Program activities; and
  • Preparing and delivering annual audited financial statements and activity reports to the ISTPP Steering Committee.

1.4.1 Partner country Delivery Organizations

The ISTPP delivers collaborative bilateral S&T cooperation activities in partnership with a number of partner country delivery organizations. Establishing programming priorities, the identification of technology areas and the management of calls for proposals is undertaken jointly with these organizations within the parameters as set out in the respective Bilateral S&T Cooperation Agreements (S&T Treaties).

Israel

The Canada–Israel Industrial Research and Development Foundation (CIIRDF) was established in 1994 to promote collaborative R&D between firms in Canada and Israel. While the CIIRDF Headquarters are located in Ottawa, it also staffs an office in Israel to facilitate the implementation of its program.

China

The Ministry of Science and Technology (MOST) of China is a central government agency under the Chinese State Council, responsible for the nation's science and technology activities. MOST is ISTPCanada's primary funding partner in China to jointly deliver and fund Canada–China research and development (R&D) projects.

The China Association for International Exchange of Personnel (CAIEP) is a nationwide government–sponsored institution under the State Administration of Foreign Experts Affairs (SAFEA) engaged in the international exchange of specialized technical and managerial personnel. In 2008, ISTPCanada signed a Memorandum of Understanding (MOU) with CAIEP for the delivery of ISTPP funded partnership development activities in China.

A tri–partite agreement also has been signed among ISTPCanada, Yangling Demonstration Area and CIIRDF for the funding of collaborative agricultural R&D projects that would support private sector organizations in Canada, Israel and China to develop and demonstrate innovative agricultural projects.

India

The Department of Science & Technology (DST) plays a pivotal role in promotion of science and technology in India, which currently has bilateral S&T cooperation agreements with over 70 countries/regions. DST, along with the Department of Biotechnology, is ISTPCanada's counterpart and primary partner in India. ISTPCanada and these two Departments jointly promote and support the development of research and technology–based collaborations between the two countries.

The Department of Biotechnology (DBT) was instituted under the Ministry of Science and Technology in 1986 to give a new impetus to the development of the field of modern biology and biotechnology in India. ISTPCanada and DBT released its first Call for Proposals for R&D projects and Partnership Development Activities (PDAs) in September 2008.

Global Innovation & Technology Alliance (GITA) is a not–for–profit organization founded by the Department of Science & Technology (DST) and Confederation of Indian Industry (CII) for the promotion and facilitation of International Science & Technology Cooperation Programs for the benefit of Indian industry and institutes. ISTPCanada and GITA jointly deliver the Canada–India Call for Proposals for R&D projects and PDAs.

Brazil

The State of São Paulo Research Foundation (FAPESP) was founded in 1962 and is linked to the State of São Paulo's Secretariat for Higher Education. It is one of the main funding agencies for scientific and technological research in Brazil. In October 2009, ISTPCanada and FAPESP launched their first joint Call for Proposals in critical sectors such as information and communications technologies (ICT), life sciences and energy.

1.4.2 ISTPP Recipients

The ultimate recipients are companies, universities/colleges and non–governmental R&D institutes who conduct joint R&D projects and receive ISTPP funding via the delivery organizations. In cases where the delivery organization is conducting only the Canadian portion of program activities, the ultimate recipients are Canadian companies, universities/colleges and non–governmental R&D institutes which operate and are headquartered in Canada. In cases such as CIIDRF, where the delivery organization is conducting activities in support of ISTPP for Canada and Israel (and receiving funding from both countries), the ultimate recipients are Canadian and Israeli companies, universities/colleges and non–governmental R&D institutes which operate and are headquartered in Canada and in Israel respectively.

1.5 Program Resources

ISTPP contribution funds have been secured through Government of Canada approval processes. ISTPP was first announced in Budget 2005 and ISTPP program funds were transferred to the ISTPP delivery agent ISTPCanada in 2007. In accordance with Government of Canada policies, the program sunset date was March 31, 2010. Given the March 31, 2010 sunset date, the conduct of this summative evaluation was crucial in an effort to ascertain the continuation of the program is support of program renewal.

In accordance with the terms and conditions of the ISTPP contributions, the program has instituted funding arrangements that allow for the distribution of funds to separate delivery organizations (ISTPCanada and CIIRDF) to support designated activities. ISTPCanada and CIIRDF charge a 15% administrative fee to cover overhead expenses. As a result of this delivery mechanism, the Department's management or internal resource base itself has been increased only marginally in capacity to support this program.

As for program administration, ISTPP can contribute at any point throughout the process and funds can be carried–over. The only condition was that all contribution funds must be expended by March 31, 2010 and, unused contribution funds by the end–date must be returned to the Crown. However, as of March 31^st, 2010, ISTPP received approval to disburse program funds in 2010–11 to cover project commitments that were approved by March 31, 2010. This is a one year extension for disbursement purposes only; not for any new funding.

Since program inception, the Government of Canada had approved only one re–profiling of funds for ISTPP. This was for the India component and it was done to accommodate the delay in establishing the bi–lateral agreement. The Department has proactively cash–managed the program in an effort to optimize the flexibility of its funding arrangements. This was being done to account for any delays that may occur in implementation. Compliance to the terms and conditions of ISTPP contributions is the subject of recipient audits that coincide with the conduct of this summative evaluation.

The following two tables present ISTPP planned and actual contribution funding to ISTPCanada and CIIRDF for the four partner countries.

Table 1:

Table 2:

* For India, this includes $500,000 TBS approved re–profiled funds made in 2005–06. A total of $250,000 was moved to 2006–07 and the remaining $250,000 was moved to 2007–08.

+ An Amendment to the Brazil Contribution Agreement re–profiled the budget by one year with the remaining $800,000 modified to $700,000 for 2009–10. This is to be disbursed in 2010–11.

1.6 Significant Findings of Past Program Evaluations

CIIRDF

CIIRDF was the subject of a summative evaluation in 2004. The evaluation found that CIIRDF's objectives were relevant to Canadian–Israeli companies. The evidence also suggested that additional public funds would be appropriate to support CIIRDF to expand this type of programming to other partner countries. The evaluation recognized that CIIRDF matchmaking, along with sufficient funding to support R&D, was a sound mechanism to link Canadian technology with new sources of technology abroad. CIIDRF had developed consortiums which solidified a positive perception of the value of Canadian companies for R&D ventures with Israel. Most firms were of the belief that they would not have conducted R&D without CIIRDF funds, as their R&D budgets were inadequate and the risk associated with conducting R&D with international partners too high to absorb.

The 2004 CIIRDF summative evaluation also projected high cumulative returns–on–investment relative to initial contribution amounts. Based on sale forecasts of 11 of 18 Canadian firms that undertook CIIRDF R&D projects, it was estimated that roughly $714M (CDN) in cumulative sales revenues and $178M (CDN) in cumulative profits would be generated by 2013. These returns would generate royalties to CIIRDF, higher business income taxes to Canada and of course, the repayment of initial contribution amounts. Not only was there a potential for economic benefit but also there were socioeconomic benefits being generated in terms of the venture projects initiated such as water treatment and purification

ISTPP

ISTPP, the subject of a formative evaluation in 2008, was found to have achieved, despite its delayed implementation, success on collaborative R&D investments (project commercialization plans) and R&D matching (introductions with R&D organizations) in China and India.

However, the formative evaluation found that more evidence was needed to assess the value–added of the service delivery model in partner countries to leverage resources as well as the capacity of ISTPCanada to leverage resources in Canada. Additionally, the evaluation pointed to a need for clarification on the roles and expectations of expert co–ordinators on evaluating project proposals; a need for comprehensive communication plans to promote ISTPP to the Canadian S&T community and, a need to target SMEs and academic communities.

The formative evaluation found that although ISTPP was still in its infancy, with only two years of operation, it was well–structured, cost–effective and early indications showed that ISTPP will likely contribute to the competitive position of Canadian science and technology organizations.

2.0 Evaluation Scope and Objectives

The objective of this summative evaluation is to provide senior management at DFAIT with a neutral, evidence–based assessment of relevance and performance of the ISTPP, and the progress made to date. The specific objectives of the evaluation are as follows:

• To evaluate the relevance of ISTPP by assessing its continued need and, the extent to which it is aligned with federal government priorities and with the Department's strategic priorities; and,
• To evaluate the performance of ISTPP, through its delivery organizations, in achieving its objectives efficiently and economically with its allocated resources since its inception.

Notwithstanding the above objectives, the evaluation will mainly focus on three key areas of inquiry: 1) the effectiveness of the governance and delivery models used, especially in the partner countries; 2) the value of the softer impacts of consortia– building relationships between partners (PDAs); and 3) the commercialization rate of CIIRDF funded projects and the economic benefits accruing to Canada and Israel.

3.0 Key Considerations

• 3.1 Rationale
• 3.2 Qualitative Impacts
• 3.3 Commercialization Rate
• 3.4 ISTPP Governance

3.1 Rationale

The ISTPP summative evaluation is being conducted to fulfill TBS requirements on assessing relevance and performance of sunsetting programs in an effort to determine program continuation. The evaluation's findings will assist to determine the continued need for this program and any expansion thereof in response to the Department's Global Commerce Strategy (GCS). It is timely in its review of whether ISTPP has been successful in reducing the gap between innovative research and commercialization by enhancing pre–export readiness and profits to Canadian firms.

3.2 Qualitative Impacts

The summative evaluation also assesses the softer impacts ISTPP has had in securing the interest and participation of Canadian advanced technology firms. This is measured by the level of support and facilitation generated on scientific, technical cooperation, and collaborative R&D projects between Canadian companies and the four partner countries. One key analysis is the relative efficacy or success of the service delivery models in the partner countries. This is valuable in explaining how the ISTPP delivery mechanisms have been able to develop and sustain effective consortiums. It was hypothesized that these consortiums had been well–targeted to reach ISTPP's objectives for mutually beneficial product development, future wealth creation and enhanced market access.

3.3 Commercialization Rate

Another evaluation aspect is to address ISTPP's performance on the commercialization rate of the research or product of completed projects. In other words, the percentage of ISTPP–funded completed projects that have reached commercialization and if profit margins were met according to projected values in their commercialization plans. This is especially timely for CIIRDF's funded projects because of their maturity. Many of CIIRDF projects have been in effect since 1994, which is a considerably longer operational period than that of the projects in China or India. It was therefore possible to review CIIRDF projects on actual sales and revenue generated to obtain economic data.

3.4 ISTPP Governance

The summative evaluation also addresses governance issues. Included in the evaluation is an assessment of those issues regarding the remuneration of expert co–coordinators responsible for project proposal evaluation. Expert co–coordinators are science sector specialists within those federal departments and agencies who are represented on the ISTPP Steering Committee. These specialists are currently contributing "in–kind" services on sector–specific co–ordination committees. Although the co–coordinator role was designed to provide an objective review and assessment of the vitality of the project proposals submitted, apparently not all expert coordinators were in agreement, only those in AAFC and NRC.

4.0 Evaluation Complexity and Strategic Linkages

The ISTPP was designed to contribute to the Department's efforts on international trade, investment and innovation, through the Government's Global Commerce Strategy (GCS) which is in place to improve Canada's competitiveness and to support Canadian firms as they pursue opportunities in the global marketplace. The GCS's guiding belief is that an expanded model of participation by Canadian businesses in world markets would contribute to an international agenda shaped to Canada's benefit and advantage in accordance with Canadian interests and values. This includes forging stronger linkages between Canada's S&T community and global innovation networks under bilateral S&T arrangements. The ISTPP is to build these international linkages while promoting the commercialization of the R&D necessary for Canadian firms to be globally competitive.

The ISTPP is also intended to increase the interaction of Canadian entrepreneurs with global business partners and promote Canada as a competitive location and partner for investment, innovation and value–added production in specific sectors of S&T. It was expected that ISTPP would contribute to greater economic benefit for Canada with its partnerships of emerging interests in sector–specific S&T markets in China, India and Brazil or in a strategically established market such as Israel. These partnerships allow for the opportunity to forge collaborative arrangements and shared benefits rather than encouraging fierce competition over market–share.

5.0 Evaluation Approach and Methodology

• 5.1 Evaluation Design
• 5.2 Sampling
• 5.3 Data Collection
• 5.4 Primary Data Collection
• 5.5 Data Analysis

The evaluation was guided by an Evaluation Advisory Committee (EAC) composed of the key ISTPP stakeholders, including representatives from DFAIT's Innovation Science and Technology Division (BBT), Industry Canada (IC), National Research Council of Canada (NRC), National Science and Engineering Research Council (NSERC), Natural Resources Canada (NRCan), Agriculture and Agri–Food Canada (AAFC), ISTPCanada, CIIRDF and an external expert advisor. The reference period for the evaluation spans approximately the five year timeframe of the program (2005–2010) and includes a longitudinal analyses of the initial 18 2004 CIIRDF projects that were considered to hold the strongest promise of commercialization. The summative evaluation was conducted over a period of three months – January to March 2010.

5.1 Evaluation Design

The evaluation employed a mix of evaluation techniques to collect both qualitative and quantitative data. Documentary content analysis was used to establish a detailed profile of the ISTPP governance structures and processes, delivery modalities and activities and outputs to date. Face–to–face and telephone interviews were conducted with the ISTPP Steering Committee members, as well as CIIRDF and ISTPCanada Board members to assess the evaluation issues related to relevance and governance issues.

Field missions were undertaken to each of the four countries to assess both relevance and performance evaluation issues through face–to–face interviews with the partner country delivery organizations and selected R&D Project partners. Questionnaires were sent to all ISTPP S&T Cooperation Sector Coordinators. Three online surveys were used in conjunction with the field missions to collect both qualitative and quantitative data on service delivery satisfaction and economic benefits from both Canadian and partner country R&D organizations that participated in the various activities of the program. Based on 70 Canadian company recipients of ISTPP–CIIRDF funds, 56 had valid contact information and interviews were held with 13 of them in addition to their written survey reply. Twenty–seven (27) interviews were held with ISTPP stakeholders; only two stakeholders were unable to participate. The data collected from these lines of evidence was compiled and analyzed using statistical and qualitative text analysis software as appropriate to facilitate data analysis and the triangulation of findings.

5.2 Sampling

5.2.1 Collaborative R&D Projects

Over the years CIIRDF has supported over 70 Collaborative R&D Investment Projects between Canada and Israel in numerous technological areas. The 2004 CIIRDF Summative Evaluation identified 18 of those projects with considerable potential for product commercialization and sales. A review of the list of active projects reveals that the CIIRDF Board has approved 17 projects since 2005. A review of the available lists of Collaborative R&D Investment Projects funded through ISTPCanada also revealed that since May 2007 a modest number of such projects were approved by the Board with China (14) and India (12), while no such projects have as yet been approved with Brazil (April 2010 is the expected date of project announcements).

Under these circumstances a Collaborative R&D Investment Project sampling strategy for this evaluation was not recommended given the recent and small overall population of ISTPP funded projects (43), the relative instability of the SME sector, and the high probably that key respondents may not be available within the evaluation's primary data collection timeframe. The evaluative technique of choice under these circumstances was an on–line survey of the ultimate recipients of ISTPP funding. For many of the same reasons, the 18 high potential projects identified in the 2004 CIIRDF Summative Evaluation were included in the project population as a distinct purposive sample and subject to a different evaluative technique, i.e., field mission site visits. The inclusion of this purposive sample gave the evaluation the needed time horizon to adequately assess the economic benefits generated by the Collaborative R&D Investment project model, thus serving as a proxy for ISTPP–funded projects.

5.2.2 Partnership Development Activities

Under current ISTPP funding, it appears that ISTPCanada began implementing R&D matchmaking activities in June 2008 while CIIRDF began in September 2008. The Network Activity Report covering the June 2008–March 2009⁽⁶⁾ period provides a narrative description of the activities undertaken, while the detailed information on the companies involved is provided in separate spreadsheet files annexed to that report: Annex 1: Partnering Missions Outcome Summary; Annex 2: Match–Making Initiative Deal Flow Table–Israel; Annex 3: Match–Making Initiative Deal Flow Table–India.

A review of the report and attachments reveals a small number of partnering missions (4), but a much larger number of matchmaking attempts involving over 150 Canadian companies. Given the adequate tracking information contained in the tables, and that email contact information was available for the Canadian companies, then a sampling strategy was not required since the performance data could be appraised using online surveys. Contributing factors and lessons learned were solicited through open–ended interviews.

5.3 Data Collection

5.3.1 Lines of Evidence

Document Review

A background document review was conducted to contextualise the ISTPP within the Government of Canada's international, economic and science and technology policy framework. Departmental Reports on Plans and Priorities, as well as S&T publications were reviewed from a policy alignment and coherence perspective. The management file review was an ongoing process as additional documentation and reports became available, e.g., ISTPP Steering Committee Meeting minutes, ISTPCanada and CIIRDF Annual Activity Reports, S&T Sector Coordinator workshop documentation, as well as R&D Project file records. Past evaluation reports of both CIIRDF and the ISTPP were reviewed as well as the draft ISTPP Financial Compliance and Audit Report.

5.4 Primary Data Collection

5.4.1 Lines of Evidence

Key Informant Interviews in Canada

Face–to–face interviews were conducted with 10 internal stakeholders, including the ISTPP Steering Committee members, ISTPP Secretariat staff, and other DFAIT senior managers. Face–to–face and telephone interviews were also conducted with 17 external stakeholders, including ISTPCanada and CIIRDF personnel, their Board of Directors, China and India Sector Coordinators, and provincial representatives (Ontario, Alberta, British Columbia) who have or intend to contribute funds to ISTPP activities. Based on 70 Canadian company recipients of ISTPP–CIIRDF funds, 56 held valid contact information and replied to the on–line survey. Interviews were held with 13 of them in addition to their written survey reply.

Field mission Site Visits

Four field visits (Israel, China, India and Brazil) were conducted in order to obtain comprehensive data on the nature and extent of ISTPP at various stages of program implementation. Given that one evaluation objective was to assess the rate of commercialization, it was necessary to collect primary data from CIIRDF which has been in operation for at least 15 years and as a result, represents a mature operation with projects completed. In–person interviews and project demonstrations were held with all but one Israeli active firm in the 2004 CIIRDF cohort. Interviews were conducted with all active Israeli firms in 2006–2009 cohort. This represents a high response rate of almost 100%. Interview and survey data was also obtained from Canadian partner firms and with the data from Israeli partner firms, enabled the evaluation to conduct matched–group comparisons. These data were complemented and supplemented by interview data from Canadian officials, the Office of the Chief Scientist and other countries with similar bi–lateral S&T agreements with Israel.

Field visits to China, India and Brazil were not as project specific as Israel but equally valid to obtain observational data and expert opinions on the relative value of ISTPP from Chinese, Indian and Brazilian key informants. These visits were valuable to gain insight from the partner country delivery organizations on their viewpoint of the ISTPP delivery mechanism during its start–up phase of implementation, in support of the evaluation objective on assessing its effectiveness.

Online Surveys and Follow–up Telephone Interviews

Three online surveys were used to collect and compile data from R&D Project partners and PDA participants. Field mission site visits and follow–up telephone interviews were used effectively to ensure both high response and completion rates. This line of inquiry generated useful data with regard to overall satisfaction with ISTPP in terms of networking, partnership and consortia–building relationships, as well as the benefits accrued in terms of increased R&D capacity, commercialization and economic benefits.

The "2004 CIIRDF Online Survey" compiled data from 20 of a possible 24 companies involved in R&D Projects prior to the launch of the ISTPP for an 83% response rate with a 95% survey completion rate; 12 companies in the sample had either ceased operations or had been acquired. Of the 20 companies from which data was collected 9 were Canadian and 11 were Israeli.

The "2006–2009 CIIRDF Online Survey" compiled data from 33 of a possible 34 companies involved in R&D Projects funded by the ISTPP for a 97% response rate with a 100% completion rate. A total of 13 companies responded to the PDA component of the survey with a 100% completion rate.

The "2006–2009 ISTPCanada Online Survey" compiled data from 19 of a possible 50 R&D organizations with approved R&D Projects funded by the ISTPP in China and India for a 38% response rate with an 84% completion rate; it should be noted however that only 8 of 25 approved R&D Projects were launched at the time of the evaluation. A total of 46 R&D organizations responded to the PDA component of the survey with a 100% completion rate.

Of the eleven (11) questionnaires sent to Canadian S&T Cooperation Sector Coordinators, five (5) were designated as China S&T Cooperation Sector Coordinators and six (6) were India S&T Cooperation Sector Coordinators. Only two coordinators returned questionnaires (for a response rate of 18%, which represented the lowest response rate of all participant groups) despite a follow–up request for their participation in completing the questionnaire.

5.5 Data Analysis

Data Sorting – Triangulation

Over 100 secondary reference documents, program reports and evaluations, interview notes and summaries of the primary data collected were compiled and loaded into a text management software package (Atlas/ti). A closed coding system was used to sort the data according to the evaluation issues and indicators contained in the Evaluation Matrix. Open coding was also used to identify and compile data on additional topics of interest not originally identified. The findings were then summarised on an indicator by indicator basis paying special attention to triangulating the lines of evidence. The evidence base to support the evaluation report findings and conclusions is easily accessible and verifiable.

6.0 Limitations to Methodology

This evaluation was undertaken in accordance with the Terms of Reference and the work–plan taking into consideration the constraints contained therein. The most significant operational constraints identified which have a bearing on the limitations of this evaluation are as follows:

• Limited performance monitoring data readily available on expected outputs and outcomes; and
• Insufficient time and resources to address all of the evaluation issues and collect data on related performance indicators.

Given the absence of ISTPP data at the Department, the evaluation employed primary data collection techniques, such as self–reported surveys, to collect performance outcome data on R&D capacity building, partnerships and consortia formed, economic benefits, and workforce impacts. There are inherent limitations to the use of self–reported measures in where it may be challenging to assess data reliability. While this limitation has been mitigated through the triangulation of or collaboration of survey data with interviews and available tracking data there nevertheless remains the possibility that respondents may not have fully disclosed the economic benefits accrued from their involvement in the ISTPP or other relevant activity being measured.

Another important consideration to note when reviewing the evaluation findings is that evaluation was not mandated to assess the effectiveness of the S&T treaty negotiation on its impact on ISTPP delivery. While not a direct limitation regarding methodology, the S&T treaties may or may not include provisions that could impact on the ability of Canadian SMEs to gain competitive advantages or to facilitate in international R&D collaborations.

There also are other limitations. ISTPP, through its delivery agent, has only been operational since 2007 and only in 2009–10, the final year of its 5 year program funding cycle, did it become operational in all four partner countries for a period of six months. As a result, this summative evaluation, conducted one year after the formative, may be considered premature to assess the cumulative effect of results achieved because many approved ISTPP projects are in their infancy. While the continued relevance and performance of ISTPP was examined in terms of increased capacity and competitiveness of Canadian R&D internationally and the formation and investment in technology consortia, and partner collaboration, given the rather early stage of ISTPP's operation in China and India, it was difficult to assess economic benefit. This issue was even more prevalent in the case of Brazil since 2009–10 was the first year of its implementation. Although ISTPP's commercialization rate of the research or product of completed projects, based on a 2004 CIIRDF proxy measure,⁽⁷⁾ assessed if profit margins were met according to values in their commercialization plans, this should not be viewed as definitive but rather as a preliminary indication on the achievement of this ISTPP commercialization objective or target.

7.0 Manamgement of the Evaluation

• 7.1 Evaluation Advisory Committee and Process
• 7.2 Evaluation Team Composition and Responsibilities

7.1 Evaluation Advisory Committee and Process

The conduct of the summative evaluation was guided by an Evaluation Advisory Committee (EAC) comprised of representatives from ISTPP and other relevant DFAIT branches/divisions, selected OGD representatives and external stakeholders as appropriate. The EAC is comprised of representatives from NSERC, NRC–NRC–IRAP, Industry Canada, NRCan, AAFC, ISTPCanada, CIIRDF, DFAIT–BBT and, an external expert advisor.

The Director of Evaluation chaired the EAC. The EAC convened upon project initiation to provide feedback on the evaluation approach (including work plan) as well as a meeting to discuss the draft report. The DFAIT division directly responsible for ISTPP, BBT, is required to provide an official management response and action plan to the recommendations presented in the evaluation report and to present the Management Response and Action Plan (MRAP) at the Departmental Evaluation Committee (DEC).

7.2 Evaluation Team Composition and Responsibilities

ZIE is the Project Authority responsible for quality assurance to ensure satisfaction and quality of the deliverables. The work was undertaken within the context of a 'mixed resource approach' wherein ZIE, through field missions, contributed significantly to the conduct of the evaluation with regard to on–site data collection from program stakeholders in Israel, India, China and Brazil as well as providing analytical expertise in the preparation of presentation and evaluation report. The consultant assumed responsibility for work plan preparation, documentary content analysis, online survey and interview data collection, data analysis and preparation of preliminary findings, attendance at the first EAC meeting and preparation of the draft evaluation report. The evaluation was managed in a highly cooperative manner between ZIE and the consultant as well as in a consultative manner with the ISTPP stakeholders and others.

8.0 Evaluation Findings

• 8.1 Relevance Issue 1: Continued Need for the Program
• 8.2 Relevance Issue 2: Alignment with Government Priorities
• 8.3 Relevance Issue 3: Consistency with Federal Roles & Responsibilities
• 8.4 Relevance Issue 4: Effectiveness of Governance Processes
• 8.5 Relevance Issue 5: Effectiveness of Delivery Models Employed
• 8.6 Performance Issue 6: Achievement of Expected Outcomes
• 8.7 Performance Issue 7: Demonstration of Efficiency & Economy
• 8.8 ISTPP Partner–Country Reports

8.1 Relevance Issue 1: Continued Need for the Program

Relevance of the ISTPP for global competitiveness

Industrial countries, such as Canada, have been pre–occupied over the last decade with their capability to innovate and to bring innovation successfully to market as a means to achieve global competitiveness. Government commerce–related policy is often developed with innovative activity as the main driver of economic progress to sustain a standard quality of life, as well as a potential factor to meet global challenges in key sectors such as environment and health. These efforts have contributed to rapid advances in scientific discovery and in general–purpose technologies such as Information and Communication Technologies (ICTs) and Biotechnology; innovations which have been accelerated by globalization.

Among industrial countries, there also appears to be a strong perception that the rise in the quality of living standards is due to economic gains from advancements in innovation. Today, innovative performance is seen as a crucial measure of competitiveness and a nation's progress on social–economic health. Moreover, innovation is important to help address global challenges, such as climate change and sustainable development. But it is the application of advances in technology, in conjunction with entrepreneurship and innovative approaches to the creation and delivery of goods and services, which translates scientific and technological advances into productive economic activity. This results in economic growth if market structures and the regulatory environment enable the more productive activities to expand.

Intellectual assets can be defined as an aggregate measure of key indicators including human capital, R&D and the capacity to conduct it, patent value and intangible assets such as the value of a brand–name value or firm–specific knowledge. When taken as a whole, intellectual assets are increasing becoming the key to creating value in a global market economy through a number of channels. For example, improvements in the skill composition of the labor market contribute to growth in productivity. Studies have also suggested that investment in R&D is associated with high rates of return thereby increasing the standard of living.

Major emerging market economies are no longer simply low value–added producers but are adding their weight to the creation and commercialization of innovative products, processes and services. Trade data on the four most significant emerging economies (Brazil, Russia, India and China or "BRIC") show that these have become more active in higher technology industries over the past decade. Figure 1 shows that between 1996 and 2004 the share of high technology goods has doubled to reach approximately 30% of total trade (exports plus imports) in manufactured goods by BRIC countries. China accounted for most of this increase. Most of China's exports of high–tech products are due to the manufacturing of these products by foreign firms located in China.

Figure 1: The Changing Structure of BRIC's manufacturing trade by technology intensity ⁽⁸⁾

The increased R&D capabilities in innovation and the availability of human capital in BRIC countries, in particular China, intensify the challenge to OECD countries for increased competition to gain a higher market share for desirable goods and services. At the same time, the emergence of BRIC economies offers major opportunities for OECD countries. One such reason is that these countries are becoming more consumer–oriented and as a result, are contributing to the market demand for innovative products while at the same time, providing access to a highly skilled workforce. These developments make it even more urgent for OECD countries to strengthen their activity in the global value chain. Many OECD countries have seen little improvement in productivity performance in recent years despite the new opportunities offered by globalization and by new technologies, especially ICTs.

In its 2008 Report Card on Canada: How Canada Performs, the Conference Board of Canada (Figure 2) provided another "D" to Canada in terms of Innovation. Canada's performance on innovation over the past three decades has remained the same according to this measure. Canada is above–average among the 17 wealthiest countries on only two indicators: scientific articles published and the export market share of the aerospace industry.

Figure 2: Canada's Innovation Report Card 2008

This is rather surprising given that Canada is considered to be well–supplied with good universities, engineering schools, teaching hospitals, and technical institutes. It produces science that is well respected in the world. But, aside from those few exceptions, Canada appears to have not taken the steps that other countries have to ensure that their science is successfully commercialized and used as a source of advantage for innovative companies seeking global market share. For example, despite current tax incentives in Canada, industry R&D expenditures appear to be among the first to be reduced or discontinued during recessions. Historically, business R&D expenditure and patent applications have moved in parallel with GDP, slowing markedly during the economic downturns of the early 1990s and early 2000s. Data on trademark applications, that reflect the creation of new goods or services, with or without technological content, shows the effect of the business cycle on a wide range of innovation. For instance, R&D expenditures vary more than GDP over the business cycle. Hence, any drop in GDP usually results in an even larger decrease in business R&D expenditures. Consequently, Canadian companies are rarely at the leading edge of new technology and too often find themselves a generation or more behind the productivity growth achieved by other global industry leaders.

In December 2009, the OECD published its Science, Technology and Industry Scoreboard. The document focuses on five themes: financing innovation, targeting new growth areas, competing in the world economy, connecting to global research and investing in the knowledge economy. Canada's highlights are presented in Graph 1.

Graph 1: Canada Relative to OECD/G7 Performance in Financing of Innovation

Financing Innovation

• R&D intensity in Canada declined from 2.1% of GDP in 2004 to 1.8% in 2008. This value is in line with the OECD (1.9%) but below the G7 average (2.2%).
• The rate of R&D tax subsidies is higher, especially for SMEs, which benefit from a 45% higher rate than large firms.
• Government financing of business R&D (2.3%) is low compared to the OECD average (6.7%).

The aforementioned suggests that Canada's approach to engage on international S&T collaborative R&D, through ISTPP, is a relevant effort:

• to stimulate S&T growth and productivity in Canada;
• to increase Canada's presence in a global economy; and,
• to sustain quality of life for Canadians.

Through ISTPP, Canadian industry and universities involved in S&T are encouraged and have an opportunity to develop and to commercialize. These are activities and principles supported in the Global Commerce Strategy.

Relevance of the ISTPP for partner countries

Israel: ISTPP in context

Israel's investment in R&D is among the highest of industrialized countries (5% of its GDP in 2008⁽⁹⁾) thereby contributing to a relatively high per–capita income base. Israel also maintains one of the highest per–capita rates of scientific patents among industrialized countries.⁽¹⁰⁾ As evidence of the country's leading role in research and development, Israel has been chosen to head the 'Eureka Initiative.' EUREKA is the leading industrial R&D programme in Europe and the largest of its type in the world.  Almost all European countries as well as the European Commission participate in EUREKA, which has been operational since 1985. Every year several hundred new R&D projects are supported under EUREKA which has had an annual budget of approximately 1.5 billion in recent years. Israel was accepted as a full member of EUREKA in 2000 and has emerged as one of its five most active members. Today, Israeli companies participate in 10% of EUREKA projects. The Israeli presidency of EUREKA is expected to significantly enhance Israel's status within the European R&D community and will allow it to promote R&D projects of particular interest to Israel as well as to host key meetings of European decision makers in the R&D field.

The Office of the Chief Scientist of Israel considers all S&T agreements as valuable since they contribute to the objectives of Israel's S&T strategy. Israel has many S&T agreements of which five are bi–national (Korea, Singapore, Canada, United States of America and Britain).⁽¹¹⁾ Canada and Israel's S&T arrangement with CIIRDF spans 20 years. While Canada offers the lowest contribution amount among the group of 5; Israel has indicated that they are willing to increase their contribution dollars should Canada match its offer. CIIRDF recipients were of the strong opinion in the value of collaborative R&D with Canadian partner firms.

China: ISTPP in context

China is growing dramatically and views collaboration on S&T as a high priority. Canada and China signed a Cooperation Agreement on Science and Technology in 2007. Technology transfers specifically with an aim at commercialization benefits both countries as it enables researchers to use R & D strengths to develop products that can be eventually marketed in both countries. For China, S&T collaboration is a high priority as technology transfer is seen as adding significant gains to their economy. China currently has S&T cooperation agreements with over 100 countries. Canada is currently in the second tier of countries that China considers important for S&T collaboration. The United States is in the first tier which represents about 1/3 of all S&T collaborations followed by the European Union, Russia and Japan. In the second tier is Canada along with the United Kingdom and Australia. China has allocated approximately $100M USD annually for S&T partnerships of which Canada represents a relatively small fraction.

Other countries to Canada have established joint research centres using a top–down government to government approach. The Chinese have just signed with the USA a five–year $150M USD accord on clean energy research in specific areas such as clean coal and electric cars. The agreement seeks to establish joint research centres: one in the USA and one in China. From the Chinese government's perspective, preferences would be given to directed government–led programs rather than to a competition for the selection of projects from an arms–length body as they appear more comfortable with the high level government relationships in the selection of activities for collaborative research. That being said, they have agreed with the ISTPP, arms–length or third–party delivery model developed by Canada.

India: ISTPP in Context

India, like China, presents a wealth of opportunity in the S&T collaboration field. Observers noted that India is where China was 10 years ago and will use technology transfers to rapidly expand its economy. The India government is predicting a 9% growth in GDP in 2010–11. Canada and India signed a Cooperation Agreement on Science and Technology in 2005. Technology transfers specifically with an aim at commercialization benefits both countries as it enables researchers to use R&D strengths to develop products that can be eventually marketed in both countries. For India, S&T collaboration is very desirable given the opportunity for technology transfer because it is seen as an advantage to generate economic growth and prosperity.

India currently has S&T cooperation agreements with 80 countries of which 60% are deemed operational. For the India government, Canada is considered an important partner however the extent of this partnership is dependent on Canada's willingness and level of engagement as India is willing to match Canadian funds. Before Canada signed its collaboration agreement there was little interest in such an agreement. This was partly because Indian researchers and scientists looked to the USA as the penultimate destination for innovation; therefore, Canada was not viewed as a strong contender.

According to the Indian Department of Science and Technology (DST), Canada is still evolving as an innovation partner and India would like to see Canada move into the top tier of countries collaborating on science and technology. The top tier represents 15 countries including the USA, UK, Australia and Germany. As an example of top tier contributions to S&T collaborations, Australia announced in 2009–10 a $100M AUS fund and signed an agreement with GITA for a $40M AUS program. Currently ISTPP has committed $6.75M CDN for its program with India.

Brazil: ISTPP in context

Brazil has identified S&T innovation as an important economic gap which needs to be reduced in order to enhance its social and economic prosperity through international engagement. Brazil's Ministry of Science and Technology has prepared a 3–year action plan (2007–2010) to enhance S&T engagement by increasing its R&D investment to 1.5% of its GDP in 2010. By comparison, in 2006, Canada invested 1.94% of its GDP in R&D, USA invested 2.66% and Sweden 3.74%.⁽¹²⁾ One key driver for this type of investment is that there is a strong correlation between the wealth of countries and investment in R&D. Brazil therefore, is committed to increasing its S&T R&D investments through international engagement and already has S&T agreements with Germany, France, USA, Britain, China, India, Korea and South Africa. Bilateral S&T agreements have accrued direct benefit to Brazil in nanotechnology, bio–technology and aerospace sectors.

However, one of Brazil's main strategic interests is to forge closer ties with Africa and while this does not preclude Canada's involvement, it would suggest that now could be an opportune time for Canada and Brazil to strengthen their engagement towards developing mutual interests in S&T. A significant aspect in advancing this relationship will be the implementation of the Canada–Brazil S&T Cooperation Agreement which was ratified by the Brazilian Parliament in March 2010. Brazil's Ministry of Science and Technology has expressed openness to engage with Canada; but, funding R&D innovation is new for Brazil and while Brazil's S&T action plan is geared to increasing R&D capacity and technology transfer, mutual collaboration with Canada through ISTPP would need to be explored further⁽¹³⁾ to assess its relevance to Brazil whose interest is in internationalizing their technology by gaining access to the North American market.

Sufficiently incremental to Canada's S&T cooperation relationship

The Government of Canada's S&T R&D relationship with Israel has for the most part taken place through the Canada–Israel Industrial Research and Development Foundation (CIIRDF). CIIRDF is a jointly funded organization that has received funding at $1.0M CDN per year since it was established in 1993. It supports up to 50% of the costs of an approved joint R&D project and also organizes and delivers a number of "match–making" events designed to bring together Canadian and Israeli companies to explore partnership potential. Federal government departments like NSERC and AAFC have also been involved in funding collaborative exchanges and research projects with Israel, such as the "Canada – Israel Cooperation in Agricultural Research Program" which fosters research between agricultural scientists from Canada and Israel. This program facilitates joint agricultural research by scientists in predetermined areas of high priority to both countries and provides agricultural researchers and institutions with opportunities to exchange scientific information and to facilitate exchange of scientists. Canadian provinces such as Ontario and Quebec have also signed agreements with Israel.

In April 2005, the Government of Ontario signed a memorandum of understanding on industrial and technological development cooperation with Israel. Ontario established a $3.0M CDN fund for three years to support R&D projects for Ontario companies, for which CIIRDF provides operational management. The Ontario–Israel fund is expected to be renewed in May 2010 at perhaps higher levels of funding. In December 2007 Quebec signed an agreement to cooperate in the areas of trade and commerce, as well as science and technology, health and education. A formal science and technology cooperation agreement is under negotiation along with the establishment of a fund to support R&D projects for Quebec companies.

Canada's S&T relationship with China is multi–faceted. While ISTPP provides a direct mechanism for delivering on the collaboration agreement between the two countries, there are other S&T collaborative arrangements that are in effect at the same time. British Columbia, Alberta, Ontario and Quebec each have collaborative arrangements between R&D organizations in Canada and counterparts in China. British Columbia, for example, has a $1.5M CDN fund to support S&T arrangements.⁽¹⁵⁾ Other governments departments are also involved such as NSERC, NRC–IRAP and AAFC.

These federal and provincial programs however, appear to operate in isolation from one other. This heightens a need for increased communication and co–ordination to reduce the potential for confusion among these various Canadian programs that promote collaborative research between the two countries. Evidence of confusion was found when a former Canadian Minister's visit to Beijing was to announce support for approved ISTPP projects. But, before the Minister's announcement, the BC government announced a list of projects supported from its own $1.5M CDN fund to support collaborative S&T research, some of which were to be announced by the Federal Minister since they were ISTPP projects. This meant that the Minister's announcement had to be quickly modified to accommodate BC's. With various S&T programs in effect at the same time, including those with provinces and other government departments promoting collaborative research and without the benefit of a centralized body coordinating the whole–of–Canada's involvement in S&T collaboration with China, a potential for confusion among programs is heightened. The Chinese government would therefore prefer that the Government of Canada take a more active role in leading the S&T relationship between the two countries. It is interesting to note that in Israel, the Canadian Embassy and CIIRDF regularly hold monthly meetings to increase awareness of each other activities on S&T in an effort to reduce confusion and promote coordination.

Canada's S&T relationship with India is also multi–faceted. While ISTPP provides a direct mechanism for delivering on the collaboration agreement between the two countries, there are other S&T collaborative arrangements in effect such as those with other government departments including NSERC, NRC–IRAP, and AAFC. Canadian provinces are also active in India, with engagement from BC, Alberta, Ontario and Quebec. Ontario has signed an agreement with ISTPCanada to pool its funds with ISTPCanada and ISTPCanada is in the final stages of discussion with Alberta to manage their S&T collaboration projects with India and other priority countries identified in the ISTP program. ISTPCanada has also signed a MOU with the Department of Bio–Technology (DBT) to support collaborative research projects involving the biotech industry. ISTPCanada is pooling funds from two (2) Canadian provinces ($1M CDN from BC and $3M CDN from Ontario) for its partnership agreement with DBT for BC, and DBT and DST for Ontario. Quebec has indicated that it will not pool its funds with ISTPCanada and will forge a separate agreement with DBT. DBT has also signed an agreement with NRC for contributions of $5M CDN to support already focussed projects.

Canada's S&T relationship with Brazil is still in its nascent stages. In 2009, Canada–Brazil was active on eleven (11) S&T activities with Quebec, Ontario, British Columbia, Environment Canada, Natural Resources Canada, and Agriculture and Agri–food Canada. ISTPCanada co–hosted one of these events, i.e., Brazil–Canada Cellulosic Bio–fuels Workshop; but was involved to some extent with the Ryerson– led workshop on Ontario Universities Opportunities Session; Brazil–Canada Chamber of Commerce Opportunities Session. The amount of dedicated federal government funding through the ISTPP for Canada's S&T relationship with Brazil nevertheless remains modest in relation to the provincial and other federal government departments involved.

Importance of ISTPP Funding to R&D Organization Recipients

The evaluation used an online survey in combination with face–to–face and telephone interviews with Canadian R&D organizations and their counterparts in the partner countries to collect data on their perceptions regarding the importance of ISTPP funding for their R&D projects. Graph 2 represents the responses from the Canadian and Israeli companies that received ISTPP–CIIRDF funding between 2006 and 2009 for their R&D projects. Graph 3 represents the responses from Canadian R&D organizations that received funding between 2006 and 2009 for their R&D projects with China and India.

Graph 2: 2006–2009 CIIRDF–Funded R&D Projects

Graph 3: 2006–2009 ISTPP–Funded R&D Projects

Relatively high proportions, 60%–80% of the funding recipients, characterized ISTPP funding as either very important or important in relation to their own financial resources, without which they would not have been able to undertake the R&D in a timely manner or at all. The fast pace of S&T innovation and rapid changes in market conditions are critical conditions that necessitate the timely funding of R&D, especially in sectors such as information communications technology; in where many companies, especially SMEs, do not have the required capital when most needed.

In relation to other government sources, Canadian R&D organizations replied that the ISTPP was the only Government of Canada program that supported international partnerships in R&D and had few conditions attached relative to funding for domestic R&D programs. By way of comparison, Israeli companies replied that their government had several bilateral R&D funding programs with other countries which they could access, as well as funding from regional programs such as the Seventh Framework Program (FP7) or EUREKA. It is also important to note that while several Israeli companies said that the ISTPP–CIIRDF administrative burden was disproportionately high for the amount of funding, they nevertheless did not see that as a strong enough deterrent given the potential of gain through international collaboration.

The perceived importance of ISTPP funding of R&D projects in relation to accessing venture capital received mixed responses from Canadian R&D organizations and their counterparts. In the range of 30%–40% of the respondents did not view ISTPP funding as an alternative to venture capital and therefore the comparison was not applicable. Others viewed that the role of venture capital is much needed to fund R&D commercialization costs, which in some cases can be quite significant. Such costs act as a barrier for success for SMEs and even larger companies without sufficient working capital or access to low interest business financing. Several cases were noted where either the Canadian or Israeli companies had sought venture capital to commercialize their R&D without success. Therefore, in the absence of available venture capital funding, ISTPP support assumes greater importance in the potential for SMEs to reach successful commercialization, although ISTPP funds are recognised as being insufficient at present to influence commercialization costs. Twenty–five (25%) of Canadian R&D organizations considered ISTPP as an important alternative to venture capital.

8.2 Relevance Issue 2: Alignment with Government Priorities

Policy coherence with national strategies and priorities

The Government of Canada's International Policy Statement (2005) established a policy framework for international commerce which addressed the need to help Canadian business access markets and opportunities abroad. The role of innovation was frequently cited as the key to improving Canada's global competitiveness and productivity as it speaks to the ability to envisage and create new products or services, or to produce existing products cost–efficiently. A key component of the economic policy framework was to strengthen entrepreneurial capacity to move from innovation to commercialization. The need to optimise the significant investments made in building domestic S&T innovation capacity in the public sector was recognised, as was the importance of linking this capacity with international commercialization efforts for the economic benefit of Canadians. While it was recognized that SMEs represented a major feature of the Canadian economy towards creating jobs and economic growth, only slightly more than a third of such companies are considered directly involved in import and export activities and less than 20% were found to participate in global supply chains.

Actively encouraging university research and the commercialization of new technologies by Canadian business was viewed as key to enhancing collaboration among Canadian organisations in order to position Canada advantageously with the new economic powers like Brazil, Russia, China and India (BRIC). Among the various initiatives subsequently identified in the International Policy Statement, the following prophetic statement was found in which a commitment is made to, "enter into a new agreement, if possible, for bilateral research collaboration through the Canada–Israel Industrial Research and Development Foundation, and explore options for similar agreements for research and development collaboration in other markets, specifically India and China."⁽¹⁶⁾

Advantage Canada: Building a Strong Economy for Canadians (2006) set out the government's economic plan to improve Canadian prosperity. It identified high rates of public and private investment in research and innovation as one of the well known determinants for economic growth. In 2003, Canada's public expenditures on research and development as a percentage of Gross Domestic Product (GDP) was the highest among the G7 countries but despite the comparatively advantageous tax incentives to both large firms and SMEs Canada's business expenditures on R&D (BERD) was the second lowest. Advantage Canada proposed to transform Canada's tax regime and create a business environment that unlocks private investment to make Canada a leader in innovation. The majority of ISTPP stakeholders interviewed expressed the opinion that the program was not only coherent and aligned with the entrepreneurial advantage of the plan but, also contributed to creating an enabling environment for business investment in R&D.

In addition, not only did innovation assume a prominent role in economic policymaking in Canada, there was also a realization that a coordinated, coherent, "whole–of–government" approach would be required. The policy commitment made in Advantage Canada was to develop and release a new science and technology strategy which was published the following year titled, Mobilizing Science and Technology to Canada's Advantage (2007). While this S&T strategy has primarily a domestic focus it nevertheless acknowledged that, "Canada must be connected to the global supply of ideas, talent and technologies – as a contributor and in order to adopt and adapt important innovations for the benefit of Canada."⁽¹⁷⁾ It sets out three S&T advantages designed to increase productivity and generate economic benefits which are summarised in the graphic presented in Figure 3.

Again, the majority of ISTPP stakeholders felt that ISTPP was primarily designed to enhance the entrepreneurial advantage by fostering a competitive business environment and by encouraging public–private R&D and commercialization partnerships. The ISTPP was also considered to be aligned with the knowledge advantage by focussing R&D on priority sectors and technology areas of national interest, as well as a human advantage in that it provided opportunities for highly qualified Canadian researchers to possibly re–connect with their countries of origin. As illustrated in Figure 3, both the knowledge and people advantage are considered to be in support of the entrepreneurial advantage.

Figure 3: Canada's S&T Advantages ⁽¹⁸⁾

Alignment with departmental priorities and outcomes

While the evaluation's review of ISTPP–related documents and interviews with ISTPP stakeholders confirm that other federal government departments have important S&T innovation mandates domestically, DFAIT is the lead department for international trade and commerce. As such, it is important that the ISTPP objectives and programming are aligned with DFAIT's strategic outcomes and expected results as stated in the 2009–10 Report on Plans and Priorities (RPP).

The evaluation examined the ISTPP Logic Model to assess whether the program expected outcomes were reasonably defined and achievable given the financial resources allocated ($18.5M⁽¹⁹⁾ CDN), programming scope (four countries and multiple sectors) and the five year timeframe. It was determined and confirmed by the vast majority of ISTPP stakeholders that the outcomes were well defined and aligned well with the Department's strategic outcomes; many however were not realistically achievable for a start–up program.

Some immediate outcomes, such as increased revenues/royalties and reduced costs accruing to Canadian companies involved in collaborative R&D Projects, assumed that a sufficient number of projects would be launched, completed and successfully commercialize their products, services or processes within the given timeframe. With the exception of perhaps the information communications technology sector, the usual timeframe for R&D projects to complete this cycle and begin generating significant revenues would take longer than 3–5 years, especially given the international partnership context of these R&D projects. Consequently, the achievement of the expected intermediate outcomes, such as, wealth creation for R&D organizations and workforce impacts in terms of permanent HQP jobs created or maintained, was determined to be unrealistic within the allocated timeframe of the program. The remaining expected outcomes related to promoting Canada as a partner for innovation, raising awareness, increased networking, consortia building were however considered by the evaluation and the majority of ISTPP stakeholders to have been realistically achievable.

8.3 Relevance Issue 3: Consistency with Federal Roles & Responsibilities

Appropriate representation of federal S&T organizations

DFAIT and Industry Canada are the primary federal government departments involved in the governance and oversight of the ISTPP. While not a science–based department/agency (SBDA), DFAIT contributes the unique "international perspective" to provide leadership in Canada, bilateral relationship building expertise and business services abroad through its network of S&T counsellors abroad. It also coordinates with Export Development Canada (EDC) and the Canadian Commercial Corporation (CCC) to ensure that Canadian companies have the necessary financial and contracting capabilities to access markets and commercialize their innovative technologies, products and processes. Industry Canada, alternatively, is considered a SBDA because it coordinates Canada's science and technology policy, conducts research in information and communications technology and promotes a strong science culture in Canada through its various portfolio research centers, institutes and programs. Representatives from NSERC, NRC, AAFC, and NRCan and an external policy advisor comprise the ISTPP Steering Committee, which is co–chaired by DFAIT and IC. These departments, including SBDAs like AAFC and NRCan, each have mandates to promote S&T innovation in Canada and ISTPP stakeholders viewed them as relevant to the governance and oversight of the program. Several ISTPP stakeholders also noted the absence of the Canadian Institutes of Health Research (CIHR) and suggested to consider including them given the significant number of R&D projects funded in the health and life sciences sector. The Government of Canada publication, Science & Technology Partnerships, presents the organization of federal S&T in Canada.

Figure 4: Federal S&T Department and Agency Partnerships ⁽²⁰⁾

While changes have occurred since 2006, most notably the disappearances of the National Science Advisor position, PM Advisory Council on S&T and Council of S&T Advisors as well as the Canada Foundation for Atmospheric and Climate Sciences,⁽²¹⁾ the structure remains current today. Most of the departments/agencies, which have S&T mandates, also have publicly funded laboratories to conduct research which is in the interest of Canada; they also sponsor numerous grant and contribution programs which target R&D organizations, both public and private, as beneficiaries.

Among most ISTPP stakeholders and some Canadian companies, there was a commonly–held perception of a fragmented S&T sector. The evaluation reviewed federal government publications which provide the policy framework and guidance for S&T and identified numerous statements which emphasis a need for all federal departments and agencies to work together and with public and private R&D organizations to connect domestic innovation efforts with international commercialization. The S&T Strategy for example makes the following statement, "Greater cooperation and alignment among federal programs, and between federal and provincial programs, could generate efficiencies and increase the effectiveness of these efforts."⁽²²⁾ Several ISTPP stakeholders viewed ISTPP as an opportunity to bring together key SBDAs and S&T parties with a common objective: to collaborate and create the needed synergy to facilitate the international commercialization of Canada's S&T for the benefit of Canadians. As such, ISTPP was considered to have some characteristics of a horizontal initiative and lends itself to being managed as one; however, ISTPP stakeholders were very much divided on whether it was a horizontal initiative given the modest resources that have been allocated to the program.

Complementarities or duplication

The evaluation examined the extent to which there are complementarities or duplication between the ISTPP and other Canadian S&T innovation programs. The following ISTPP design characteristics were considered important for comparison purposes: its international mandate, its funding of partnership development activities, its funding of either Canadian industry or universities involved in R&D partnerships, its funding of public–private R&D partnership projects on the verge of commercialization, and facilitating the creation of international partnerships with R&D organizations in priority countries. The inventory of similar S&T programs presented below was then developed through documentary research and stakeholder interviews.

Figure 5: Canadian S&T Programs Excluding ISTPP

• DFAIT's Going Global S&T Programprovides funding to Canadian researchers from private companies, universities and non–government research centres to participate in international R&D activities and development partnerships with key players in other countries.
• Industry Canada's Strategic Aerospace and Defence Initiative (SADI) is an example of domestic funding of strategic R&D and demonstration projects in Canada's aerospace, defence, security, and space industries.
• NSERC's Ideas to Innovation (I2I) Phase 2B program is designed to support technology innovation projects which are closer to the cusp of commercialization in an attempt to foster the commercialization of university research with an industry partner.
• The Canadian Institutes of Health Research's (CIHR) International Opportunities Program (IOP) offers two types of one–time funding grants, the Development/Planning Grant to initiate projects and the Collaborative Research Projects Grant to sustain collaboration.
• The Canada Foundation for Innovation's (CFI) International Access Fund is designed to foster international R&D partnerships between Canadian and international researchers and businesses.
• The NRC–Industrial Research Assistance Program (NRC–NRC–IRAP) provides a range of technical and business advisory services to Canadian–based small and medium–sized enterprises (SMEs) whose R&D projects are at the pre–commercialization stage. The program is delivered by an extensive network of 260 Industrial Technology Advisors (ITAs) in 100 communities across the country. NRC–NRC–IRAP has also been engaged for the past 10 years in R&D programming with Canadian companies in China and for 5 years with India.

Based on the aforementioned descriptions, many S&T organizations have international programs that share similar characteristics with the ISTPP, such as: DFAIT's Going Global which funds international partnership development activities; NRC–IRAP which funds industry R&D projects at the pre–commercialization stage; or NSERC which funds university R&D projects at the pre–commercialization stage. While ISTPP stakeholders acknowledged for the most part that these programs occupy the same S&T space as the ISTPP, many did not view this as duplicative but rather complementary. What apparently distinguishes the ISTPP from other Canadian S&T programs is its mandate to promote S&T internationally as the instrument⁽²³⁾ of the S&T Treaties. Several ISTPP stakeholders pointed to the importance of promoting S&T program integration, synergy and to pool resources together to support the ISTPP in fulfilling its international mandate. Canada's S&T Strategy also has underlined the need to streamline and consolidate the range of available funding mechanisms for S&T innovation support programs as several other countries, including Austria, Finland, Germany, and the Netherlands, already have done.

Appropriate Provincial involvement

The S&T Strategy was developed in consultation with the provinces and territories and a commitment was made by the federal government to work with them to improve commercialization outcomes. Among Canadian provinces, the largest provincial R&D investors are Ontario, Quebec and Alberta which together account for the vast majority of the expenditures. A review of ISTPP documents, internet research and stakeholder interviews reveals below a portrait of provincial interest in international R&D partnerships, the ISTPP and its delivery organizations.

Ontario

While international trade is a federal government responsibility these provinces have signed MOUs and agreements to promote collaborative S&T cooperation activities with countries such as Israel and China. Subsequent to the signing of such an agreement with Israel, the Government of Ontario contributed $3.0M CDN to CIIRDF which was matched by Israel to give effect to the agreement. The funds are used to support Ontario companies that did not receive support through the ISTPP national call for proposals and evaluation process. To date, four (4) R&D projects have been funded with several others pending. CIIRDF acts as an operating agent for the Ontario Israel Research and Development Monitoring Committee (OIRDMC) which was established to monitor the performance of the program. The OIRDMC Chair reports directly to the Deputy Minister of the Ministry of Research and Innovation and the Israeli Chief Scientist. In 2009, the Government of Ontario launched a competitive bidding process seeking a delivery organization for a joint $6.0M CDN S&T collaboration fund with India that had been previously announced. ISTPCanada was awarded the $3.0M CDN contract as the Canadian delivery organization for the Ontario–India Fund. The Fund's program is composed of PDA activities designed to generate interest among companies to submit R&D proposals, as well as an Ontario only call for proposals for joint R&D projects.⁽²⁴⁾

British Columbia, Alberta and Quebec

ISTPCanada has actively solicited the engagement of the provinces in international S&T cooperation activities by organizing information sessions and meetings with provincial representatives in British Colombia, Alberta and Québec. In 2008, the Government of British Colombia awarded ISTPCanada a $1.0M CDN contract to implement an S&T collaboration program with India to undertake PDA activities and R&D projects.⁽²⁵⁾ Similar to the Ontario–CIIRDF arrangement, the funds are used to support B.C. companies that did not receive support through the ISTPP national call for proposals and evaluation process. One R&D project has been approved to date and several PDA activities sponsored, most notably the Canada–India B2B Partnership Summit in Life Sciences held in January 2010.

In 2008, the Government of Alberta entered into an agreement with ISTPCanada to deliver an S&T cooperation program with China which also included both a PDA and R&D project component. The program was funded through contributions from the Government of Alberta, Western Diversification (WED) and the ISTPP each for an amount of $400K CDN for a total of $1.2M CDN. A dedicated call for proposals for Alberta companies was held and five (5) companies were awarded R&D projects. Alberta Advanced Education in Technology and ISTPCanada are presently in negotiations for the Alberta Global Technology Fund which would expand S&T cooperation programming to India, Brazil and Israel. A similar joint funding model would be used with expected total contributions of approximately $3.0M CDN.

8.4 Relevance Issue 4: Effectiveness of Governance Processes

Effectiveness of governance structures

The evaluation assessed the effectiveness of ISTPP governance structures and processes as defined in the program's approval documentation. The composition, oversight and monitoring responsibilities of the ISTPP Steering Committee were described in a consistent manner in several key documents including the Terms and Conditions (T&Cs), TBS submissions, Audit, Risk and Accountability Framework (ARAF), and in the respective contribution agreements for the delivery organizations.

The evaluation found that the S&T Treaty Joint Committee structures were not referenced in any of the approval documents. The evaluation had expected a reference because ISTPP was created as the instrument to promote industrially–oriented international partnerships under bilateral science and technology cooperation agreements with the priority countries.⁽²⁷⁾

The composition and operational management responsibilities of the delivery organization's Board of Directors (BoD) were also presented in a consistent manner in the ARAF and in their respective contribution agreements. The respective roles and responsibilities of these two governance structures were consistent with the principles and objectives of using "arms–length or third–party" delivery organization to implement the ISTPP. The respective contribution agreements also made provisions for:

• The ISTPP Steering Committee's participation on the delivery organization's BoD;
• DFAIT could be represented on the CIIRDF's BoD because of CIIRDF's bilateral mandate and in due regard to the Value and Ethics Code for the Public Service; and,
• In the case of ISTPCanada, government officials could participate as observers at BoD meetings.

These provisions enhanced ISTPP's Steering Committee's opportunity to exercise its oversight responsibilities while respecting the delivery organizations' decision–making authority over operational management, such as, the selection and approval of proposals for PDA activities and R&D projects.

Effectiveness of governance processes

While the role and responsibility of ISTPP Steering Committee was presented and discussed at its inaugural meeting, the ISTPP stakeholders interviewed held mixed understandings of the purpose of the Steering Committee. Some believed it to be an ad hoc meeting of interested parties, others referred to it as a sounding board or advisory committee, while only few stakeholders acknowledged its role as a Government of Canada oversight committee. The evaluation found the Steering Committee was infrequently engaged in strategic planning discussions such as on the involvement of the provinces. They were minimally consulted for advice on programming issues such as the involvement of Sector Coordinators and NRC–IRAP–ITA in proposal evaluations and they were not provided with adequate documentation to exercise their performance management oversight role. Several stakeholders, for example, misunderstood that the ISTPP Steering Committee through the ISTPP Secretariat, and not the respective BoDs, had oversight responsibility to ensure that the delivery organizations operated as per their respective contribution agreements. Some also expressed discomfort with attributing this level of accountability to the ISTPP Steering Committee and its members.

Based on available meeting records and stakeholder interviews, the ISTPP Steering Committee membership was at the Director General level but evolved and fluctuated considerably between December 2006 and December 2009. There were 6 meetings held during this period for the following reasons: 1) a launch meeting to approve the selection of ISTPCanada, 2) to meet with ISTPCanada representatives to discuss how weaknesses identified in their proposal could be addressed, 3) to receive a briefing from ISTPCanada on the first year of operations, 4) to receive a second briefing from ISTPCanada on the second year of operations, 5) to receive a third briefing from ISTPCanada on the third year of operations, and 6) to discuss the renewal/expansion of the ISTPP.

Notwithstanding the yearly progress briefings, the evaluation did not find any evidence that the Steering Committee received annual audited financial statements and activity reports from the respective delivery organizations as per the T&C reporting requirements.⁽²⁸⁾ The committee was briefed on the conduct of the 2008 formative evaluation, although there was also no indication from the subsequent meeting records or stakeholder interviews that the evaluation report was tabled for discussion. Two decisions were recorded in meeting records: 1) approval of the CIIRDF–Precarn proposal for the delivery of the ISTPP in China, India and Brazil, and 2) approval of the ISTPCanada project approval process, both decisions taken as per the ISTPP Terms and Conditions during the first two meetings.

Role of S&T Sector Coordinators

The 2008 formative evaluation of the ISTPP recommended that the role of the S&T Sector Coordinators be clarified in relation to the ISTPP since there was some confusion as to whether they should be involved in one or more of the following activities: supporting proposal applicants at the proposal preparation stage, assessing or facilitating the assessment of proposals, organising partnership development activities and/or promotional activities in general.

The summative evaluation was asked to revisit this issue since several stakeholders felt that the involvement of the S&T Sector Coordinator was sub–optimal and remained to be clarified. The evaluation reviewed the S&T Treaty documentation for each country to identify references to the role of the S&T Sector Coordinators and found that while the position is not cited in the S&T Agreements themselves, the functions of the Joint Committee are in each case, most notably to identify priority forms of S&T cooperation activities for each calendar year. The evaluation reviewed the Joint Committee meeting minutes. It found S&T Sector Coordinators are appointed by the Co–chairs to lead thematic working groups that are tasked:

• to identify specific technology areas of common interest and expertise; and,
• to identify concrete research projects.

As such, the S&T Sector Coordinators are in principle linked to the governance and administrative structures of the S&T Treaties, recognising that the Joint Committee structure and process evolve differently for each partner country.

The ISTPP Secretariat acted on the 2008 formative evaluation recommendation to clarify the role of the S&T Sector Coordinators. A workshop was held with the S&T coordinators in November 2008 at which time role clarification was discussed at considerable length. Shortly thereafter, in February 2009, an S&T Sector Coordinator Role document was widely disseminated first as a draft for comment and then formally. It was distributed by the designated Assistant Deputy Minister (DFAIT) to his counterparts in the other relevant S&T organizations. This document highlighted the importance of the S&T Sector Coordinator role and suggested that their functions be incorporated into their job description to provide recognition for their role and be allocated internal resources to support their function.

Stakeholder interviews also indicated that the ISTPP processes for identifying and undertaking PDA activities and R&D projects is not fully integrated with the key functions of the S&T Sector Coordinators; that is, to contribute to the delivery of the S&T collaboration under the S&T Agreements. This was viewed as problematic given that the ISTPP was approved as the dedicated instrument⁽²⁹⁾ for these agreements. Interviewees offered the following explanations for this apparent disconnect:

• the Joint Committees and their working groups are still evolving and have provided limited guidance to date;
• the role of the S&T Sector Coordinators in providing strategic and technical guidance to the Joint Committees and the ISTPP is not integrated into their formal position descriptions and therefore not recognised by their organizations' management;
• S&T Sector Coordinators have limited time and resources to be personally engaged in the ISTPP or to engage their scientific personnel due to the financial constraints under which their organizations are operating;
• federal S&T organizations are reluctant to allocate human and financial resources unless internationalising S&T is a part of their overall plans and commitments (i.e., MRRS/PAA); and,
• the ISTPP delivery organizations have not actively communicated with or engaged all of the S&T Sector Coordinators, other than to request assistance in the assessment of funding proposals.

Interview data revealed that the latter issue apparently has not been resolved to the satisfaction of a number of stakeholders. The evaluation heard that without resolution there is a risk in losing the involvement and support of the S&T Sector Coordinators until adjustments are made to ensure a better integration of the S&T Sector Coordinator role in the planning and delivery of the ISTPP.

Performance management practices

Advantage Canada (2006) emphasized the importance of rigorous and objective measurement of R&D investments with a focus on results and improved accountability, as does the S&T Strategy which states, "Canada's federal government will increase its accountability to Canadians by improving the way that we measure and report the results of federal S&T expenditures."⁽³⁰⁾ The evaluation assessed the extent to which standard practices in performance management were used during the implementation of the ISTPP in terms of establishing baseline data, monitoring and reporting on expected outcomes and risks as required by the T&C and defined in the ARAF. Based on the various Excel files received during the course of the summative evaluation, it is clear that the delivery organizations have implemented a system for tracking partnership development activities and R&D projects. The latter are tracked from the date of an expression of interest (EOI) to payments to the R&D organizations upon receipt of the required technical reports and the submission of commercialization reports.

Annual audited financial statements and activity reports were submitted to DFAIT in compliance with the method of payments requirement contained in each contribution agreement. A content analysis of the 2009–10 Activity Reports confirms that progress was reported on the program objectives and the status of all approved PDA activities and R&D Projects for each country as required by the contribution agreements. However, these reports did not integrate the available progress tracking information on R&D projects and no variances in program implementation or adjustments were reported. These were found to be inconsistent with the ISTPCanada cash flow management issues identified in the preliminary recipient audit.

The evaluation found little evidence that performance and risk monitoring systems had been established to capture and document the outcomes of the ISTPP and to identify and mitigate risks, as had been previously recommended by the 2004 CIIRDF summative and 2008 ISTPP formative evaluations. Some stakeholders felt that it was premature for ISTPCanada to report on outcomes given the short time it has had to become fully operational, while the same would not apply to CIIRDF which has successfully launched (11) and completed (9) a significant number of R&D projects. In the absence of immediate outcome data available from the delivery organizations, the summative evaluation had to undertake primary data collection activities in order to assess performance for this evaluation report on ISTPP.

8.5 Relevance Issue 5: Effectiveness of Delivery Models Employed

Appropriateness of the alternate service delivery model

The summative evaluation was asked to re–asses the effectiveness of using a third–party arms–length organization to implement the ISTPP. This issue was first examined in the 2008 formative evaluation which prepared a comparative analysis of strengths and weaknesses. It reported that "ISTPCanada, as a third–party delivery agent, appears to be the most cost–effective way to deliver the program."⁽³¹⁾ The current summative evaluation, while not disputing the earlier finding, found that there is a strong need to further study the role and cost–effectiveness of third–party delivery agents for ISTPP. Evaluation interviews in partner countries suggested the mandate for the delivery agent should be operational and its cost–effectiveness determined by its efficiency to deliver programming without adding more cost than value.

The 2008 formative evaluation noted that the ISTPP T&C, which were prepared prior to the selection of ISTPCanada, had defined eligible delivery organizations as non–government organizations incorporated in Canada This suggests that an assessment may have been initially done during the program design phase, although no documentation was found to this effect. The present summative evaluation, conducted one year after the formative, based largely on stakeholder interviews, confirmed much of what was reported in the formative evaluation. Table 3 replicates the analyses on strengths and weaknesses of delivery models presented in the formative evaluation. Included within are additional aspects that were found during the summative evaluation.

Table 3: Analysis of Delivery Models

The significant additions to the table were as follows:

• The preferences expressed by partner country delivery organizations to deal directly with Government of Canada representatives;
• The perceived need to compensate federal S&T organizations for their contributions to the program (i.e., NRC–IRAP ITAs and S&T Sector Coordinators);
• The perceived need for sufficiently robust resident expert capacity such as an ITA Network or Resident Chief Scientist; and,
• The requirement to work within a 15% administrative cost structure.

In the absence of other bilateral agreements, the S&T Treaties have an important role in bilateral relationship–building. Several stakeholders expressed a preference that R&D bi–lateral relationship–building would be best accomplished government–to–government rather than through a third–party delivery agent. The reasons for this preference for heightened government involvement were:

• The reliance on the NRC–IRAP – ITA Network, estimated at a value of $1.0M CDN to date, was magnified as an important network especially during times of fiscal restraint when each federal S&T organization is managing its resources prudently.
• The perceived restricted in–house technical capacity as well as appropriateness of non–governmental representatives, to organize and lead partnering missions (PDAs) with the sole responsibility to interface effectively with international researchers was considered a constraint by some stakeholders and Canadian companies.

However, from the delivery organization's perspective a 15% administrative cost structure is considered a comparative disadvantage when the administrative and performance management expectations of the federal government remain constant. ISTPCanada has stated that during its evaluation review process, it includes broad communities of academics, industry experts, and reviewers identified by proponents to obtain R&D proposal evaluations.⁽³³⁾ ISTPCanada would also have resident expertise in some fields of science relevant to the S&T priorities to facilitate in the proposal evaluation process to determine eligibility. These factors could mitigate positively the use of third–party agents to deliver international programming such as ISTPP but, may not be sufficiently strong to mitigate the perception held by many involved in ISTPP on the need for a defined presence of Government representatives for leading bi–lateral relationship–building and partnering missions in support of S&T Treaties. The evaluation therefore found a strong need to further study and examine the delineation of ISTPP from its operational mandate on program delivery (including its proposal review process and to address the issue of cost–recovery when calling upon NRC–IRAP's network of experts).

Appropriate allocation of resources

The ISTPP is a modestly funded international S&T innovation program given that it is the dedicated instrument⁽³⁴⁾ for the four S&T Treaties with Israel, China, India and Brazil, especially when viewed against other countries with similar sized economies. For example, the 2009 Annual Innovation Report of the UK Department for Innovation and Skills reported that in 2008 British R&D organizations received at total of 909 M equivalent to 13.7% of the total Seventh Framework Programme for Research and Technological Development (FP7) funding.⁽³⁵⁾ The UK Technology Strategy Board⁽³⁶⁾ is the national contact point for the FP7 and reported in its 2008 Annual Report that British R&D organizations led 413 FP7 projects, and a total of 2,546 British organizations were involved in 1,545 projects.⁽³⁷⁾ Canada does not have a dedicated source of funds to foster S&T collaboration with EU countries and companies. This was identified as a major barrier to future collaboration in the 2008 joint study, "Assessing the costs and benefits of a closer EU – Canada economic partnership."

Based on interviews conducted with representatives of China's MOST, China has S&T cooperation agreements with over 100 countries. Canada is in the second tier of countries China considers important for S&T collaboration. The United States (USA) represents about one–third of all S&T investments followed by the European Union, Russia and Japan. The second tier includes UK, Australia and, Canada. China has allocated approximately $100M USD for S&T partnerships of which Canada represents only a tiny fraction with ISTPP funding of $5.5M CDN.

According to India's Department of Science and Technology (DST), Canada is evolving as an innovation partner and they would like Canada in the top tier of countries collaborating on science and technology. The top tier represents 15 countries including the USA, UK, Australia and Germany. As an example of top tier contributions to S&T collaborations, Australia in 2009–10 announced a $100M AUS fund. Currently ISTPP has committed $6.75M CDN for its program with India. The total amount of funding that has been allocated to the ISTPP to engage with these countries as Canada's dedicated instrument⁽³⁸⁾ for S&T cooperation is disproportionately small in relation to the investments by other countries.

The majority of ISTPP stakeholders indicated that ISTPP is severely underfunded and unsustainable at these funding levels. It was raised that partner country delivery organizations may lose interest due to high transaction costs associated with managing such a small amount of funding and, Canadian R&D organizations may also lose interest in a responsive program that can only fund less than 5% of the proposals received. A commonly–held opinion among those ISTPP stakeholders who review proposals is that a significant number of high quality proposals cannot be approved due to a lack of funds. An indication that this may very well be the case is the fact that some provinces are willing to use their own R&D funding to support unfunded proposals from their jurisdictions.

Appropriate allocation of resources

Total ISTPP contributions of $18.4M CDN to ISTPCanada and CIIRDF were made in accordance with the ISTPP T&Cs. Based on a total ISTPP contribution of $13.4M CDN for China, India and Brazil programs, final recipient audits as of March 2010 revealed that $5.2M CDN had been disbersed, while $8.2M CDN was left as an uncommitted balance.

By contrast CIIRDF, a more mature operation, was more balanced. Of the $5.0M allocated to CIIRDF for the Israel program, $4.4M CDN was disbursed as of March 2010⁽³⁹⁾ and $613K in operating expenditures. This amounts to a slightly higher total of $5.1M because of the total PDA and R&D disbursements. Table 4 presents the variation in ISTPP programming.

Table 4: ISTPP Actual Contributions⁽⁴⁰⁾

ISTPP stakeholders provided several reasons to explain disbursement delays. They were:

• ISTPP was a start–up program during this first phase of operation and needed time to establish agreements;
• ISTPCanada was a new organization created for the purpose of implementing the ISTPP and needed time to establish operations;
• ISTPP experienced delays in the S&T bi–lateral negotiation, signature and ratification of the S&T Treaties; and,
• Additional time was required to develop bilateral relations with partner country delivery organizations, to agree on common platforms for program delivery; to create, launch and implement a call for proposal process; and to negotiate the selection of proposals of common interest with the partner country delivery organizations.

The evaluation also examined the delivery of the R&D Project component of the program and found that there were significant delays in the launch of R&D projects following initial approval by the BoD of the two delivery organizations. The evaluation team calculated the time–lag between the BoDs approval date and the date of first payment made to the Canadian recipient organization based on the files provided by ISTPCanada and CIIRDF for all approved R&D projects from April 2004 to December 2009. These dates were selected to determine the efficiency of the ISTPP administrative processes based on the assumption that the Canadian R&D organizations would begin work on the project upon receipt of first payment. The average time–lag by partner country was as follows: Israel – 86 work days, India – 171 work days and, China – 238 work days.

The differential performance of the two delivery organizations (ISTPCanada and CIIRDF) can be attributed to the added complexities faced by ISTPCanada in ensuring that industry–university partnership agreements are signed on the Canadian side prior to the signing of contribution agreements, and the challenges faced by the Canadian R&D organizations of negotiating in detail the respective roles and responsibilities with the partner country R&D organizations prior to project launch, including the subsequent signing of intellectual property rights agreements. The evaluation has noted that in the case of China the average time–lag decreased to an average of 105 work–days based on three R&D projects approved in the second call for proposals; three of the six projects approved had not yet been launched at the end of the evaluation's reference period.

Coherence with sectoral priorities

The evaluation examined the appropriateness of ISTPP resource allocations with respect to sectors considered to be a priority for Canada. The S&T Strategy identified four broad sectors of focus which were also highlighted in DFAIT's Innovation Brochure; they are:

1. Environmental science and technologies;
2. Natural resources and energy;
3. Health and related life sciences and technologies; and,
4. Information and communications technologies.

Two studies had been conducted prior to the signing of the S&T Agreements with China and India, both of which made recommendations regarding specific technology areas of common interest and expertise. The evaluation reviewed the S&T Treaties documentation to identify the agreed–upon sectors and technology areas for S&T cooperation activities. A comparative analysis between these two studies and the results showed a high degree of sector coherence with Canada's sector priorities, with the exception of agricultural foods and bio–products with China and the aerospace sector with India.

The evaluation noted that partner countries preferred to be more specific with regard to the technology areas that they were interested in within each sector. For example, renewable or alternate energy technologies, e.g., hydrogen and fuel cells, as well as nanotechnology applications to life and environmental sciences. In the case of Israel the evaluation found no evidence of commonly agreed upon sectors of focus for S&T cooperation activities. Based on the delivery organizations' 2009–10 Activity Reports the evaluation analysed the approved R&D project distribution by sector and dollar, which are presented in Figure 6.

Figure 6: ISTPP R&D Project Distribution by Country and Sector, 2009–10 ⁽⁴³⁾

Appropriateness of the program delivery model used

As discussed in this evaluation report, the delivery of ISTPP has a number of characteristics which were not found combined together in any other Canadian S&T innovation program. Consistent with the S&T Strategy, partnership development between Canadian industry and universities to enhance the commercialization of R&D is facilitated through both of its main programming components: PDA activities and collaborative R&D projects. The ISTPP T&Cs provide the ISTPP with considerable flexibility in the choice of activities, projects and eligible recipients, the latter defined as, "Canadian companies, universities/colleges and other research and development institutes which operate and are headquartered in Canada." Consequently, it has had the flexibility to fund industry–industry or university–industry partnerships through either the Canadian company or the university depending on the proposal.

This flexibility also allows ISTPP to adjust to the circumstances and preferences of partner country delivery organizations in terms of determining the eligibility of the R&D organizations involved in proposals. ISTPP stakeholders supported the need for such flexibility to establish and maintain bilateral relations based on mutual respect for different circumstances and priorities. International R&D partnerships in the CIIRDF–Israel program are industry–to–industry as the norm given the advanced development of its private sector with considerable R&D capacity, while in China the circumstances are quite different where the still nascent private sector has little R&D capacity, a role that the university design institutes have traditionally fulfilled. Industry involvement in international R&D partnerships with Brazil is also considered a realistic requirement, while for India the pairing of universities and institutes of technology with industry is preferred and the norm in their ISTPP R&D projects. In contrast, it is interesting to note that in evaluation interviews with BIRD, it was found not to advocate university–to–industry collaborations given challenges in contractual arrangements between two very distinct and different focuses. Industry is focussed on profit and commercialization, university research less–so and, as a result, leads to challenges upon commercialization of science or technology.

As previously noted in this evaluation report, the linking of domestic S&T innovation capacity with international commercialization efforts for the economic benefit of Canadians has been a consistent Government of Canada priority despite changes in leadership, economic plans and renewed S&T innovation strategies. ISTPP stakeholders were of the opinion that while not all ISTPP objectives were being reached, ISTPP's the main objective was being achieved in the forging of international partnerships to contribute to an acceleration of R&D commercialization. This corresponds most closely to ISTPP's delivery, at least for the R&D project component which identifies and funds those proposals where partners are developing new technologies and products that are on the verge of international commercialization and seek funding and support to enter into the market in an effort to assess demand.

The need to use responsive, rigorous and competitive processes for assessing the potential of R&D projects to deliver value for Canadian taxpayers has also been a consistent Government of Canada theme since Advantage Canada was first published. The successful mobilisation of NRC–IRAP's vast network of Industrial Technology Advisors (ITA) has ensured a high level of scrutiny and due diligence on three assessment criteria: scientific/innovation, market opportunity and business potential.⁽⁴⁴⁾ Most ISTPP stakeholders believed that the assessment process is managed efficiently, fairly and transparently by the ITAs and the delivery organizations' BoDs.

The international peer review comes in the form of a parallel assessment process undertaken by the partner country delivery organization where the perceived level of effectiveness and transparency varies among partner countries. China is considered as the most opaque, while the bilateral CIIRDF–Israel model is considered the most transparent. However, the cost–effectiveness of the competitive calls for proposals and assessment model used for China, India and Brazil is put into question when the respective short lists upon comparison have had too few R&D proposals in common for both Canada and the partner country. This issue was encountered on the China program. It was found then that considerable federal government scientific and technical human resource hours are expended in implementing the competitive proposal assessment process, when at times less than a half dozen R&D projects were funded.

The ISTPP delivery organizations also implement PDAs, referred to as R&D Matchmaking in the approval documents, using the responsive call for proposals and assessment model, as well as a proactive programming approach. Based on a review of related ISTPP programming activities, it was found that PDA activities could be grouped into the following categories:

• International technology–based matchmaking service;
• Technology focussed workshops;
• Partnering missions;
• Distinguished International Speaker Series, and
• Conferences, seminars and roundtables.

The responsive call for proposals programming approach tends to be broad in scope and solicits proposals for seminars, workshops, lectures and other forms of information exchange activities. Since China's MOST does not fund PDA type activities, ISTPCanada partnered with the China Association for International Exchange of Personnel (CAIEP) as the partner country delivery organization for PDA activities. Similarly, ISTPCanada has worked with India's Department of Biotechnology (DBT) to manage joint PDA calls for proposals and other pro–active programming activities. While ISTPP is actively delivered with more than one partner and in various applications, these multiple partner country delivery organizations and/or calls for proposals (PDA and R&D Projects) contributed to confused and increased tensions in understanding ISTPP programming (as noted in evaluation interviews conducted in China, India and Brazil). Furthermore this multiplicity was viewed as heightening a risk of diluting the effects of bilateral collaboration when a small amount of funds is distributed too widely.

The proactive programming approach is used for the remaining activities, most notably the development of programming "platforms" focussed on specific technology areas or regions within a country. The use of partnering missions and roundtable summits has been used effectively to establish the bilateral relations and programming modalities to bring Canadian and partner country R&D organizations together. Although ISTPP stakeholders frequently cited the Yangling partnering mission and the Canada–Israel–China Trilateral Roundtable on Agri–innovation as an effective use of PDA resources, the call for proposals did not solicit as much interest as initially expected, possibly due to the trilateral requirement, thus the deadline for the call had to be extended.

ISTPP stakeholders held mixed views of the purpose of these PDA activities. Some believed that these activities were designed to create partnerships between Canadian R&D organizations, such as industry–university partnerships, while others believed that they served to brand Canada as a partner of choice for international collaboration in R&D innovation. While both are legitimate objectives, some stakeholders questioned the need for the latter given Canada's already enviable reputation for its S&T infrastructure, high quality research personnel and given steadily increasing international collaborations and joint authored research in natural sciences and engineering. For example, based on 2009 OECD indicators⁽⁴⁵⁾ on connecting to global research, foreign students account for almost 40% of doctoral students in Canada, which is one of the highest rates in the OECD, and Canada has the highest rate of internationally co–invented patents among the G7.⁽⁴⁶⁾ Nevertheless, Canada still underperforms on key commercialization indicators such as R&D funds invested in Canada from abroad and its technology balance of payments.

8.6 Performance Issue 6: Achievement of Expected Outcomes

Partnerships and Consortia Formed

The evaluation used an online survey in combination with face–to–face and telephone interviews with the Canadian R&D organizations and their counterparts in the partner countries to collect data on the number and characteristics of the partnerships and consortia that were formed. A comparative analysis of the three R&D project cohorts presented in Graphs 4–6 indicates a continuous improvement in the percentage of partnership agreements signed between the Canadian partner(s) and counterpart R&D organizations; however, the number of consortia agreements reported was insignificant. The evaluation did not examine the partnership agreement documents, but evaluation interview data and the narrative responses to the online survey characterized the general nature of these partnerships.

Graph 4: 2004 CIIRDF R&D Project Cohort Project Agreements Signed

Graph 5: 2006–2009 CIIRDF R&D Project Cohort Project Agreements Signed

Graph 6: 2006–2009 ISTPP R&D Project Cohort Project Agreements Signed

The roles and responsibilities of the companies involved in the 2006–2009 CIIRDF R&D project cohort were characterized by the respondents as complementary (70%) and important (50%), with only one "tangential" response. Upon closer examination of these characteristics, it was found that twice as many Israeli companies had the lead in conducting innovative scientific research and retained the intellectual property rights to the scientific discoveries than the Canadian companies. Both were equally involved in applying the science discoveries to product development and assembling or integrating product components. On the other hand it was found that almost twice as many Canadian companies conducted or provided market access research, and had the lead for marketing and distribution of the product, particularly in North America. Two joint venture and three supply agreements were reported, the latter involving two Canadian companies providing product components to their Israeli partners.

The roles and responsibilities of the R&D organizations in the 2006–2009 ISTPCanada R&D project cohort involving China and India were characterised by predominantly Canadian respondents as complementary (58%) and important (79%), with only one 'tangential' response. Upon closer examination of these characteristics, it was found that Canadian R&D organizations are equitably involved in conducting innovative scientific research and in applying the science discoveries to product development along with their international partners. While both China and India are identified as target markets for commercialization, many of the products under development are also intended for wider markets, like Canada and North America. These findings are provisional since these R&D projects are still nascent and the partnerships evolving.

R&D Capacity Building

The evaluation used online survey in combination with face–to–face and telephone interviews to collect data on the R&D capacity building benefits of the ISTPP accrued to the Canadian R&D organizations and their counterparts in the partner countries. The results of a comparative analysis of three R&D project cohorts (Graphs 7–9) provide some insight into the extent of outcome achievement in specific areas of need.

Graph 7: 2004 CIIRDF R&D Project Cohort

Graph 8: 2006–2009 CIIRDF R&D Project Cohort

Graph 9: 2006–2009 ISTPP R&D Project Cohort

When two CIIRDF R&D Project cohorts (2004 and 2006–2009) are compared over time, there is a significant improvement in the perceived non–financial contribution that CIIRDF has made on several aspects. The companies involved in the 2006–2009 R&D project cohort reported having a better understanding of technology market conditions, enhanced R&D partnering strategies and approaches, and having acquired and adopted innovative technologies or having developed them with partners.

ISTPP–CIIRDF was credited with making a significant contribution in these areas, as well as enhancing the R&D credibility and image of these companies. However, protecting intellectual property rights, overcoming barriers to market access, and establishing multi–partner industry–university consortia were areas of little significant improvement when comparing the two cohorts but these are important areas where ISTPP–CIIRDF was perceived to have made a minor or no contribution to the R&D capacity building of the companies involved.

Keeping in mind the nascent character of the 2006–2009 R&D projects with China and India, the perceived contribution of the ISTPP to capacity building is nevertheless quite significant in a number of relevant areas. The companies involved in this cohort reported having a better understanding of technology market conditions, enhanced R&D partnering strategies and approaches, being matched with the right partners and having made better strategic decisions and avoided difficulties. The ISTPP was credited with making a significant contribution in these areas, as well as enhancing the R&D credibility and image of these companies.

The remaining areas for capacity building were rated as "N/A" (not applicable) since many of these projects were either not operational or had not reached a stage where the respondents would be able to provide an informed opinion. There are two areas however where this should not have been the case, i.e., establishing multi–partner, industry–university consortia and protecting intellectual property rights. These are areas where the ISTPP would have been expected to have made a contribution since they are closely associated with R&D project preparation phase.

Enhanced International Reputation

As indicated earlier in this report, Canada, among G7 and OECD countries, is considered to have an enviable reputation for the high quality of its R&D infrastructure found primarily in its universities supported by an array of granting councils, foundations and institutions. Indicative of this reputation is Canada's high ranking among G7 and OECD countries for the number of jointly held international patents. Although the rate of co–authored papers and international collaboration in Canada has declined slightly with the United States since 1996 (–7%), it has increased with several other countries (see Graph 10). The most notable increase has been with China, whose share of Canada's international collaboration has increased significantly over the past decade.

Graph 10: Canada relative share of bi–lateral collaboration

Source: Calculated by Science–Metrix using the Scopus database

Note: Some values were corrected by Science–Metrix

While many of Canada's S&T Agreements with the European Union, France, Germany, California, Chile, Japan and South Korea are without dedicated funding, the extent to which the ISTPP with its modest $20M CDN allocation can be expected to influence Canada's reputation as an S&T innovation partner of choice with countries such as Israel, China, India and Brazil must be kept in perspective. Evaluation interviews with ISTPP stakeholders in Canada and with the partner country delivery organizations held the view that the ISTPP is severely under–funded relative to other potential S&T partners (e.g., Australia, United Kingdom and Germany).

The evaluation found that ISTPP partner countries have suggested⁽⁴⁷⁾ that they would be prepared to match any funding increases that the Government of Canada is prepared to allocate to their respective S&T cooperation agreements. Several Canadian stakeholders raised the concern that limited ISTPP funding allocated to each partner country may be perceived by the ISTPP partner country delivery organization as indicative of the relative importance of their S&T cooperation relationship. If such is the case as anecdotal data would suggest, then the impact of Canada's reputation could suffer rather than be enhanced by ISTPP's funding levels.

The ISTPP is viewed as an important instrument⁽⁴⁸⁾ to engage the partner countries in bilateral discussions on topics related to S&T cooperation and others, especially in the absence of bilateral trade and commerce agreements. However, partner countries to some extent view the use of an "arms–length" alternate service delivery organization to suggest that S&T bilateral relationship building may not necessarily be important for Canada.

Evaluation interviews also suggested the leadership role of ISTPCanada as the principal interface with the partner country delivery organizations has at times caused confusion. It has contributed to a perception that the ISTPP works independently from government as opposed to being an instrument of the Government of Canada. This would suggest of a need to clarify the responsibilities of the ISTPP and the role of ISTPCanada especially in regard to the roles of the Canadian Trade Commissioner Service through its S&T Counsellors at Missions.

Economic Benefits

The evaluation collected data on the economic benefits accrued to recipients resulting from their participation in collaborative R&D projects.⁽⁴⁹⁾ The 2004 CIIRDF Summative Evaluation included a sample of 18 CIIRDF R&D projects (involving 34 companies) funded prior to the ISTPP that were considered to have high commercialization potential. The summative evaluation assessed this potential through an on–line survey and interviews. Complete responses were received from 20 of 24 companies still known to exist. The remaining 11 companies' ceased operations merged or were acquired. This represents about 30% of firms in the sample which is not an unreasonable proportion given the volatile nature of risk in the SME sector in conjunction with the global economic downturn experienced since the launch of the ISTPP, most notably in the information communications sector, in where company survival rates were generally low.

Figure 7: 2004 CIIRDF R&D Project, by Organization Status

The performance indicator data in Table 5 is a proxy for the ISTPP since the current cohort of ISTPP funded collaborative R&D projects between Canada and Israel (2006–2009), has not had sufficient time to fully complete their project and commercialization cycle. Contrary to the projected estimates made in the 2004 CIIRDF evaluation, the cumulative commercial sales revenues attributable to the CIIRDF–funded R&D projects were found to be considerably less than expected and, showed significantly less economic benefits accruing to Canadian companies. The evaluation did not find evidence to support earlier project estimates that had specifically stated that "3 of 18 CIIRDF projects, initiated since 1999, will have to achieve commercialization" in order to be considered successful.⁽⁵⁰⁾ The commercialization figures for the Israeli companies were primarily from one Israeli company's response, with the exception of $7.0M CDN reported by another. The commercialization figures for Canadian companies were primarily from one Canadian company's response, with the exception of $1.0M CDN reported by another. These two companies were not involved in the same R&D project. It is interesting that the above figures were cross–referenced with the reported "royalties paid" using ISTPCanada project tracking data and while they align perfectly, the total amount of $264,889 paid in royalties is significantly lower than what would have been expected.

While one very successful R&D project of the sample of 18 may be a reasonable outcome given the high risks involved in funding R&D projects, it is the marked imbalance in the economic benefits accrued to the partners that warrants further examination. The combined annual projected revenues from now through to 2013 from the continued commercialization and licensing of any acquired or developed technologies directly or indirectly attributable to the R&D projects in the sample does not surpass $2.0M CDN.

Table 5: 2004 CIIRDF–Funded R&D Project Commercialization

* The value of joint venture equity investments made in the Israeli company was not disclosed for reasons of confidentiality.

The evaluation also surveyed the 2006–2009 cohort of ISTPP–funded R&D projects. Given the recent launch of R&D projects with China and India, it was expected that there would be no commercial sales, licensing revenues, cost savings or investments reported by any of the project partners which was in fact the case. However, several R&D projects with Israel were launched in the first few years of the ISTPP and consequently some project partners did report the successful commercialization of new technologies and products. The evaluation found there was an imbalance in the economic benefits accrued to the Israeli partners ($17.2M CDN) as compared to their Canadian counterparts ($1.3M CDN). It is also noteworthy that the Israeli companies collectively forecasted commercial sales and licensing revenues ($2.1 billion) over the next five years at a rate of six times higher than their Canadian counterparts ($316M CDN). While these figures appear to reflect a healthy optimism on the part of the innovators and entrepreneurs involved, they appear unrealistic if compared to the actual revenues generated from the 2004 CIIRDF–funded cohort of companies.

Notwithstanding the already known volatile nature of high–risk joint S&T R&D collaborations and in recognition of the challenge in implementing such a program internationally, the evaluation assessed the results of joint S&T R&D collaborations that preceded the 2004 CIIRDF project cohort to identify factors that could impact success. ISTPP defines success as commercialization.⁽⁵¹⁾ This examination was a longitudinal retrospective analysis that compared two pre–selected pre–1999 CIIRDF projects⁽⁵²⁾ which showed significant revenues to Canadian companies achieved at a much later point in time, on average 10 years later, after CIIRDF project closure. The analysis is based on several sources of information: CIIRDF project files and, attestations from former company officials including their replies to the summative evaluation's 2004 CIIRDF survey.

A Review of CIIRDF Project Files and Attestations

In addition to the 2004 CIIRDF cohort findings, the evaluation was provided with attestations on pre–1999 CIIRDF–funded R&D projects⁽⁵³⁾ which were not within the parameters of the purposive sample. Former company officials issued these recent attestations.⁽⁵⁴⁾ Two R&D projects were selected that had been supported by CIIRDF⁽⁵⁵⁾ to some degree at project initiation. Subsequently, over the course of the last decade, these Canadian companies successfully developed innovative technologies in the information communications technology sector which contributed to high sales revenues prior to the market decline in 2000.

The Canadian businesses involved in these two projects had grown substantially through the successful commercialization of their products. The reports attest to the two Canadian companies involved in one project realising combined revenue streams of $550M CDN⁽⁵⁶⁾ over a 10 year period, while the Canadian company involved in the other project was merged and then acquired in a business transaction worth $650M CDN over the same time frame. Both reports attested to the instrumental role of CIIRDF in facilitating partnerships and necessary access to financial and legal expertise in the late–1990s.

However, in one of the two projects CIIRDF funded a start–up company comprised of two companies which when merged with another company, dissolved its Israeli presence, while still active in CIIRDF. Given that the initial start–up company owned both companies, it was able to integrate technology it developed in Israel into its product line. CIIRDF's contribution was discontinued due to the nature of the merger in that the Israeli firm was dissolved. Records show that CIIRDF agreed to absorb a loss of 25% of the total contribution amount paid, while the new merged company repaid 75% to CIIRDF. This was repaid in short order, one year after the merger, absolving the legal claim to any subsequent profit from commercialization.

This repayment approach was external to the parameters of the existing provisions in the contribution agreements and as a result, it may suggest a need for further analyses of any additional provisions needed in contribution agreements with companies to secure to the greatest extent possible potential economic benefits from secondary or tertiary partnership arrangements in terms of contribution repayment/royalties. As it stands now, the relative contribution of CIIRDF funds to the overall net value of eventual commercial sales as presented in the attestations is confounded given that the initial CIIRDF contribution was repaid before actual commercialization and hence, no obligation to acknowledge future success. Another issue put forward by this case is to consider how much success should be attributed, or officially recorded, to CIIRDF funding (or ISTPP) given that it is possible that these companies may have been as successful commercially without CIIRDF funds.

A Review of Evaluation Survey Replies

To more precisely delineate the attribution of CIIRDF funding over time towards commercialization, the evaluation team provided the authors of these attestations the opportunity to complete the same on–line survey⁽⁵⁷⁾ provided for the 2004 CIIRDF cohort. On–line surveys were completed for both of the selected pre–1999 CIIRDF projects. The replies are treated separately in this report to maintain the integrity of the evaluation by not introducing bias to the purposive sample.⁽⁵⁸⁾

Based on the survey replies, it was found that the total duration of projects was relatively short: one project was 2 years, while the other was one year. In both projects, partnership agreements were signed but neither project reported additional spin–off R&D projects. It was indicated that CIIRDF funds were important for these projects because CIIRDF was the only support available for collaborative projects, which are usually more complex and costly relative to other funding sources that provide for direct project execution by the company. CIIRDF awarded grants in the amount of $600K to each project, which was disbursed in full to one of the two projects. Royalty payments were issued for one project and for the other project, the grant was repaid because the non–Canadian partner was dissolved.

In addition, it was reported that CIIRDF provided non–financial assistance such as partnership development to facilitate relationship–building and, through these support services, significant contributions were reported towards understanding technology, market conditions, and finding the right corporate partner. The Canadian companies signed 11 commercial sales agreements; representing $100K for one company which was later sold and $100M for the other. The latter noted that the product had been augmented over time and it would be difficult to know the approximate sales revenue generated from the CIIRDF project. There was no report of licensing or purchasing agreements signed to source innovative technologies as a direct or indirect result of the CIIRDF–funded projects. A joint venture of $1M CDN was reported and one subsidiary was established to produce goods and/or services for export as a direct or indirect result of the CIIRDF–funded project. Only a single job was retained.

Summary Analyses

While there are wide differences in reporting between the attestations and their replies to the on–line survey, especially when reviewing commercialization, both data sets (attestation and survey replies) acknowledge the complementary role of CIIRDF in providing assistance in the preliminary stages of technological development. Although the survey replies showed the difficulty in ascertaining the nature and extent of direct attributable CIIRDF results, it is logical to assume that synergies were created and compounded over time in a business environment, whether it be through secondary or tertiary partnering in nature, which when taken together, could yield a degree of economic benefit.

Some Other Practices

The evaluation noted in its stakeholder interviews with other foundations and granting agents active in supporting international joint S&T R&D collaborations the challenges in defining success when dealing with high–risk industry–based R&D. Included in these challenges is whether commercial success should include provisions for repayment or royalty and at what stage in the business cycle repayment be considered. Most other foundations or granting agents interviewed for this evaluation factor in a repayment or royalty provision initiated upon commercialization. BIRD focuses on industry–to–industry R&D collaborations and for example, assumes that 30% of its grants will yield commercial success and its general rule is whichever company commercializes first is responsible or supports repayment.

Others view SME company failures in commercialization as the norm with the value being the overall process to form partnerships and continued engagement. As a granting agency, they would continue to consider proposals of merit, provide on–going support to those projects if merit is sustained, and absorb losses. The evaluation did not find an international standard or benchmark on how to define success for joint S&T R&D industry collaborations based on the few interviews but, these interviews and the results from the longitudinal analysis of the CIIRDF experience do strongly suggest a need for further study on what would be considered success for ISTPP programming.

Workforce Impacts

The evaluation, based on survey and interview data with Israeli companies and Canadian companies, assessed the workforce impacts of ISTPP–funded R&D projects. Sixteen (16) Canadian companies involved in R&D projects with Israel reported 63 full–time equivalent highly qualified personnel (HQP) employed on their projects, while 15 Israeli companies reported 101 full–time equivalent highly qualified personnel employed on the same projects. Eight (8) Canadian R&D organizations, which reported on their ISTPP–funded R&D projects with China and India, intended to employ 24 full–time equivalent highly qualified personnel on their projects.

The evaluation also found that ISTPP had contributed to supporting about 3–4 HQP positions in Canadian companies and organizations through the funding of its R&D projects, which lasted anywhere from 18–24 months. In addition, evaluation interviews found that this financial support was considered to be quite significant for SMEs to sustain their operations over a period of 18–24 months especially during economic downturns.

Longer term employment of the same highly qualified Canadian personnel is largely dependent on the successful commercialization of technologies and products developed by ISTPP–funded R&D projects and/or any "spin–off" projects. Four (4) Canadian companies that either had achieved commercialization or launched "spin–off" projects reported the retention of six (6) highly qualified personnel.

8.7 Performance Issue 7: Demonstration of Efficiency & Economy

Disbursements and administrative overhead costs

The evaluation examined the demonstration of ISTPP efficiency and economy with regard to percent of funds disbursed and percent of administrative costs as presented in Tables 6 and 7.⁽⁵⁹⁾ As previously mentioned, the shortfall in the disbursement of funds can be in large part attributed to the fact that the ISTPP is a start–up program which had only reached its full operational capacity mid–term in 2009–10.

The evaluation factored in that establishing relations, negotiating programming modalities with partner country delivery organizations, and assisting in building institutional capacity in the case of India's Global Innovation & Technology Alliance, has slowed ISTPP programming activity and disbursements with China, India and Brazil. By comparison, CIIRDF programming with Israel has been well established for many years. It has not encountered any of the challenges faced in establishing ISTPP efforts in other partner countries and consequently has met its disbursement estimates and incurred only 12% administrative costs relative to other partner country programming disbursements in compliance with the conditions of its contribution agreement.

Table 6: ISTPP 2005–⁽⁶⁰⁾

ISTPCanada contribution agreements for China, India and Brazil each stipulated the following obligations with regard to administrative overhead costs.

"The Institution will limit its overhead and operating expenditures to no more than 15% of the total annual contribution. Expenses incurred in support of partnering/matchmaking activities are not considered overhead expenditures."

While the administrative overhead costs incurred for establishing ISTPP programming with these countries is within the established parameters, as a percentage of the contributions received from DFAIT, they nevertheless appear high as a percentage of total disbursements as presented in Table 7. However, as country programming becomes fully operational, the ratio of program to administration costs usually improves or normalizes over time. Evidence of this trend was found for the China program.

Table 7: ISTPP 2005 ⁽⁶⁴⁾

Partnership Development Activity Efficiency⁽⁶⁶⁾

The evaluation examined the demonstration of ISTPP's efficiency and economy with regard to its two main programming components, collaborative R&D projects and partnership development activities (PDA). Although not specified in the ISTPP Terms and Conditions or in the respective contribution agreements, the understanding by DFAIT was that disbursements would reflect an 80%–20% split between the R&D project and PDA components. Table 8 shows the relative imbalance between these two programming components by country.

Table 8: ISTPP 2005 ⁽⁶⁷⁾

While it is logical to assume that a start–up program would funnel a high proportion of time and effort into partnership development at its onset in an effort to lead towards highly–targeted joint R&D collaborations, the evaluation could not determine if this trend was evident in ISTPP's delivery for two reasons. First, at this early stage of ISTPP implementation, it is premature to assess this linearity and secondly, this approach was not explicit in ISTPP's objectives or program design.

Instead, the evaluation found the need for PDAs varied among partner countries. PDAs appeared less valued in Israel because of the existence of well–established research consortia, other fora for networking and, Israel also hosts several scientific bodies. Israelis who study abroad also return to Israel for academic or scientific positions with established networks and contacts with international scientific bodies; therefore the view was that there was less need for PDAs. Additionally, CIIRDF has established a Technology Matchmaking Service which has been useful in processing many partnership requests. Several Israeli respondents to the online survey have benefited from this service and expressed its contribution in 'finding the right partner'.

Several areas of concern with regard to ISTPP PDA programming with China emerged based on evaluation interviews. To begin with, the Ministry of Science and Technology (MOST) does not have funds available for activities envisaged under the PDA component which have to date included networking activities such as partnering missions, workshops, roundtables and conferences. Consequently, a second partner country delivery organization in China is now involved which increases the demand for ISTPP's on communications, coordination and liaison efforts required to undertake partner–country programming.

PDAs represented 31% of the ISTPP funds disbursed for S&T activities with China. Of the 18 PDA projects approved since 2007 through the call for proposal process, 7 have been launched and are ongoing, 3 have been completed and the remainder on–hold or terminated. Interview respondent views from both Canadian and Chinese perspectives were concerned that this responsive mechanism (i.e., through a call for proposal for PDA activities) may generate interest in non–priority areas or not result in funded collaborative research projects. The number of projects precluded the need for PDAs unless specifically targeted. For instance, the evaluation found only one Canadian R&D organization reported that its involvement in a PDA activity resulted in a funded R&D project. If the purpose of PDAs is to generate awareness, networking or collaborative R&D projects, then it would seem that with the response rate to proposal calls there is no need to generate further interest unless it is directly targeted to a specific technology area. Both Chinese and Canadian respondents believed that better results would be achieved from PDAs if they were targeted and directed to meet specific needs. It was also believed that PDAs should be led as well as initiated through the S&T Sector Co–ordinators in both countries, if a specific need was identified.

For India, ISTPP funded PDAs such as "Distinguished International Speaker Series" workshops, an international matchmaking service, partnering missions and roundtables and summits. Expenditures on PDAs represented up to 29% of ISTPP funds disbursed for collaborative research projects in India. Of the two calls for proposals held in 2008 and 2009 only 3 proposals were submitted in total, all were approved and 2 have been completed.

For the 3 projects reviewed in India, the collaborative partnership between Canadian and Indian organizations did not result from a PDA and only one Canadian R&D organization reported through the online survey that its involvement in a PDA activity resulted in a funded R&D project. Although it may be too premature at this stage of ISTPP implementation to assess the effectiveness of PDAs towards encouraging joint R&D industry collaborations, some believed that PDAs were important means to bring Canada and India together but would only a expend a small proportion of funds, in the range of 10–15%, to engage in such activity in the future.

Collaborative R&D Project Efficiency

ISTPP Project Proposal Evaluation: Efficiency

ISTPP efficiency and economy was examined with regard to collaborative R&D projects. The evaluation compiled statistics on the call for proposals process and project outputs⁽⁶⁹⁾ (see Table 9). With the notable exception of ISTPP programming with Israel where Expressions of Interest (EOI) is not a feature of the call for proposal process, the ISTPP had generated considerable interest among R&D organizations, particularly for projects with China. This interest may also have been influenced to some degree by already close relations between China and Canada considering that there are many highly qualified Chinese researchers who have made Canada their home. Alternatively, Canada may not have the same advantage with India where graduate students prefer study in the United States, Australia or the United Kingdom which are considered premier partners by India in S&T innovation. The interest in R&D projects with India by comparison has declined from an already modest level since the first call for proposals in 2007.⁽⁷⁰⁾

The evaluation interviews also suggested that the proposal review process is time consuming. Since 2008, ISTPCanada has reviewed 612 EOIs to determine eligibility and strategic alignment with ISTPP objectives. Once eligibility is determined, selected proponents are then invited to submit full proposals. These proposals are forwarded to S&T Sector Coordinators with a request that they be evaluated by appropriately qualified S&T personnel in their departments or agencies in Canada. Although this process was deemed to be fair and transparent albeit lengthy, previous efficiency assessments of the ISTPP call for proposals evaluation process did not take into account costs associated with the time and effort of federal public servants to review 260 project proposals over the course of three years. Some ISTPP stakeholders expressed this concern over work demand that needed to be resolved for example, by way of cost recovery. It has prompted some SBDAs to withdraw their proposal evaluation services unless compensation is available. One agency, NRC–IRAP, estimated that the value of coordination and proposal evaluation services provided by their Industrial Technology Advisors (ITAs) to be worth in excess of $1.0M CDN. If this were to be considered as a baseline, then the review cost per approved project would be in the order of $15,000 CDN based on 65 approved projects as one point of analyses. Table 9 details the R&D project call for proposals.

Table 9:

NB: There is no EOI stage in the ISTPP–Israel R&D Project Call for Proposal process.

* Projects are on hold pending completion of agreements, while one project has been terminated.

Again, with the exception of ISTPP programming with Israel where the CIIRDF Board of Directors meet to discuss and jointly select the best R&D project proposals, a parallel proposal evaluation process was established with the three other partner country delivery organizations. While the same R&D project proposal application form is used in all cases, both China and India use their own criteria to assess proposals. Each partner country prepares a shortlist of projects that have been deemed eligible for funding which are then exchanged and compared with the Canadian selections. Proposals on both short–lists are automatically approved; however, the discussions and ensuing negotiations to consider their acceptance of the remaining proposals is time–consuming and labour intensive. Evaluation interviews in Canada and during site visits in China found that there was a perception of a lack of transparency regarding the decision–making process and certain impatience with the process.

In the case of China, for example, fewer projects were approved than the pre–established ISTPP budget allocated for R&D joint collaborations which lead to uncommitted funds. Such pressures during project approval negotiations were less evident with India. Brazil, at the time of this evaluation, was in the midst of its first project approval process but did comment on the lengthy and laborious process.

For China, MOST did not wish to request Expressions of Interest (EOI) for the first round of R&D project selection. Full proposals were to be submitted. As there is no peer review system paid evaluators assess the proposals. Interview respondents noted that the first call for proposals was a lengthy process but subsequent calls improved in timeliness because of increased familiarity with procedures. Having two parallel processes in both countries also complicates funding arrangements and the timing of R&D project activities. One example was cited where the Chinese partner received their funding from MOST in October 2008 but the Canadian partner did not receive their first instalment until April 2009. The Chinese partner could not start their activities because they were dependent on the Canadian partner for data so they had to request MOST to re–profile their funding arrangement.

The parallel proposal evaluation model also has been employed with India for the approval, funding and monitoring of collaborative projects. The implementation of ISTPP can be complex in that it requires political as well as operational considerations. For instance, ISTPP was delayed because of challenges encountered with the establishment of the Indian partner agency GITA. As of March 2010, the Parliament of India had not yet ratified GITA as an independent funding body.

On an operational level, interview respondents noted that it took some time to deliver ISTPP. Table 10 shows the elapsed time between various stages in the call for proposal and payment process for India.

Table 10: Time Elapsed for R&D Process in India

Interview respondents noted that the ISTPP start–up was time consuming in its first year following the announcement of ISTPP but seemed to improve in the second year of operation with only 12 months elapsing between the first call to the second call. The amount of time between the call date and the board approval dates remained relatively stable at 8 and 9 months respectively⁽⁷¹⁾ but, delays were more evident in the administration of the contribution agreements. As of March 2010, only 6 of the 12 projects approved in these first two calls received their first project payments. Three other recipients were still waiting for their first payment, and it is uncertain if or when the remaining three projects will begin.

ISTPP Administrative Process: Average Time–lag

Once the R&D project proposals are approved by the ISTPCanada and CIIRDF Boards of Directors their respective staff attend to the administration of preparing project funding (contribution) agreements and to release the first payment to the Canadian R&D organization. The evaluation compiled time–lag statistics from the available tracking data provided by ISTPCanada and CIIRDF. Time–lag was operationally defined as the number of work days⁽⁷²⁾ between the approval date of R&D project by their respective Boards of Directors and the issuing date of the first payment to the Canadian organization.⁽⁷³⁾ The average time–lag was as follows: Israel – 86 work–days; India – 171 work–days; and, China – 238 work–days (Table 11).

When the average time–lag is combined with a lengthy proposal evaluation and approval process of approximately a year, it was not surprising that many Canadian R&D organizations expressed impatience with the ISTPP process which has taken close to two years from proposal submission to first payment in some instances.

Table 11: Administrative Process⁽⁷⁴⁾ showing average elapsed time for R&D Projects, 2004–2009

ISTPP Project Management

The evaluation found that the Board Approval dates varied considerably among and between partner countries. Although this variation is reasonable given program implementation delays, the evaluation expected more regularity or efforts towards establishing a systematize method to deliver the program in an effort to address efficiency. That is a standard timeline for Board Approval dates among partner countries; for example, to hold biannual Board approval meetings at the same time to approve projects. Interestingly, this appears to have occurred somewhat in China and India but less evident in Israel which is considered a more mature and stable operation. Establishing a standard and regular time for Board Approval meetings could reduce the burden of holding overly frequent meetings to address routine business items such as in this case, project reviews for decision on acceptance.

Another point to consider is the time between project approval and execution. Projects approved for Israel, as expected, had all been executed in a relatively short time–line but, project execution was more delayed in China and even more evident in India. There was also considerable time–lag between Board Approval and the First Canadian Payment. The average elapsed time was 4.3 months for Canadian–Israeli projects, 8.5 months for Canadian–Indian projects and 11.9 months for Canadian–Chinese projects for an overall average of 8.3 months. Brazil was excluded from this analysis because no projects were officially announced at the end of the reference period for this evaluation (March 2010). All in all, these issues demonstrate a need for further examination of more efficient mechanisms to ensure expedient program delivery of funds.

Although Canada's ISTPP third–party delivery agent model has been seen to be ideal in many ways for R&D project collaboration leading towards commercialization, partner country delivery organizations as well as Canadian respondents remarked that the S&T collaboration field is becoming increasingly competitive and they would be better served if a more nimble and efficient process for the selection and implementation of projects was considered. With a more rapid and timely project selection, Canada could increase its probability to be at the forefront of S&T innovation. Many who participated in the evaluation were also of the opinion that a more focussed top–down strategic targeted approach with the S&T Joint Committees would achieve more value–added for Canada and partner countries in the selection of projects than the funding of smaller scale projects that were not visible to or fall below the radar of Canadian and partner country government officials.

Leveraging of Federal Funds for S&T Innovation

As previously indicated in this evaluation, CIIRDF and ISTP Canada have successfully engaged Canadian provinces as delivery organizations for their joint S&T cooperation initiatives with Israel, China and India. Table 12 presents the total funds leveraged from the provinces for S&T cooperation activities including matching contributions from the country partners totalling $15.6M CDN.⁽⁷⁷⁾

Table 12: Total Provincial Funds Leveraged

ISTPCanada and CIIRDF also have leveraged funds from the Canadian R&D organizations involved in R&D projects, since the ISTPP only funds 50% of the total planned project budget. Based on actual R&D disbursements to date of $6,414,900,⁽⁷⁸⁾ the matching contributions from Canadian R&D organizations would amount to approximately $6.5M CDN, to which an additional $6.5M could be added from the partner country delivery organizations.

Canadian respondents to the online survey reported an additional $4.4M CDN in funds leveraged for their R&D projects from other sources, excluding in–kind contributions. These other sources included foundations, drug companies, institutional and commercial partners and venture capital. The evaluation estimates that the total amount of funds leveraged from all sources to date,⁽⁷⁹⁾ therefore, is approximately $33M CDN. Based on total disbursements to date of $10.6M CDN this would represent a leveraging ratio of approximately 3:1.

8.8 ISTPP Partner–Country Reports

Israel

The current relationship is industry–to–industry based. Its mandate is to provide collaborative bi–lateral R&D between Canadian and Israeli firms for mutual benefit. Given that research consortia and other fora already exist in Israel, it would appear that less focus should be placed on PDAs. Israelis who study abroad and return to Israel for academic or scientific positions return with established networks and contacts with international scientific bodies; therefore, there is less of a need for PDAs. Israel also hosts several scientific bodies. CIIRDF also has established a TECHNOLOGY MATCHMAKING SERVICE which has been useful in processing many requests and is being upgraded through the design of a portal. Canadian Trade Commissioners and the Office of the Chief Scientist occasionally are also involved in CIIRDF–related PDAs and match–making. It should be noted here that CIIRDF regularly meets with Canadian Trade Commissioners to discuss events, share information and coordinate activities.

Israel has a clearly defined S&T strategy mandated through the Chief Scientist of Israel. Its focus is to encourage S&T in life sciences, electronics, ICT, biotechnology, clean technology and other related science sectors through the awarding of grants. Israel does not have natural resources to export; its export is science and among the OECD, Israel has the highest per capita rate of science–related experts and IP patents.

Evaluation observations clearly show that many Israeli firms developed the science with considerably fewer projects showing significant mutual R&D collaboration with Canadian partner firms to develop science. In many cases, Israeli firms had already found Canadian partners that had a crucial element to add to the Israeli development such as access to research samples or highly specialized technology.

The overwhelming majority of Israeli firms reported that their Canadian partner was responsible for market access or addressing the business of marketing, distribution and procurement of maintenance. It was not surprising to hear that many Israeli firms owned the IP rights. Therefore, many remained in business and could market their science with other firms; while in some instances Canadian partner firms had either dissolved or experienced bankruptcy. Israel's unemployment rate was a constant 7% over the same time frame. Israel's S&T industry is primarily based on science development. Israel is willing to increase its share of financial support to CIIRDF and has encouraged Canada to consider increasing its share.

Since CIIRDF funds contribute to the development of science it appeared that Canadian firms were at a higher risk of not reaching success or direct benefit from the CIIRDF funds relative to the Israeli firms. Of the 18 CIIRDF projects⁽⁸⁰⁾ funded before 2004, 40% of Canadian partner firms were dissolved, bankrupt or in one case, acquired by another firm. Alternatively, over the same time frame, 25% of Israeli partner firms were found "not in business" but only one of them was dissolved; the others were acquired by other firms. Although it appears that Canadian partner firms were more often unable to fulfill their commitment to their Israeli partner, the Israeli firms also were more resilient in that many held the R&D lead and retained the IP rights to the science developed.

It is also important to note that Israel's S&T strategy creates the drive for science and it measures its performance by the number of IP patents on science contributions, royalties, increases in jobs, increased tax revenue, higher proportion of direct foreign investment – all of which have increased over the last several years. The Israeli government's approach is highly tolerant of risk, and the likelihood of projects not succeeding. In fact, funded projects that fail are seen as lessons–learned, part and parcel of the R&D process, and investment in future projects. This positive and forward looking attitude came across in interviews with companies.

PDAs seem to be less of an interest because there are many already in place; however, tri–lateral arrangements with other countries such as ones on agriculture may be advantageous for Canadian and Israeli participants. It depends on the theme of the tri–lateral arrangement because they often open the opportunity for synergies. The challenges are to align best–fits in that each country has its own values, regulations, standards and uses of new and developing technologies which need to be addressed.

Canada–Israel maintains a strong bi–national relationship for over 60 years. However, among all the multi–national and bi–national S&T agreements, CIIRDF offers the lowest dollar amount of annual contributions. The challenge is for Canada to identify its international S&T approach and its importance to commit and sustain those partnerships over time.

CIIRDF deals with high–risk developments in many projects for the following reasons:

• many have not reached commercial potential;
• many technologies have been too closely dependent on market current trends which have shown to shift and have easily been out–dated quickly in the market;
• many firms do not wish to join with VCs because of attention to bottom–line expectations for profit–generation.

Israeli firms seem to be more resilient to the market than Canadian firms in that there were few Israeli firms that dissolved over the time frame the evaluation examined. In many cases, Israeli firms had developed the science (with the contribution of CIIRDF funds), owned the IP and as a result, appeared to be more flexible and agile to evolve with the market in that they could diversify. The evaluation found less of a return than initially expected for Canadian firms based on commercialization; but those Canadian firms in mutual R&D development with Israeli firms continue with their partnerships.

Another challenge is that CIIRDF funds up to 50% of approved project budget costs at project initiation and in some cases, companies absorb additional expenses on their own. This can be problematic for small firms who do not have the operating budget to assume additional costs and as a result, places the completion of a project at risk.

Other Comments:

The BIRD Foundation (Israel–USA) is a robust bi–national industrial research and development foundation that is industry–to–industry focussed, with partnering between large to SME enterprises. It has been in operation for 33 years (started in 1977), and has been incorporated as a government corporation. It operates on the interest accrued from a $100M USD endowment, which is indexed to annual inflation rates. It has 10 staff (8 in Israel; 2 in USA).

BIRD's activities are similar to CIIRDF as both are involved in match–making companies together. In BIRD's opinion, partnering industry–to–academia is problematic because of nature of the contractual transaction. Their preference is not to partner industry with academia.

Grants are limited to $1M USD. BIRD defines success rate as 30% of grant repayment. Whoever sells the product first, would repay BIRD. BIRD stops monitoring once repayments of the initial payment and interest has been reached.

Technology–based partnerships account for the highest percentage of failure or non–repayment of grants given the volatile nature of the market. BIRD considers repayments as a bonus. BIRD Energy is open to PDAs with academia but not in other sectors. In fact, BIRD has experienced that large companies in partnership with small ones or with other large companies are more likely to succeed because the risk ratio is better. Therefore, they prefer to advocate such partnerships.

Table 13:

Table 14:

China

Cooperation with Canada and China on science and technology is based on the Cooperation Agreement signed between the two countries in 2007. This agreement spells out the terms of cooperation and identifies the priorities for engagement between the two countries. There is a joint committee which meets every year to review the nature of cooperation and to decide on the priorities for the coming year.

ISTPP is one element of the delivery mechanism for fostering cooperation on research and development between Canada and China. Other mechanisms include NRC–IRAP, AAFC and CIHR and provincial funding for joint R&D projects notably B.C., Alberta and Quebec.

ISTPP began operations in China in 2007 and has negotiated an agreement with the Ministry of Science and Technology (MOST) to identify and fund collaborative research and development projects. Both ISTPCanada and MOST develop calls for proposals to fund collaborative research projects between research and development organizations from both countries. Since inception there have been 3 calls for proposals with 6 projects funded in 2007/08 and 8 projects funded in 2008/09. For 2009/10, proposals are presently being evaluated from which it is expected that up to 10 projects will be funded. Because of the emerging nature of Chinese industry in fostering innovation, most partnerships with China are with academic and research organizations.

ISTPCanada has also signed a tri–partite agreement between Yangling Demonstration Area and CIIRDF for the funding of collaborative R&D projects of an agricultural nature that would support private sector organizations in Canada, Israel and China to develop and demonstrate innovative agricultural projects. The first call for proposals was issued in January 2010 and was extended to the end of June 2010 due to lack of proposals received by the initial deadline.

ISTPCanada also funds PDA and partners for these activities with the China Association for International Exchange of Personnel. To date ISTPCanada has funded numerous PDAs including 8 workshops, an international technology matchmaking service, 3 partnering missions and one roundtable/summit.

The Federal S&T Strategy as outlined in Mobilizing Science and Technology to Canada's Advantage 2007 identifies the Environment, Natural Resources and Energy, Health and Life Science, and Information and Communications Technology as R&D priority areas. The Collaborate S & T agreement between Canada and China identifies the following as priority areas for R & D collaboration: Energy, Environment, Health and Life Sciences/Biotechnology and Agricultural Foods and Bioproducts. Of the 14 projects funded to date in China three (3) are in the Energy field, one (1) within the energy environment field, four (4) in Health and Life Sciences/Biotechnology, four (4) in agricultural and two (2) in ICT. Within the ICT field, respondents from both China and Canada felt that there was sufficient exchange between Canadian and Chinese companies that it should not be a focus of the ISTPP.

ISTPP projects funded to date were not considered to be not high profile in nature. This may be in part due to the early stages of project implementation but is also due to the lack of communication between ISTPCanada and officials at the Canadian Embassy. It was noted that information on the implementation of ISTPP has not been forthcoming and it was only on the eve of a Canadian ministerial visit that Canadian officials received information on the projects being funded under ISTPP.

While Canada is seen by the Chinese to have an ideal model for collaboration on research leading towards commercialization unlike relationships established with other countries, they, as well as Canadian officials, remarked that the S&T collaboration field is becoming increasingly competitive and that a more nimble and efficient process for the selection and implementation of projects should be established to ensure that Canada is at the forefront of collaboration efforts. Many observers remarked that a more top down targeted approach would achieve better value added for both countries than the funding of smaller scale projects that are falling below the radar.

For China, science & technology collaboration is a high priority as they see technology transfer as the fastest way to fuel their rapidly expanding economy while creating significant value and benefits for China. China currently has S&T cooperation agreements with l00 countries. Canada is in the second tier of countries China considers important for S&T collaboration. The United States is in the first tier representing about one third (1/3) of all S&T collaborations followed by the E.U., Russia and Japan. In the second tier is Canada along with the UK and Australia. China has allocated approximately $100M USD annually for S&T partnerships of which Canada represents only a tiny fraction.

From the Chinese government perspective, preference would be given to directed government led programs rather than to a competition for the selection of projects from an arms–length body as they appear more comfortable with the high level government relationships in the selection of activities for collaborative research. However, they have agreed with the model Canada has developed but would like to see a process that is more nimble and efficient and involves more engagement by the Government of Canada as the lead in the science and technology arrangement.

Leading up to the cooperation agreement on S&T between Canada and China, a complementary study was conducted which identified the areas in which collaboration in R&D could be effectively established. The cooperation agreement itself is managed by a joint committee meeting annually to establish priority areas for engagement in the coming year. Interviews in China and Canada have indicated that the mechanisms established for the delivery of projects under the S&T cooperation agreement are not necessarily coordinated nor tied to the processes established for managing the agreement.

Chinese and Canadian respondents felt better results would be achieved from PDAs if they were targeted and directed to meet specific needs. In China, PDAs are developed in association with another governmental organization, the China Association for International Exchange of Personnel (CAIEP). MOST is not involved in these activities with ISTPCanada and do not consider the PDAs funded by ISTPP as adding value to the relationships being forged for collaborative partnerships. Officials at MOST argue that higher value is achieved through government led rather than industry led exchanges. PDAs, they argue, should be at a higher level that is more capable of providing information for the establishment of long term priorities and relationships such as the complementary study undertaken as part of the process emanating from the cooperation agreement. Others in China are of the view that tendering PDAs based on a call for proposals is a complex process that does not necessarily add value or result in an efficient and effective means for establishing collaborative relationships. The current projects funded under the ISTPP were based on relationships that preceded the program. ISTPCanada anticipates that future calls will include partnership projects that have developed from the PDAs implemented to date. However, with the high degree of interest expressed in each call for proposal, there are some that would argue that in the China context, PDAs are not necessary to build collaborative relationships. It was noted further that approving PDAs through a call might not be the most efficient or economical means for establishing collaborative relationships as they are relatively costly with little so far to show for the expense. A more directed or targeted approach whereby priority areas and organizations are identified and the PDAs are used at the final stages in developing collaborative research projects may deliver better results than the current process.

The challenges for ISTPP going forward is to balance the competition for ideas achieved through a call for proposals with the need for greater nimbleness and efficiency in selecting collaborative research projects that will ultimately lead towards commercialization. With limited funding and competition from other nations, the challenge for ISTPP is to build a program that is recognized by the Chinese as unique and contributing to the achievement of rapid commercialization of innovation products. To do this, ISTPP will need to be more adept at coordinating its program with other initiatives sponsored by Canadian agencies and provinces and to communicate better with Canadian officials so that Canada is being represented in a coherent fashion in its S&T relationship with China.

India

Cooperation with Canada and India on science and technology is based on the Cooperation Agreement signed between the two countries in 2005. This agreement spells out the terms of cooperation and identifies the priorities for engagement between the two countries. There is a joint committee which meets every year to review the nature of cooperation and to decide on the priorities for the coming year.

Canada's S&T relationship with India is multi–faceted. While ISTPP provides a direct mechanism for delivering on the collaboration agreement between the two countries, there are other S&T collaborative arrangements that are in effect. Other governments departments are also involved such as NSERC, NRC–IRAP, and AAFC. All of these programs operate in isolation from one another. Canadian provinces are also active including BC, Alberta, Ontario and Quebec. Ontario has signed an agreement with ISTPCanada to pool its funds and ISTPCanada is in the final stages of discussion with Alberta to manage their S&T collaboration projects with India and other priority countries identified in ISTPP.

ISTPP has negotiated agreements with the Global Innovation and Technology Alliance (GITA), an arms–length institution established by the Department of Science and Technology (DST) as well as with the Department of Biotechnology (DBT) to identify and fund collaborative research and development projects. Each organization issues calls for proposals to fund collaborative research projects between research and development organizations and industries from both countries. Since inception there has been one (1) call for proposals in 2007 with GITA and one (1) call for proposals with DBT in 2008. Eight (8) projects of which four (4) are currently operational were approved in the GITA call while four (4) projects of which two (2) are in progress were approved in the DBT call.

Partnership Development Activities are to represent up to 20% of the funds allocated for collaborative research projects in India. PDAs are funded through competitive calls for proposals. Others are developed and implemented by ISTPCanada with partners. Types of activities funded under PDAs in India have included a Distinguished International Speaker Series, workshops, an international matchmaking service, partnering missions and roundtables and summits.

The Federal S&T Strategy as outlined in Mobilizing Science and Technology to Canada's Advantage 2007 identifies the Environment, Natural Resources and Energy, Health and Life Science and Information and Communications Technology as R&D priority areas. The Collaborate S & T agreement between Canada and India identifies the following areas for R & D collaboration: Nanoscience and Nanomedicine, Information and Communications Technology, Biotechnology and Health Research and Medical Devices, Sustainable and Alternative Energy, Environmental Technologies and Earth Sciences and Disaster Management. Of the twelve (12) projects approved to date six (6) are in the Health Research and Medical Devices/Biotechnology field, three (3) within the Energy, Aerospace, and Environment fields, two (2) in ICT, and one (1) in Agriculture focusing on veterinary diagnostics. No projects have been selected in the Nanoscience and Nanomedicine fields, however a Nanotechnology Partnering Mission that took place in August 2008 may lead to collaborative research projects funded in future calls.

ISTPP projects funded to date are not high profile in nature. This may be in part due to the early stages of project implementation but is also due to the lack of communication between ISTPCanada and officials at the Canadian High Commission. For many on the Indian side, there is confusion and concern about whose program this is: The Government of Canada's or ISTPCanada's. The roles of ISTPCanada and the High Commission are not clear and without consultation and ongoing communication, the two organizations are working in parallel and not as a synchronous unit. As such, the Government of Canada does not appear to be providing the necessary direction and oversight to ensure the effective implementation of its investment in the science and technology collaborative research field in an effort to provide results for Canadian businesses and to advance the Canada–India innovation relationship.

While Canada is seen by the Indians to have an ideal model for collaboration on research leading towards commercialization unlike relationships established with other countries, they, as well as Canadian officials, remarked that the S&T collaboration field is becoming increasingly competitive and that a more nimble and efficient process for the selection and implementation of projects should be established to ensure that Canada is at the forefront of collaboration efforts.

India currently has S&T cooperation agreements with 80 countries of which 60% are deemed operational. For the India government, Canada is considered an important partner however the extent of this partnership is dependent on Canada's willingness and level of engagement as India is willing to match Canadian funds. Before Canada signed its collaboration agreement there was little interest in such an agreement. This was partly because Indian researchers and scientists looked to the USA as the penultimate destination for innovation; therefore Canada was barely considered. The Government of Canada is still evolving as an innovation partner and India would like to see Canada move into the top tier of countries collaborating on science and technology. The top tier represents 15 countries including the US, UK, Australia and Germany. As an example of top tier contributions to S&T collaborations, Australia has just announced a $100M AUS fund and signed an agreement with GITA for a $40M AUS program. Currently ISTPP has committed $6.75M CDN for its programs with India.

A number of respondents in India questioned the relevance of using two delivery partners given the limited resources available in the ISTPP. It was noted that there are 10 ministries associated with DST, but only one ministry, DBT, that has become a delivery partner for ISTPCanada. However, there is a risk of diluting the effects of collaboration if a small amount of funds is spread too thinly. DST did not see any problem in having two partner organizations in India as both GITA and DBT fall within the authority of DST. They would prefer that more funds are made available so that the Canada–India partnership could be enhanced. As it is, the small amount of projects being funded will only result in small incremental changes to the S&T landscape in India rather than the transformational changes that a much larger fund might generate.

Leading up to the cooperation agreement on S&T between Canada and India, a complementary study was conducted which identified the areas in which collaboration in R&D could be effectively established. The cooperation agreement itself is managed by a joint committee meeting annually to establish priority areas for engagement in the coming year. Interviews in India and Canada have indicated that the mechanisms established for the delivery of projects under the S&T cooperation agreement are not necessarily coordinated nor tied to the processes established for managing the agreement.

From the Indian viewpoint, PDAs are considered to be very expensive and are not needed to generate interest in collaborative projects. It was pointed out that a recent call for proposals with Spain generated over 160 proposals. On the other hand, some Indian officials view PDAs as important for bringing the two nations closer but would only allocate 10–15% of funding for PDAs in the future.

The challenges for ISTPP going forward is to balance the competition for ideas achieved through a call for proposals with the need for greater nimbleness and efficiency in selecting collaborative research projects that will ultimately lead towards commercialization. With limited funding and competition from other nations, the challenge for ISTPP is to build a program that is recognized by the Indians as unique and contributing to the achievement of rapid commercialization of innovation products. To do this, ISTPP will need to be more adept at coordinating its program with other initiatives sponsored by Canadian agencies and provinces and to communicate better with Canadian officials so that Canada is being represented in a coherent fashion in its S&T relationship with India.

Brazil

Canada is seen as relatively unknown S&T partner to Brazil. The reasons for this perception include:

• The implementation of ISTPP in Brazil through ISTPCanada only began, in the last year of the ISTPP program, which was 2009–10. It consisted of three (3) calls for proposals (a bi–lateral call with FAPESP; a unilateral R&D; a unilateral PDA).
• The S&T bi–lateral agreement between Canada and Brazil was recently ratified (March 2010) at the Parliament (Brazil Congress). This agreement will allow for a framework on future engagement. Brazil is now in a position to name a federal agency to be the partner service delivery agent. Early indications seem to suggest FINEP in Brazil will be selected to fill this role. It is also unknown how this will impact the relationship between ISTPCanada and FAPESP, the Sao Paulo state research agency currently involved. FINEP's⁽⁸¹⁾ mandate is to decentralize federal government S,T&I⁽⁸²⁾ policies towards integration to state and local levels.

Therefore, the current Canada–Brazil relationship would be considered new and emerging. In addition, based on the evaluation interviews held with the Brazilian Ministry of Science and Technology, science innovation in Brazil is in its infancy. Regulatory and legal instruments as well as funds for supporting scientific and technological enhancements have only been in effect since 2005, when Brazil set into force the Innovation Law:

1. To strengthen the interaction between university and industry;
2. To promote the shared use of S&T infrastructure by firms and institutions;
3. To stimulate the creation of new technology; and,
4. To create new financial mechanisms for grants to R&D innovation to firms.

In support, Brazil's Ministry of Science and Technology prepared a 3–year action plan (2007–2010) to enhance S&T engagement by increasing its R&D investment to 1.5% of its GDP in 2010. By comparison, in 2006, Canada invested 1.94% of its GDP in R&D, the US invested 2.62% and Sweden 3.73%.⁽⁸³⁾ It is interesting to note that there is a strong correlation between the wealth of countries and investment in R&D. Israel's investment in R&D is among the highest of industrialized countries (5% of its GDP in 2008) thereby contributing to the relatively high per capita income base.

Brazil also has S&T agreements with Germany, France, the US, UK, China, India, Korea and South Africa and one of Brazil's strategic interests is to develop closer ties with Africa. During the evaluation interviews, the impression was that efforts with China and India have been somewhat less successful than with other countries, although progress continues.

By way of follow–up to the ratified S&T treaty with Canada, Brazil anticipates that a joint committee with private, academic and public sectors will be established to develop an integrated S&T approach for Brazil. This joint committee would decide on risk tolerance and would define the success of these endeavours. The evaluation found that bilateral agreements have accrued direct benefit to Brazil in Nanotechnology, Bio–technology and Aerospace sectors.

The evaluation also found that the issuance of multiple calls for ISTPP–related proposals in a relatively short period of time (over 4 months) contributed to confusion within the Brazilian community of interest in that it was difficult for the potential recipients to obtain information on submission guidelines and on proposal status reports from ISTPCanada or the Canadian Trade Commissioners in Brazil. It was also difficult for the trade commissioners to reply to these requests as they were not aware of project review status.

The evaluation also noted the need to clarify the roles of ISTPCanada, ISTPP and the Canadian Trade Commissioner Service in the delivery of ISTPP given that ISTPCanada held PDAs independently; for example, agriculture–related PDA with EMBRAPA.⁽⁸⁴⁾ This PDA, held in March 2010 under the Memorandum of Understanding between Agriculture and Agri–Food Canada (AAFC) and EMBRAPA, brought together Canadian and Brazilian researchers and experts to explore R&D opportunities in biofuels, cellulosic ethanol technology in agriculture and forestry. While this was the first PDA hosted by ISTPCanada, in total, Canada and Brazil were involved in eleven (11) S&T activities in 2009–10. These included: Brazil and Ontario, Quebec, British Columbia, Environment Canada, Natural Resources Canada and Agriculture and Agri–Food Canada.

Brazil's partnering with Canada on R&D S&T collaboration is of interest given that it generates opportunity for technology transfer, learning and for increased funding. One of Brazil's targets is the internationalization of their technology; therefore, enhanced market access is a relevant feature. It also was raised that funding options for R&D in Brazil are challenging because, as it stands currently, the private equity market is quite small for R&D, tax incentives are not seen as effective and, high interest rates for business loans often discourage applications. However, research granting bodies such as FAPESP are willing to offer long term grants without ceiling limits to R&D projects of merit.

The interviews left the impression that Brazil is open to innovation because it is seen as contributing to social and economic propensity. This need for innovation has been identified as a gap to commercialization. Brazil's Ministry of Science and Technology, as a result, has set clear targets⁽⁸⁵⁾ to increase the number of patents issued, the number of researchers and the prevalence of PhD level research in Brazil (training of researchers, publishing research).

FAPESP's view was that ISTPP has been the only program which has come to their attention to encourage joint R&D collaboration on an international level. This approach is of interest because it supports Brazil's overall objective for internationalization with an R&D focus. Many companies in Brazil do not include R&D; therefore, partnering allows for such an opportunity to experience the complementary role of R&D to industry.

For example, S&T–related activities on renewable energy and hydrogen fuel cells technology offered the opportunity for Brazilian researchers (industry) to network with Canadian researchers (industry) to stimulate the development of new technologies and to apply technology transfer. Through these Canada–Brazil transactions, a potential for collaboration on hydro–power was identified but which has yet to materialize. Brazilian researchers who attended those S&T forums and were interviewed for this evaluation saw R&D match–making to be much easier done between universities. The reason for this was because of the difficulty to find and match two companies together in specialized fields such as fuel cell development. This perception of difficultly may be due to the reluctance of industry partnering given the highly competitive nature of IP and the presence of relatively few specialized companies in the market.

The evaluation interviews suggested the need to have a co–ordinated approach to develop S&T needs and targets for mutual benefit for Canada and Brazil. It had suggested that a Canadian S&T co–ordinator (scientist) for Brazil could be an asset to offer a strategic direction to Canada's S&T interests.

On an operational level, Brazilian researchers who attended S&T related activities had suggested that to encourage industry matchmaking it may be valuable to determine sound criteria for match–making companies and to have clear guidelines on IP rights. In fact, it would be preferred if ISTPCanada, as a granting agency or other granting agencies, could consider pre–proposals to be drafted for review rather than full proposals given the sensitivity of some science sectors to the level of disclosure requested in the application process.

For highly specialized fields, such as hydrogen fuel cells, it was raised that there was limited opportunity to develop synergies within such a small economy that has a highly competitive need to retain IP; therefore, it would be advantageous for Canadian and Brazilian companies to build a relationship, to encourage the development of a trusted partnership by sharing technology and information. Some of the researchers in Brazil proposed that perhaps ISTPP could become a vehicle for this type of collaboration.

The Brazil – Canada relationship is emerging on S&T. Foreign direct investment is welcome in Brazil and to encourage a stronger relationship between the two countries, there was a sense gleaned from evaluation interviews that Brazil and Canada would benefit by furthering their understanding of each others' business cultures; for example, national consumer protection is not prevalent in Brazil as it so in Canada so could such a difference impact R&D collaboration?

In summary, the interviews left the following impressions:

• Canada should be more pronounced in its promotion of S&T interests in Brazil.
• R&D partnerships with universities may be easier to establish than industry–to–industry collaborations given the challenges of operating in a business environment with a focus on commercialization.
• A need for Brazil and Canada to mentor SMEs to determine successful market access by following and supporting them throughout their engagement.
• S&T proposal guidelines and status reports (ISTPCanada) should be made available on a regular basis and, efforts continue to work more closely with Canadian Trade Commissioners on S&T approaches in order to avoid duplication of effort and to promote efficient functioning of this relationship.

Some voiced the opinion that ISTPP may appear too focussed on selling Canadian technology when the emphasis is on commercialization and if the objective is to promote joint R&D collaboration then that message should be made clear without emphasizing commercialization. There was also confusion over the difference between ISTPCanada, ISTPP and the Canadian Trade Commissioner Service. More effort was suggested to refine their respective roles.

9.0 Conclusion of the Evaluation

The following conclusions were derived from evaluation findings and are as follows.

ISTPP is a relevant government instrument that remains coherent with national economic and S&T innovation priorities that seeks to enhance Canada's productivity and global competitiveness by optimising our entrepreneurial and knowledge advantages. It is an important complement and supplement to current R&D tax subsidies in that it directly supports Canadian industry and universities to commercialize their joint R&D internationally. While it does provide an opportunity to engage partner countries in S&T innovation activities, its level of funding does not commensurate with its mandate as the designated instrument for implementing the S&T Treaties. It appears insufficiently incremental to federal departmental and provincial initiatives and disproportionately small in relation to competing countries; therefore, ISTPP is at risk of becoming irrelevant in the international drive to create new global value chains through S&T innovation.

Although ISTPP approval documents clearly mandate the ISTPP as the dedicated instrument for the implementation of the S&T Treaties, they neglected to make explicit the relationship between the Joint Committees and the ISTPP Steering Committee. Recognising that the Joint Committee structures and processes are still evolving, the evaluation has nevertheless noted that they have had little influence to date on strategic decisions that have shaped the program.

The S&T Sector Coordinators who provide advice to and take direction from the Joint Committees have also had little opportunity to provide strategic guidance, for example, on the identification of high potential technology areas for collaboration, or to provide technical support other than to evaluate R&D project proposals. While the ISTPP Steering Committee composition is suggestive of a whole–of–government approach, the appropriate financial mechanisms to facilitate interdepartmental collaboration were not established. Some departments and their S&T Sector Coordinators therefore have not been fully engaged which has reduced the potential to optimize the available resources across government to maximize the benefit of ISTPP programming.

Strengthening economic ties with the emerging economies of China, India and Brazil has been a Government of Canada priority for over a decade. The S&T Treaties provide an opportunity to engage these countries in bilateral relationship building with a focus on the international commercialization of S&T innovation. ISTPP represents Canada's first endeavor at establishing international partnerships between the respective R&D communities in technology areas of common interest and expertise. The use in this context of an "arms–length" delivery organization (in this case ISTPCanada) to provide such leadership has caused some confusion among stakeholders both in Canada and abroad. While the Governments of China, India, and Brazil have all approved the use of an "arms–length" organization to deliver this program (and, in the case of India, they established a mirror arms–length organization to partner with ISTPCanada), their preference is to work directly with official Government of Canada representatives in order to continue to build the bilateral S&T relationship. At the same time, Canadian Trade Commissioners and Diplomats abroad were not always aware, informed of or involved in many ISTPCanada activities and projects in the jurisdictions for which they are responsible. This led to an inability for Trade Commissioners and Diplomats to adequately fulfill their representational role. The overall impression left with key stakeholders could lead to a misinterpretation of the Government of Canada's intentions contributing to a risk of diminishing returns rather than enhancing Canada's reputation as an S&T partner of choice.

While the overall ISTPP program delivery model is appropriately flexible to accommodate the various domestic and international interests in undertaking collaborative S&T cooperation activities, the various proactive and responsive modalities used are insufficiently focussed and streamlined to effectively accelerate the international commercialization of R&D that would benefit Canada. The PDA component included a wide variety of activities with some intended to "brand" Canada's S&T innovation capacity and others to facilitate international partnership building among R&D organizations. DFAIT's Going Global Program and other government department programs that facilitate the mobility of highly qualified research personnel working in the higher learning institutions appears duplicative of the responsive modalities used in the PDA component. The purpose and expected outcomes of PDAs were not always clear to the stakeholders involved, especially among the partner country delivery organizations due primarily to their high cost, but also in relation to the results achieved. The proactive and responsive processes used by the delivery organizations to identify and undertake PDA activities have essentially been the same despite previous evaluation recommendations to better integrate the S&T Sector Coordinators. Little concrete action appears to have been taken on this issue other than to clarify and communicate the job position description.

ISTPP has been successful in facilitating collaborative R&D partnerships between Canadian industry and universities, and with international R&D organizations located in partner countries. It has produced guidelines for industry–to–industry partnerships between small and medium enterprises, and successfully involved them in both PDA and R&D Project components. It has made a significant contribution to R&D capacity building in several important areas but, protecting intellectual property rights and overcoming barriers to market access were cited as two areas in need of improvement to enhance Canada's competitive advantage and position.

The modest funding allocated to the ISTPP and the use of an alternate service delivery model for China, India and Brazil appears to contribute to the risk of diminishing rather than enhancing Canada's reputation as an S&T partner of choice. With the small number of R&D projects that can be supported at the current funding levels, the number of delivery organizations engaged in each country, with a lengthy lapse of time between the call for proposals and the delivery of the first Canadian payment, it is unlikely that ISTPP will reach the requisite critical mass required to generate significant economic benefits and workforce impacts from successful R&D commercialization. Based on a small proxy sample of CIIRDF–funded R&D projects pre–dating the launch of the ISTPP, the evaluation found that the expected economic benefits accrued to the partners was much less than expected and moreover favoured the country partner.

The ISTPP Steering Committee composition has evolved since the program's inception and functioned moreover as an informal group of interested departmental representatives than a governance committee with oversight responsibilities for strategic guidance and performance monitoring as presented in ISTPP's approval documents. The Steering Committee received annual verbal briefings from the delivery organizations on the progress of ISTPP but without written documentation and analyses in the form of Annual Activity Reports that would have permitted the members to objectively appraise performance.

While the ISTPP delivery organizations tracked and reported on process and output data to DFAIT, there was little evidence that the information was analysed and used to improve performance. For example, the evaluation was able to use the tracking data to calculate the estimated time required to launch approved R&D projects on a country basis and found lengthy delays making the program inefficient and unresponsive to the needs of Canadian R&D organizations and their international partners. Although almost a dozen R&D projects had been completed or were near completion, there was no evidence that any measurement systems were established by CIIRDF to collect and analyse data on expected outcomes or to monitor risks as recommended in the previous evaluations. Evaluators normally rely on performance measurement systems and data established by program management. However, given the absence of such systems and data for ISTPP, the evaluation had to generate data on the results of the program to fulfill its reporting mandate. The evaluation was able to systematically collect and analyse outcome data in a relatively short period of time from the ultimate recipients of ISTPP funding.

It is important to note that any areas for management improvement for ISTPP could have been identified earlier to the Steering Committee and the Department if they had been provided ongoing regular program analysis of ISTPP and CIIRDF performance. The absence of such programming analysis reduces the confidence of the ISTPP Steering Committee and the Department to comment on whether ISTPP objectives such as Canadian commercial and research interests are maximized and its programming (e.g., PDAs) is effectively targeting sectors of need and interest. The evaluation found that DFAIT would also have benefitted from a more closely analysed review of the provisions in the ISTPP contribution agreements including but not limited to such issues as ensuring the effectiveness of repayment and royalties in the short–term and over the long–term, as well as IP rights. Regular dialogue on such strategic issues with the ISTPP Steering Committee, the Joint S&T Committees and others would offer the opportunity for more accountability and enhanced engagement.

10.0 Recommendations

The following recommendations are derived from the evaluation findings and conclusions. They take into account a long–term perspective that bilateral S&T cooperation focussed on the international commercialization of R&D will play an increasingly important role in Canada's economic prosperity.

The evaluation found Canada's reputation as an S&T partner of choice for joint R&D collaboration appears limited in effectiveness unless current ISTPP funding increases. The evaluation did not pursue any line of questioning on the nature or extent of increase but partner countries representatives, companies and ISTPP stakeholders voiced this preference for attention to increased funding. It was recognized that ISTPP funds, albeit limited, fills a much–needed gap in the pre–commercialization of S&T innovation in terms of sustaining some SMEs through challenging economic fluctuations, offering others the opportunity to assess market demand for their products and in other cases, offering to risk–share the prospect of scientific discovery.

The evaluation found that while there is a need for considerable improvement in the implementation and management of ISTPP, attention should also be given to engage in a strategic dialogue with Canadian departments and agencies with an S&T mandate to ensure that ISTPP delivery does not duplicate other federal programs with an international S&T focus. Given that the overall responsibility and accountability for the ISTPP rests with the Minister of International Trade including its development, implementation and administration, it would appear reasonable to suggest that DFAIT lead on this dialogue.

It is the view of this evaluation that the ISTPP has been designed and governed in a manner which does not serve the long term purpose of the S&T Treaties, i.e., to encourage, develop and facilitate Cooperative Activities between the Parties in S&T innovation. The evaluation interviews had suggested now is an opportune time to consult with established S&T Joint Treaty Committees and refine the delivery of ISTPP in partner countries while it is still in the early stages of implementation. Specific areas of common interest and expertise will evolve over time, and quite rapidly as the pace of technological change advances so it would be imperatative for ISTPP to employ and strategically engage with Joint Committess to ensure Canada is seen as active in a competitve S&T market. The Joint Committees with the support of the S&T Sector Coordinators should be at the forefront of these changes and should provide strategic guidance on how best to invest ISTPP's resources for mutual economic and social benefit of all parties involved.

The evaluation found that the role of an ISTPP third–party delivery agent should be clearly defined and agreed–to as primarily supportive with accountability for the operational management of the ISTPP. Its independence in managing the competitive R&D Project proposal and evaluation process is of paramount importance to ensure fairness and transparency. However, the discretionary use of funds to support Partnership Development Activities (PDA) would better serve the ISTPP if it were directed by the Joint Committees through the S&T Coordinators. The ISTPP Secretariat should establish the financial mechanisms to manage the PDA funds in conjunction with the S&T Coordinators, while ISTPCanada or a third–party delivery agent, should fulfill an operational mandate to deliver ISTPP; a supportive role to facilitate the involvement of S&T Coordinators in selected activities.

The evaluation found that ISTPP delivery organizations have not given sufficient attention to managing the performance of their programs to demonstrating accountability.⁽⁸⁸⁾ The evaluation also found the need for the Department (and the ISTPP Steering Committee) to assume a sufficiently robust oversight role in the management of ISTPP by monitoring, reporting and analysis of the performance of ISTPP. The evaluation at the onset recognized the absence of reasonable performance expectations and did in the course of this evaluation conduct primary data collection and program analysis to report on performance. It also developed and validated a relevant Logic Model and where possible, collected information on other countries S&T programs on the measurement of success for S&T pre–commercialization. Therefore, this evaluation strongly encourages the ISTPP Secretariat or other relevant entity conduct an international benchmarking study to determine reasonable performance expectations.

As noted, the evaluation reported on many issues relating to the attribution of and the measurement of success of S&T endeavors that could be considered in the conduct of such a benchmarking study. Furthermore, ISTPP may wish to consider conducing, a "programming and results capacity check" to identify strengths and weakness in the current performance measurement and reporting systems and recommend improvements. One other option to consider is the use of conditions in the administration of ISTPP in an effort to assist Departmental program management to target areas for improvement.

11.0 Management Response and Action Plan

12.0 ISTPP Programming Activities

Israel: ISTPP Programming Activities

NB: There is no EOI stage in the CIIRDF R&D Project Call for Proposal process.

Partnership Development Activities

International Technology Matchmaking Service

CIIRDF focused early efforts to establish a platform to provide a systematic matching of Canadian and Israeli technology based SMEs through an approach that matches the needs of the requesting company with the features of the prospective partner. Through this program, CIIRDF provided Canadian SMEs better visibility into the R&D landscape and new partnering opportunities.

Newfoundland and Labrador, Oceans Technology Visit

In June 2006, CIIRDF hosted a delegation of 15 representatives from Newfoundland. One–on–one meetings were arranged between Canadian and Israeli companies and a Roundtable on Marine Technologies was convened. A map of the Israeli industry was commissioned for the purposes of the Roundtable. As a result of this activity, Newfoundland commissioned a map of its oceans technologies industry and a return visit of Israeli companies to Newfoundland in June 2009 was planned, culminating in a Roundtable in St. John's, NFLD. To date, there is one CIIRDF project, currently in progress, as a direct result of this activity.

Convergent Medical Technologies (CMT) Workshop

In May 2007, a workshop on convergent medical technologies was held at the MaRS facility in Toronto in collaboration with Health Technology Exchange (HTX) and the Ontario Centres of Excellence (OCE). The workshop attracted strong industry participation and also included science counsellors representing the embassies of India and China. The objectives of the workshop were to evaluate the state of the technology (innovation, investment and diffusion trends) in Israel, Canada and globally to establish a sound platform for optimizing cooperation in convergent medical technology and to consider various options for bi–lateral and multi–lateral cooperation. As a direct result of this workshop, HTX and OCE commissioned the Israeli company that developed these technology/ industry roadmaps to produce a systematic road mapping of Ontario capabilities in medical and assistive technologies. In addition, the workshop led to a delegation of 4 individuals from Ontario, representing more than 6 different Ontario companies, attending BioMed Israel in June, 2007 and the submission and approval of a CIIRDF project between an Ontario company and an Israeli company.

CleanTech Workshop

In October 2007, a CleanTech Workshop was organized, in partnership with OCE, at MaRS, to showcase Israeli capabilities in the cleantech sector. Over 70 participants representing a substantive core of Ontario's presence in environmental, energy and pollution control industries were in attendance. The objectives of the workshop were to showcase Israeli capabilities and establish a platform for Ontario Israel cooperation and collaboration. Among the features of the workshop was a presentation of an Israeli Magnet consortium consisting of leading Israeli industry and academic institutions, seeking Ontario participation.

Canadian Cleantech Delegation

In December 2007, a delegation of cleantech companies was organized. A Roundtable was organized with key Israeli leaders and one to one meetings were arranged for the participants with Israeli cleantech companies for the purposes of potential partnerships and collaborations.

Canadian University President Delegation

In May 2008, CIIRDF coordinated a delegation of Canadian University Presidents to Israel, to conduct an intensive study of Israel's strong success in the commercialization of university–based research and the rapid creation of innovation based companies that strengthen the country's technological expertise and international competitiveness. The visit has resulted in ongoing discussions on expanding platforms for Canada–Israel S&T cooperation.

Canada–China–Israel Agricultural Technology Partnering Mission to Yangling, China

In November 2008, CIIRDF and ISTPCanada broke important new ground by organizing a tri–lateral partnering mission aimed at promoting research cooperation involving Israel, China and Canada for the development of new agriculture and agri–foods technologies. CIIRDF and ISTPCanada brought a delegation of over 30 representatives from various agricultural technology companies in Canada and Israel to attend bi–lateral and tri–lateral matchmaking events during the Yangling Agricultural Hi–tech Fair (CAF 2008). A roundtable meeting of Agriculture officials was organized to discuss the value propositions of establishing a tri–lateral consortium/ center of excellence in agriculture science and technology as a platform for multilateral cooperation. This meeting resulted in the signing of the Yangling Declaration to ensure the advance of this trilateral initiative. CIIRDF and ISTPCanada held a follow on roundtable on Agri–Innovation in Ottawa in July 2009.

Second Canada–China–Israel Roundtable on Agri–Innovation

On July 28, 2009 delegates from China, Israel and Canada convened at the Second Canada–China–Israel Roundtable on Agri–Innovation in Ottawa, Ontario to advance groundbreaking trilateral research and development cooperation in the critical technological area of agriculture and agri–foods. Building on the outcomes of the Yangling Declaration emerging from the first trilateral meeting hosted in Yangling, China on November 5, 2008, the delegates agreed to engage in continued open dialogue on cooperation in agriculture and agri–foods R&D; and to launch a trilateral funding program that promotes cooperation among researchers and developers from the three participating countries.

China: ISTPP Programming Activities

* One project is on hold pending completion of funding agreement and one terminated.

Partnership Development Activities

NB: There is no EOI stage in the PDA Call for Proposal process.

* One project is on hold pending completion of funding agreement and one terminated.

International Technology Matchmaking Service

ISTPCanada has not yet established a platform to provide a systematic matching of Canadian SMEs and Chinese R&D organizations through an approach that matches the needs of the requesting company with the features of the prospective partner. The numbers of matches identified below show the result of Partnership Development Activities such as technology partnering missions to China.

Workshops Program

1. A 2–day workshop in Beijing and Shanghai on increasing productivity and protection of corn and rice.
2. A 3–day workshop in Chongqing on eutrophication.
3. A 3 day symposium on airway smooth muscle and asthma with participants from Chongqing, Beijing, Chengdu and Shanghai.
4. A 2–day workshop in Beijing on climate change, agriculture and biofuels.
5. A 2–day workshop in Guangzhou on network formation for advanced polymer processing and system technologies.
6. A 2 day workshop in Taiyuan and a 6 month student exchange on Clean Utilization of Coal
7. A 2–day workshop in Shenzhen on identifying opportunities for process improvement at municipal wastewater treatment plants in Shenzhen.
8. A 3–day workshop in Tianjin on low–power ultra–wideband integrated circuits for medical sensing and imaging applications.
9. Development of surveillance program of Johne's disease in livestock species by visits, meetings and joint pilot studies.
10. Sensor network systems for Aqueous Environmental Monitoring

Partnering Mission to Jiangsu Province, China

In November 2008 ISTPC, in partnership with the Jiangsu provincial government, organized a SME match making mission for Canadian SMEs to Jiangsu, China. Eight Canadian SMEs participated in this mission form across Canada covering various technologies such as ocean technologies, ICT, environment and food. The match making was held in the periphery of Jiangsu's first annual international technology transfer event. The conference turned out to be an excellent opportunity for Canadian companies to network not only with Chinese companies but also with companies from across the globe. Jiangsu province played a very active role in this mission, covering the accommodation and food for all Canadian SMEs for three days during the conference. The match making event attracted over 100 delegates from 30 Chinese companies interested in meeting Canadian companies. The follow up survey to the Canadian mission participants indicated that most of the Canadian companies are in close touch with their new partners in China and are developing successful working relationships.

Partnering Mission to Yangling Agricultural Fair, Yangling, China

In November 2008, together with Agriculture and Agri–Food Canada and the Yangling Agricultural Demonstration Zone Administrative Committee, ISTPCanada/CIIRDF organized the Canada–China–Israel Agricultural Technology Partnering Mission held at the 15th China Yangling Agricultural High Tech Fair (CAF) in Yangling, China. Canadian delegates participated in the technology focused partnering seminar, company presentations and the multiple one–on–one meetings with Chinese and Israeli companies. Following this mission, some Canadian SMEs have visited the site of the new partners in China, and others are at various stages of developing R&D collaboration with the partners in China and actively expanding the newly found market.

ICT Partnering Mission to Jinan, Chongqing, China

In April 2008 ISTPCanada, in consultation with Industry Canada and the Communications Research Centre Canada, led an ICT (Information and Communications Technologies) partnering mission to Jinan and Chongqing. Sixteen Canadian organizations, including R&D based companies, academia and associations, were able to pursue opportunities, partners and technology cooperation in China, as well as learn of the local policies and programs to support International S&T partnerships.

Second Canada–China–Israel Roundtable on Agri–Innovation

On July 28, 2009, delegates from China, Israel and Canada convened at the Second Canada–China–Israel Roundtable on Agri–Innovation in Ottawa, Ontario to advance groundbreaking trilateral research and development cooperation in the critical technological area of agriculture and agri–foods. Building on the outcomes of the Yangling Declaration emerging from the first trilateral meeting hosted in Yangling, China on November 5, 2008, the delegates agreed to engage in continued open dialogue on cooperation in agriculture and agri–foods R&D; and to launch a trilateral funding program that promotes cooperation among researchers and developers from the three participating countries.

India: ISTPP Programming Activities

* All projects are on hold pending completion of funding agreement.

Partnership Development Activities

NB: There is no EOI stage in the PDA Call for Proposal process.

International Technology Matchmaking Service

ISTP Canada established a platform to provide the systematic matching of Canadian SMEs and Indian R&D organizations through an approach that matches the needs of the requesting company with the features of the prospective partner. Through this program, ISTPCanada provided Canadian SMEs better visibility into the R&D landscape and new partnering opportunities.

Distinguished International Speaker Series

In June 2009 ISTPCanada and the Ontario Centres of Excellence (OCE) hosted DST Government of India as the featured presenter in the jointly hosted Distinguished International Speaker Series in Ottawa, Ontario. In the presentation, "Adapting an ancient civilization to modern ways of development in a knowledge economy…this is truly innovation" DST noted key opportunities and challenges facing India as it transforms politically and economically, and aims to create a resilient innovation ecosystem that connects higher education, research and policy to foster sustainable wealth creation. Engaging more than 150 representatives from government, industry and academia from across Ontario, the role and value of global R&D and technology alliances was underscored as part of this strategy.

Canada–India Biofuels Workshop

McGill University, Department of Biotechnology Government of India and ISTPCanada organized a roundtable discussion with key scientists and researchers from Canada and India in the area of biofuels on July 18, 2008. The topic of discussion was "Canada–India R&D collaboration for the Next Generation of Biofuels." Attending were DBT and 15 prominent scientists and experts from both countries in the area of biofuels. Building on previous bilateral discussions to discover areas of complementarities, the workshop was conducted via video conferencing between Delhi, Ottawa, Montreal and other Canadian cities.

Indo–Canada Workshop on Electricity Generation using Renewable Energy

This workshop was held at the Institute for Technology in Delhi, India from October 9–10, 2009. The goal was to strengthen collaboration between Indian industry and academic partners and their Canadian counterparts to develop practical solutions and recommendations in this crucial sector both for grid fed and off grid situations. At the end of this Partnership Development Activity (PDA), appropriate joint projects and teams comprising industry, academia, utilities and user agencies (for each project) from India and Canada were identified for follow up in liaison with local and global funding agencies. This was organised by the Institute for Sustainable Energy, Faculty of Engineering, University of Waterloo.

ISTPC Clean Technology Mission to India

In December 2008 ISTPC in collaboration with NRC–IRAP, DFAIT and GITA organized a clean technology matchmaking mission to New Delhi. With the active participation of NRC–IRAP this initiative received a tremendous response in Canada and sixteen companies registered for the mission. All companies were recruited from the NRC–IRAP network across Canada. The mission took place three days after the Mumbai attacks; consequently, a half of the registered companies declined at last minute because of security concerns. Eight Canadian companies, one venture capitalist and three clean technology experts still joined the mission. The response from the Indian companies was overwhelming. In certain cases there were five Indian companies interested in meeting each Canadian company. Two Canadian companies signed MOUs with companies in India.

There was major programmatic accomplishment during the mission where ISTPC held various meetings with its counterparts GITA and CII (Confederation of Indian Industry) to formalize the matchmaking process. GITA and CII have now jointly appointed a dedicated resource in India who will be available to ISTPCanada and will handle the match making activities in India. This includes both handling ISTPCanada requests to best fit Indian companies for Canadian SMEs and to help Indian companies connect to Canadian SMEs through ISTPCanada.

Nanotechnology Partnering Mission

In August 2008, ISTPC in collaboration with GITA organized a nanotechnology collaboration and match making mission in Canada. This mission was lead by Canadian and Indian expert coordinators. Ten of India's top nano–scientists and experts came to Canada and held joint information sharing sessions in Waterloo, Toronto, Montreal and Edmonton. Canadian attendees included researchers, scientists and SMEs. NRC–IRAP marketed the event to its SME network.. During this mission, a MOU was signed between Alberta and India and a workshop was organized in collaboration with the following local Canadian partners: University of Waterloo in Waterloo, University of Toronto in Toronto, Nano Quebec, Quebec Government and University of Montreal in Montreal, and the University of Alberta, NINT (National Institute of Nano Technology) and Alberta Government in Edmonton.

The Canada–India B2B Partnership Summit on Life Sciences

In January 2010, ISTPCanada brought together key stakeholders from industry, academia and government in British Columbia and India to forge new partnerships leading to bilateral R&D and business collaboration. The event was hosted by the Ministry of Small Business, Technology and Economic Development, Government of British Columbia, and ISTPCanada. The India delegation was led by Department of Biotechnology (DBT) and Confederation of Indian Industry (CII). Canada–India B2B Partnership Summit on Life Sciences brought together more than 100 key stakeholders from industry, academia and government in British Columbia and India to forge new partnerships leading to bilateral R&D and business collaboration. The event featured a roundtable discussion between experts from BC and India to discuss what is important to SMEs in the life sciences when developing bilateral R&D or business partnerships. It also provided a wealth of networking and matchmaking opportunities for participating SMEs, in conjunction with BioPartnering North America.

Key outcomes of the Summit were to include the development of new B2B partnership–development guidelines that can be used by Canadian and Indian innovation agencies and other organizations to develop effective matchmaking programs that promote strong R&D alliances and accelerate business collaboration. These guidelines, and other outcomes from the Summit, also directly support the development of a BC–India B2B Partnering Service for SMEs in life sciences.

Brazil: ISTPP Programming Activities

* All projects are on hold pending completion of funding agreement.

Partnership Development Activities

NB: There is no EOI stage in the PDA Call for Proposal process.

* All projects are on hold pending completion of funding agreement.

The Canada–Brazil Bilateral Workshop on the Efficient Production of Cellulosic Biofuels

Together with Agriculture and Agri–Food Canada (AAFC), the NSERC Green Crop Network, the Cellulosic Biofuels Network and EMBRAPA (Agricultural Research Corporation of Brazil), ISTPCanada convened more than 50 scientists and delegates from academia, industry and government to explore opportunities for bilateral R&D and business cooperation at the Canada–Brazil Workshop on the Efficient Production of Cellulosic Ethanol on March 13, 2010 in Ottawa, Ontario. With a focus on the development and application of novel biofuels technologies, this initiative could help to stimulate increased trade, foreign investment and business opportunities that create competitive advantage for both countries.

The workshop brought together a Brazilian delegation led by EMBRAPA Agroenergy, and Canadian representatives from eight companies, 11 universities, and more than a dozen government departments and research networks with expertise in cellulosic biofuels. The event enabled participants to meet with prospective partners, identify key opportunities for Canada–Brazil R&D cooperation, discuss potential bilateral R&D projects and prospective sources of investment; and define the next steps. The group plans a follow–up meeting in Brazil in July 2010 to explore initial implementation of select bilateral initiatives.

1. The ISTPP Accountability, Risk and Audit Framework (March 2006) fulfilled this requirement.

2. Data Source: Results–Based Management and Accountability Framework and Risk–Based Audit Framework, 2006

3. The membership of the Steering Committee is at the Director–General level. It is comprised of Industry Canada, Natural Resources Canada, National Research Council of Canada, Agriculture and Agri–Food Canada and, Natural Sciences and Engineering Research Council.

4. ISTPP planned contribution allocation was $1.5M for Brazil; however, the actual ISTPP contribution delivered to ISTPCanada for Brazil was $1.4M as cited in an amended contribution agreement in 2009–10 and in the final recipient audit of March 31, 2010.

5. Data Source: Recipient Audit, March 31, 2010

6. This appears to be a report to NRC–IRAP for networking services. It is the only report received to date with regard to the R&D Matchmaking component.

7. CCIRDF is a mature operation in effect since 1994; therefore, data on actual sales and revenue generated was collected to assess economic benefit. The 2004 CIIRDF summative evaluation noted that based on 18 projects, commercialization plans indicated an estimated $715M CDN in sales revenue and $178M CDN in profits by 2013.

8. OECD, Bilateral Trade Database.

9. Office of the Chief Scientist Israel.

10. Office of the Chief Scientist Israel.

11. (http://www.moit.gov.il/NR/rdonlyres/CD3AF19B–2619–415B–B2F4–B747101C5202/0/TheIntellectualCapital3550.pdf)

12. World Development Indicators, Organization for Economic Co–operation and Development (OECD) as cited in Ministry of Science and Technology, Action Plan 2007–2010 ( Brazil). Note: the latest 2009 OECD data tables on % GDP on R&D data is for 2007: Can –1.89%; USA –2.68%; Swe – 3.63%. Year over year changes have been slight on final expenditures on R&D as a % of GDP. For example, in 2006, OECD reported CAN–1.94%; USA –2.66%; Sweden–3.74%. USA and Sweden were selected to show expenditures on R&D investment relative to economic prosperity.

13. Next steps will need to be developed between Brazil and Canada on the composition of the Joint Steering Committees, sectoral interests and needs.

14. As presented in Government of Canada approval documents.

15. Note that the Government of BC funding is for collaborative research not allocated to companies; therefore, not industrially–oriented.

16. Government of Canada International Policy Statement (2005) pg 22.

17. Mobilizing Science and Technology to Canada's Advantage (2007), pg 85.

18. Data Source: Science, Technology and Global Innovation: Trends, Best Practice and Future Prospects for Canada, DFAIT.

19. Of the $20 Million CDN, $18.5M was allocated for ISTPP programming and administrative costs and $1.5M was allocated for Departmental resources.

20. Data Source: Science & Technology Partnerships: The Canadian Way (2006)

21. These positions were discontinued in 2008.

22. Canada International Policy Statement (2005) pg 59.

23. As presented in Government of Canada approval documents.

24. ISTPCanada levies a 15% administrative overhead charge on the contract.

25. ISTPCanada levies a 15% administrative overhead charge on the contract.

26. As presented in Government of Canada approval documents.

27. As presented in Government of Canada documents.

28. Annex A in the contribution agreements list the undertakings.

29. As presented in Government of Canada approval documents.

30. Advantage Canada: Building a Strong Economy for Canadians (2006) pg 22.

31. 2008 Formative Evaluation, page 35.

32. Industry recipients of IRAP funds may partner with universities.

33. Volunteer peer review is an accepted practice among many disciplines so it is not uncommon to request such reviews from R&D proposal proponents.

34. As presented in Government of Canada approval documents.

35. Annual Innovation Report (UK Department for Innovation and Skills, 2009) pg 18.

36. The Technology Strategy Board has been established by the Government and operates at arm's length as a business–led executive non–departmental public body. It is sponsored and funded by the UK Department for Business, Innovation and Skills. The budget available for the Technology Strategy Board's work is currently around £200m per year.

37. Data Source: 2008 Annual Report, United Kingdom Technology Strategy Board, pg 31.

38. As presented in Government of Canada approval documents.

39. Monthly unaudited financial report submitted to DFAIT, March 2010.

40. Data Source: ISTPP for China, India and Brazil, Final Recipient Audit, March 2010.

41. Includes Administrative costs. Recipient audit, March 2010.

42. Data Source: ISTPCanada (CIIRDF) unaudited figures as of March 2010, subject to adjustment pending final audit.

43. Data Source: ISTPP evaluation on–line surveys 2009–10.

44. Proposals are assessed with regard to commercialization, manufacturing and marketing perspectives using the following criteria: clear statement of customer need being addressed; evidence of market demand (e.g., conditional orders, market research); initial target customers (include specific names if possible); size and growth rate of target market; and competitive alternatives.

45. OECD Science, Technology and Industry Scoreboard 2009: Canada Highlights.

46. OECD Science, Technology and Industry Scoreboard 2009: Canada Highlights.

47. Based on data collected from the summative evaluation interviews and documents from joint meetings as well as other fora.

48. As presented in Government of Canada approval documents.

49. Based on the results from online surveys, in–person interviews with Israeli companies and telephone interviews with Canadian companies,

50. 2004 CIIRDF Summative Evaluation.

51. Refer to the ARAF and Logic Model.

52. ISTPCanada selected these two projects in their capacity as holding expert corporate knowledge of transactions over time to be able to illustrate successful collaborations that are attributed to CIIRDF's efforts.

53. EAC asked that these attestations be included in the evaluation as supporting evidence of CIIRDF's efforts.

54. February 2010 was the authored date of these attestations.

55. For one project, royalties were paid. For the second project, apparently two–thirds of the contribution was repaid and 25% was absorbed as a loss by CIIRDF. The evaluation did not review audited statements.

56. Due to the wide variation between the online survey accounts and the commercialization reports, the evaluation was not able to ascertain the accuracy of these accounts to the same extent as with the 2004 CIIRDF cohort.

57. The on–line survey was designed to collect precise answers to address the commercialization aspects agreed–to for the summative evaluation to ascertain results for ISTPP–CIIRDF and therefore, if completed would provide a more accurate delineation of attribution.

58. The validity of non–probability samples can be increased by eliminating as many sources of bias as possible. Non–probability samples are limited with regard to generality.

59. Data Source: ISTPCanada (2009–10) CIIRDF recipient audit financial reports as of March 31st, 2010.

60. Data Source: Recipient Audits and Financial Reports: ISTPCanada and CIIRDF, March 2010.

61. Includes administrative costs.

62. Data Source: ISTPCanada (CIIRDF) unaudited figures as of March 2010, subject to adjustment pending final audit

63. ( –– ) denotes shortfall due to PDA and R&D costs.

64. Data Source: Recipient Audits and Financial Reports: ISTPCanada and CIIRDF, March 2010.

65. Data Source: ISTPCanada (CIIRDF) unaudited figures as of March 2010, subject to adjustment pending final audit.

66. A description of PDA activities undertaken with each country is provided.

67. Data Source: Recipient Audits and Financial Reports: ISTPCanada and CIIRDF, March 2010.

68. Data Source: ISTPCanada (CIIRDF) unaudited figures as of March 2010, subject to adjustment pending final audit.

69. ISTPCanda–CIIRDF confirmed the accuracy of these statistics.

70. It may be possible that the volume of proposals was lower in the second call for proposals was targeted on Bio–Health and was not an open call.

71. Data Source: ISTPCanada, Board Approved Projects, Approval and Payment Schedule. Dates were confirmed by ISTPCanada.

72. Excludes statutory holidays.

73. Organization includes company, university and research institute/think–tank.

74. Data Source: ISTPCanada, February 2010.

75. Average Elapsed Work–Days is a composite indicator based on the total number of work days (excluding statutory and Ontario civic holidays) for each project executed (Board approval date to First Canadian payment date) divided by the total number of projects executed by Board approval date. All calculations of Totals are rounded up. The average elapsed work days indicator is a mid–point analyses of high and low values in a range of scores; therefore, the resulting average elapsed work–days has been normalized. Individual average elapsed work–days vary by actual project.

76. Average elapsed work–days, when converted to months, are derived based on the Canadian standard of an average of 20 work days per month. Again, this calculation normalizes the actual number of available work–days per month which can vary depending on statutory holidays and other factors.

77. Based on formally signed and agreed to transactions that was collected during the reference period for data collection for this evaluation (January to March 2010).

78. Includes total R&D disbursements for India (15%), China (20%) and Israel (65%). Data Source: Recipient Audit, Financial Reports: ISTPCanada and CIIRDF, March 2010.

79. Based on formally signed and agreed to transactions that was collected during the reference period for data collection for this evaluation (January to March 2010).

80. Two projects represented the same firms. One project was cancelled.

81. Brazil Innovation Agency – Ministry of Science and Technology

82. Science, Technology and Innovation. Source: Second International Workshop Presentation Deck, President of FINEP, FINEP, April 2007, Rio de Janerio

83. World Development Indicators, 2007. Organization for Economic Co–operation and Development (OECD) as cited in Ministry of Science and Technology, Action Plan 2007–2010 ( Brazil). Note: the latest 2009 OECD data tables on % GDP on R&D data is for 2007: Can –1.89%; USA –2.68%; Swe – 3.63%. Year over year changes are slight.

84. Brazil Agriculture Research Cooperation

85. Brazil Ministry of Science and Technology, Action Plan, 2007–2010.

86. As presented in Government of Canada approval documents.

87. As presented in Government of Canada approval documents.

88. "Accountability is a relationship based on the obligation to demonstrate and take responsibility for performance in light of agreed expectations", defined by the TBS in Modernising Accountability Practices in the Public Sector, http://www.tbs–sct.gc.ca/rma/account/oagtbs02_e.asp

Office of the Inspector General

^* If you require a plug–in or a third–party software to view this file, please visit the alternative formats section of our help page.

Primary navigation

Date Modified:

2013-01-21