Language selection

Evaluation of the Partnerships Contribution Program - April 2008

ACKNOWLEDGEMENT

Audit and Evaluation Division acknowledges the time and effort given by departmental managers and staff to provide information associated with this report.

Executive Summary

The Partnerships Contribution Program (PCP) was created from the Voluntary Sector Initiative (VSI), a five-year joint initiative between the Government of Canada and the voluntary sector that was launched in June 2000. The overall goal of the initiative was to improve the quality of life in Canada. Specifically, the VSI focused on strengthening the relationship between the voluntary sector and the government, and, enhancing the capacity of the voluntary sector.

The PCP allowed Veterans Affairs Canada (VAC) to enter into Contribution Agreements, with a range of targeted recipients such as non-profit organizations, museums, educational, research and health institutions, and the provinces, territories and municipalities in order to work in a collaborative way to further VAC's mandate in providing and improving Veterans benefits and services as well as in undertaking commemorative activities.

VAC plans to seek renewal of the PCP's Terms and Conditions (Ts and Cs) in the Spring of 2008. Prior to seeking renewal VAC committed to conducting an evaluation of the Program. The emphasis of the evaluation was on assessing the relevance and the achievement of objectives, with program efficiency, economy and cost-effectiveness also to be assessed.

The evidence supports a need for VAC to continue to support the organizations interested in Remembrance through the Community Engagement Partnership Fund (CEPF) and the Cenotaph/Monument Restoration Program (CMRP). The use of the Program is growing, the target groups have evolved to include youth and CF Veterans and the Program continues to be congruent with the priorities of both VAC and the Government of Canada.

The evidence was mixed on the rationale for Policy, Programs and Partnerships ( the health, social and economic programs branch of the Department) to maintain involvement in the PCP. Although the objectives of the PCP are relevant to the work PPP performs, PPP chose other means besides partnering with the voluntary sector to accomplish their mandate. There were two projects completed through the VSI/PCP. Currently, PPP Branch has expressed an interest in using the PCP to assist in a variety of research related initiatives that could be supported by the PCP.

Organizations that have benefited from PCP support stated that the funding received has been important to the success of the project. Also, the majority of projects funded stated the financial assistance either allowed the project to proceed or the funds had an incremental impact on the scale and quality of the project.

All applications for funding follow the same processes and are delivered in the same manner. However, the scope and resource requirements for projects can differ greatly, with contributions ranging from less than $1000 to more than $100,000. Streamlining the requirements and approval process for smaller dollar projects amounts would lead to improvements in efficiency, cost-effectiveness and economy.

There are opportunities for VAC to improve the cost-effectiveness, efficiency and economy of the Program now and over time as performance information is collected and analyzed. With the implementation of the evaluation's recommendations and emphasis placed on implementing a performance measurement strategy, VAC would be able to realize improvements in cost-effectiveness, efficiency and economy.

Summary of Recommendations

  • R1 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, in consultation with the Senior Assistant Deputy Minister, Policy, Programs and Partnerships Branch, update the Program's objectives to reflect both the use of the Program to date and the expected future use of the Program.
  • R2 It is recommended that the Director General, Program and Service Policy Division, determine the extent the Branch will use the PCP in coming years. If PPP Branch plans to continue to participate in the PCP, along with revising its objectives, PPP Branch should develop a plan to seek funds in the near future.
  • R3 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, assess whether or not repeat funding should be allowed for exceptional projects. Projects often need more than one year of "seed money" to help build a brand/reputation and seek other funding sources to sustain and grow the project.
  • R4 It is recommended that the Director General, Canada Remembers Division, improve the availability of Program information and Program access by:
    • Developing a standardized application form for the CEPF. Also, the form should be available on the VAC website;
    • Improve the Program's presence on VAC's website. Ensure detailed information is available, including the application form, program and process details, including eligibility criteria, key processes, and approximate time from application submission to funding decision;
    • Develop and distribute nationally, marketing and communications materials for the Program, to help ensure universal access to the Program;
    • For the CMRP, develop standards for typical improvements (i.e. for flag poles, wheelchair accessibility, drainage etc.) and post these online to assist organizations in their project planning, costing, and implementation;

      and
    • List and put links to other potential funding sources online to assist organizations in their funding efforts.
  • R5 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, seek other funding framework options for the Juno Beach Centre Association (JBCA). By making large ongoing payments to a single organization, the effectiveness of the Program is restricted.
  • R6 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, require the JBCA to provide more complete performance data and financial data to increase accountability.
  • R7 For the CEPF, it is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, define more precisely in PCP policy, what financial contribution organizations must make to their project, both in-kind and from cash sources (for example, the CMRP funds up to 50% of the cost, requiring organizations to provide 50% of funding).
  • R8 It is recommended that the Senior Assistant Deputy Minister, Policy, Programs and Parternerships, seek alternative arrangements for three non-compliant contribution agreements currently in effect.
  • R9 It is recommended that the Director General, Canada Remembers Division, ensure the CMRP's Review Committee keep Records of Decisions with signatures of Committee members when decisions are made.
  • R10 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch:
    • design new streamlined requirements for smaller dollar projects (i.e. funding $5000 and less) to ease the burden on both VAC resources, and organizations that do not have the resources to easily comply with all of the standard requirements, and/or CRDshould consider establishing a grant component for qualified projects; and
    • Delegate approval authority to the ADM, SD&C (i.e. for funding $5000 and less).
  • R11 It is recommended that the Director General, Canada Remembers Division, implement quarterly application deadlines and develop more precise criteria for determining project selection.
  • R12 It is recommended that the Director General, Canada Remembers Division, prioritize the projects by type to maximize program results, and clearly define what types of projects are eligible for funding.
  • R13 It is recommended that the Director General, Canada Remembers Division, assess and ensure its funding is proportionate to the benefits received when funding all projects and especially large national scope projects.
  • R14 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, implement a performance measurement strategy to support managing for results, proportionate to VAC's investment, and for determining if outcomes and targets are being achieved. Also, VAC should establish a "dashboard" of measures that should be reported upon yearly and be used to improve the performance of the Program.
  • R15 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, ensure the funds CRD notionally allocates to the Regions for CEPF projects are sufficient to meet the demands on the Program.

1.0 Introduction

1.1 Background

The Partnerships Contribution Program (PCP) was created from the Voluntary Sector Initiative (VSI), a five-year joint initiative between the Government of Canada and the voluntary sector that was launched in June 2000. The overall goal of the initiative was to improve the quality of life in Canada. Specifically, the VSI focused on strengthening the relationship between the voluntary sector and the government, and enhancing the capacity of the voluntary sector. The VSI was based on the recognition that the voluntary sector is one of the three pillars of Canadian society, equal in importance to the public and private sectors. As a horizontal initiative the VSI formally ended on March 31, 2005, however the government continues to support ongoing work in individual departments.

The PCP allowed Veterans Affairs Canada (VAC) to enter into Contribution Agreements, in accordance with Treasury Board Secretariat's Transfer Payment Policy, with a range of targeted recipients such as non-profit organizations, museums, educational, research and health institutions, and the provinces, territories and municipalities in order to work in a collaborative way to further VAC's mandate in providing and improving Veterans benefits and services as well as in undertaking commemorative activities.

1.2 Description and Evolution

The initial funding for the PCP was approved by Treasury Board (TB) in July, 2001, under the VSI's Sectoral Involvement in Departmental Policy Development. The PCP's Terms and Conditions (Ts and Cs) were approved in November 2001. The PCP is delivered and managed by both Canada Remembers Division (CRD) which forms part of Service Delivery and Commemoration Branch (prior to a reorganization April 1, 2008 the Branch was called Public Programs & Communications), and Programs & Service Policy Division (PSPD) which forms part of Policy, Programs and Partnerships Branch (prior to the reorganization the Branch was called the Veterans Services Branch). The initial funding was $37,500 for 2001-02 and $262,500 for 2002-03 (to assist the Royal Canadian Legion in a housing study). Additional funding was dependent upon availability of funds from the VSI and Departmental reference levels.

A further $118,000 for VSI initiatives was granted to VAC by TB in 2002 for a project by the Victorian Order of Nurses (VON), an organization that provides home and community care for seniors, to assist VAC with the development of health care policy for Veterans. This was the extent of Policy, Programs and Partnerships' funding requests for VSI/PCP initiatives. Subsequent PCP requests for funding have been exclusive to the Canada Remembers Division.

The Ts and Cs were approved by TB in November 2001 and the objectives of the PCP outlined in the Ts and Cs were to:

  1. identify and understand the evolving needs of existing and new clientele;
  2. aid in the adaptation and delivery of VAC programs (benefits and services) to respond to the needs of clients;
  3. aid in the integration and co-ordination of the multiple players (agencies, institutions, associations and other levels of government) who have a role in the provision of 'quality of life' services and benefits to VAC clients; and
  4. ensure that the achievements and sacrifices of Canada's citizens in the defence of freedom are recognized, and the historical significance of these accomplishments are nationally promoted and celebrated.

PCP's expected results and outcomes include:

  • improved information about current and future clients and their needs, upon which pro-active and responsive policy and services can be developed and delivered;
  • improved information, integration and coordination of benefits and services relating to the financial, physical, mental and social health and well-being of VAC clients, resulting in collaborative, seamless and transparent services to VAC clients; and
  • commemorative events, materials and monuments which sustain and improve Canadian's appreciation for and awareness of the contribution of VAC clients to our national well-being and heritage.

When the PCP's Ts and Cs were developed, it was expected that the uptake would be widespread with a focus on activities surrounding Veteran's health and well-being as stated in three of the four PCP objectives. However, the use of the PCP has been almost exclusive to pursuing the objective relating to commemoration. VAC's Canada Remembers Division has used the PCP to further its mandate of endeavouring to keep alive the achievements and sacrifices made by those who served Canada in times of war and peace; to engage communities in remembrance of these achievements and sacrifices; and, to promote an understanding of their significance in Canadian life as we know it today. CRD's use of the PCP has evolved and grown, with the Program's funds directed to two major initiatives: The Community Engagement Partnership Fund (CEPF) and the Cenotaph/Monument Restoration Program (CMRP).

1.3 Community Engagement Partnership Fund

In 2003-04, the PCP became an integral part of CRD's plan to engage local communities in commemorative initiatives. In September 2003, with the implementation of a government-wide remembrance policy, TB approved PCP funding in the amount of $350,000 for fiscal year 2003-04, and $700,000 per year for ongoing funding.

The Government of Canada proclaimed 2005 as the Year of the Veteran, a year that would commemorate the 60th Anniversary of the end of the Second World War. To support additional commemorative projects throughout Canada in 2005 TB approved a one-time increase to the PCP's reference levels by an additional $400,000 in 2004-05 and an additional $1 M in 2005-06.

PCP funding of community engagement projects has increased substantially from 15 organizations receiving funding in 2003-04 to 112 organizations receiving funding in 2006-07. This rise is due in part to VAC's outreach strategy to increase Canadians' awareness of commemoration and the visibility of this contribution funding at the community level. Commemorative activities that have received funding include reunions, ceremonies, parades, theatrical performances, and the development and distribution of educational materials.

From its inception in July, 2001 until April, 2005, the PCP was managed and delivered out of Veterans Affairs Head Office in Charlottetown. However, commencing April 1, 2005, the VAC's four Regions (Atlantic, Quebec, Ontario, and Western) were delegated the authority by CRD to recommend and administer individual CEPF proposals up to a maximum of $5,000 each. This has enabled VAC to further enhance the Program through the Regions' knowledge of and experience interacting with local organizations and their ability to outreach to the community. The Regions currently have been notionally allocated a budget of $40,000 per year, with additional funding available from Head Office if the Regions require more funds.

The CEPF includes significant funding to the Juno Beach Centre Association (JBCA). Currently, VAC has entered into an ongoing contribution agreement with the JBCA for $500,000 per year to cover ongoing expenses. The Juno Beach Centre (JBC), a 10,000 square foot interpretive centre in Normandy France, was opened in June 2003 and it presents the war effort made by all Canadians, civilian and military alike, both at home and on the various fronts during the Second World War. The Centre was constructed by the JBCA, a non-profit organization with a Board of Directors based in Burlington, Ontario. The JBCA anticipated the Centre would be a self-funding entity with revenue generated by admissions and a gift shop, and coupled with a memorial brick campaign, revenues were expected to be sufficient to cover annual expenses. Millions of dollars were raised by the JBCA, however, revenues did not meet expectations, and the Centre requested VAC's assistance on numerous occasions to complete the Centre and to provide ongoing funding support.

Recognizing the Centre's commemorative importance as Canada's only Second World War Memorial in Europe. VAC, through the PCP, has provided financial assistance to the JBCA since 2002 to assist with start-up costs and to support the Centre's operations. The following table outlines the funding VAC provided to the JBC.

Table 1 Funding to the JBCA
Year Description Amount
2002-03 Partnership Contribution Agreement (PCA) $1.025M
2003-04 PCA $1.775M
2004-05 PCA

Grant
$1.16M

$0.34M
2004-05 to 2006-07 PCA $.2M
2007-08 and ongoing PCA $.5M

Note: Additional funds totaling $0.285M were provided for displays and commemorative events over these five years.

Exemptions to the PCP's Ts and Cs were sought by VAC and granted by TB to allow for the ongoing funding to the JBCA as the Ts and Cs do not allow for support to initiatives if they represent or establish a requirement for ongoing funding. The exemption was granted for this by TB in September 2003. Another exemption to the Ts and Cs to accommodate JBCA included allowing an exemption so that a payment in excess of $500,000 could be made from the PCP. These exemptions were only for the JBCA.

When VAC decided to support the funding requests from the JBCA, it looked into options for supporting the Centre. One option was to establish a grant to the JBCA. A grant appeared to be the favoured mechanism to support the Centre, however, due to time constraints and the requirements for establishing and paying a grant, which include developing Ts and Cs for the grant program and listing the grant in the Public Accounts, consultation between VAC and TBS led to contribution agreements being established through the PCP framework. However, the JBCA contribution agreements are very different from other agreements. As mentioned, multiple exemptions to the Ts and Cs were necessary prior to developing the contribution agreements. Also, the funding is ongoing, supports annual operational expenses, and are of a dollar amount significantly larger than other contribution agreements.

1.4 Cenotaph/Monument Restoration Program (CMRP)

In May 2005, VAC established the CMRP to assist communities across the country to properly conserve cenotaphs/monuments that honour Canada's War Dead and Veterans. There are an estimated 6,000 local cenotaphs/monuments that are dedicated to Canada's War Dead and Veterans across the country. The vast majority of monuments in Canada were erected 60 to 90 years ago by community groups, provinces, private sponsors, regimental associations, or Veteran's organizations. Over time many of these cenotaphs/monuments have fallen into disrepair and either the groups that built them have been unable to provide the care required to restore them or the groups no longer exist.

VAC had been approached by Veterans stakeholder groups, Parliamentarians and the Canadian public on many occasions to assist communities and organizations with their cenotaph/monument conservation projects, but VAC did not have a program that could support these requests. The CMRP was created and it would be delivered through the PCP with the objective of providing financial assistance to communities and non-profit organizations across the country to properly conserve these silent reminders to the high standard of care and dignity that they deserve.

The program was designed with a cost sharing aspect in which VAC may provide financial support of up to 50% of eligible expenses, not exceeding a maximum of $25,000. The organizations seeking funds are responsible for the remaining expenses, of which a minimum must be 25% in cash from sources other than Veterans Affairs Canada, and up to 25% can be contributions made by in-kind donations. This formula helps to ensure that there is tangible community support for the project and provides a method to engage the public in the project. Also, VAC would provide expert advice to organizations to ensure the conservation work stands the test of time, and technical materials were to be available on VAC's web site to assist groups in selecting proper methods and materials for their particular needs.

In order to deliver the CMRP through the PCP, two amendments to the Program's Ts and Cs were granted by the TB. Capital costs were not eligible for funding under the PCP and the restoration of non-government cenotaphs and monuments was considered to be a capital expenditure. Therefore, TB amend the Ts and Cs to allow capital costs as an eligible expense for the restoration of existing cenotaphs/monuments only. Also, the Ts and Cs were amended (for cenotaph/monument projects only) to limit VAC's contribution to 50% of funding to ensure community support and engagement. The CMRP is expected to grow as it becomes more known, and it is marketed and communicated by VAC to stakeholders. Additional PCP funding was approved when the CMRP was added to the Program: $1M in 2005-06, $2.0M in 2006-07 and $1.5M 2007-08 and ongoing.

1.5 Other VAC PCP Initiatives

VAC used the PCP to fund The "Seniors' and Veterans' Home and Community Health Interests" project conducted by the Victorian Order of Nurses and the "Seniors' and Veterans' Housing on a Continuum of Care Spectrum" project conducted by the Royal Canadian Legion (RCL. These projects, conducted between 2002/03 and 2003/04, met the objectives of the PCP and supported or resulted in changes to VAC's policy and/or programs.

The project supported by a contribution agreement with the VON was to conduct a study to determine the present and future home and community care needs of a growing segment of the Canadian population, Veterans and seniors. This study resulted in policy recommendations to VAC to address the concerns raised in the study. For example, the report recommended a role for each of the three levels of government to play as well as how to better communicate with seniors. VAC continues to work with other levels of government and stakeholder organizations to ensure client needs are met.

The project supported by a contribution agreement with the RCL examined various policy issues affecting the housing of seniors and Veterans across Canada. Many recommendations were made, including the continuation of VAC's support for the Legion's Centre for Excellence in Housing in order to continue to provide expert advice and guidance to all Legion Branches developing housing for Veterans and seniors across Canada. This Centre of Excellence continues to be operational at VAC in support of this initiative.

As a result of an organizational restructuring, the Division that managed the PCP for the Policy, Programs and Partnerships Branch was disbanded. As a result, Management decided to not continue with the use of the PCP as resources were limited, other major initiatives were underway, and other mechanisms, such as contracting, were used to obtain information on client needs and adapting programs and services. However, VAC's Policy, Programs and Partnerships Branch currently has contribution agreements with three other organizations. These are three-year agreements that are to provide services or ensure services are available to VAC clients. VAC is pursuing other funding options with these organizations as using the PCP was only to be an interim measure.

The following Table depicts PCP expenditures and Contribution Agreements since inception:

Table 2 PCP Expenditures and Number of Contribution Agreements
Year Expenditures* Number of Contribution Agreements*
01/02 $18k 0
02/03 $1.25M 8
03/04 $2.28M 15
04/05 $2.29M 41
05/06 $2.32M 173**
06/07 $2.20M 166

* Includes Juno Beach Centre

** 2005/06 was the Year of the Veteran and there was more funds available for CEPF projects

2.0 Study Approach

2.1 Evaluation Context

As part of the TBS's Transfer Payment Policy, there is a requirement to evaluate all Grants and Contribution programs, including the PCP. The emphasis of the evaluation was on assessing the relevance and the achievement of program objectives, with program efficiency, economy and cost-effectiveness also to be assessed. VAC plans to seek renewal of the PCP's Ts and Cs in the Spring of 2008. Prior to seeking renewal VAC committed in the VAC 2007/08 Audit and Evaluation plan to conducting an evaluation of the Program.

The intent of the evaluation was to provide VAC management with findings, analysis and recommendations regarding areas for improvement in the administration and delivery of the Program. The results of the evaluation will be used to assist VAC management in updating the Terms and Conditions, provide management and central agencies with an assessment of the Program, and to improve the delivery of the program.

The Objectives of the evaluation were to:

  • Evaluate the relevance of the Partnerships Contribution Program;
  • Evaluate the success of the Partnerships Contribution Program;
  • Evaluate the effectiveness of the Partnerships Contribution Program;
  • Evaluate the economy of the Partnerships Contribution Program; and
  • Evaluate the efficiency of the Partnerships Contribution Program.

The Terms of Reference for the evaluation were approved by the Audit and Evaluation Committee. The evaluation's scope included all major aspects of the Program's activities from inception in 2001/02 to September 30, 2007.

The evaluation issues that the evaluation encompassed were:

  • Relevance: The extent to which there is a need for the PCP, and the program is consistent with VAC's and government-wide priorities.
  • Success: The degree to which the program is effective in meeting its objectives.
  • Effectiveness, Economy and Efficiency: The extent to which the most appropriate and efficient means are being used to achieve objectives, relative to alternative design and delivery approaches.

Details regarding the evaluation issues and questions are provided in Annex 1.

A compliance audit of the PCP was conducted by VAC's Audit and Evaluation Division in 2005/06. The audit recommended areas for improvement and Management Action Plans were developed in response to address the recommendations.

2.2 Methodology

An evaluation framework was developed in June, 2007 followed by background research and other planning activities. Fieldwork was conducted between September, 2007 and January, 2008. First, interviews were conducted with both Head Office and Regional Office personnel, followed by a file and document review and analysis, including a review of eighty Program files. Also, a number of funding recipients were interviewed for feedback on both the Program and key administrative processes.

2.2.1 File and Document Review and Analysis

The File and Document review was undertaken on VAC's involvement with the PCP. Two types of files were reviewed: 1) a general review of program files (such as administrative records, program records, financial records, communications and media); and 2) a more systematic review of select recipient files (Program files).

The review covered the following:

  • Administrative records (includes information such as the number of participants/rejected applicants per region, type of participants, type of projects funded, duration of projects, cost of projects (PCP and in-kind), etc.);
  • Applicable VAC policy and legislation;
  • TBS documents (TB submissions, Terms and Conditions, Transfer Payment Policy, etc.) Information that describes how and why the program was created and what changes and influences (if any) since the program began.
  • Financial records provided information on the program funds allocated and expended.
  • Program files (proposal, review committee assessment, letters, contribution agreement, invoices, monitoring, reporting, feedback etc.). The review covered PCP recipients (National and Regional) and rejected applicants.
  • Strength of the partnerships with volunteer organizations (i.e. constraints, funding).
  • Other grant and contribution programs available to organizations from federal, provincial, municipal governments, and from the private sector were researched and reviewed;
  • Media coverage (e.g. television, newspaper, radio, Internet); and
  • Communications functions.

The objective of the file review was to develop an understanding of the types and variety of projects and organizations the PCP supports, to assess the key steps and timeliness of the program delivery, and to help assess the need for, success and effectiveness of the program.

The information collected in the project file review allowed for the analysis of:

  • types of projects funded and denied;
  • types of organizations seeking funding;
  • range of dollar amounts requested;
  • scope and potential reach of projects;
  • key turnaround times (e.g. proposal date to approval date, proposal date to Review Committee recommendation date, recommendation date to signed contribution date, etc);
  • expected deliverables, outputs and outcomes of individual projects; and
  • other funding sources.

2.2.2 Key Informant Interviews

Interviews were conducted with management and staff of VAC's Program, Policy, Research, Finance, and Communications Divisions. Also, 31 recipient organizations were interviewed by the evaluation team. The interviews obtained feedback on a variety of evaluation issues, including the rationale and continued relevance of the Program, achievement of objectives, and program delivery.

2.2.3 Sampling Strategy for File Review and Recipient Interviews

The strategy used to select projects that would provide both an understanding of the initiatives undertaken and evidence to address the evaluation issues was guided by the following principles:

  • Random samples were initially drawn from the populations of the CEPF and the CMRP. The population of CEPF files was significantly greater than CMRP files since the CMRP has only been in existence for two years. Forty-five CEPF and 35 CMRP files were reviewed. Also, from these samples, recipient organizations were interviewed (18 interviewed for the CEPF and 13 interviewed for the CMRP). For PPP Branch files, all five agreements were reviewed. The sample of files reviewed had minor adjustments to account for the factors noted below.
  • Samples were reviewed to ensure adequate representation: geographically, in scope (national and regional), in nature (various types of projects would be reviewed), and approved versus denied for funding; and
  • Only project files with a significant amount of information on file were reviewed (i.e. the files had to be completed or at least had a recommendation for funding approval or denial) to enable the evaluation to assess the evaluation issues.

2.2.4 Challenges/Limitations to the Evaluation

At the time of the evaluation, a number of factors combined to limit the nature and scope of the evaluation and these are discussed below.

The lack of baseline data and performance information, including indicators, outcomes, and targets makes assessing impacts and success difficult, along with assessing cost-effectiveness, economy and efficiency. Therefore, specific performance information used for the evaluation had to be researched or causal inference was used to draw conclusions. The Canada Remembers Division (CRD) does not have a fully implemented performance measurement strategy and there is not a comprehensive database from which to draw data for analysis.

With the lack of baseline and performance data, and the limited public opinion research, it is difficult to draw a direct causal link between the impact of PCP projects and any increased remembrance-related awareness, understanding, or engagement by Canadians. Instead, the evaluation draws upon information gathered through project files, interviews and program information reports to draw reasonable inferences of whether or not these activities contributed to the expected result to sustain and improve Canadians' appreciation for and awareness of the contribution of VAC clients to our national well-being and heritage.

Three of the PCP's four objectives had very little uptake and determining the overall success of the Program was challenging. The evaluation team sought to determine if the three objectives were relevant to VAC as whole and the reasons why there was limited activity with respect to three of the four objectives.

3.0 Key Findings

3.1 Relevance

3.1.1 Need for the Program

Based on the interviews with key informants and an analysis of Program documents and files, there is evidence that considerable interest exists in commemorating Canada's War Dead and Canada's military achievement and sacrifices. For example, the CMRP was added to the PCP after many requests from the public to assist communities with cenotaph/monument restoration, and the CMRP has continued to grow.

The PCP is the primary mechanism available to VAC to provide funds to the voluntary sector. Considering the scope of the Canada Remembers mandate, the Department's position is that it cannot stand alone in delivering remembrance initiatives and activities. A need exists for a program which helps get Canadians and communities engaged in keeping alive the achievement and sacrifices of Veterans. This correlates to the PCP sub-objectives of community engagement (to promote partnerships with, and build capacity within Canadian communities) and citizen engagement (to promote the involvement of Canadians in public policy and program/service delivery). The voluntary sector has a broad reach that allows remembrance programming to touch ordinary Canadians in a relatively economic manner. Also, the voluntary sector takes ownership and pride in their work, which is reflected in the quality of the projects funding by the PCP.

Recognizing the achievements of Canada's Veterans is one of the four objectives of the PCP. The other three relate to understanding the evolving needs of VAC's clientele, co-ordinating the work of multiple players to support their quality of life, and adapting VAC programs.

These three objectives have a continuing relevance. The needs of VAC's clients are evolving and the demands on the Department are increasing simultaneously with respect to its two main distinct client groups: War Veterans (i.e. Second World War and Korean War), and Canadian Forces Veterans. To deal with these changes, programs and services are being adapted to best meet client needs. For this purpose, however, the Department has mostly used mechanisms other than the PCP. Notably, VAC has focused much time and resources on developing and implementing the New Veterans Charter, a suite of programs designed for CF Veterans, created after significant research and consultation to respond to their needs. Brief descriptions of some other VAC initiatives that align with these three objectives of the PCP are outlined below.

VAC has and continues to conduct client surveys to help identify and understand evolving client needs. The Department is in the process of completing the Veterans Health Services Review, an extensive review of health care provided to clients that will allow VAC to better meet client needs. Also, VAC has either participated in or supported a number of studies that focused on mental health of Canadian Forces personnel that supported the development of a new Mental Health Policy.

VAC has undertaken initiatives that have aided in the adaptation and delivery of VAC programs to respond to the needs of clients, including:

  • Research in numerous health issues that can affect policy decisions; and
  • Care giver studies, which have influenced VAC's extension of the Veterans Independence Program benefits to survivors and primary care givers.
  • Initiatives that pertain to the PCP objective of aiding in the co-ordination of the multiple players (agencies, institutions, associations, and other levels of government) who have a role in the provision of "quality of life" services and benefits to VAC clients have included:
  • The Continuing Care Research Project, a partnership between VAC and the Government of Ontario looking at care issues related to seniors in institutional and non-institutional settings;
  • The Federal Healthcare Partnership, in which VAC is an active participant; and
  • The Gerontological Advisory Council, formed by Veterans Affairs Canada to advise on policies, programs, services and trends impacting Canada's aging veteran population.

Projects that were funded through the PCP included the Royal Canadian Legion's Seniors' Housing Study and the Victoria Order of Nurses' Giving Voice to Seniors and Veterans' Interests Study. These projects, described on page 6 made recommendations that impacted positively on VAC clients.

There is a potential for VAC to re-establish its involvement relative to the other three objectives of the PCP. VAC's research objectives often coincide with those of community and non-profit organizations, especially in areas such as characteristics of client populations and models of service delivery. Seeking the engagement of partners is a way to leverage the Department's research capacity. The Department could also use the PCP to ensure that VAC's expertise is shared in the broader academic community, to develop research capacity in, for instance, the area of military and Veteran health and to support the Department's commitment to the mental health of all VAC clients. In this context, there is a continuing relevance and need for the PCP to support Veterans' programming.

It should be noted that the March 2006 report of the Audit of the Partnerships Contribution Program, conducted by VAC's Audit and Evaluation Division, identified an issue regarding the attainment of program objectives. In essence, the report observed that three of the PCP's objectives were not being actively pursued, and recommended that those objectives be revised to ensure they have ongoing relevance. Management agreed with the recommendation and committed to seek approval from Treasury Board for revised Terms and Conditions.

3.1.2 Changes in clients, target groups and/or stakeholders

VAC's client populations are split into two major groups; 1) Second World War, Korean War Veterans, and their dependents, and 2) Canadian Forces Veterans and their dependents, and members of the Royal Canadian Mounted Police. The former group is declining in number, while the latter is increasing and is expected to continue to increase for the foreseeable future. This dynamic has led to significant changes in VAC's programs and services.

One change in target groups occurred in 2005, when the Government of Canada proclaimed 2005 as The Year of the Veteran. One theme of "The Year of the Veteran" was "Pass the Torch", in which Veterans ceremoniously passed on to Canada's youth the "remembrance torch" to ensure the sacrifices and achievements of Veterans are not forgotten. This is a significant change for VAC as increased emphasis is placed on involving youth in Remembrance. The CEPF often places emphasis on youth being involved in funded initiatives or activities and when proposals are reviewed, one of the criteria of the review is to identify a youth component for the project, if possible.

Similar to the efforts to involve youth in Remembrance activities, projects that involve CF Veterans, either as the main focus or through other involvement, are being promoted by VAC. Increasing the participation of CF Veterans and personnel in initiatives/activities and celebrating their sacrifices and achievements are of importance to VAC, its clients and Canadians alike.

VAC's benefits and services for CF Veterans have undergone much change in recent years, the most significant being the implementation of the New Veterans Charter (NVC). As discussed on page 16, this major initiative results in improved benefits and services for CF Veterans and their families.

3.1.3 Client and recipient needs the Program meets, and evolving and unmet needs

Veterans and Canada's War Dead are recognized, remembered, and honoured through the restoration of cenotaphs/monuments, and through commemorative events and activities held in communities, nationally and internationally. Much of the emphasis of VAC has rightly been placed on VAC's traditional clients, for commemoration purposes, First World War, Second World War and Korean War Veterans as these have been the vast majority of VAC's clients for many years and are the largest group of Veterans. Most of the contribution agreements have involved projects with a focus on these Veterans. The PCP, as only part of the overall Remembrance Programming, meets client and recipient needs through the involvement of a wide variety of organizations dispersed throughout the country that engage a wide variety of demographics.

Currently, and in the recent past, attention is also being placed on Veterans who served in Peacekeeping missions and current day CF Veterans. There is a need to remember these Veterans and their sacrifices and achievements. VAC is seeking to increase Remembrance of those who served Canada in peace time and the engagement of CF Veterans in Remembrance initiatives.

As mentioned in the Introduction, most of the cenotaphs and monuments were erected to commemorate First World War, Second World War and Korean War Veterans. Canada Remembers has received many requests to support the construction of new memorials to honour Canadian Veterans and Canada's War Dead, including CF Veterans. These requests come from new and established communities that want to recognize their Veterans and from communities that want to add to their current memorials. These requests are not eligible for funding under the current Ts and Cs of the PCP.

The Ts and Cs state that activities will be ineligible if they represent or establish a requirement for ongoing funding or create a dependency. However, interviews with key informants, recipients, and the file review revealed a potential unmet need to provide repeat funding for the same project. Several project files revealed that funds are being provided to some organizations to repeat successful initiatives or to assist organizations that host yearly events (i.e. annual reunions) or celebrations of a milestone. Many of the projects involving Veterans organizations are run by volunteers who are passionate about honouring and celebrating Canada's Veterans. Their resources are limited and their memberships often are in decline. Many of the projects are held in November, during Veterans' Week. There is potential for pressure to continue to support these initiatives, including Veterans' Week and Remembrance Day events. See 4.2.3 for further discussion.

Also, improving access to the Program has been identified as an issue. VAC mainly relies on organizations coming to the Department in search of funds for their projects. According to TBS's Communications policy, "institutions of the Government of Canada must provide the public with open access to information about policies, programs, services and initiatives. Information for public use must be disseminated or readily available in all regions of Canada using all forms of media practical.Footnote 1 However, the PCP is not actively communicated, actually, it can be difficult to find information on the Program. VAC's website only provides limited information and the Program is not evenly promoted across the country. As a result, VAC may not be receiving the highest quality applications and VAC may not be exposing all Canadians to benefits of the Program. Increased access and knowledge of the Program would likely lead to an increase in the number of applications for funding. To accommodate higher public awareness, developing more precise criteria for selecting the projects that have the greatest chance for success and are of the highest quality is important to ensure the Program is administered cost-effectively, economically and successfully.

3.1.4 Congruency with federal government priorities and VAC's role

In Canada's Performance Report 2006-07: The Government of Canada's Contribution, a strategic outcome in the Social Affairs area of the Government of Canada is "A Vibrant Canadian Culture and Heritage in which the Government of Canada ensures that Canadians have the opportunity to participate in and benefit from a broad range of artistic experiences". The government celebrates and commemorates Canadian history, our citizens, and their stories and contributes to the protection, restoration, preservation, and presentation of Canada's built heritage.Footnote 2,Footnote 3

The VAC Strategic Outcome that support the federal government outcome stated above is: Canadians remember and demonstrate their recognition of all those who served in Canada's efforts during war, military conflict and peace. Remembrance is also stated in VAC's Mission statement: To provide exemplary, client-centred services and benefits that respond to the needs of Veterans, our other clients, and their families, in recognition of their services to Canada; and to keep the memory of their achievements and sacrifices alive for all Canadians.

Under Privy Council Order 1965-688, the Minister of Veterans Affairs Canada has been assigned primary responsibility for all matters relating to the "commemoration of the War Dead and the recognition of the achievements of former members of the Canadian citizens-in arms in the defence of freedom and the development of Canada as a nation".Footnote 4

The Commemoration portion of the PCP is in line with both federal government and VAC priorities as funding is provided to organizations that engage Canadians in celebrating and commemorating Canadian Veterans and Canada's war history. Also, the PCP furthers Canada Remember's mandate as a mechanism to engage Canadians who have the desire to celebrate, preserve and raise awareness of the contribution of Veterans to our national well-being and heritage.

VAC commissioned Ipsos-Reid to conduct public polling throughout 2005, the Year of the Veteran. One of the objectives of the polling was to gauge the importance of activities commemorating Veterans for their sacrifices and achievements among Canadians. Over 85% of Canadians felt it was very important or somewhat important that Canadians recognize War Veterans and Canadian Forces Veterans for their accomplishments and the sacrifices they made for their country. Public polling commissioned in 2006 by the Dominion Institute, a national charitable organization dedicated to creating active and informed citizens through greater knowledge and appreciation of the Canadian story, revealed that there has been a decline in attendance at Remembrance Day ceremonies with only 41% of Canadians planning on attending in 2006, down from 50% in 2001. Also, only 36% of Canadians and one-in-four youth were aware of Vimy Ridge, while 60% failed a test on the First World War.Footnote 5

These polling results confirm the importance of Remembrance programming. Key informant interviews indicated that the PCP's mandate is still relevant to VAC. However, the Program is not well known, both internally at VAC, and externally, although through increasing outreach and communication activities, awareness of the commemorative aspects of the Program are building.

A second outcome in the Social Affairs area that pertains to VAC is "healthy Canadians". The Government of Canada is committed to enhancing the health and well-being of Canadians. The PCP is in line with both federal government and VAC priorities as funding was and is expected to be provided to organizations that will enhance the health and well-being of clients.Footnote 6

The VAC Strategic Outcome that supports the federal government outcome of healthy Canadians is: Canada's War Veterans, eligible CF Veterans and still-serving members, RCMP clients, qualified civilians and their families receive benefits and services in a fair and timely manner. VAC's mandate stems from laws and regulations. Among the more significant is the Department of Veterans Affairs Act, which charges the Minister of Veterans Affairs with the following responsibilities:
"The care, treatment, or re-establishing in civil life of any person who served in the Canadian Forces or merchant navy or in the naval, army or air forces or merchant navies of Her Majesty, of any person who has otherwise engaged in pursuits relating to war, and of any other person designated . . . and the care of the dependants or survivors . . . "Footnote 7

R1-R4 Recommendations

R1 Management Response

R1 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, in consultation with the Assistant Deputy Minister, Programs, Policy, and Partnerships, update the Program's objectives to reflect both the use of the Program to date and the expected future use of the Program. An assessment of the Program's objectives suggests they are too broad given the resources available to the Program and the past use of the Program. Canada Remembers Division (CRD) should develop more specific objectives for the Community Engagement Partnership Fund (CEPF) and the Cenotaph/Monument Restoration Program (CMRP) and Policy, Program and Partnerships Branch (PPP) should develop an objective pertaining to their expected use of the PCP.

Management agrees with this recommendation. As part of the renewal of the Program in July 2008, the objectives of VAC's PCP have been updated to better reflect the use of the Program for the Community Engagement Partnership Fund (CEPF) and the Cenotaph/Monument Restoration Program (CMRP). Policy, Programs and Partnerships Branch (PPPB) will not be developing specific objectives as part of the new revised Terms & Conditions of PCP, at this time, and may put forth a TB Submission in the near future to address additional PCP objectives, directly relating to the Branch, and funding sources for their prospective projects.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Terms and Conditions of the PCP have been amended to ensure ongoing relevance. Director General, Canada Remembers Division June 1, 2008

R2 Management Response

R2 It is recommended that the Director General, Program and Service Policy Division, determine the extent the Branch will use the PCP in coming years. If PPP Branch plans to continue to participate in the PCP, along with revising its objectives, PPP Branch should develop a plan to seek funds in the near future.

Management agrees with this recommendation. The PPP Branch may put forth a TB Submission in the near future to address additional PCP objectives, directly relating to the Branch, and a funding source for their prospective projects.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Consider submitting a TB submission to allow PPP Branch involvement with PCP Director General, Program and Service Policy Division Ongoing

R3 Management Response

R3 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch assess whether or not repeat funding should be allowed for exceptional projects. Projects often need more than one year of "seed money" to help build a brand/reputation and seek other funding sources to sustain and grow the project.

Management agrees with this recommendation. This is the current practice, however, the guidelines addressing this will be examined to ensure consistent application for repeat funding and will be shared with the Regional Offices. Organizations seeking funding must indicate in their proposal/application any previous funding received from VAC. Review Committee members are advised of this when making their assessment and, if appropriate, will recommend repeat funding or decreased funding for a similar project.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
The Proposal Assessment Sheet will continue to ensure that Review Committee members are aware, and take into account all previous funding to the organization while making their recommendation Director General, Canada Remembers Division Current and Ongoing
Update the guidelines and distribute to the Regional Offices Director General, Canada Remembers Division March 31, 2009

R4 Management Response:

R4 It is recommended that the Director General, Canada Remembers Division, improve the availability of Program information and Program access by:

  • Developing a standardized application form for the CEPF. Also, the form should be available on the VAC website;
  • Improve the Program's presence on VAC's website. Ensure detailed information is available, including the application form, program and process details, including eligibility criteria, key processes, and approximate time from application submission to funding decision;
  • Develop and distribute nationally, marketing and communications materials for the Program, to help ensure universal access to the Program;
  • For the CMRP, develop standards for typical improvements (i.e. for flag poles, wheelchair accessibility, drainage etc.) and post these online to assist organizations in their project planning, costing, and implementation; and
  • List and put links to other potential funding sources online to assist organizations in their funding efforts. This would complement VAC's role as a catalyst.

Management agrees with this recommendation. A standardized application and detailed program guidelines have been prepared for the CEPF and will be available on the VAC Web site. A new and improved Web site for CEPF will complement the more "user friendly" CMRP Web site which is currently in place. The Communications' Division has developed a draft communication strategy and communication materials for national distribution. An on-line guide for typical improvements to cenotaphs and monuments is being developed to assist potential organizations in their project planning. Research will be completed to locate other potential funding sources and the results will be posted on the VAC Web site.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
A CEPF application form and guidelines have been created and will be available on the VAC Web site. Director General, Canada Remembers Division August 1, 2008
An updated, more detailed, and more user friendly, presence on the VAC Web site for the PCP. Director General, Canada Remembers Division August 1, 2008
Finalize Communication Strategy and communication materials Director General, Communications September 30, 2008
Develop a guide for typical improvements to cenotaphs and monuments (i.e. for flag poles, wheelchair accessibility, drainage etc.) and make it available on the VAC Web site. Director General, Canada Remembers Division March 31, 2009
Research and post on the VAC Web site other potential funding sources for CEPF applicants. Director General, Canada Remembers Division March 31, 2009

3.2 Success

3.2.1 Objectives achievement

The Terms and Conditions for the Program state that a key result of the Program was "commemorative events, materials and monuments which sustain and improve Canadians' appreciation for and awareness of, the contribution of VAC clients to our national well-being and heritage". The evaluation assesses whether or not the Program has had any incremental impact on appreciation and/or awareness. For example, if most of the funded projects would have been carried out as planned in the absence of PCP funding, then one could conclude that the program has had little incremental impact. The evidence indicates that the PCP Program has had an incremental impact on funded projects. Had PCP funding not been available, most projects either would not have proceeded or would have proceeded at a reduced quality or scale.

Organizations that had received funding were asked about the impact of the Program on projects. 32% of recipients interviewed in the course of the evaluation said that their project would not have proceeded without PCP funding. Recipients who indicated that their project's would have proceeded or who were unsure if the project would have proceeded without PCP support were then asked what benefits the PCP funds provided. 58% of recipients stated that the projects quality and/or scale would have been reduced, resulted in the organization accumulating debt or resulted in the organization attempting to raise more funds (which may or may not have been successful). Benefits of the funding to the remaining 10% who said their projects would have proceeded as planned included requiring less fundraising. The evaluation's survey results support the view that PCP funding has an incremental impact on the delivery of commemorative related projects. These findings were supported by the file review. In many cases, recipients were small, non-profit and volunteer organizations, and this was their first time planning and delivering a commemorative related project, which include the restoration of a cenotaph/monument. Also, the evaluation's findings correlate with the findings in the VAC 2005/06 internal audit of the PCP.

Interviews with key informants and the file review covered the subject of success. A key finding is that VAC does not have a clear understanding of the appropriate measure(s) of success for the Program. This may be partly due to the variety of projects that are funded, the wide range of criteria for approval and the difficultly in measuring engagement or increased awareness. Project success is measured at the output level, i.e. the project deliverables, number of participants, number of attendees, number of copies distributed, etc. Program success is measured at the outcome level, however, further outcome identification and measurement is required before evidence-based determinations can be made.

The number of projects funded or the number of participants attending events is somewhat useful to identify the reach (i.e. the number of participants) however, if the CEPF's goal is the engagement of Canadians in Remembrance then other measures are needed as the number of participants is an absolute measure but does not indicate with certainty the number who are engaged or their level of engagement. Some projects involving only a few dozen people may have a greater impact on participants regarding engagement and appreciation versus a larger project that has an attendance in the hundreds. Further public polling research is needed to appropriately measure Canadians' level of awareness and appreciation from which CRD can set targets and deliver the Program to help achieve the targets. Nevertheless, applying causal inference, attendance or participation at a commemorative event would often result in engagement, or lead to appreciation being sustained or improved. Another factor influencing the level of engagement is the quality of the event. For example, if youth attend a military display and presentation, it is reasonable to infer that a number of the youth would increase their knowledge of Canada's military history and achievements. If the quality of the event is high, youth may be inspired to learn more and/or share their experience with family and friends, which in turn may lead more people to become engaged in Remembrance or sustain or improve their level of appreciation and awareness. However, the impact or level of engagement or appreciation cannot be determined as only limited performance information was available.

Based on applicant information, expectations for remembrance/engagement or sustaining or improving awareness and appreciation can be set at the beginning of each project, in accordance with the type of project undertaken, and that these estimates could be later confirmed by program managers against actual results through monitoring and review of deliverables, reports and communication activities.

The three non-commemorative objectives of were achieved by VAC, however, many of the initiatives were not supported by the PCP. The PCP's RCL and VON studies were completed and their recommendations impacted on VAC's policy. Other initiatives that VAC has undertaken that relate to the PCP's objectives include conducting client surveys to help identify and understand evolving client needs, completing an extensive review of healthcare provided to clients that will allow VAC to better meet client needs. Also, VAC has either participated in or supported a number of studies that focused on mental health of Canadian Forces personnel to support the development of a new Mental Health Policy.

3.2.2 Achievement of VAC's goals/mandate

The ongoing priorities of the Portfolio (VAC and the Veterans Review and Appeal Board comprise the Portfolio) include:

  • Enhancing and adapting programs and services to meet the needs of our Veterans, serving members, other clients and their families; and
  • Engaging Canadians in community-based remembrance activities with an emphasis on Canada's youth.

Additionally, Canada Remembers Mandate states: The Canada Remembers Program endeavours to keep alive the achievements and sacrifices made by those who served Canada in times of war and peace; to engage communities in remembrance of these achievements and sacrifices; and, to promote an understanding of their significance in Canadian life as we know it today.

A further priority for CRD is the re-focusing of remembrance activities on the involvement of Canada's youth in community-based activities within Canada, while respecting overseas responsibilities. The importance of involving youth stems from the fact that Canada's War Veterans are declining in number yet there is a desire amongst stakeholders and the Canadian public to continue to remember the sacrifices and achievements of our War Veterans. The CEPF portion of the PCP places emphasis on the involvement of youth in projects. One of the preferred criteria when assessing proposals is youth involvement and youth is actually one of the target groups for the CEPF. CRD personnel will work with organizations to try and involve youth in the projects.

Two strategic objectives for VAC include: improving and expanding services for Canadian Forces Veterans and still-serving members and transforming service delivery and promoting innovation in policies and practices. The PCP can help VAC achieve these objectives through partnering with organizations that have the ability to impact VAC's policy and programs.

3.2.3 Unintended outcomes

One unintended outcome identified by key informants and the file review is the potential of "creating a dependency" on PCP funding. There are many organizations that repeatedly apply to the program, some for the same project and some for a different project. The Ts and Cs of the PCP state that initiatives/activities are ineligible for funding if they represent or establish a requirement for ongoing funding or create a dependency. The file review has identified several initiatives that have received repeat funding and this was confirmed in recipient interviews, while other first-time recipients indicated they would reapply. Reasons for reapplying include the event was such a success the organizations want to repeat the project or the project is an annual event.

Since all of the organizations are not-for-profit, government, or educational institutions, funding is always an issue and raising funds can consume much time and effort. One way to address the concerns about dependency would be to enforce the Ts and Cs and not provide repeat funding. Another approach would be to revise the Ts and Cs to clarify that repeat funding may be approved for a limited number of projects that VAC deems to be highly effective at engaging communities in remembrance.

The Program is intended to be a catalyst to encourage and support remembrance initiatives/activities. However, when VAC's contribution is a significant percentage of the funding and the organizers want to repeat the project due to its success, without VAC's financial assistance the organizers may not have the funds to repeat the project, either to the same quality and scale or potentially not at all. Key informants indicated that funding should be available for at least two years for projects or have the ability to fund projects for up to three years with decreasing amounts each year. Such projects would have to be supported by a plan that demonstrates how they would become self-sufficient within that term, and then VAC would exclude those organizations from further accessing the fund for the same project after the end of the agreement. This approach would allow organizations some time to build their infrastructure and reputation, and secure other sources of funds to be able to continue to deliver the initiative/activity without VAC's financial support.

3.2.4 Are resources effectively targeted to meet the objectives?

Overall, the projects CR funds through the PCP meet the objectives of the Program. A wide variety of projects have been funded, in Canada and internationally, with many types of organizations. Canada Remembers should continue to focus on community-based organizations that undertake projects that engage Canadian communities, and specifically with respect to the specified target groups. However, with the number of applications for funding increasing, administering the Program on the current first-come first-serve basis does not lend itself to maximizing the effectiveness of resources.

Similar funding programs have established deadlines for the receipt of applications. Instituting such deadlines would assist VAC in ensuring the selection of projects that are of high quality and offer value-for-money, and ensure Program goals are met through analysis of a group of projects. Also, the change in process would improve VAC's planning, decision-making and budgeting processes, and better manage the expectations of recipient organizations. The deadlines would allow organizations to better plan their projects, including securing the necessary funds from other sources and provide them with an approximate timeframe for when they can expect notification of a decision. Also, the absence of clear key performance information (e.g. criteria and indicators) prevents CRD from maximizing impact and reach when determining success and selecting projects to fund.

The contribution agreement with the JBCA consumes a significant portion of the PCP funding. The agreement is unique in that multiple exemptions to the Ts and Cs were necessary to provide funding to the JBCA, funding that is ongoing and supports operational expenditures. The PCP was not designed for making large, ongoing payments to an organizations, and supporting its operating costs. VAC should ensure the JBCA provides more complete performance data and financial data to increase accountability and VAC should regularly receive copies of the JBCA's financial position, visitor statistics and feedback, and annual reports to monitor and account for VAC's contribution.

The initial two projects that Policy, Programs and Partnerships Branch funded through the PCP provided VAC with valuable information on client needs, however, the three contribution agreements currently in effect under the PPP Branch are not in compliance with the PCP's Terms and Conditions. The organizations involved have an ongoing and continuing responsibility to provide the care and services that are being specifically covered by the Partnerships Contribution Agreements. This cannot comply with Article 7 of the Terms and Conditions which indicates that activities will be ineligible when they are ongoing or continuing programs that would have occurred without VAC funding. Similarly with the above observation is the requirement, as set forth in Article 9 of the Terms and Conditions, that the activities covered by the agreements be finite in nature. However, the project undertakings described in the Agreements mostly do not begin upon initiating the agreements nor will they end when the agreements are terminated. Currently, VAC has not made the 2007-08 payment to the organizations and VAC is assessing funding options to continue working with the three organizations.

3.2.5 Annual Program goals - Are they realistic and are they achieved?

The overall goal for the CRD's use of the PCP has continually been to engage Canadians in Remembrance through partnerships with organizations. The CEPF has a goal to increase the involvement and participation of youth and CF Veterans and Members in the initiatives/activities. Key informant interviews and the file review confirmed that this goal is being achieved with emphasis continuing to be placed on engaging these groups.

Another annual goal for both the CEPF and the CMRP is to promote the Program to stakeholders. Initially, the limited funds available restricted promotion and communication of the PCP. VAC did not want to be inundated with applications, only to have to deny many of the requests. With increased ongoing funding, CR personnel at Head Office and Regional Offices have been more proactive in communicating the Program. When VAC personnel do presentations on CR Programs to various organizations and groups, a profile of the PCP is presented. Also, VAC issues a news release for each funded project that includes a description of the initiative/activity, the amount of Government assistance towards the project and that the funding was provided by VAC through the PCP or specifically, the CEPF or CMRP.

The goals described above have not been quantified and annual targets have not been established. Without quantifiable goals and targets that relate to performance indicators, the determination of achievement of goals and whether or not the goals are realistic is difficult. As stated above, based on the research conducted during the present evaluation and the expertise of the evaluation team, the Program is meeting its current goals, although the goals are broad and not well defined.

R5-R8 Recommendations

R5 Management Response

R5 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, seek other funding framework options for the Juno Beach Centre Association (JBCA). By making large ongoing payments to a single organization, the effectiveness of the Program is restricted.

Management agrees with this recommendation. Management will assess other funding frameworks for the JBCA.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Consider funding frameworks for JBCA in the context of the Strategic Review or, if possibly, as a later TB Submission. Director General, Canada Remembers Division March 31, 2009

R6 Management Response:

R6 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, require the JBCA to provide more complete performance data and financial data to increase accountability. VAC should regularly receive copies of the JBCA's financial position and visitor statistics and feedback and annual reports to monitor and account for VAC's contribution.

Management agrees with this recommendation. VAC will request the JBCA to provide annual copies of their financial position, visitor statistics, visitor feedback and annual reports.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Ensure the JBCA provides annual copies of their financial position, visitor statistics, visitor feedback and annual reports. Director General, Canada Remembers Division March 31, 2009

R7 Management Response:

R7 For the CEPF, it is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, define more precisely in PCP policy, what financial contribution organizations must make to their project, both in-kind and from cash sources (for example, the CMRP funds up to 50% of the cost, requiring organizations to provide 50% of funding). This would help ensure CRD's role as a catalyst for the projects, that the organizations are committed to and engaged in the projects and potentially allow for more projects to be funded.

Management agrees with this recommendation. It should be noted that all funding sources, including in-kind support, are required to be provided in the new CEPF application. The PCP policy will be updated to clarify the expected contribution that organizations must make to their project, both financially and in-kind. For some particularly worthwhile remembrance initiatives, community groups are, on occasion, unable to make significant financial contributions. In these cases provision of up to 100% funding is in keeping with the stacking provisions of the Terms and Conditions of the PCP.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Update PCP policy Director General, Canada Remembers Division March 31, 2009

R8 Management Response:

R8 It is recommended that the Senior Assistant Deputy Minister, Policy, Programs and Partnerships, seek alternative arrangements for three non-compliant contribution agreements currently in effect.

The Policy, Programs and Partnerships is seeking alternative arrangements to provide funding to these organizations.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
The PPP Branch is seeking alternative arrangements to provide funding to these organizations. Senior ADM, Policy, Programs and Partnerships December 31, 2008

3.3 Cost-effectiveness/Efficiency/Economy

3.3.1 Improving Program delivery

One focus of the evaluation was on the delivery of the Program. The majority of projects are $5000 and under, and there are opportunities to improve the delivery, with benefits for both VAC and recipients. All applications for funding follow the same processes and are delivered in the same manner. However, the scope and resource requirements for projects can differ greatly, with contributions ranging from less than $1000 to more than $100,000. Also, the abilities and resources amongst organizations are wide-ranging, from the community volunteers to larger not-for-profit organizations with extensive resources.

For smaller dollar amounts funded (i.e ≤$5000), the requirements placed on VAC and the requirements VAC places on organizations are extensive, even burdensome. From January 1, 2006 to September 30, 2007 approximately 60% of funded projects were $5000 or less. The amount of time and resources VAC personnel spend assessing and approving applications, preparing contribution agreements, amending agreement end dates and ensuring the proper receipts are received for every expense, may at times exceed the amount funded. Accountability and program integrity is important, and VAC has to perform due diligence, however, alternate means are available to improve cost-effectiveness and economy.

One option for improving the cost-effectiveness and economy of the delivery of the PCP would be to streamline the requirements for small dollar amount projects. Requirements that could be lessened include:

  • Requiring recipients to keep receipts on file for two years post completion of the event for auditing purposes. Annually, a sample of organizations would be asked to submit receipts to ensure compliance and assess
  • consider allowing projects to span fiscal years to minimize the need for amendments. Currently all CMRP contribution agreements are only effective to the end of the fiscal year in which they are approved, while many of the projects are not completed with that initial time frame; and
  • the Minister's Commitments and the Recipients Commitments section. There is an opportunity to review the Commitments for their relevance as currently they are the same for all projects and some items may not be necessary for smaller dollar projects.

The streamlined requirements would only apply to projects that VAC, upon assessment of the project and organization, determines would qualify. An example where economy could be improved is the process of amending the agreement end date in the CMRP. The end date is always the fiscal year end of the current year. Many files reviewed had amendments to extend the end date another full fiscal year. Assessing the specific length of time it takes to complete a project would help establish a more suitable end date and considering the time of year is important as most of the restoration work needs to be undertaken outside of the winter months.

A second option would be to institute a grant component for funding projects. This would involve more effort than streamlining the PCP's Ts and Cs. The grant component would need separate, but similar Ts and Cs as it would be recognized as a distinct program. There are several funding programs, in both the public and private sector that offer grants to organizations or either a grant or a contribution. Grants are mostly for smaller dollar projects and contributions for larger dollar projects. These programs offer funding to organizations for a variety of initiatives/activities, some of which PCP recipients are eligible to apply for funds.

Establishing a grant component to the delivery of the Program would improve the cost-effectiveness, along with improving efficiencies. The advantage of a grant includes reduced requirements on both VAC and the recipient organizations, leading to decreased administration time and cost for VAC and recipient organizations. Before a grant is approved to an organization VAC would apply 7.5.1 iv of TBS Transfer Payment Policy and assess whether or not "there is a reasonable expectation that the recipient of a class grant will use the funds for specified purposes or to meet specified objectives."Footnote 8

3.3.2 Duplication or overlap with other existing programs (federal, provincial, municipal or private sector)

Program overlap occurs when programs have the same goals, the same activities or strategies to achieve them, and/or the same targeted recipients. The file reviews and key informant interviews indicated that a number of PCP funded projects received funding from other Federal Government departments, Provincial Governments, Municipal Governments, other non-profit organizations as well as the private sector, however, the PCP is unique in that it focuses on Veterans and Canada's military achievements and sacrifices. This is evidence of the collaborative nature of the PCP in which several entities work together towards a common goal. There was no evidence that duplication of expenses occurred (two or more entities paying for the same thing) because VAC requires original receipts of eligible expenses prior to payment.

While VAC has a very clear mandate in terms of commemoration for Veterans, other levels of government as well as non-profit and the private sector also have a role to play. The Ts and Cs do not require recipients to provide funding from any other source(s) (with the exception of the CMRP), although virtually all funded projects have more than one source of funding. The CMRP requires the recipient to raise at least half the funds from other sources, ensuring broader support for the project.

VAC sometimes notifies organizations of other funding programs to assist the organization in securing the necessary funds, however, this is done on an ad-hoc basis and there is not a list of other programs available that VAC personnel can post on its website or distribute to organizations. It is important that funding from various sources is well co-ordinated to ensure the funds are best used to achieve the overall objectives of the project and to ensure the project will be viable. VAC needs to examine what its role should be when recipients receive funding from multiple sources (i.e. a catalyst role, major contributor or to have a presence) and VAC's funding should be proportionate to the expected benefits, for both projects that exclusively align with the VAC mandate and for projects that have a broader scope that incorporate VAC's mandate.

3.3.3 Effectiveness of VAC's relationship with recipients

Key informants, both VAC personnel and recipients indicated that the relationship between VAC and recipients is effective and is one of mutual appreciation. VAC's role is not to delivery all Remembrance programming on its own, but the amount of, and quality of, initiatives/activities and organizations that are funded through the PCP demonstrates to VAC that the Program meets a need in society and is of value. VAC personnel work closely with recipients to ensure projects are successful. Recipients spoke very favourably of the Program and of their interactions with VAC. Every recipient interviewed said that they would apply for further funding again if they had a different project that was eligible. Many organizations had already received funding in the past for the same or other projects, and through this ongoing relationship, a positive rapport has developed.

3.3.4 Demonstration of improved efficiencies, economies, or cost-effectiveness related to annual Program goals

One method to determine operational efficiency would be to measure and analyze key turn around times. One such turn around time measure would be from the time receipts are submitted for payment to the time the payment is issued. These could be analyzed against standards and trends, in order to determine if PCP was efficient using these measures. The evaluation team did analyze the time from receipt of application to approval date for a number of files. For the CEPF, since January 2007, the average time of three months was on par with other grants and contribution programs. For the CMRP, the turnaround was over four months for the files reviewed.

In January 2007 there was a significant change in the approval process for projects and the communication of partnerships. The Minister of Veterans Affairs started to exercise his authority to approve all projects over $1000. Authority to approve projects under $1000 remained with the ADM, SD&C. Other process changes included changing the composition of the CEPF Review Committee to include more senior CRD personnel and a representative from Communications Division. Also, all projects previously recommended for approval in the ROs were now being reviewed a second time by the HO Review Committee. Additionally, a news release was to be written and distributed for every funded project. The results of these changes were processes that have become more layered and lengthy. The typical turnaround time from application receipt to official notification of funding approval has increased from two or three weeks to approximately three months. This is a significant increase in time and demands placed on VAC personnel and the applying organizations (especially organizations that had applied in the past and expected a quicker turnaround time).

According to the PCP's Ts and Cs, all projects are to be reviewed by the Review Committee. There is an opportunity for VAC to determine what is the appropriate level of representation on the Committee and what are the requirements to send a project to the Committee. If an application is obviously ineligible, Management could deny or recommend denial of the application and having the Committee review the application would not be necessary. Also, there is an opportunity to streamline the recommendation and/or approval process by delegating applications that seek a relatively small dollar amount to Management.

The majority of projects for the CEPF are administered in the Regions. As the Program grows, and with a focus on community partnerships, the Regions would be expected to have increased activity and they are in the best positions to know the local organizations and communities that apply for funding. Currently, CRD notionally allocates each Region $40,000 per annum, however since the level of activity varies among Regions, the budget allocations should reflect expected use in each Region.

For the CMRP, 2006-07 was the first full year for the program so there is a lack of data to determine if improvements have been made. Also, processes are not fully documented and are being refined with experience. However, there is very limited documentation on file regarding Review Committee dates, discussion, recommendations or records of decision.

Cost-effectiveness analysis is when one or more appropriate interventions (i.e. eligible projects funded) are compared in terms of both their associated costs and benefits. Such analysis examines the extent to which the balance between inputs (the costs of resources used) and outputs (effects/benefits/outcomes) of the interventions represent value for money.

The main point to discuss that would demonstrate improved cost-effectiveness of the Program would be knowledge of the types of projects that offer the best value for money (i.e. are the most cost-effective at sustaining or increasing awareness in commemoration). Similarly, the concept of opportunity cost, the cost of pursuing one choice instead of another, has not been an issue for the PCP as enough funds have been available for all eligible projects and VAC has not had to choose between eligible projects. However, as the program continues to grow, VAC will likely be confronted with this challenge in the near future and it is important to have better processes and methods to assess which project(s) to fund. The current first-come first-serve process for funding projects would not allow VAC to sufficiently ensure the best projects are funded.

The data was not available to evaluate this component and appropriate performance measures need to be developed and implemented to measure cost-effectiveness. More precise criteria are needed to help in the assessment of applications, and these criteria should be based on a review of Program data and public polling data.

3.3.5 Process to award funds includes a qualified assessment of merit

The process is clear for the awarding of contributions. All applications for funding are to be reviewed by a Review Committee consisting of at least three members. Several criteria are to be used to assess the eligibility and merits of the applications. In response to a recommendation in the 2005-06 Audit of the PCP, virtually all projects have since been reviewed by the Committee, who then recommends approval or denial to the Minister (when >$1000). However, during the file review analysis it was observed that some projects (mostly from the first few years of the Program and before the audit) were not assessed by the Committee. In some cases, CR management approved funding of eligible projects, or denied applications that were obviously not eligible (e.g. funding application was by an individual).

As stated in the previous section, establishing more precise criteria when assessing project merits, and having annual goals to help guide project selection would improve the assessment of applications/proposals. Some of the current criteria are broad and have not been assessed for their continued relevance and importance. Reviewing and revising the criteria to be more precise and to ensure their relevance would assist in assessing project merits.

3.3.6 Oversight practices that provide sufficient knowledge of recipient activities

The application process and final report collects information from organizations to provide VAC with knowledge of recipient activities. The level and quality of the information received varies, from not providing sufficient information to full and complete. When applications are not complete, VAC personnel contact the organization to ensure all required information is received.

VAC should always ensure that its funding will not result in organizations profiting from the project, therefore it is necessary to be informed of all other sources of support and all expenses incurred. VAC must ensure that it is only providing the necessary amount to undertake the project successfully. There should be an expectation that the federal government will only contribute a certain amount to an individual project, especially since VAC considers the PCP funding to be a catalyst to the projects.

Other oversight practices include the provision of original receipts prior to payment. Receipts are reviewed by VAC personnel and only eligible expenses are paid. Also, a final report is required to be submitted by the recipients that details the results and accomplishments of the project. A good example of oversight practices is for the CMRP before and after pictures of the cenotaph/monument are required. The visual depictions are of great value and really allow VAC to appreciate the condition of the cenotaph/monument both before the restoration and afterwards.

3.3.7 Collection and reporting of Program and recipient performance data

VAC's 2006/07 Departmental Performance Report reported the number of contribution agreements and the dollars expended along with an overall summary of the Program. The information related mostly to outputs. There is an absence of a performance measurement strategy, the foundation of which is a set of expected outcomes, followed by a plan for reviewing performance against stated expectations and a "dashboard" of performance measures for the Program for reporting purposes.Footnote 9

Attribution or program impact is also difficult to substantiate and management has not collected data and/or analyzed the data to measure the impact of the PCP. Also, several other commemoration initiatives and groups are involved in commemoration (e.g. other CR programming, RCL and Dominion Institute involvement, etc.), with each having an impact on sustaining and raising awareness, and engaging Canadians in Remembrance.

In order to properly measure the effectiveness of the PCP, distinct from other remembrance programming, a set of measures would have to be developed. These measures would allow for determining the expected impact of a type of project and an annual assessment of the impact(s) of completed projects. VAC's Program Activity Architecture had identified a target of 60% of Canadians who are aware of the contributions and sacrifices of Canadian Veterans and Canada Remembers Division will be conducting polling in 2008 to measure the current level of awareness.

The measurement strategy should be designed to support management in its efforts to manage for results, as well as fulfill accountability and performance reporting requirements. The strategy would enable VAC to determine if it is achieving the expected outcomes for the targeted beneficiaries within the set time frame and at a reasonable cost. Other benefits of a performance measurement strategy include:

  • Provides timely information;
  • Provides managers/stakeholders with opportunities for self-assessment;
  • Provides on-going support for management decisions; and
  • It can provide up-to-date information with respect to client and recipient satisfaction.Footnote 10

R9-R15 Recommendations

R9 Management Response:

R9 It is recommended that the Director General, Canada Remembers Division, ensure the CMRP's Review Committee, similar to the CEPF Review Committee, keeps Records of Decisions with signatures of Committee members when decisions are made. Currently, the project files are lacking documentation and signatures from the Review Committee meetings.

Management agrees with this recommendation. This practice has been in place since October 2007.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
N/A N/A N/A

R10 Management Response

R10 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch:

Management partially agrees with this recommendation. Canada Remembers Division is not currently considering establishing a grant component for this program. Management will consider and propose ways to streamline delivery of all aspects of the program. The Minister's approval to delegate authority to the ADM, for approval of regional and national applications of contributions of $5,000 or less, will be sought.

  • Design new streamlined requirements for smaller dollar projects (i.e. funding of $5000 and less) to ease the burden on both VAC resources, and organizations that do not have the resources to easily comply with all of the standard requirements, and/or CRD should consider establishing a grant component for qualified projects;
  • Seek delegated approval authority to the ADM, SD&C for funding of $5000 and less. Currently, the ADM, SD&C has approval authority for projects less than $1000, but during Veterans Week, 2007 authority for projects $5000 and less was granted to the ADM, SD&C and prior to January 2007, the ADM, SD&C had full approval authority for CEPF projects. The change in process during Veterans Week was reported to be successful and more efficient by both HO and RO personnel.
Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Efficiencies for program delivery will be examined and implemented. Director General, Canada Remembers Division March 31, 2009
Seek Ministerial approval to delegate authority to the ADM, SD&C for contributions under $5,000. Director General, Canada Remembers Division September 30, 2008

R11 Management Response

R11 It is recommended that the Director General, Canada Remembers Division, implement quarterly application deadlines and develop more precise criteria for determining project selection. This is commonplace for programs of this nature and would serve both VAC and the organizations with whom it partners. This would assist VAC in the planning and delivery of the Program. Also, in having more precise criteria for project selection, VAC can better ensure the projects selected would best meet the Program's objectives (e.g. for when there is an opportunity cost and CRD must chose to fund one project over the other due to limited funding, VAC needs to know which one of the two should be funded). The applications deadline would assist organizations in planning their projects and would provide them with an approximate timeframe for when they can expect notification of a decision. Also, the deadline would replace the current "first come first" serve process that does not ensure selection of projects that are of high quality, and offer value-for-money.

Management partially agrees with this recommendation. The new application form for CEPF, states that applications must be received at least 3 months prior to the project date to allow for processing and notification of funding. Imposing quarterly deadlines would often require small community groups (of frequently elderly volunteers) to develop their plans as much as 6 months in advance of their event. This is an unrealistic expectation for the majority of PCP applicants. The now required 3 month advance application will meet the objectives of this recommendation. More precise criteria for project selection will be developed.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Develop more precise criteria for project selection Director General, Canada Remembers Division March 31, 2009

R12 Management Response

R12 It is recommended that the Director General, Canada Remembers Division, prioritize the projects by type to maximize program results and clearly define what types of projects are eligible for funding. There are many types of projects that are funded, which increases diversity, however, VAC should determine what projects are of most importance for any given year (eg. Special anniversary projects, CF, youth, etc.) and give clear direction to HO and RO personnel on what types of initiatives/activities are acceptable.

Management partially agrees with this recommendation. With respect to prioritizing the type of project, it is less important to catagorize the type and more important to examine the initiative's accomplishment of the program objectives. There is currently a list of 11 types of projects which can only be prioritized after examining each project's merits in regards to meeting the program's objectives. For example, a conference may or not be a higher priority than a display/exhibit depending on its content. Management will review the criteria for project selection. Management will ensure the current guidelines explaining the parameters for acceptable initiatives/activities (for example: reunions/dinners) are reviewed and provided to HO and RO partnership staff.

Management will ensure that significant anniversaries or special VAC initiatives are identified on the partnerships section of the CRD Web site. While submitting an application focusing on one of these initiatives, will not guarantee funding approval by the Minister, the posting of significant anniversary dates and initiatives will assist in the Canadian public in focusing on current remembrance priorities for the coming months.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Develop more precise criteria for project selection and update current guidelines Director General, Canada Remembers Division March 31, 2009

R13 Management Response

R13 It is recommended that the Director General, Canada Remembers Division, assess and ensure its funding is proportionate to the benefits received when funding all projects and especially large national scope projects. A significant portion of the CEPF's budget is consumed from partnering with organizations that seek relatively large amounts of money.

Management agrees with this recommendation. The Review Committee will continue to assess all funding applications, including large national scope projects, to ensure the amount recommended is proportionate to the remembrance outreach and benefits received.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Review Committee members will continue to be advised, and take into account while making their recommendation, the funding amount and the anticipated remembrance outreach and benefit from each application. Director General, Canada Remembers Division current and ongoing

R14 Management Response

R14 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, implement a performance measurement strategy to support managing for results, proportionate to VAC's investment, and for determining if outcomes and targets are being achieved. Also, VAC should establish a "dashboard" of measures that should be reported upon yearly and be used to improve the performance of the Program.

Management agrees with this recommendation. Management will implement a performance measurement strategy, proportionate to VAC's investment, to assist in assessing the success of the PCP.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date
Implement a performance strategy, proportionate to VAC's investment, to assess the outcomes and success of PCP projects. Director General, Canada Remembers Division March 31, 2009

R15 Management Response

R15 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration Branch, ensure the funds CRD notionally allocates to the Regions for CEPF projects are sufficient to meet the demands on the Program. The majority of projects are administered in the regions. As the Program grows, and with a focus on community partnerships, the Regions would be expected to have increased activity and they are in the best positions to know the organizations and communities that apply for funding.

Management agrees with this recommendation. Currently, if the notional allocation to a region is not sufficient the Region can request additional funding from Head Office. Management will review the CEPF fund allocation to the Regions.

Management Action Plan
Corrective Action to be Taken OPI (Office of Primary Interest) Target Date

Complete an analysis of annual notional CEPF funding provided to the Regions in comparison to the annual CEPF contributions used in each Region

Director General, Canada Remembers Division

September 30, 2008

4.0 Program Profile

4.1 PCP Current Funding Levels

The current and ongoing reference levels of the PCP now total $2.2 M per year for three specific commemorative uses by VAC and is notionally allocated as follows: community engagement projects ($700,000), funding to the JBCA ($500,000) and the restoration of cenotaphs/monuments throughout Canada ($1.0 M). All current PCP funding has been approved for the objective relating to commemoration. PPP Branch has not requested PCP funding since 2002.

4.2 Application and Approval Processes

Proposals/applications are received year round on a first-come first-serve basis. Selection criteria and guidelines are used to determine eligibility, review proposals/applications and evaluate projects in a fair and equitable manner. The evaluation is performed by Review Committees that meet periodically to make decisions on applications/proposals.

The major steps in the application and approval processes for both the CEPF and the CMRP are similar and are outlined below:

  1. Application/Proposal is received by VAC
  2. VAC personnel review the application/proposal for completeness
  3. Review Committee reviews and makes a recommendation to the Minister
  4. Program Officer prepares and sends Head Office Recommendation Package to the Minister
  5. Communications Division drafts communications materials
  6. Minister's decision - approval or denial
  7. If denied, Assistant Deputy Minister, SD&C sends letter of denial
  8. If approved, Minister sends letter of approval
  9. Program Officer prepares Contribution Agreement for ADM, SD&C signature and sends to the recipient organization
  10. Organization signs Contribution Agreement and returns to VAC
  11. Funds paid by VAC Finance upon acceptance of invoices/receipts and final report.

Exceptions to the above process are:

  • for CEPF projects that are Regional Office based and are $5,000 and less, Steps 1, 2 and 3 are first performed at the Regional level, then repeated at HO; and
  • CEPF projects that are less than $1000 are approved by the ADM, SD&C, and the Minister is notified of the decision.

4.3 Eligibility Criteria

Eligible organizations include:

  • non-profit organizations;
  • registered charitable organizations and associations;
  • museums and historical organizations;
  • the provinces territories and municipalities; and
  • education, research and health institutions.

Individuals, the for-profit sector, and other federal government departments are not eligible for funding.

Several types of activities are eligible for funding, including:

  • research studies,
  • the development of education and information products;
  • the provision of education and information;
  • the organization, development, construction and/or maintenance of commemorative events, materials and remembrances; and
  • the restoration of cenotaphs/monuments that are located in Canada and were erected to commemorate Canadian War Dead or Veterans.

Initiatives/activities are ineligible for funding, unless exemptions or amendments were granted, if they:

  • represent or establish a requirement for ongoing funding;
  • create a dependency;
  • would have taken place without the support of VAC;
  • could not be expected to contribute in a cost-effective manner to the objectives of the PCP; and
  • involve capital costs (except for CMRP projects)

Also, specific to the CMRP, funding is ineligible for:

  • Creation of new cenotaphs/monuments;
  • Cenotaphs/monuments outside of Canada;
  • Cenotaphs/monuments that are dedicated solely to an individual; and
  • Projects which have received previous funding from VAC within the last three years for the purpose of restoration.

4.4 Funding and Payment Terms

The value of a contribution shall not exceed $500,000 and the duration of an agreement to complete a project cannot exceed three years (with the exception of the JBC). Payment can be made as a lump sum upon completion of a project, periodically based on expenditure claims submitted to VAC, or based on the achievement of milestones. Unless paid in full upon completion, a final payment of at least 10% of the estimated contribution is withheld until after successful completion of the project.

5.0 Conclusions

The main conclusions of the Evaluation of the Partnerships Contribution Program on the main issues of relevance, success, and cost-effectiveness, efficiency and economy are described below.

5.1 Relevance

The rationale for the Program was borne out of a government-wide horizontal initiative to recognize the voluntary sector as one of the three pillars in society, to strengthen the sector's capacity, and to improve the quality of life for Canadians. Also, the Program would assist VAC in improving its programs and services.

The evidence supports a need for VAC to continue to support the organizations interested in Remembrance through the CEPF and CMRP. The use of the Program is growing, the target groups have evolved to include youth and CF Veterans and the Program continues to be congruent with the priorities of both VAC and the Government of Canada. Key informants, being VAC personnel and funding recipients, expressed a highly favourable desire to maintain and enhance Program, and spoke of the need for Canadians to be aware and appreciate the sacrifices and achievements of Veterans.

The evidence was mixed on the rationale for PPP Branch to maintain involvement in the PCP. Although the objectives of the PCP are relevant to the work PPP performs, PPP chose other means besides partnering with the voluntary sector to accomplish their mandate. Currently, VAC has expressed an interest in using the PCP to assist in a variety of research related initiatives that could be supported by the PCP. VAC wants to expand its research capacity, and with additional funding, plans to use the PCP to further its mandate.

5.2 Success

VAC has undergone significant change since the Program began. CRD has grown significantly, while PPP Branch has developed, implemented and adapted many new programs and services to better meet the needs of clients.

Organizations that have benefited from PCP support stated that the funding received has been important to the success of the project, which in turn is important to communities. Also, the majority of projects funded stated the financial assistance either allowed the project to proceed or the funds had an incremental impact on the scale and quality of the project.

The vast majority of recipients indicated in telephone interviews that they would seek funding again if they had a different project. Recipients overwhelmingly declared their projects to be very successful, however, some relied heavily on PCP funds and since the Ts and Cs do not allow for repeat funding for the same project, some recipients were unsure how they would continue to conduct the project the following year to meet the demands of the public.

The Program assists VAC in its programming efforts in a relatively effective manner. There are also benefits to the organizations who develop and implement the projects, and to communities and Canadians as a whole as the projects may have a lasting impact on organizers and participants.

5.3 Cost-Effectiveness/Efficiency/Economy

The emphasis of the evaluation was on relevance and achievement of objectives. With the implementation of the evaluation's recommendations and emphasis placed on collecting and analyzing performance data, VAC would be able to better assess the cost-effectiveness, efficiency and economy. There are opportunities for VAC to improve the cost-effectiveness, efficiency and economy of the Program now and over time as performance information is collected and analyzed.

All applications for funding follow the same processes and are delivered in the same manner. However, the scope and resource requirements for projects can differ greatly, with contributions ranging from less than $1000 to more than $100,000. Streamlining the requirements and approval process for smaller dollar projects amounts would lead to improvements in efficiency, cost-effectiveness and economy.

Cost-effectiveness of the Program could be improved through enhanced knowledge of the types of projects that offer the best value for money (i.e. are the most cost-effective at sustaining or increasing awareness in commemoration). As the program continues to grow it is important to know which are the best project(s) to fund. The current first-come first-serve process for funding projects does not allow VAC to sufficiently ensure the best projects are funded.

6.0 Distribution

  • Deputy Minister
  • Chief of Staff to the Minister
  • Senior Assistant Deputy Minister, Policy, Programs and Partnerships Branch
  • Assistant Deputy Minister, Corporate Services Branch
  • Assistant Deputy Minister, Service Delivery and Commemoration Branch
  • A/Director General, Coordination Secretariat
  • Deputy Coordinator, Access to Information & Privacy
  • Comptrollership Branch (TBS)
  • Office of the Auditor General
  • Program Analyst, Treasury Board of Canada, Secretariat

7.0 Annex 1

Evaluation Framework
Partnerships Contribution Program (PCP)

Background

Veterans Affairs Canada's (VAC's) Partnerships Contribution Program (PCP) was established with TB approval in November, 2001. The PCP was created to allow VAC to enter into contribution agreements with organizations in order to work collaboratively to further the Department's mandate in both providing Veterans benefits and services, and undertaking commemorative activities.

The objectives of VAC's Partnerships Contribution Program are to:

  • Identify and understand the evolving needs of existing and new clientele.
  • Aid in the adaptation and delivery of VAC programs (benefits and services) to respond to the needs of clients.
  • Aid in the integration and co-ordination of the multiple players (agencies, institutions, associations, and other levels of government) who have a role in the provision of "quality of life" services and benefits to VAC clients.
  • Ensure that the achievements and sacrifices of Canada's citizens in the defence of freedom are recognized and the historical significance of these accomplishments are nationally promoted and celebrated.

The PCP allows VAC to extend its reach in partnering with the stakeholders in the development and delivery of its programs. The expected results and outcomes include:

  • Improved information about current and future clients and their needs, upon which pro-active and responsive policy and services can be developed and delivered
  • Improved information, integration and co-ordination of benefits and services relating to the financial, physical, mental and social health and well-being of VAC clients, resulting in collaborative, seamless and transparent services to VAC clients
  • Commemorative events, materials and monuments which sustain and improve Canadians' appreciation for and awareness of the contribution of VAC clients to our national well-being and heritage

It was expected that the uptake on VAC's PCP would be widespread with a large focus on activities surrounding Veterans' health and well-being. However, to date use of the PCP has focused on commemorative initiatives with community organizations and the restoration of cenotaphs.

The Terms and Conditions for the PCP were to expire on July 31, 2006, however, under the Minister's authority, pursuant to section 7.3.2 of the Transfer Payment Policy, a one-year extension to this date was granted by the Minister of VAC in July, 2006. In June, 2007, TB approved an additional one-year extension to the PCP's Terms and Conditions. During this one-year extension, VAC is committed to: completing a full program evaluation of the PCP; revising the Terms and Conditions of the PCP; and updating the RMAF and RBAF to reflect changes in the Program.

Study Approach

The evaluation will address several issues, however, the focus will be on relevance and the achievement of objectives. Knowing that there has been very limited program activity in pursuing three of the four objectives, the evaluation will seek to determine the reasons why the program has not been utilized as originally intended.

The evaluation will encompass three main evaluation issues, with emphasis on Relevance and Success:

  • Relevance: The extent to which there is a need for the PCP, and the program is consistent with VAC's and government-wide priorities.
  • Success: The degree to which the program is effective in meeting its objectives.
  • Effectiveness, Economy and Efficiency: The extent to which the most appropriate and efficient means are being used to achieve objectives, relative to alternatives design and delivery approaches.

The evaluation team will review the history of the PCP and the Voluntary Sector Initiative (from which the PCP was created), interview key stakeholders and review related documents. Also, through a literature review, program comparison and interviews, the evaluation will assess alternate program delivery mechanisms.

In addition, the evaluation team will seek to determine the reasons for the lack of activity against three of the PCP's objectives: are the objectives relevant to VAC's mandate; have other initiatives been used in place of the PCP to achieve similar objectives; and have organizational changes impacted on the use of the Program, etc.

The evaluation will encompass FY 2001-2002 to September 30, 2007 inclusive.

The evaluation is expected to be completed by February 28, 2008. The minimum evaluation issues to be examined will include relevance, success, cost-effectiveness, economy and efficiency. Also, through the course of the evaluation, the PCP's performance measurement system will be reviewed. The estimated cost of the evaluation will be $115,000.

Methodology

The methodology for this study includes a file and document review, and interviews. A brief description is provided below:

File and Document Review

A File and Document review will be undertaken on Canada Remember's and PPP Branch's involvement with the PCP. Two types of files will be reviewed: 1) a general review of program files (such as administrative records, program records, financial records, communications and media); and 2) a more systematic review of select client or recipient files (program files).

The review will cover the following:

  • Administrative records (includes information such as the number of participants/rejected applicants per region, type of participants, type of projects created, duration of projects, cost of projects (PCP and in-kind), etc.).
  • TBS documents (TB submissions, Terms and Conditions, etc.) Information that describes how and why the program was created and what changes and influences (if any) since the program began.
  • Financial records can provide information on the cost to deliver the program and the program funds allocated and expended.
  • Program files (proposal, review committee assessment, letters, contribution agreement, invoices, monitoring, feedback etc.). The review will cover PCP recipients (National and Regional) and rejected applicants.
  • Strength of the partnerships with volunteer organizations (i.e. constraints, funding).
  • Media coverage (e.g. television, newspaper, radio, Internet).
  • Communications functions.

Interviews

Interviews will be conducted with management and staff.
Interviews will be conducted with regional staff, program recipients.

Annex 1 - Partnerships Contribution Program - Evaluation Issues and Questions

Relevance

The extent to which there is a need for the program; that the program is consistent with VAC's and government-wide priorities (Program focus, Partnership, Public interest)

Relevance
Key Issues and Questions Indicators Data Source Data Collection Methodology
1. Is there a need for the program? Needs identified

Activities aligned with needs
Program data
  • Document review
  • Interviews
2. Is there a change in clients, target groups and/or stakeholders?
  • Demographics
  • Regulatory Changes
  • Census
  • Program data
  • Public polling
Statistical data
3. What client and recipient needs are met by the program?
  • Are needs evolving?
  • Are there unmet needs?
Needs/interests identified
  • Survey results
  • Program data
  • Interviews
  • Statistical data
  • Survey results
4. Is the program congruent with federal government priorities?
  • Is there a need for government involvement?
  • What role should VAC play?
  • Does the program serve the Public Interest?
  • Program objectives support government objectives
  • Overlap with other stakeholders
    Government priorities
  • Government priorities
  • Canadians understanding of the country's history
  • GofC data, acts, policies, throne speech
  • Stakeholders
  • Acts, policies
  • Survey data
  • GofC documents
  • Document review
  • Interviews
  • Document review
  • Document review
  • Interviews
  • Statistical data
  • Public Polling
  • Document review
Success

The extent to which the program is effective in meeting its objectives, within budget and with expected outcomes.

The extent to which the activities and outputs of the program are consistent with the program mandate and are linked to achieving the objectives

Success
Key Issues and Questions Indicators Data Source Data Collection Methodology
1. To what extent has the program achieved its objectives?
  • Improved information, programming and policy
  • Public polling results
  • Program data
  • Survey results
  • Recipients
  • Document review
  • Survey results
  • Interviews, testimonials, feedback sheets
2. How does the program assist VAC in achieving its goals/mandate? Alignment with VAC's goals/mandate Reports on Plans and Priorities, DPR, Mission Statement Document review
Interviews
3. What, if any, unintended outcomes has the program had? Unintended outcomes identified
  • Program monitoring
  • Departmental Performance Reports
  • Document review
  • Interviews
4. Are the program's resources effectively targeted to meet the objectives of the program?
  • Program usage
  • Program marketing
  • Contribution agreements
  • Communications materials
Document review
5. Does the program have annual goals? What are they? Are achieved? Are they realistic?
  • Program usage
  • # participants
  • # projects
Departmental Performance Reports Document review
Cost-effectiveness, efficiency and economy

The extent to which the most appropriate and efficient means are being used to achieve objectives, relative to alternative design and delivery approaches

Cost-effectiveness, efficiency and economy
Key Issues and Questions Indicators Data Source Data Collection Methodology
1. To what extent are there alternative delivery mechanisms to deliver the program more cost-effectively? Costs factors Business cases
Options analysis
Document review
Program data
2. To what extent is there duplication or overlap with other existing programs (federal, provincial, municipal or private sector)?
  • Program usage
  • Alternative programs
  • Literature search
  • Program data
  • Document review
  • Interviews
3. How effective is the collaborative relationship between VAC and the recipients?
  • Program usage
  • Feedback from partners
  • Survey results
  • Feedback documents
  • Client satisfaction survey results
  • Review of feedback
4. Does the program demonstrate improved efficiencies or cost-effectiveness in achieving program goals each year?
  • Expenditures
  • Reporting to government
  • Expenditure data
  • Program data
  • DPR
  • Review of program data
  • Review of expenditure data
  • Document review
5. Are funds appropriately spent?
  • Are contributions awarded based on a clear process that includes a qualified assessment of merit?
  • Does the program have oversight practices that provide sufficient knowledge of recipient activities?
  • Does the program collect recipient performance data on an annual basis and make it available to the public in a transparent and meaningful manner?
  • Application package
  • Evaluation Committee Assessments
  • Reporting to government
  • Program documents
  • TB submission
  • Ministerials
  • Complaints
  • Program monitoring
  • Feedback from clients/partners
  • DPR
  • Public Accounts
  • Document review
  • Document review
  • Interviews
  • Document review
Date modified: