2.0 Scope and Methodology

2.0 Scope and Methodology

This evaluation has been conducted due to departmental risk and need. During the preparation of VAC’s Departmental Evaluation Plan 2019-24, the CRB was identified (through interviews and a documentation review) as a program that would benefit from an evaluation, primarily to assess program performance and effectiveness. In addition, this evaluation was conducted in accordance with Treasury Board of Canada’s Policy on Results.

2.1 Evaluation Scope and Questions

The evaluation focussed primarily on the time period of April 1, 2018 to September 30, 2019 (post implementation for the CRB).

The objective of this evaluation was to assess the relevance, performance, effectiveness, economy and efficiency of the CRB. Specific questions assessed during the evaluation are identified in Table 1.

Table 1: Evaluation Questions

To what extent does the program continue to address a demonstrable need, now and in the future? (Relevance)

To what extent does the program align with Government of Canada priorities and with federal roles and responsibilities? (Relevance)

To what extent is the Program responsive to the needs of its intended recipients? (Relevance)

Are processes currently in place to ensure the program is being administered as indicated in departmental guidance? (Performance)

To what extent is the program achieving its outcomes? (Performance)

  • Immediate
  • Intermediate
  • Ultimate

Are there opportunities to improve the efficiency and economy of the program? (Efficiency/economy)

Are there any unintended impacts (positive or negative)? (Efficiency/economy)

2.2 Multiple Lines of Evidence

Multiple lines of evidence have been used to support the evaluation findings. The methods undertaken to support these lines of evidence are identified in Table 2.

Table 2: List of Methodologies
Methodology Source
Departmental Documentation and Secondary Research Review Departmental documentation/information has been reviewed to understand the program objectives/intent, authorities and requirements, complexity, context and any key issue areas. Documents included departmental planning documents, policies, mandate letters, business processes, records of decisions, strategic documents, performance reports, research papers, survey results, and correspondence.
Non-Departmental Document Review Various non-departmental documents were reviewed, including, Parliamentary reports, Budget Speeches/Plans, and Speeches from the Throne.
Interviews Interviews were conducted with VAC staff in the following areas: program policy, stakeholder engagement, strategic policy, field operations, program management (service delivery), benefits processing, as well as other subject matter experts within the Department including Senior Management.
Statistical Analysis Statistical analysis included:
  • VAC Facts and Figures.
  • financial and operational data collected by VAC.
  • financial and operational data for other VAC programs that may relate to the CRB.
File Review A file review was completed to assist in evaluating the performance and efficiency of the CRB. A random sample of 180 decisions were reviewed (of 1,806 total decisions), providing a confidence level of 95% +/- 7%.
Survey A survey was disseminated to all current CRB recipients. Recipient feedback has been used as a data source. The survey was completed by 166 caregivers (of 705 contacted), providing a confidence level of 95% +/- 7%.

2.3 Considerations, Strengths and Limitations

  • The CRB is a relatively new program, replacing the previous FCRB: neither have been previously evaluated.
  • The evaluation team conducted the CRB Survey to better understand the experiences and perspectives of caregivers. The 705 caregivers in receipt of the benefit as of September 30th, 2019 were contacted. This was the first time contacting and requesting feedback from the caregivers limiting ability to complete trend analysis.
  • The evaluation team did not assess other interventions that may be available for supporting caregivers.
 
Date modified: