Language selection

2.0 Scope and Methodology

The scope of the evaluation was from April 2010 to March 2013. The evaluation focused on two main eligibility groups: CAF Veterans who were medically-released from the military and applied within 120 days of release or who have a physical or mental health problem resulting primarily from service that is creating a barrier in their transition to civilian life. The evaluation was conducted between June 2013 and December 2013.

The scope of the evaluation included reviews of:

  • the processes and costs to fund Program services and benefits;
  • the eligibility process for the Program; and
  • the partnership between VAC and the national vocational rehabilitation service provider as it relates to service outcomes.

2.1 Multiple Lines of Evidence

The lines of evidence used to evaluate the Program’s relevance and performance are outlined in Table 1 below:

Table 1 - List of Methodologies
Methodology Source
Literature Review
  • Departmental reports (Acts, Regulations, program and planning documents)
  • Previous VAC Evaluations
  • Program documents and data from United Kingdom, United States, and Australia
  • Recipient survey results
Documentation Review
  • A review of departmental policies, regulations and procedures, forms, etc.
Research Studies
  • VAC Research Directorate studies
  • Studies conducted by other federal government departments
  • Evidence-based non-VAC literature
Key Informant Interviews
  • A combination of 96 in-person, telephone and video interviews with VAC staff, CanVet staff, and departmental subject matter experts
File Review Two File Reviews (See Appendix B):
  • Statistically valid random sample of 134 eligible recipients (confidence level of 90%, with a margin of error of ± 7 %)
  • Review of all 73 Rehabilitation participants who released after 2006 and had active plans in the 2010 file review conducted during the New Veterans Charter Evaluation Phase III
Statistical/Program Data
  • VAC Finance Division
  • Program Performance Measurement
  • Re-establishment Survey
  • Contractor health claims processing data

2.2 Limitations and Analytical Challenges

The following limitations were identified during the evaluation:

  • The evaluation team did not consult with recipients or providers directly; they used results from previous surveys.
  • The team did not have the required qualifications or access to subject matter experts or health professionals and therefore were unable to assess participants’ health outcomes or case management activities.
  • The main performance measure for Program outcomes is based on the Re-establishment Survey which contains self-reported health measures. Analysis of the data is current as of March 31, 2012, and the overall response rate for the 2011-12 Re-establishment Survey was 33% (697 responses from the 2,110 rehabilitation decisions). The evaluation team attempted to mitigate this limitation by acquiring other lines of evidence to support or refute the survey results.
  • NVC Financial Benefits were reviewed at a high level as Audit and Evaluation Division plans to evaluate these benefits in 2015-16 at which time it is expected that an in-depth evaluation of Earnings Loss Benefits will occur.
  • The evaluation team was unable to estimate with an acceptable level of confidence Program resource utilization costs for the period under review, due to limitations with the current methodology.

These limitations should be considered when reading the evaluation findings.

Date modified: