2.0 Scope and Methodology

The evaluation team examined the Program over fiscal years 2009-2010 to 2011-2012. The evaluation focused on the Program pertaining to war service and CAF recipients. The evaluation does not include VAC’s Rehabilitation Program, which uses the provision and delivery model of this Program for its recipients, but has a separate set of eligibility criteria and approval authorities.

The five core issues cited in the Treasury Board Directive on the Evaluation Function were examined to assist senior departmental management in making future decisions regarding the design and delivery of health care benefits and services. The five core issues are outlined below.

Table 1 - Five Core Evaluation Issues

Relevance:	Performance:
Continued Need for the Program Alignment with Government Priorities Alignment with Federal Roles and Responsibilities	Achievement of Expected Outcomes Demonstration of Efficiency and Economy

The evaluation was conducted between May 2012 and March 2013. The study employed multiple lines of evidence to assess the Program’s relevance and performance. Table 2 below provides a list of the methodologies used.

Table 2 - List of Methodologies

Methodology	Source
Qualitative
Literature Review	Departmental reports (Acts, Regulations, program and planning documents, and audit and evaluation reports) Program documents from other countries (United Kingdom, United States and Australia) Other federal departments’ audit and evaluation reports Canadian health care reports and studies
Research Studies	VAC Research Directorate studies Studies conducted by other federal government departments who deliver health care programs
Key Informant Interviews (semi-formal, structured interviews)	Combination of 50 in-person, telephone and video interviews with: VAC staff Medavie staff Other federal and provincial government departmental subject matter experts
Quantitative
File Review	Random sample recipient file review (270 files provided a confidence level of 90% with a margin of error of ± 5 %)
Statistical/Program Data	VAC Statistics Directorate data Program performance measurement data VAC National Client Survey Contractor claims processing data Previous Veteran and public opinion polls and focus groups

Evaluation Considerations/Limitations

The evaluation team acknowledged the following when developing the evaluation design:

• This Program evaluation was designed to cover all major program components at a high level, providing a base-line for future studies.
• At the time of the evaluation, the Program was undergoing significant changes (policy, processes, delivery) and the resources involved in administering the Program were also changing.
• The evaluation team did not consult with recipients or providers directly. The team relied on results from previous public opinion surveys, papers produced by Veteran stakeholder groups, a recipient file review, as well as interviews with VAC field staff.
• As the Department is in the process of re-tendering a new third-party processing contract, the evaluation team did not conduct a detailed review of associated processes.