2.0 Study Approach

2.1 Evaluation Context

This is the first evaluation of the NVC and the findings will assist Veterans Affairs Canada (VAC) program managers to improve the design and delivery of the NVC programs and provide an assessment of the achievement of desired outcomes. In addition, this evaluation fulfills a Departmental commitment to complete a comprehensive evaluation of the NVC before December 2010.

Prior to commencing this evaluation, an evaluation framework and a detailed plan were developed. Given the size and complexity in conducting this comprehensive evaluation, the work will be completed in three phases with a report developed for each phase. Phase I, completed in December 2009, focused on the relevance and rationale of the NVC and its programs. This report, Phase II, focuses on the efficiency and effectiveness of VAC’s outreach, application process, and the service delivery framework. The final report, Phase III, will focus on unintended impacts and the success in achieving desired outcomes.

While this is the first evaluation of the NVC, other related work has recently been completed or is currently underway. The Audit and Evaluation Division recently completed a review of the NVC Redress process and a comparative analysis of the Disability Pension and Disability Award programs. In addition, VAC has completed an internal review of case management and an evaluation of case management is planned for 2011-2012. As a result, all the areas noted above were not included in the scope of this evaluation.

2.2 Scope of Work

The core evaluation issues and specific evaluation questions which will be covered by this evaluation are presented in the table below.

Table 2.1 - NVC Evaluation Questions

Core Evaluation Issues

Continued Need for the Program

Phase I Evaluation Questions

• What is the need for each of the NVC programs?

Alignment with Government Priorities

Phase I Evaluation Questions

• To what extent does the NVC align with Government priorities?

Alignment with Federal Roles and Responsibilities

Phase I Evaluation Questions

• To what extent does the NVC align with Government priorities?
• Is there duplication or overlap with other programs or services?

Achievement of Expected Outcomes

Phase I Evaluation Questions

• Are there unmet needs which should be covered by the NVC?
• How do the NVC programs compare to similar programs offered by other countries?
• How is program performance measured?

Phase II Evaluation Questions

• Are there appropriate supports in place for operational staff delivering the NVC programs?
• Are roles and responsibilities clearly defined and appropriate?
• How efficient and effective is the application process?

Phase III Evaluation Questions

• How efficiently and effectively does VAC assist clients in transitioning from military to civilian life?
• Are the NVC programs adequately supporting all NVC clients?
• How satisfied are clients with the NVC programs?
• How does the effectiveness of VAC’s NVC programs compare to other similar programs?
• What are the unintended impacts that have occurred?

Demonstration of Efficiency and Economy

Phase II Evaluation Questions

• How efficient and effective is VAC’s outreach?
• How efficient and effective is the application process?

Phase III Evaluation Questions

• How efficiently and effectively does VAC assist clients in transitioning from military to civilian life?
• What are the inputs and outputs?

2.3 Methodology

To strengthen the governance of this project, a Steering Committee consisting of VAC’s senior management was formed to monitor progress and provide strategic direction to the evaluation. In addition, an Advisory Committee was formed consisting of Directors from Head Office, regional and district offices across the country. The role of the Advisory Committee is to support the fieldwork and provide input into the development of data collection tools and evaluation findings. These two committees will support all three phases of the evaluation.

The scope of Phase II covers the period from April 1, 2006 to March 31, 2010. The data collection methodology incorporated included: key informant interviews, discussions with Client Advisory Groups, focus groups with clients, documentation review and data analysis. Utilizing these multiple lines of evidence ensured the reliability of information collected and results reported. Fieldwork for this report was conducted between November 2009 and April 2010.

For this phase, key informant interviews served as a primary source of information. These interviews provided qualitative input from staff involved with NVC across the country and also provided context to the documentation review and data analysis. For NVC, outreach activities, program application and service delivery occurs primarily in VAC’s 27 districts, including Integrated Personnel Support Centres (IPSCs), which meant that findings could vary by office due to differing site characteristics. As a result, input was collected from a broad cross section of staff across the country. In total, 165 interviews were conducted with staff from Head Office, all regional offices, 12 district offices, four IPSCs, Operational Stress Injury Support Service and the Canadian Forces. The site selection criteria as well as the specific offices included are presented in Annex C.

Client Advisory Groups were a key source of information regarding the client’s perspective. Prior to this evaluation, VAC had already established a NVC Advisory Group and Special Needs Advisory Group responsible for reviewing VAC programs and providing advice. These two groups consisted of representatives from Veterans organizations, academics, researchers, service providers and clients. As part of this role, these groups have prepared several reports summarizing their findings and recommendations. These reports were reviewed as part of the analysis conducted by the NVC evaluation team. In addition, after reviewing these reports, discussion groups were conducted separately with both groups to collect their input specific to the NVC evaluation.

Focus groups were also utilized as a key source of client input. An external consultant was engaged to ensure the confidentially of client input. A set of twelve focus groups were conducted in four locations. Four groups were conducted in Halifax and Quebec City (French), and two groups were conducted in each of Cold Lake and Mississauga. The target audiences were: CF members who recently participated in a transition interview, Veterans of the Regular Force released prior to the implementation of the NVC, Veterans of the Regular Force released after the implementation of the NVC, Veterans of the Reserve Force, spouses or survivors and participants of the Job Placement Program (now called Career Transition Services). The consultant’s report, including detailed methodology, is presented in Annex G. These focus groups collected valuable client input in relation to outreach, the application process and service delivery with key findings summarized in sections 4.1 to 4.3. The focus groups collected the client’s perspective on recognition, community integration and satisfaction with services provided. This information is also presented in the report and will support the analysis of outcomes in Phase III.

A documentation review was conducted to provide context and understanding of the current processes. Primary sources for the documentation review included: NVC legislation, VAC policies and business processes, developed frameworks or draft documents, the results of relevant internal reviews and program process maps. Data analysis was used to quantify findings and provide additional information to support qualitative comments made during interviews or focus groups. This information was derived with the support of VAC’s Statistics Directorate.