Language selection


Search veterans.gc.ca

3.0 STUDY APPROACH

3.1 Evaluation Context

This is the first evaluation of the NVC. The findings will assist VAC program managers by providing an assessment of the achievement of desired outcomes to improve the design and delivery of the NVC programs. In addition, this evaluation fulfills a departmental commitment to complete a comprehensive evaluation of the NVC by December 2010.

Prior to commencing this evaluation, an evaluation framework and a detailed plan were developed. Given the size and complexity in conducting this comprehensive evaluation, the work was completed in three phases with a report developed for each phase. Phase I, completed in December 2009, focussed on the relevance and rationale of the NVC and its programs. Phase II, completed in September 2010, focussed on the efficiency and effectiveness of VAC’s outreach, application process, and the service delivery framework. This final report, Phase III, focusses on the success in achieving desired outcomes.

An evaluation of disability pensions (DP) and disability awards (DA) was conducted concurrently with the NVC evaluation, and was finalized in August 2010. This report also incorporated elements examined Earnings Loss (EL) and Permanent Impairment Allowance (PIA). EL and PIA are both benefits paid to eligible recipients who are participating in the Rehabilitation Program. The purpose of the Disability Pensions and Awards Evaluation was to assess similarities and differences in program design, delivery, activities, outputs and outcomes. The evaluation also considered relevant elements of other NVC programs in the comparisons, where appropriate. The Evaluation of Disability Pensions and Awards should be read in conjunction with the NVC Evaluation reports. Further information as to the conclusions and recommendations from these reports can be found in appendices B, C and D.

While this is the first evaluation of the NVC, other related work has been completed recently or is currently underway. The Audit and Evaluation Division recently completed a review of the NVC Redress process. Additionally, VAC has completed an internal review of case management and an evaluation of case management is planned for 2011-2012. As a result of the ongoing or planned work, these areas noted above were not included in the scope of this evaluation.

3.2 Scope of Work

The scope of Phase III covers the period from April 1 2006 to 31 March 2010. In addition to information collected from earlier phases, a number of data collection methodologies were utilized.

The core evaluation issues and specific evaluation questions which were covered by the NVC and DA/DP evaluations are presented in Appendix E.

3.3 Methodology

To strengthen the governance of this project, a steering committee consisting of VAC’s senior management was formed to monitor progress and provide strategic direction to the evaluation. An advisory committee was formed consisting of directors from Head Office, regional offices and district offices across the country. The role of the Advisory Committee was to support fieldwork and to provide input into the development of data collection tools and evaluation findings. These two committees supported all three phases of the Evaluation.

Working in collaboration with VAC’s Statistics Directorate, significant data analysis was conducted. Large amounts of data were received across all NVC programs which required Statistics to review available data, and in some cases, perform additional manipulation based on requirements specific to this evaluation.

A file review completed by an external occupational therapist provided an overall assessment of progress for participants in the Rehabilitation Program. The sample population for this file review consisted of individuals who entered the Rehabilitation Program between April 1, 2006 and October 31, 2009, and who had participated in the program for at least six months. A stratified random sample of 350 participants was drawn. This descriptive sample allowed for the analysis of the sample’s characteristics and a comparison among various groups, but did not allow for the results to be generalized for the entire participant population. However, the findings from the File Review remarkably mirror the results and percentages gathered when studying the entire population using departmental statistical information.

Since neither baseline participant information nor control group data was available, Audit and Evaluation Division partnered with VAC’s Research Directorate, DND and Statistics Canada on a multi-year, four-part research study called the Life After Service Study (LASS). The findings from this study will support VAC in meeting its mandate in the care, treatment and reestablishment in civilian life of Veterans, and fill gaps in understanding military to civilian transition in Canada.

This research is composed of four parts. The first part of the LASS is an Income study of CF Members who released over a ten year period from 1998 – 2007. This income study was done longitudinally to collect Veteran income information for the year prior to release, the year of release, and all available post-release years up until 2007. Income information was collected by matching the study population data to the general family file (T1FF) tax records using social insurance numbers contained in both databases. Confidentiality of an individual’s information was maintained by Statistics Canada and only summary results were shared with VAC. VAC’s Research Directorate led the development of this study and wrote a summary report which informed the NVC evaluation.

The second part of the LASS is the Survey on Transition to Civilian Life (STCL). STCL information was drawn from a nationally representative sample of 4,721 of 32,015 Regular Force Veterans released from 1997 to 2008 was contacted by Statistics Canada during February and March 2010. The majority responded to the survey (71 percent), and the great majority of those (94 percent) agreed to share their responses with VAC and DND. The study included Veterans participating under the older Pension Act, and the first of those participating under the Canadian Forces Members and Veterans Compensation and Re-establishment Act (CFMVRCA) of 2006. The sample was stratified into three groups: individuals participating in New Veterans Charter programs (NVC participants), Veterans and spouses in receipt of disability pensions but not participating in NVC programs (DP recipients), and Veterans not participating in VAC programs. The information from the STCL report was used to inform the Re-establishment section of this report.

Parts three and four (mortality and cancer) will be the subject of another report on the CF Cancer and Mortality Study and do not inform this analysis.

Additionally, focus groups were utilized as a key source of input for the NVC evaluation. An external consultant was engaged to ensure the confidentially of participant input. A set of twelve focus groups were conducted in four locations across Canada. Details can be found in the Phase II report.

Lastly, key informant interviews and documentation reviews were utilized to provide important context to the information collected from data analysis and other methodologies. A number of interviews were conducted with staff from various levels of the organization. Primary sources for the documentation review included: NVC legislation, VAC policies and business processes, developed frameworks or draft documents, as well as the results of relevant internal reviews.

3.4 Limitations

3.4.1 Performance Measurement and Outcome Information

As described in the Phase I evaluation, VAC was still developing specific program logic models and performance measurement plans for the NVC programs. During the course of this evaluation, Audit and Evaluation Division provided input into these logic models and performance measurement plans which have now been finalized and implemented (for information on data not being reported, please refer to Appendix F). However, limited historical performance information has been collected which could be utilized as part of the analysis.

A key tool developed to collect outcome information was the “Re-establishment Survey.” This tool includes the SF-12 survey (a tool which measures health related quality of life) and was chosen as the best tool to meet the needs of the Department. In addition to the SF-12, the Re-establishment Survey includes other questions developed in collaboration with various areas within VAC. This additional survey content collects data related to economic security, community and family integration, employment status, and perceived recognition. The Re-establishment Survey data is self-reported. VAC does not collect clinical findings on outcomes.

As noted in the Phase I report, the response rate to the Re-establishment Survey was 44 percent in 2007-2008, making the information of limited utility. As a result, it was recommended that opportunities be identified and implemented to improve response rates. Program Management actioned this recommendation by revising the survey and developing a new approach to improve response rates. However, the evaluation was able to utilize some information from the 2008-2009 Re-establishment Survey.

Also, the extent and nature to which performance measures can be defined is limited due to the constraints imposed by VAC’s current Performance Activity Architecture.

3.4.2 Absence of a baseline or comparison group

The Evaluation Plan identified that this was the first evaluation of the NVC and a baseline was not established in advance of implementation. In addition, as mentioned in the evaluation of DA/DP, no comparable suite of programs existed within Canada, and a control group could not be developed as all individuals who apply and are eligible for NVC programs and services generally receive them. Furthermore, VAC serves a group of individuals with very complex needs, not all of whom will be able to successfully re-establish. Where the evaluation team had sufficient evidence to make conclusions on effectiveness and efficiency, they have done so. However, some general conclusions regarding the overall effectiveness were limited given limited baseline or comparison group.

3.4.3 Availability of data

The evaluation team worked closely with the Statistics Directorate to collect and analyze all available information relating to the NVC outcomes. However, there were gaps in the information that was available. One significant gap, as described in Section 4.2, was the inability to identify accurate costs for treatment in the Rehabilitation Program. This limited the evaluation team’s ability to assess the efficiency of the program as specific costs could be isolated only through assumptions.

Another gap is that no barriers (a barrier to re-establishment in civilian life is that which limits or prevents the individual’s reasonable performance of his/her roles in the workplace, home or community), were recorded for medically released Veterans who apply for the Rehabilitation Program within 120 days of release. In total, there were 1,285 Veterans who have accessed the Rehabilitation Program under medical release eligibility so the information relating to their barriers is not documented. This limits the evaluation’s ability to assess health progress for approximately 30 percent of all rehabilitation participants. In April 2010, modifications were made to the Rehabilitation Record of Decision (RROD), allowing data entry concerning barriers to re-establishment for medically releasing Veterans.

VAC has developed an earnings loss benefit to temporarily provide income replacement for individuals while in the Rehabilitation Program and offers vocational or Career Transition Services to enhance employment opportunities. For the period under review, VAC did not have sufficient information regarding post employment income to determine if this employment is providing a level of income deemed to meet basic needs. Recent changes announced by the Minister of Veterans Affairs have rectified this shortcoming for those who are seriously disabled.

The file review noted that in baseline (and subsequent) assessments performed by case managers (CM), the degree of information captured on the extent of community participation is often limited. The individual may be participating, however, it is difficult to capture the degree of participation because the information is not always documented and the relevant information is not captured concisely in one location.

3.4.4 Time period

Many injured CF members require several years of support from VAC to successfully transition and re-establish to civilian life. The NVC was implemented in April 2006 providing only four years of data and only a limited number of Veterans have had enough time to complete the programs. This posed limitations on the assessment of overall program effectiveness over the long term. Future evaluations will have additional information available to analyze the various outcomes over a longer post-program time period.

Date modified: