Language selection

4.0 Performance

4.1 Achievement of Expected Outcomes

The effectiveness and success of a program are generally measured through the use of performance measures and program outcomes. This section of the report addresses the progress realized towards achieving the Program outcomes.

Immediate Outcome: Eligible Veterans and other program recipients have access to medical, psychosocial, and vocational rehabilitation services

Veterans receive eligibility decisions to the Program within VAC standards but access to rehabilitation services is not always timely.

Generally, the process for access to the Program is:

  1. An application for the Program is received and eligibility is confirmed;
  2. An assessment is completed by the Case Manager;
  3. A plan is developed to determine which services/ benefits will be accessed; and
  4. The services and benefits identified in the plan are funded.

As of March 31, 2013, the Program had a 96% favourable rate for eligibility decisionsFootnote 12. VAC has a service standard for processing applications for entry into the Program. The standard states that "a decision will be made within 2 weeks of receiving all information in support of your application". The target (80%) has been achieved every quarter since March 2011. As per legislation, an application for entry into the Program is required. Given the high approval rates for the Program, the Department is looking at streamlining the application process.

The breakdown of eligible Program recipients by fiscal year for 2010-11 to 2012-13 is outlined below in Table 2. Overall, there was a 29.9% increase (4,515 to 5,866) in recipients eligible for the Program from 2010-11 to 2012-13. Although the medically-released recipients increased the most on a percentage basis (39.8%), the most populated eligibility type, rehabilitation need recipients, continues to drive the increase in overall Rehabilitation participants in terms of numbers.

Table 2 - Total Eligible Program Population at Year End (March 31st)
Eligible Program Recipients 2010-11 2011-12 2012-13 Population Change
(2010-11 to 2012-13)
Rehabilitation Need 3,081 3,554 3,883 26.0%
Medically-Released (120 Days) 1,338 1,589 1,871 39.8%
Spouses and Survivors 96 113 112 16.7%
Total 4,515 5,256 5,866 29.9%

Source: VAC Statistics Directorate

VAC policy does not require Case Managers to conduct an in-depth assessment of the Veterans' need for the Program prior to an eligibility decision. The intent of this process is to ensure eligibility decisions are made quickly to ensure Veterans receive needed rehabilitation services or vocational assistance in a timely manner. For example, in situations where the Earnings Loss BenefitFootnote 13 is payable, Veterans will receive the benefit once eligibility for the Program is approved. As an in-depth assessment is not conducted at the application stage, participants can only be provided with general information regarding the Program and its requirements. Detailed information cannot be provided until an assessment is completed, re-establishment barriers are identified and the Plan is developed.

A participant's Plan defines the resources, benefits, and services required for their successful completion of the Program. For fiscal year 2012-13, 58% (3,380 of 5,866) of participants (target 80%) received a Plan within 90 days of their eligibility decision.Footnote 14 The delay in development of the Plan means that access to and provision of rehabilitation services is also delayed. New service standards regarding service delivery have been added which require regular follow-up with participants within specified time periods. These standards may address the situation but were out of the evaluation's scope and will be reviewed during the next Program evaluation.

Further, the delay in creating the Plan can mean a delay in consultation between Case Managers and health/rehabilitation professionals. Interviews with staff at varying levels indicated that Case Managers often do not consult with internal health and rehabilitation professionals until the later stages of the participant's Plan. According to the file review, there was no consultation with a VAC health or rehabilitation professionalFootnote 15 on 31% (28 of 90) of the files.Footnote 16 When consultation did occur, on average it happened approximately 5 months after approval of the recipient's eligibility. This file review did not measure if consultation with external service providers had occurred; however, a file review of Rehabilitation participants undertaken in the Audit of Delegated Decision-Making in December 2013 identified that consultation with external service providers occurred in 44% (53 of 120) of the files reviewed.

The delay in consultation with health and rehabilitation professionals, in addition to any time lapsed from date of injury, negatively impacts the possibility of addressing the recipient's rehabilitation need.Footnote 17 The time lapse is a major challenge and further supports the need for early consultation with health and rehabilitation professionals as well as subject matter experts.

In 2009, VAC's Medical Advisory Committee identified that health and rehabilitation professionals are often consulted too late in the process. Evidence-based international research favours early and comprehensive disability intervention guided by multi-disciplinary professionals.Footnote 18 It suggests that a lack of consultation at the entry-level stage can affect the consistency of decisions and subsequently the effectiveness of the developed plans. As each type of professional (e.g., occupational therapist, doctor, nurse, or vocational specialist) brings their expertise, a more holistic approach to the creation of the participant's Plan can be developed. Appendix D illustrates the relationship between each of the various Rehabilitation elements and the wide range of professionals that may contribute to the development of the Plan.

Intermediate Outcome: 1. Eligible Veterans and other program recipients experience improved health and functional capacity

To determine if Program participants experience improved health and functional capacity, additional data/information is required.

Currently outcome measurements are based on self-reported scores from the March 2012 Re-establishment SurveyFootnote 19 which also contains the Short Form-12 Health Survey.Footnote 20 Based on responses from the Survey, participants who have completed the Program show moderate improvements and/or declines in stress levels, physical health and mental health. As noted in the evaluation limitations, the participation rate for this survey is below adequateFootnote 21 (33% or 697 of 2,110), the most recent results are as of March 2012, and survey participants may not have been representative of the overall program participant population.Footnote 22

As part of the NVC Phase III Evaluation, an occupational therapist, external to the department, conducted a file review which provided an overall assessment of progress for participants in the Program. The sample population for this file review consisted of individuals who entered the Program between April 1, 2006, and October 31, 2009, and who had participated in the program for at least six months. The Evaluation Team reviewed assessments of 73 of these participants, who continued to have open Plans in 2010, to obtain further information on the perceived change in health and functional capacity. 53 participants in the file review had an assessment completed before October 2009 and another assessment completed between November 2009 and September 2013 (the end of file review data collection for the current evaluation).

The findings from the file review are similar to that of the Re-establishment Survey. More detailed results of the file review can be found in Appendix E. It should be noted that there are limitations to the file review as assessments by a Case Manager are not conducted on a regular basis and some or all of the above information is self-reported by the recipient. The Re-establishment Survey also has some limitations. Self-reported health measures are a predictor of health status but should be used to complement other health status indicators.Footnote 23 In addition, research indicates respondents may inflate the incidence and severity of health problems and disability in order to rationalize labour force non-participation and/or receipt of disability benefits.Footnote 24 Additional measures from alternate sources would assist in performance measurement for this outcome.

Intermediate Outcome: 2. Eligible Veterans and other program recipients have the knowledge, skills, and abilities to achieve an appropriate occupational goal

Eligible Veterans are acquiring the knowledge, skills, and abilities to achieve their occupational goal.

The purpose of vocational rehabilitation is to assist Veterans with a health problem become employable given their education, training, and experience. Vocational services and benefits include vocational assessments, training, career exploration, job placement, and follow-up support.

According to the Re-establishment Survey, for fiscal year 2011-12, 54% (91 of 168) of recipients felt prepared to find employment at the beginning of the Program compared to 82% (116 of 141) who were prepared when the Program was completed. Also, most participants who completed the Program indicated that their knowledge/skills increased in résumé writing, interview techniques, conducting a job search, labour market information, employment planning, and self-marketing.Footnote 25

The CanVet Quarterly Client Training Report, for fiscal year 2012-13, reported that of the 218 participants who enrolled in training, 69% (151) of recipients successfully completed one or more of the training programs that were included in their Individualized Vocational Rehabilitation Plan (IVRP).Footnote 26

While these results are positive, i.e., participants in vocational rehabilitation are acquiring the knowledge, skills, and abilities to achieve their occupational goal, over the three year period from 2010-11 to 2012-13, the proportion participating in vocational rehabilitation funded by VAC has decreased from 27% (1,207 of 4,515) to 22% (1,309 of 5,866) respectively.Footnote 27

Ultimate Outcome: 1. Eligible Veterans and other program recipients actively participate in the civilian workforce

The number of eligible Veterans achieving their employment goals is below the Program's target.

From 2010-11 to 2012-13, the employability outcome (Veterans attaining their employment goal) at the end of a recipient's IVRP increased from 20% (9 of 46) to 28% (84 of 297). This remains below the 80% target. Footnote 28

As of March 31, 2013, 22% (1,309) of the eligible 5,866 Program participants were accessing vocational rehabilitation. As 32% (1,883 of 5,866) were eligible for Earnings Loss and in receipt of SISIP (VAC has no authority to provide Vocational Services) and a further 16% (914 of 5,866) have been determined TPI it is unlikely that these individuals would be participating in vocational rehabilitation. The remaining 30% (1,760 of 5,866) of eligible Program participants were not accessing vocational rehabilitation as it was determined they were not currently suitable or they had no interest. This information is represented below in Figure 1.

Figure 1 – Breakdown of Vocational Rehabilitation, SISIP, and TPI Participants - March 31, 2013

Figure 1
Figure 1 - Breakdown of Vocational Rehabilitation, SISIP, and TPI Participants - March 31, 2013 Table
Category Number of Participants
Participants Not Accessing Vocational Rehabilitation 1760
Participants Considered TPI 914
Participants Eligible for EL and in Receipt of SISIP 1883
Participants Accessing Vocational Rehabilitation 1309

Source: Rehabilitation and Vocational Assistance Program Performance Snapshot and Client Profile, 2012-13

Figure 2 below displays the number of participants who accessed vocational rehabilitation services, submitted an IVRP, completed an IVRP, completed training, became employed, and achieved their employment status goal. Although the number of participants becoming employed or achieving their employment status goals has increased annually, this number is small in comparison to the number of participants who accessed vocational rehabilitation and submitted an IVRP.

Figure 2 – Overall Achievement of Employment Goals by Eligible Recipients

Figure 2
Figure 2 - Overall Achievement of Employment Goals by Eligible Recipients Table
Employment Goals 2010-11 2011-12 2012-13
Accessed Vocational Rehabilitation Services 1207 1326 1309
IVRP Submitted 452 623 660
IVRP Completed 25 113 182
Training Completed 19 91 151
Employed - IVRP Completed 12 76 123
Achieved Employment Status Goal 9 58 84

Source: CanVet Quarterly Reports

One benefit available to rehabilitation participants is the Earnings Loss Benefit. This benefit recognizes the economic impact that a service-related disability may have on the participant's ability to earn income following release from the CAF. When combined with other potential benefits (e.g., Permanent Impairment Allowance, Permanent Impairment Allowance Supplement)Footnote 29, a recipient can receive a substantial monetary benefit while not participating in the workforce.

Employment has an impact on physical, mental, and social health; provides compensation; as well as a sense of identity, purpose, and social contacts.Footnote 30 It is important that the monetary benefits received by Program participants do not detract from the outcome of participation in the workforce. Research supports the notion that increases in monetary benefits can have a detrimental effect on program completion rates and outcomes. MacLean and Campbell (2013) state “... the generosity of benefits can be a financial disincentive undermining employment goals ...”.Footnote 31 Recent research also identifies that Veterans' success in the workplace contributes to a successful transition to civilian lifeFootnote 32 yet many participants may never reach the reasonable standard of living as defined by the Earnings Loss Benefit (currently a total pre-tax income of at least $42,426 as of 2013). If a participant can earn more income while receiving the Earnings Loss Benefit than they would from a job, there would be a lack of incentive to participate in the workforce. Interviews with Case Managers and other VAC staff supported the notion that monetary benefits can impact a participant's motivation to participate in and complete the Program.

In most cases, when a participant is confirmed to have a health problem which does not enable him/her to reach 66 2/3% of their pre-release earnings, he/she may be deemed TPI. A participant who is declared TPI may receive monetary benefits from VAC for life without need for employment. While the majority of TPI Veterans reported being permanently unable to work or not being in the active labour force, 27% had been working in the year following release from the military. A TPI classification does not recognize the varying degrees of work capacity which can encourage labour market engagement by Veterans. Evidence-based research identifies that with the increase in monetary benefits and the increase in the income threshold introduced in 2011, the number of TPI Veterans has tripled.Footnote 33

In summary, these findings raise questions about whether the monetary benefits available under the Financial Benefits Program have created an unintended outcome for the Program by acting as a deterrent to active participation in the Program. As financial benefits may be a reason to remain in the Program for a longer period of time, it may affect a participant's motivation to actively engage in the labour market. These issues may be reviewed under the current NVC Review and are part of the upcoming NVC Financial Benefits Evaluation which Audit and Evaluation Division plans to begin in 2015-16.

Ultimate Outcome: 2. Eligible Veterans and other program recipients actively participate and are integrated into their communities

Eligible Veterans are experiencing positive family relationships but limited integration within their communities.

According to the 2011-12 Re-establishment Survey results, 75% (246 of 326) of participants reported they were very or somewhat involved in the day-to-day activities of their families and 73% (238 of 326) of participants reported having a positive relationship with their families when they complete the Program. Responses in regards to community involvement were less favourable as only 33% (106 of 321) of recipients reported satisfaction with their level of community involvement and 26% (85 of 328) of recipients reported having a positive sense of belonging to their local community in the 2011-12 fiscal year.Footnote 34

File review results from the most recent assessments by a Case Manager indicated 34% (25 of 73) of participants reported a somewhat strong sense of belonging to their community, followed by somewhat weak (29% or 21 of 73), and very weak (21% or 15 of 73).Footnote 35 In addition, the Life After Service Survey on Transition to Civilian Life noted that NVC participants reported a very low rate of community belonging (39%) compared to Disability Pension participants (56%) and non-participants (62%).Footnote 36 These findings support the mixed sense of belonging that recipients feel within their communities.

4.2 Demonstration of Economy and Efficiency

4.2.1 Program Expenditures and Participants

Expenditures have increased annually and are forecast to continue to increase.

According to the initial Program forecast, the Department did not anticipate the current level of expenditures or number of participants in the Program. The forecast was based on industry standards for rehabilitation that assumed, on average, participants would require rehabilitation treatment between 18 and 24 months.Footnote 37

Figure 3 - Rehabilitation Participants and Expenditures (Actuals and Forecasts), 2006-07 to 2017-18

Figure 3
Figure 3 - Rehabilitation Participants and Expenditures (Actuals and Forecasts), 2006-07 to 2017-18 Table
Year Actuals and
2013 Forecast
($)
Original
Forecast
($)
Actuals and 2013 Forecast
Eligible Participants
Original Participant
Forecast
2007 $599 $14,900 1139 767
2008 $2,300 $30,300 1897 1536
2009 $4,836 $30,900 2591 1539
2010 $7,868 $22,000 3417 1067
2011 $11,662 $12,900 4515 596
2012 $13,926 - 5256 -
2013 $18,360 - 5866 -
2014 $24,345 - 6700 -
2015 $25,742 - 7400 -
2016 $32,717 - 8100 -
2017 $33,623 - 8700 -
2018 $36,187 - 9100 -

Sources: Original forecast from the Integrated Results-based Management Framework and Risk-based Audit Framework (RMAF/RBAF) for the NVC Memorandum to Cabinet. Actual and 2013 Forecast data is from the 2013 VAC Client and Expenditure Forecast document.

Figure 3 above illustrates the original forecast, actuals, and recent forecasts of both participants and expenditures. At inception, the Department expected a sharp increase and corresponding decline in the number of participants in the Program. In actuality, participants and expenditures have continued to increase year over year.

Data compiled since the Program's inception shows that the length of time an individual remains in the Program can vary considerably. The evaluation did not examine if the participants' plans included discussions regarding the timeframes for identified resources, benefits, and services as this is a case management activity. Case management was not included in the scope of the review. Figure 4 below shows the number of participants who entered the Program each fiscal year and those who are still eligible for the Program. Of the participants who entered the Program in 2006-07, approximately 33% (324 of 989) are still in the Program.

Figure 4 - Rehabilitation Applications and Eligibility by Type, 2006-07 to 2012-13 Table

Figure 4
Figure 4 - Rehabilitation Applications and Eligibility by Type, 2006-07 to 2012-13 Table
Program Status Year
2006-07 2007-08 2008-09 2009-10 2010-11 2011-12 2011-12
Still Eligible - Medically-Released 64 83 143 165 275 411 615
Still Eligible - Rehabilitation Need 260 222 268 443 700 767 1019
Total Applications 989 784 867 1091 1415 1468 1770

Note: Due to small reporting numbers, unknown and survivor applications were excluded from the number of still eligible applications. Any applications that were cancelled, not eligible, withdrawn, had invalid coding, or belonged to a deceased participant were excluded.

The number of participants still in the Program is further supported by the findings of the file review. It identified that of the 73 participants who had open plans in 2010, 40% (29 of 73) are still in the Program.

As shown below in Table 3, between 2010-11 and 2012-13 Program expenditures have increased by 57.3% ($11.7 million to $18.4 million). In addition, the number of recipients participating in the Program has increased by 29%, from 4,515 in 2010-11 to 5,866 in 2012-13.

Table 3 - Program Expenditures at Year End (March 31st)
Program Information 2010-11 2011-12 2012-13 Change from
(2010-11 to 2012-13)
Program Expenditures $11.7M $13.9M $18.4M 57.3%
Program Participants 4,515 5,256 5,866 29.9%
Average Program Expenditures per Participant $2,591 $2,645 $3,137 21.1%

Source: VAC Facts and Figures

While both participant numbers and Program expenditures are increasing, Program expenditures are increasing at a higher rate. This increase is mostly due to a growth in the usage of related health servicesFootnote 38 and the cost of vocational rehabilitation services which has resulted in increasing expenditures per participant per year.

As of March 31, 2013, 2,342 recipients have completed the Program since its inception. On the contrary, there are a large number of Veterans (2,863 or approximately half of the 5,866 total participants)Footnote 39 that have remained in the Program for longer than originally expected. Of the participants that have been eligible since 2006-07, 44% (143 of 324) of them have been deemed TPI. In subsequent years, the majority of eligible participants deemed TPI continue to be comprised mostly of participants eligible through a rehabilitation need. For a detailed annual breakdown of the eligible TPI participants that have remained eligible for longer than expected, please refer to Table 4.

Table 4 - Historic Number of Participants Deemed TPI, 2006-07 to 2010-11
Rehabilitation Eligibility Type Description Year of Eligibility
2006-07 2007-08 2008-09 2009-10 2010-11
Participants Becoming Eligible Each Year # of Participants 324 306 415 612 983
# Who Are TPI 143 102 105 148 194
% Who Are TPI 44% 33% 25% 24% 20%
Medically-Released within 120 Days # of Eligible
Participants TPI
26 18 38 49 67
% of Eligible
Participants TPI
18% 18% 36% 33% 35%
Rehabilitation Need # of Eligible
Participants TPI
117 84 67 99 127
% of Eligible
Participants TPI
82% 82% 64% 67% 65%

Source: VAC Statistics Directorate

Many of these Veterans require long-term support. This is affecting the Program's ability to achieve the outcomes as designed. The Audit and Evaluation Division's New Veterans Charter Evaluation - Phase III (February 2011) recommended the Department develop a strategy to address the needs of participants who require long-term support; however, the recommendation has yet to be addressed and may be reviewed in the context of the NVC Review or in response to the ACVA report entitled The New Veterans Charter: Moving Forward. With these participants remaining in the Program for extended periods of time, expenditures should continue to climb sharply. As of September 2013, the Department has forecast an increase in expenditures from $18.4 million to $36.2 million (97% increase) between 2012-13 and 2017-18.Footnote 40

4.2.2 Program Expenditures and Participants

The methodology for calculating the allocation of costs for the Program needs to be revised.

Assessment of resource utilization is concerned with the degree to which a program demonstrates efficiency and/or economy in the usage of resources. The costs associated with delivering a program include salaries, operating and maintenance, employee benefits, and contract administration costs. Usage of program resources is driven by a number of factors, including:

  • program expenditures;
  • recipient population;
  • delivery mechanisms; and
  • eligibility criteria.

The evaluation team attempted to obtain the actual program resource utilization costs for the Program for 2010-11, 2011-12 and 2012-13 but noted limitations in the quality of data available related to costs. For example, each year VAC's Finance Division used varying methodologies to determine how costs were allocated when calculating program resource costs.

The evaluation team also determined that resource utilization costs attributed to the Program may be understated as supported by qualitative information gathered during interviews with staff. Case Managers interviewed noted that a significant percentage of their time was allocated to Program related activities. This would indicate that a large portion of Case Manager's salaries should be allocated to the Program. Based on information from VAC Finance Division, this salary amount is not being reported as program resource utilization costs.

Due to the data currently available, the evaluation team could not estimate, with an acceptable level of confidence, the resource usage in relation to progress toward expected outcomes.

4.2.3 Efficiency

Recent initiatives have improved the efficiency of the Program.

Recent implementation of initiatives have improved efficiency within the Program. As part of Transformation, VAC has streamlined the Program policies to assist decision-making and reduce complexity for staff.

In 2011, VAC implemented enhancements to delegated decision-making, which provided Case Managers with more authority to make medical/psychosocial decisions for Program participants in order to reduce the time to deliver benefits to participants. Audit and Evaluation Division's Delegated Decision-Making Audit of this process showed that Veterans were receiving more timely and effective responses from decision-makers due to the increased delegation of authority.

In addition, recent changes to policies for vocational services allow for more flexibility in education/training amounts and eligible expenses. VAC's national contractor, CanVet, has the authority to recommend and pay for more training expenses, without a maximum stipulated per benefit, for a maximum of up to $75,800 for needed training expenses arising out of an approved Plan.

Date modified: