4.1 Performance Tools and Measures
There is a Performance Measurement Strategy in place for the Career Transition Services Program and performance targets are defined. Limited reporting information has been made available.
A Performance Measurement Strategy (PMS), including performance measures and program outcomes, is created upon program introduction and updated annually. These measures and outcomes are used to gauge program effectiveness and success.
The Program’s PMS, establishing a framework for monitoring activities and outcomes, was last updated in July 2015. Some data have been reported and made available to Program Management on a monthly basis; i.e., number of Veterans in the program, number of applicants, and summary expenditures. However, while more detailed information relating to factors such as age, release status, and release type is available, there is no evidence it has been used for decision making.
The 2015-2016 Report on Plans and Priorities planning highlights for the CTS Program indicate “The Department will amend the Program’s Performance Measurement Strategy to ensure that measures are put in place to monitor achievement of outcomes and that published service standards are communicated accurately.”
4.2 Achievement of Expected Outcomes
The outcomes are partially met. Veterans and survivors have access to funds. Program participation is low. The Ultimate Outcome is not measureable.
Program theory surmises that eligible Veterans and survivors with access to funding to support career transition will use available services to fulfill their career needs, thus contributing to the Program’s ultimate outcome, namely, “Eligible Veterans and survivors actively participate in the civilian workforce.” The following sections address the progress realized towards achieving the Program’s immediate, intermediate and ultimate outcomes, as laid out in the Program’s logic model (see Appendix D).
Immediate Outcome: Eligible Veterans and their survivors have access to funds for career transition services.
Funds are available for Veterans and their survivors to access; therefore, the immediate outcome is achieved.
Funds are available for eligible Veterans and their survivors. Via a September 2012 TB Submission, funding was allotted to the Program for fiscal years 2012-13 to 2016-17, as shown in Table 6.Footnote 27
| 2012-13 | 2013-14 | 2014-15 | 2015-16 | 2016-17 | Total | |
|---|---|---|---|---|---|---|
| Allocation (000s) | $139 | $285 | $296 | $37 | $50 | $807 |
At the lifetime maximum reimbursement level of $1,000, allocated funding for fiscal years 2013-14 to 2014-15 was sufficient to support 580 Veterans. Based on the average reimbursement for that period ($585.23), the program capacity for the funds allotted would have been 992 Veterans.
Intermediate Outcome: Eligible Veterans and survivors utilize available funds to obtain career transition services.
While some Veterans accessed funds to obtain career transition services, the low program uptake would indicate the intermediate outcome has not been met.
Table 7 shows Program participation and funding use over the past five years. Note that the number of eligible participants dropped from 581 in 2011-12 to 27 in 2012-13, corresponding with Program changes which eliminated access for those still in uniform. Such low participation rates are evidence that the Program’s intermediate objective is not being met.
| 2010-11 | 2011-12 | 2012-13 | 2013-14 | 2014-15 | |
|---|---|---|---|---|---|
| Veterans Eligible | 548 | 581 | 27 | 92 | 233 |
| Veterans Receiving Reimbursement | 17 | 23 | |||
| Expenditures (000s) | $1,613 | $1,096 | $1,003 | $12 | $13 |
Ultimate Outcome: Eligible Veterans and survivors actively participate in the civilian workforce.
There is currently no program measure assessing the degree of success in achieving this outcome.
4.3 Demonstration of Efficiency and Economy
4.3.1 Efficiency
The Program is administered in an efficient manner.
The evaluation team reviewed the entire data history of CTS and found no areas requiring improvement with respect to efficiency; however, the analysis is limited in what it can say or project due to the low participation rate. For Veterans who used the Program, it was administered efficiently throughout its various stages, from a timely response with respect to rendering eligibility decisions through to eventual adjudication and reimbursement of claims for services attained as well as appropriate adjustments made to claims exceeding the $1,000 limit.
Despite demonstrated Program efficiency, it does little to contribute toward achievement of the ultimate outcome. The Program has lived up to neither projected participation rates nor expenditure forecasts. Although a January 2013 change allowed for a lifetime maximum reimbursement of $1,000 for transition employment services from any qualified provider, participation remains extremely low.
4.3.2 Economy
The Program has not matured to achieve projected participation or expenditures; therefore, it has not proven to be economical.
Between 2013-14 and September 30, 2015, Program expenditures have increased by a small amount and the number of Program participants has also increased.
Table 8 below reflects the VAC Client and Expenditure Forecast 2015-16 (as per the Statistics Directorate) which shows increases for both eligible CTS recipients and expenditures from now until fiscal year 2018-19.
| Actual | Forecast | |||||||
|---|---|---|---|---|---|---|---|---|
| 2011-12 | 2012-13 | 2013-14 | 2014-15 | 2015-16 | 2016-17 | 2017-18 | 2018-19 | |
| Expenditures (000s) | $1,096 | $1,003 | $12 | $13 | $32 | $49 | $61 | $73 |
| Veterans Eligible | 581 | 27 | 92 | 233 | 360 | 470 | 560 | 650 |
Source: VAC Finance Division
To evaluate economy, it is useful to compare forecasted with actual uptake and expenditures, as shown in Table 9 below. The forecast information is based on the in-yearFootnote 28 forecasts prepared by VAC Statistics. Differences between forecast and actual amounts can be attributed to the absence of historical trend information (program delivery method changed in January 2013) and limited participation to-date. Even a small change to the number of Veterans or average costs can have a significant effect on the final expenditures.
| Year End 2013-14 |
Year End 2014-15 |
YTD Sept 2015 | Year End 2015-16 |
|
|---|---|---|---|---|
| Eligible Veterans at March 31 | ||||
| Forecasted | 86 | 150 | N/A | 360 |
| Actual | 92 | 233 | 307 | N/A |
| Veterans Reimbursed at March 31 | ||||
| Forecasted | 57 | 35 | N/A | 54 |
| Actual | 17 | 23 | 19 | N/A |
| Expenditures at March 31 (000s) | ||||
| Forecasted | $57 | $25 | N/A | $32 |
| Actual | $12 | $13 | $14 | N/A |
Notes:
VAC does not complete half year forecasts, therefore forecasted participation and expenditures are not available for YTD Sept 2015. Project scope ends at September 2015 so actuals for 2015-16 not reported. Source: VAC Finance Division
Resource utilization relates to the costs associated with delivering a program, including salaries, operating and maintenance, employee benefits, and contract administration costs. One measure of economy is how much it costs for the outcomes achieved.
Table 10 depicts the number of applicants who were reimbursed through the CTS Program from April 1, 2013 – September 30, 2015, the program expenditures, and how much was spent in delivery. During this time, only 59 people received reimbursements via the Program; therefore, program expenditures were relatively low. With respect to salary, the number of FTEs required to administer the program has been reduced from 7.1 to .4Footnote 29. Further analysis of administrative costs was not undertaken due to the low risk and low materiality of the Program.
| 2013-14 | 2014-15 | Y-T-D Sept 2015 | Total | |
|---|---|---|---|---|
| Veterans Receiving Reimbursements | 17 | 23 | 19 | 59 |
| Program Expenditures (000s) | $12 | $13 | $14 | $39 |
| Administration (000s) | $883 | $1,127 | $20* | $2,030 |
| Total Costs (000s) | $895 | $1,140 | $34 | $2,069 |
* In 2015-16, the Department adjusted its model of allocating Full Time Equivalent (FTE) staff and administrative costs across programs and services to more accurately reflect where resources are located to serve Veterans. As a result, expenses allocated to some programs, such as CTS, vary significantly from previous year. Annual administration costs are allocated at fiscal year-end; to cover the period of this evaluation (April 2015 to September 2015), the 2015-16 planned amount of $39,666 was divided by two. Also, in 2014-15, administration costs include a one-time advertising expense.
4.3.3 Alternative Approaches
As described in Section 3.3, releasing CAF members have a variety of needs related to employment and transition, impacted by factors such as release type, years of service, branch of the Forces and occupation, physical and mental health, family situation, province of resettlement and economic climate at the time of release. The interplay of these factors and individual goals suggests the need for flexible employment supports and transition services.
Evidence points to some likely overlap in needs between medically-releasing and voluntarily-releasing CAF members. Thirty-six percent (36%) of Veterans who applied for CTS had also applied, and were deemed eligible for, Vocational Rehabilitation. As interviews with CAF and VAC personnel revealed, employment is the driving need and Veterans eagerly embrace whatever means would enable them to achieve that end. Interviewees report that releasing members and Veterans exhibit a sense of “I don’t need rehab; I need a job.” These findings suggest an employment strategy to address the needs of all releasing CAF members is required.
As described in the Senate Subcommittee’s 2014 reportFootnote 30, the military fully manages a member’s career. Therefore, since many releasing personnel have been in the military their entire careers, they have not acquired the career development skills common to most civilians which could put them at a disadvantage in the civilian job market.
VAC research has revealed a certain segment of Veterans meet with serious barriers to transferring and translating their military skills and general service experience into terms understandable to civilians. There is some evidence to back the concept that support offered over a longer period of time, pre- and post-employment, achieves greater success in employment transition than the traditional "train then place" programs most common today. The United Kingdom (UK) Career Transition Partnership, for example, which provides on-going support up to two years post civilian employment, yields a high degree of success. There are also some domestic models which have adopted variations of this continued support model and are experiencing similar results.
United States Veterans Affairs (USVA) offers Transition GPS (Goals, Plans, Success), developed jointly with the Department of Defence and other agencies, which provides comprehensive services to service members to assist in the transition to work, life, and home after the military. The Veterans Employment Center connects Veterans and their families with meaningful employment and career development opportunities. Personnel are eligible for all programs prior to release.
These international examples of transition-employment assistance for Veterans suggest that early intervention (pre-release) and a longer length of support contribute to successful outcomes. A full comparison of Canadian, American and United Kingdom career transition services programs can be found in Appendix B.