Language selection

Evaluation Framework for the New Veterans Charter - March 2008

ACKNOWLEDGEMENT

The Audit and Evaluation Division would like to thank staff of the Veterans Affairs Canada Quality Management Team for their input concerning the ongoing performance measurement for the Department.

Secondly, the Manager of the New Veterans Charter programs provided valuable assistance in reviewing this program profile section of this document.

Executive Summary

New Veterans Charter (NVC) programs offer releasing Canadian Forces (CF) members, Veterans and their families a modernized, comprehensive and integrated range of compensation and wellness programs to support their rehabilitation and re-establishment from military to civilian life.

The purpose of this evaluation framework is to outline how the New Veterans Charter programs could be evaluated in the future. It provides a profile establishing the origin, legislative authority, benefits, objectives and results, and delivery structure of the program. Secondly, it provides an evaluation framework which establishes potential evaluation issues, information and methodologies required, and inputs to the evaluation planning process.

Preparation of the evaluation framework has taken into account the information needs of the government in the management of its resources and programs. It complements and utilizes the information produced by the ongoing performance measurement established for the NVC programs. Consultation on the development of this evaluation framework included the Veterans Affairs Canada NVC Program Manager and the Quality Management Team.

The nature and timing of future evaluations will be dependent upon:

  • requirements of government in the management of resources;
  • expectations created with central agencies upon program approval;
  • the maturity of the program; and
  • senior management's information requirements for program modification, resource allocation and results accountability

The ability to evaluate the NVC program in 2009-10 as projected in the NVC Results-based Management Framework will be constrained in outcomes measurement by the lack of clients who will have completed the longitudinal ongoing performance measurement survey by that time. This survey, which measures client outcomes at the time of application, program benefit completion (estimated at two years), and two years subsequent to completion of benefit delivery, commenced only on April 1, 2007.

Given the slower than expected take-up of the program and timing of commencement of ongoing outcome performance measurement, it would be desirable to delay evaluation of NVC outcomes until at least 2011-12. If this delay is not possible, some indication of outcome performance may be possible using the limited data available in 2009-10 from the ongoing performance measurement plus a one-time client survey. An evaluation in 2009-10 could include issues relating to program rationale and alternatives.

There are no specific recommendations in this report.

1.0 Introduction

New Veterans Charter (NVC) programs offer releasing Canadian Forces (CF) members, Veterans and their families a modernized, comprehensive and integrated range of compensation and wellness programs to support their rehabilitation and re-establishment from military to civilian life. It represents a new social contract between Canadians, CF members and CF veterans.

The purpose of this evaluation framework is to outline how the NVC programs could be evaluated in the future. It provides a profile establishing the origin, legislative authority, benefits, objectives and results, and delivery structure of the program. Secondly, it provides an evaluation framework which establishes potential evaluation issues, information and methodologies required, and inputs to the evaluation planning process.

Preparation of the evaluation framework has taken into account the information needs of government in the management of its resources and programs. It complements and utilizes the information produced by the ongoing performance measurement established for the NVC programs. Consultation on the development of this evaluation framework included the Veterans Affairs Canada NVC Program Manager and the Quality Management Team.

2.0 Program Profile

2.1 Background

In addition to the traditional war Veterans who served in both World wars and Korea, Veterans Affairs Canada (VAC) is responsible for providing benefits and services to serving and former Canadian Forces (CF) members and their families. It had become apparent that the VAC programs designed in the 1980s and 1990s to meet the needs of its more elderly clientele were not able to meet the needs of the much younger Canadian Forces clients.

The size of the population to be served by the NVC programs may be characterized as follows:

  • 671,000Footnote 1 + Estimated CF peace time population (CF Veterans and serving members)
  • 4,000 + CF members releasing from the Forces every year
  • 63,000 Forecast number of CF clients getting support from VAC by 2013

VAC's CF clients have higher health care utilization and more long-term health problems than does a matched comparison group from the general population. CF clients have significant health issues relating to pain, arthritis and psychological distress.

The average age at release of CF members of 36 years is noteworthy in that it implies at least two more decades of employable years. A significant number of VAC's CF clients face challenges in obtaining and maintaining employment due to physical and/or psychological health problems and a lack of education, resulting in inadequate levels of income.

In response to the identification of these needs, VAC established a Service and Program Modernization Task Force in 2003 which culminated in the development of a suite of modernized programs which would come to be called the NVC programs. On April 1, 2006, VAC began the process of implementing the programs.

During the program development and design phases, working relationships and partnerships were developed with various federal departments including DND, Social Development Canada, and Human Resources and Skills Development Canada (HRSDC).

2.2 Legislative Authority

In May 2005, the NVC received Royal Assent. With a focus on wellness, independence and dignity, the Canadian Forces Members and Veterans Re-Establishment and Compensation Act (CFMVRCA) offers releasing Canadian Forces members, Veterans and their families a modernized, comprehensive and integrated range of compensation and wellness programs to support their rehabilitation and re-establishment from military to civilian life.

2.3 Elements

The NVC programs and services are a blend of case management services, financial benefits and financial assistance for the purchase of services which augment existing federal and provincial programs in meeting the needs of releasing CF members and veterans.

2.3.1 Case Management

Case management is a process which ensures that clients with complex needs access timely and appropriate resources and services to optimize their level of independence and quality of life. All releasing CF members are offered a VAC transition interview to assess the member's readiness for transition and to identify potential barriers to civilian re-establishment. This is followed by: assessments to establish the nature of barriers; case planning to identify re-establishment goals; resource coordination; ongoing monitoring; and evaluation to ensure that goals have been achieved.

2.3.2 Disability, Death and Detention Benefits

A Disability Award recognizes and compensates CF members and Veterans, and in some cases surviving spouses/common-law partners and surviving dependent children, for the non-economic effects of a service-related disability, including pain and suffering, functional loss and the effects of permanent impairment. The Award is a monetary, tax-free, lump sum payment.

The Death Benefit recognizes and compensates a surviving spouse/common-law partner and/or surviving dependent children of a member for the non-economic impacts of a sudden service-related death, which includes the member's loss of life, the resulting loss of guidance, care and companionship; and the impact of the member's death on the functioning of the household.

A Clothing Allowance is provided to CF members or Veterans to recognize the costs associated with wear and tear of clothing, and specially made apparel related to disabilities compensated by a Disability Award.

The Detention Benefit recognizes and compensates for the non-economic effects arising from the period of time the CF member or Veteran was detained by an enemy, opposing force or criminal element, was engaged in evading capture, or escaped from such power.

2.3.3 Rehabilitation Services and Vocational Assistance

The Rehabilitation Program follows VAC's client-centred case management model focusing on the reasonable restoration of client functioning in five major areas - mental and physical functioning of the client, family relationships, financial security, employment and personal productivity, and community participation. Needs are determined, goals agreed, and a plan established for the provision of services and benefits that will be needed.

2.3.4 Financial Benefits

The Earnings Loss Benefit ensures income replacement of up to 75% of the Veterans' gross pre-release military salary while the Veteran is participating in an active Rehabilitation or Vocational Assistance Plan. It is taxable and indexed annually.

The Supplementary Retirement Benefit is a lump sum taxable benefit that is payable to a Veteran who received an Earnings Loss Benefit due to total and permanent incapacity and whose Earnings Loss Benefit has ceased. This benefit recognizes the lost opportunity to contribute to a retirement pension due to a severe career-ending or service-related disability.

The Canadian Forces Income Support (CFIS) Benefit is available to CF Veterans who have successfully completed a rehabilitation program, are capable of working, but are not yet employed.

The Permanent Impairment Allowance (PIA) is a taxable monthly benefit payable for life to eligible CF Veterans who have one or more physical or mental health problems that are creating a permanent and severe impairment for which the Veteran has received a Disability Award.

2.3.5 Health Benefits

The Health Benefits Program fills gaps in post-release health coverage for eligible medically released CF Veterans and CF Veterans with a rehabilitation need who are currently ineligible for health coverage after release. Certain survivors are also included. Access is provided to group family health insurance through the Public Service Health Care Plan (PSHCP) or a comparable VAC program.

2.3.6 Job Placement

In cooperation with current CF programs and services, the Job Placement Program offers post-release job-finding support to eligible CF Veterans. Those CF Veterans releasing for medical reasons or those with a rehabilitation need may receive job placement assistance through the Vocational Rehabilitation Program.

The aim is to fill gaps that may occur within the current DND, HSRDC and VAC programs and services, and to help Veterans become established in a civilian career. The Job Placement Program provides job-search training, career counseling and job-finding assistance. The program may also be provided to survivors.

2.4 Objectives, Guiding Principles and Expected Outcomes

The aim of the NVC programs have been expressed in terms of overall program objectives, founding guiding principles, and the expected outcomes of the program.

2.4.1 Overall Program Objectives

The overall NVC program objectives are:

  • To assist CF members, Veterans and their families, making the transition from military to civilian life, to successfully re-establish into civilian society, and to achieve and/or maintain an optimum level of health, independence and quality of life in their communities;
  • To facilitate the effective and efficient use of re-establishment related supports and resources through a structured, culturally sensitive case management process to determine service needs and develop an effective and responsive case plan; and
  • To ensure that all CF members, Veterans and their families who have an assessed need for assistance in making the transition from military to civilian life have access to transition supports.

2.4.2 Guiding Principles

The Service and Program Modernization Task Force: Principles and Policy Directives (2004) provides further delineation of the guiding principles for the program as follows:

  • A successful transition from military to civilian life is essential for CF Veterans and their families to participate fully as citizens in all segments of Canadian society;
  • Service in Canada's military represents a special and unique contribution to Canada, both overseas and at home. Thus, the Government's long-standing commitment to contribute to the re-establishment of CF clients and their families in civilian society will be reflected in programs for Veterans and families;
  • Support should be based on client assessed need and the assessed needs of family members;
  • Services and programs should encourage and provide incentives for independence and wellness while maximizing potential for participation in community life. For optimal effectiveness, support should begin while the member is still serving;
  • VAC should strive to provide seamless, comprehensive access to and coordination of the required services and programs to CF Veterans in their own communities, while utilizing available community resources to the extent effective. VAC will avoid duplication with other government programs by filling gaps and building on existing resources where available and effective;
  • Post-release services and programs should enhance CF recruitment and retention.

2.4.3 Expected Outcomes

The expected outcomes from the NVC programs are that CF Veterans:

  • experience improved health and functional capacity as a result of access to health benefits and rehabilitative service;
  • actively participate in the civilian workforce (unless totally disabled or retired) as a result of access to employment-related supports in the form of Vocational Assistance training and Job Placement assistance;
  • have a level of income adequate to meet basic needs as a result of enhanced employment opportunities provided by Job Placement assistance, and access to employment enhancement supports such as re-training opportunities as part of vocational rehabilitation;
  • actively participate in and are integrated into their communities; and
  • feel recognized for their contribution to the security of the country.

2.5 Program Logic Model

A program logic model describes the essence of a program by displaying its key elements and the causal relationships among its elements. Appendix A shows how the activities of the NVC programs produce outputs (products and services) which lead to the achievement of expected outcomes.

2.6 Delivery Structure

The first contact with VAC staff for a medically-releasing CF member is an Area Counsellor available at CF Bases and Wings across Canada. They offer a transition interview and work with the DND Clinical Case Manager for the six month period prior to release.

In the VAC District Offices, case management services are provided by a interdisciplinary client service team composed of Area Counsellors, Client Service Agents, Pension Officer, District Nursing Officer, and District Medical Officer, under the direction of a Client Service Team Manager. Through a transition interview and assessment, needs are assessed, information and support are provided, and for more complex needs, case management services are provided.

Decisions for rehabilitation services and vocational assistance are made at the District Office. Decisions for Job Placement eligibility, Disability Pensions, Disability Awards, Death Benefits, Earnings Loss and Health Benefits are made at VAC Headquarters.

VAC Regional Offices provide a National Contact Centre and functional and policy direction to the District Client Service Teams.

VAC Headquarters provides NVC program policy direction, quality management and someNVC benefit decisions within the existing VAC program delivery structure.

Private contractors are utilized to deliver Rehabilitation Services and Job Placement Services. The Health Insurance program is delivered by a private contractor managed by Treasury Board.

2.7 Resources

The Integrated Results-based Management Framework and Risk-based Audit Framework (RMAF/RBAF) for the NVC Memorandum to Cabinet provided estimates of program clients and expenditures (Figure 1) and projected administrative costs (Figure 2.)

Figure 1: Projected Clients, Average Costs and Expenditures (Millions) by Program

(Source: Results-Based Management Framework, NVC Programs)

Figure 1: Projected Clients, Average Costs and Expenditures (Millions) by Program
Program 2007-
2006
2007-
2008
2008-
2009
2009-
2010
2010-
2011
Total
Disability Awards, Death:
- Clients 6,034 5,454 5,004 4,554 4,105 -
- Average Cost $35,593 $36,726 $38,045 $39,471 $41,033 -
- Expenditures $214.8 $200.3 $190.4 $179.8 $168.4 $953.6
Rehabilitation Services and Vocational Assistance:
- Clients 767 1,536 1,539 1,067 596 -
- Average Cost $3,836 $3,951 $4,070 $4,192 $4,318 -
- Expenditures $14.9 $30.3 $30.9 $22.0 $12.9 $111.0
Economic Loss
Temporary Earnings Loss:
- Clients 767 1,536 1,539 1,067 596 -
- Average Cost $19,416 $19,739 $20,059 $20,650 $21,690 -
- Expenditures $14.9 $30.3 $30.9 $22.0 $12.9 $111.0
Economic Loss
Extended Earnings Loss / Supplementary Retirement Benefit:
- Clients 0 0 132 264 396 -
- Average Cost $0 $0 $21,853 $22,268 $22,703 -
- Expenditures $0.0 $0.0 $2.9 $5.9 $9.0 $17.8
Economic Loss
Canadian Forces Income Support:
- Clients 0 0 59 86 113 -
- Average Cost $0 $0 $23,631 $24,340 $25,070 -
- Expenditures $0.0 $0.0 $1.4 $2.1 $2.8 $6.3
Economic Loss
Permanent Impairment Allowance:
- Clients 2 6 11 17 25 -
- Average Cost $8,979 $9,249 $9,526 $9,812 $10,106 -
- Expenditures $0.0 $0.1 $0.1 $0.2 $0.3 $0.6
Health Benefits:
- Clients 904 1,453 2,001 2,335 2,669 -
- Average Cost $1,333 $1,505 $1,448 $1,369 $1,318 -
- Expenditures $1.2 $2.2 $2.9 $3.2 $3.5 $13.0
Job Placement:
- Clients 2,080 2,080 2,080 2,080 2,080 -
- Average Cost $2,125 $2,189 $2,255 $2,322 $2,392 -
- Expenditures $4.4 $4.6 $4.7 $4.8 $5.0 $23.5
Total Expenditures: $238.2 $243.4 $239.4 $222.3 $204.3 1,147.6

Figure 2: Represents the resource profile associated with salaries, O&M, transfers to other partners and capital costs.

(Source: Results-Based Management Framework, NVC Programs)

Figure 2: Projected Administrative Costs (Millions) 2005-06 to 2010-11
Exercice 2005-2006 2006-2007 2007-2008 2008-2009 2009-2010 2010-2011
Salaries 6.145 6.100 3.700 2.700 2.700 2.700
Employee Benefits 1.229 1.22 0.74 0.54 0.54 0.54
PWGSC 0.798 0.793 0.481 0.351 0.351 0.351
Other Operating 8.827 5.887 5.079 2.409 2.409 2.409
Total 17.000 14.000 10.000 6.000 6.000 6.000

Actual NVC clients and expenditures for 2006-07, as shown in Figure 3., reflect the slower than projected take-up and implementation of the program. Most of the variance between the actual program activity in 2006-07 and forecast estimates can be attributed to timing. The forecast estimates assumed all new program applications received in 2006-07 would be paid in 2006-07. The Memorandum to Cabinet estimates did not include a provision for the gradual intake of applications over the year or the time to process and adjudicate new applications. Program specific variances from forecast include:

  • The over forecast of clients and expenditures for Disability Awards was due to fewer than expected new award clients. Prior to implementation of the NVC, there was an unexpected increase in applications for disability pensions from CF clients preferring a lifetime pension rather than a lump sum award;
  • Earnings Loss expenditures were lower than expected due to a higher percentage of clients being in receipt of DND's SISIP LTD and having other income such as a disability pension or superannuation in excess of the VAC payment; and
  • Job Placement intake was much lower than forecast as DND releases have been attending CF Career Transition Workshops until VAC completed the process for a national job placement contract in the Fall of 2007.

Figure 3: Actual Clients and Expenditures 2006-07Footnote 2

Figure 3: Actual Clients and Expenditures 2006-07
Program Clients Expenditures Average Cost
Disability Awards & Allowances
Disability Awards 999 $33,385,131 $34,510
Disability Award and Pension 397 $10,623,683 $27,634
DA Death Benefit 15 $3,771,485 $251,432
DA and DP - Death over 30 days 2 $502,865 $251,432
DA and DP Death Benefit 7 $1,760,026 $251,432
Total Disability Awards 1420 $50,043,190 $35,242
Financial Counseling 4 $1,315 $329
Rehabilitation
Health Care - Med/Psych/Social 271 $309,355 $1,142
Vocational Rehabilitation 135 $289,520 $2,145
Total 1139 $598,875 n/a
Loss of Earnings
Temporary Earnings loss 857 $3,485,619 $13,980
Extended Earnings Loss 62 $311,221 $32,308
Total 919 3 485 619 $ 13 980 $
Health Insurance
Total 235 n/a n/a
Job Placement
Counseling services n/a $553 n/a
Job Finding Services n/a $0 n/a
Total 15 $553  
Total NVC Programs 2584 $54,129,552 $20,948

2.8 Performance Measurement Strategy

An integrated approach to performance measurement of the NVC programs has been established within the larger context of quality management and organizational performance measurement. It includes:

  • intermediate outcomes related to health, employment, economic security, community integration and perceived recognition are measured by a VAC Re-establishment Survey which is administered to clients as they enter the program, leave the program, and two years later. Incorporated in this survey are questions taken from a short survey, SF-12, copyrighted by Quality Metric and licensed to VAC, which targets self-reported health status and resulting activity limitations.
  • outcome measurement is inclusive of all clients who complete either the Rehabilitation Program or Job Placement Program;
  • the Quality Management Program, in partnership with other VAC organizations, is responsible for developing, monitoring and reporting on organizational performance indicators including outcome measures related to outputs, program outcomes and client outcomes. These include Quality Assurance Reviews of a sample of client files and a Client Satisfaction Survey measuring service quality.
  • performance measurement information is stored electronically in the Client Service Delivery Network (CSDN) and the Reporting Data Base (RDB)

This evaluation framework utilizes the data generated by the NVC programs performance measurement strategy and provides the opportunity to collect collaborative information through one-time data collection as required to assess program outcomes.

3.0 Evaluation Framework

3.1 Introduction

This section of the report outlines how the NVC programs could be evaluated to provide information to assist parliamentarians in the oversight of government spending and management, and to assist the Deputy Minister in management of the program for effectiveness / efficiency and demonstrating accountability for program performance.

It identifies the evaluation issues which would cover the information requirements of parliamentarians and departmental management, provides the basic methodology required to address these issues, and concludes with inputs to the evaluation planning process.

3.2 Evaluation Issues

Evaluation issues in this evaluation framework have drawn upon the issues identified in the NVC Results-Based Management Framework and the questions posed by government in the management of its resources and programs. These questions include:

  • Public interest - Does the program continue to serve the public interest?
  • Role of government - Is there a legitimate and necessary role for government? Is the current role appropriate? Is the program a candidate for realignment?
  • Partnership - What could be transferred in whole or part to the private/voluntary sector?
  • Value for money - Are Canadians getting value for their tax dollars?
  • Efficiency - If the program or activity continues, how could its efficiency be improved?
  • Affordability - Is the resultant package of programs and activities affordable? If not, what programs or activities would be abandoned?

Evaluation issues are grouped following the basic program evaluation issues of program relevance and rationale, objectives achievement and impact, and alternatives. Each issue shown below has several criteria / indicators which help to describe the focus of the issue. Based upon the information needs at the time of evaluation, the evaluation may only consider some of these issues or criteria/indicators.

The evaluation issues for the NVC programs are as follows:

Relevance and Rationale

Issue 1: The extent to which the NVC programs address the needs of clients and are consistent with VAC and government priorities.

  1. Program take-up
  2. Program attrition
  3. Unmet client needs
  4. Alignment with VAC priorities
  5. Alignment with Government priorities
    • Public interest
    • Role of government
    • Partnership
    • Value for money
    • Efficiency
    • Affordability

Issue 2: The extent to which the activities of the program are consistent with its mandate and linked to program objectives.

  1. Activities within scope of mandate
  2. Logical linkage of activities to objectives

Objectives Achievement and Impact

Issue 3: The extent to which the NVC programs are meeting their objectives within their budget.

  1. Overall success in transition from military to civilian life
  2. Expected outcomes are achieved for at-risk veterans/families
    • Needs are identified
    • Access to required services
    • Improved health and functional capacity
    • Participation in civilian workforce
    • Income adequate to meet basic needs
    • Participate/integrate into their communities
    • Feel recognized for their contribution
  3. Expenditures are within projected budgets

Issue 4: What unintended impacts have occurred?

  1. Exclusion due to eligibility criteria
  2. Creation of dependency
  3. Unrealistic expectations
  4. Other positive or negative unintended impacts

Alternatives

Issue 5: Are there improvements for cost-effectiveness within the current delivery model?

  1. Integration and coordination of departmental and government-wide benefits
  2. Collaboration with DND, HRSDC and other partners
  3. Seamless and transparent service to clients
  4. Quality of service improvements
  5. Cooperation between delivery and policy elements
  6. Use of performance measurement information
  7. Value added by each step of delivery process
  8. Environmental scanning of future client needs

Issue 6: Are there more cost-effective delivery models?

  1. Federal vs Provincial vs private sector
  2. Other Federal government instruments
  3. Foreign government delivery models

3.3 Methodology

The data collection methodologies contemplated for the evaluation of the NVC programs are related to the evaluation issues /criteria /indicators and information required as shown in Annex B.

To the extent possible, data from the ongoing program management systems and performance measurement systems would be utilized. Additional one-time data collection will be required where necessary to support the data from the ongoing systems and to address the issues not supported by ongoing systems.

The actual data collection used at the time of the evaluation will be dependent upon the issue focus of the evaluation. If all issues were to be addressed, the data collection methodologies would include:

  • file review of program and client data on the Reporting Data Base;
  • file review of Re-establishment Survey outcome measures;
  • personal interviews and/or focus groups of clients;
  • survey of clients (if Re-establishment Survey not timely)
  • personal interviews and/or focus groups of VAC and partner staff;
  • personal interviews of veterans organizations;
  • personal interviews of private sector delivery organizations;
  • document review of government policy, NVC plans and budgets;
  • mapping and analysis of delivery process;
  • case file review of client files; and
  • review of Statistic Canada data.

Data sources arising from program delivery instruments and computerized databases are described in Annex C and include:

  • Transition Interview Record;
  • Screening Record;
  • Application Forms;
  • Area Counsellor Assessment;
  • Third Party Assessments;
  • Re-establishment Survey;
  • Contractor reporting;
  • VAC Client Satisfaction Survey;
  • CSDN Reporting Data Base (RDB); and
  • Freebalance financial system

3.4 Evaluation Planning

The nature and timing of future evaluations will in part depend upon:

  • Requirements of Parliamentarians for evaluation input;
  • Expectations created with central agencies upon program approval;
  • Maturity of the program; and
  • Senior management's information requirements for program modification, resource allocation and results accountability.

The Results-based Management Framework submitted for program approval identified full evaluations of the NVC programs (Disability Award, Earnings Loss, Canadian Forces Income Support, Job Placement, Rehabilitation, and Health Benefits) in fiscal year 2009-10.

The ability to evaluate the NVC program in 2009-10 will be constrained in outcomes measurement by the lack of clients who will have completed the longitudinal ongoing performance measurement survey by that time. This survey, which measures client outcomes at the time of application, program benefit completion (estimated at two years), and two years subsequent to completion of benefit delivery, commenced only on April 1, 2007.

Given the slower than expected take-up of the program and timing of commencement of ongoing outcome performance measurement, it would be desirable to delay evaluation of NVC outcomes until at least 2011-12. If this delay is not possible, some indication of outcome performance may be possible using the limited data available in 2009-10 ( an estimated 600 clients will have completed their program but not the desired subsequent two years of benefit from their new circumstances). This limited data could be augmented by a directed one-time client survey which would have the added benefit of capturing NVC clients who were not involved in Rehabilitation, Earnings Loss and Job Placement programs.

In the normal course of evaluation of programs, the first evaluation, or formative evaluation, could take place within two years of program commencement and consider issues which would help in modifying program design and delivery for better program effectiveness and efficiency. In the case of the NVC programs, these issues / questions may include:

  • Program take-up;
  • Unmet client needs;
  • Review of early ongoing performance information;
  • Eligibility criteria; and
  • Improvements within current delivery model.

Some of the evaluation issues / questions could be addressed at any time once the program is well established include:

  • Alignment with VAC priorities;
  • Alignment with Government priorities;
  • Activities within scope of mandate;
  • Logical linkage of activities to objectives;
  • Objectives and outcomes achievement.

Evaluation issues targeted specifically at the needs of government for the management of its resources and programs would typically come several years after program commencement as government policy and the program's environment might have changed. These include:

  • Public interest;
  • Role of government;
  • Partnership;
  • Value for money;
  • Efficiency; and
  • Affordability.

A final input in the choice of issues / questions of future evaluations is the consideration that certain data collection methodologies for required issues mayfacilitate the study of additional issues at very little extra cost. For example, focus groups of clients targeted at outcomes measurement may also easily consider un-met client needs.

In conclusion, the actual choice of issue / questions for any of the evaluation work done on the NVC programs will be dependent upon availability of data and the needs of senior management and external stakeholders as determined in the departmental evaluation planning process.

4.0 Distribution

  • Deputy Minister
  • Associate Deputy Minister
  • Chief of Staff to the Minister
  • Chair, Veterans Review and Appeal Board
  • Assistant Deputy Minister, Veterans Services Branch
  • Assistant Deputy Minister, Corporate Services Branch
  • Assistant Deputy Minister, Public Programs and Communications Branch
  • A/Director General, Coordination Secretariat
  • Executive Director and Chief Pensions Advocate, BPA
  • Regional Director General, Western Region
  • Deputy Coordinator, Access to Information & Privacy
  • Comptrollership Branch (TBS)
  • Office of the Auditor General
  • Program Analyst, Treasury Board of Canada, Secretariat

5.0 Appendix A - Program Logic Model

For further information regarding the Program Logic Model of the Evaluation Framework for the New Veterans Charter. please contact Veterans Affairs Canada.

6.0 Appendix B - Evaluation Methodology

Issue 1: The extent to which the NVC programs address the needs of clients and are consistent with VAC and government priorities.
Criteria / Indicator Information Required Collection Methodology Data Source
a. Program take-up
  • # CF leaving service
  • # CF program applications
  • # CF receiving benefits
  • Type /number & value of benefits
  • File review
  • DND statistics
  • VAC Reporting Data Base (RDB)
  • VAC RDB
  • VAC RDB
  • Freebalance
b. Program attrition
  • # CF clients entering program
  • # CF clients completing program
  • Reasons for client withdrawal
  • File review
  • VAC RDB
  • Job Placement Contractor Reporting
  • Rehabilitation Contractor Reporting
c. Unmet client needs
  • Nature of needs not covered
  • Magnitude of occurrence
  • Document review
  • Interviews
  • VAC Client Satisfaction Survey
  • DND / VAC counsellors
  • NVC clients
d. Alignment with VAC priorities
  • NVC purpose
  • VAC mandate and priorities
  • Document review
  • NVC legislation
  • VAC legislation
  • VAC planning documents
e. Alignment with Government priorities
  • NVC purpose
  • Government priorities
  • Document review
  • NVC legislation
  • Government policy
- Public interest
  • productive entrants to workforce
  • well being of disabled citizens
  • stable family environment
  • enhanced recruitment to CF
  • Document review
  • Interviews
  • Re-establishment survey results
  • DND staff
- Role of Government
  • Veterans a Federal responsibility
  • Needs created in service to country
  • Lack of coordinated service alternatives
  • Document review
  • Federal policy
  • Alternative service directories
- Partnership
  • Private sector capacity
  • Voluntary sector capacity
  • Extent current private sector participation
  • New partnership opportunities
  • Document review
  • Interviews
  • Alternative service directories
  • Industry / voluntary sector representatives
- Value for money
  • Program results vs costs
  • Cost per program benefit
  • Comparison to provincial or foreign government programs
  • Comparison to equivalent private sector programs
  • Document review
  • Interviews
  • Re-establishment survey results
  • VAC RDB and Freebalance
  • Provincial or foreign government staff
  • Private sector representatives
  • Veterans Associations
- Efficiency
  • See Issues 5 and 6.
- Affordability
  • Priority within VAC and government
  • Within VAC and government budget envelopes
  • Document review
  • Government and VAC documents setting priorities
  • VAC Freebalance
  • Government and VAC budget documents
Issue 2: The extent to which the activities of the program are consistent with its mandate and linked to program objectives.
Criteria / Indicator Information Required Collection Methodology Data Source
a. Activities within scope of mandate
  • Understanding of program activities as implemented
  • Understanding of program mandate
  • Degree of coincidence
  • Activities outside of mandate
  • Personal interviews and/or focus groups
  • Document review
  • VAC staff
  • NVC program documentation
b. Logical linkage of activities to objectives
  • Linkages activities and objectives
  • Document review
  • NVC program documentation
Issue 3: The extent to which the NVC programs are meeting their objectives within their budget.
Criteria / Indicator Information Required Collection Methodology Data Source
a. Overall success in transition from military to civilian life
  • Summary of expected outcomes achievement data
  • Document review
  • Rapports sur la mesure du rendement dans la BDR
a. Overall success in transition from military to civilian life
  • Customer feedback
  • Client file review
  • Client personal interview and/or focus groups
  • Client survey (If required for clients outside, or to augment, the Re-establishment Survey)
  • Client files
  • Re-establishment Survey results on RDB
  • NVC clients
a. Overall success in transition from military to civilian life
  • Delivery staff and client representative feedback
  • Personal interviews and/or focus groups
  • VAC / DND staff
  • Veterans organizations
b. Expected outcomes are achieved for at-risk veterans/families
-Needs are identified
  • Transition issues identified
  • Both client and family issues
  • Culturally sensitive process
  • Issues added over time as trust established
  • Client feedback
  • Case file review
  • Personal interview and/or focus groups
  • Transition and Screening interview notes in RDB
  • Area Counsellor Assessments and Case Plans in RDB
  • NVC clients
  • VAC delivery staff
- Access to required services
  • Extent services accessible in timely manner
  • Availability in own community
  • Utilize existing government services
  • Why needed services not utilized
  • Personal interview and/or focus groups
  • District Office staff
- Improved health and functional capacity
  • Self-rated overall health
  • Extent of limitations on everyday activities
  • Extent of physical health impact on work or daily activities
  • Extent of emotional health impact on work or daily activities
  • Extent of pain interference with normal work
  • Extent of time feeling calm, energized, depressed
  • Extent of time health or emotional problems interfered socially
  • File review (Longitudinal on specific client)
  • Document review
  • Individual responses to SF-12 portion of Re-establishment Survey results on RDB
  • Aggregate responses on Re-establishment Survey reports on RDB
- Participation in civilian workforce
  • Client current employment
  • Client preferred employment status
  • Client match current position to education, training, skills and experience
  • Client preparedness to find suitable job
  • Client preparedness to keep suitable job
  • Client knowledge of job-finding skills
  • Program percent registered who achieve employment
  • Program percent eligible who registered
  • Program percent needs met by alternative training
  • Program customer satisfaction rates
  • Labour market conditions
  • File review (Longitudinal on specific client)
  • Document review
  • Document review
  • Individual responses to Re-establishment Survey results on RDB
  • Aggregate responses on Re-establishment Survey reports on RDB
    (Note: Some job placement activities occur at DND prior to application to VAC.)
  • Job Placement Contractor reporting
  • Statistics Canada and other government sources
-Income adequate to meet basic needs
  • Ability to provide for basic needs of food, shelter, clothing, healthcare, transportation
  • Self-described financial situation
  • Expected financial situation in six months
  • File review (longitudinal on specific client)
  • Document review
  • Individual responses to Re-establishment Survey on RDB
  • Aggregate responses on Re-establishment Survey reports on RDB
-Income adequate to meet basic needs
  • Annual income
  • Accepted cohort income
  • Personal interview
  • File review
  • Document review
  • NVC clients
  • Application Form for Earnings Loss and CFIS (Income Support clients only)
  • Statistics Canada data
- Participate / integrate into their communities
  • Feel recognized for their contribution - Overall extent feel recognized
  • Recognized by whom and extent - File review (longitudinal on specific client)
  • Document review - Individual responses to Re-establishment Survey on RDB
  • Aggregate responses on Re-establishment Survey reports on RDB
  • File review (longitudinal on specific client)
  • Document review
  • Individual responses to Re-establishment Survey on RDB
  • Aggregate responses on Re-establishment Survey reports on RDB
- Feel recognized for their contribution
  • Overall extent feel recognized
  • Recognized by whom and extent
  • File review (longitudinal on specific client)
  • Document review
  • Individual responses to Re-establishment Survey on RDB
  • Aggregate responses on Re-establishment Survey reports on RDB
c. Expenditures are within projected budgets
  • NVC expenditures by program
  • NVC budgets
  • Cost per client actual vs projected by program
  • Reasons for variances
  • File review
  • Document review
  • Personal interviews and/or focus groups
  • VAC Freebalance
  • Annual budgets
  • VAC staff
Issue 4: What unintended impacts have occurred?
Criteria / Indicator Information Required Collection Methodology Data Source
a. Exclusion due to eligibility criteria
  • Identification of problem
  • Extent of exclusion
  • Impact of exclusion
  • Program design options
  • Personal interviews and / or focus groups
  • Document review
  • NVC clients
  • VAC staff
  • Veterans organizations
  • Reasons for negative decisions for applications on CSDN
b. Creation of dependency
  • Nature of dependency
  • Extent of problem
  • Impact of dependency
  • Program design options
  • Personal interviews and / or focus groups
  • NVC clients
  • VAC staff
  • Veterans organizations
c. Unrealistic expectations
  • Identification of problem
  • Extent of problem
  • Consequences for client
  • Program design options for consideration
  • Personal interviews and / or focus groups
  • NVC clients
  • VAC staff
  • Veterans organizations
d. Other positive or negative unintended impacts
  • Identification of impact
  • Extent of impact
  • Consequences for client
  • Program design options for consideration
  • Personal interviews and / or focus groups
  • NVC clients
  • VAC staff
  • Veterans organizations
Issue 5: Are there improvements for cost-effectiveness within the current delivery model?
Criteria / Indicator Information Required Collection Methodology Data Source
a. Integration and coordination of departmental and government-wide benefits
  • Identification of government and VAC programs with similar objectives, clients and delivery processes
  • Extent of current integration and coordination by NVC programs
  • Rationale for separate NVC program and delivery system
  • Opportunities for improved cost-effectiveness
  • Document review
  • Personal interviews and/or focus groups
  • Government program documents
  • Staff in VAC and other Federal departments
b. Collaboration with DND, HRSDC and other partners
  • Nature of current collaboration
  • Problems and opportunities for improvement
  • Personal interviews and/or focus groups
  • VAC staff
  • Partner staff
c. Seamless and transparent service to clients
  • Primary point of contact for client
  • Information requested only once from clients
  • Clients understand benefits and delivery process
  • Communication with client is effective at all levels
  • Effective coordination among delivery staff
  • Problems and opportunities for improvement
  • Personal interviews and / or focus groups
  • Document review
  • VAC staff
  • NVC clients
  • VAC procedures and delivery instruments
  • VAC Client Satisfaction Survey
d. Quality of service improvements
  • Access
  • Timeliness
  • Responsiveness
  • Clear communications
  • Courtesy, respect
  • Personal interviews and / or focus groups
  • Document review
  • NVC clients
  • VAC Delivery staff
  • VAC Client Satisfaction Survey
e. Cooperation between delivery and policy elements
  • Formal and informal feedback mechanisms
  • Timely policy responses
  • Clarity of policy direction
  • Mechanism for handling special cases
  • Personal interviews and/or focus groups
  • VAC staff
f. Use of performance measurement information
  • Extent of dissemination
  • Quality of analysis of data
  • Use as input to program policy or delivery
  • Document review
  • Personal interviews and/or focus groups
  • Performance reporting documents
  • VAC staff
g. Value added and cost for each step of delivery process
  • How each step contributes to delivery of benefit
  • Administration cost of each step
  • Cost and time impact of transfers and loops in the delivery process
  • Opportunities to delegate decisions to front line staff as program experience gained
  • Process mapping
  • Personal interviews and/or focus groups
  • VAC staff
h. Environmental scanning re future client needs
  • Existence of activity
  • Use of results
  • Personal interviews and/or focus groups
  • VAC staff
Question 6: Are there more cost-effective delivery models?
Criteria / Indicator Information Required Collection Methodology Data Source
a. Government versus private sector
  • Existence of private sector alternatives
  • Cost effectiveness of alternatives
  • Consequences of devolution for client and government
  • Document review
  • Personal interviews
  • NVC and Modernization Task Force planning documents
  • VAC staff
  • Private sector representatives
b. Other government instruments
  • Use of Federal tax system
  • Devolution to provinces
  • Document review
  • Personal interviews
  • NVC and Modernization Task Force planning documents
  • VAC staff
  • Provincial representatives
c. Foreign government delivery models
  • Existence of similar programs
  • Context of foreign programs
  • Applicability to Canadian situation
  • Document review
  • Personal and telephone interviews
  • NVC and Modernization Task Force planning documents
  • VAC staff
  • Foreign government veterans representatives

7.0 Appendix C - Evaluation Data Sources

Evaluation Data Sources
Data Source Program Coverage Location Notes
Transition Interview Notes
  • All NVC
  • Reporting Data Base
  • For C.F. releasing or Reserves.
  • Offered to all. Ninety-one percent medically releasing accept. Approx. Sixty percent voluntary releasing accept.
Screening Notes
  • All NVC
  • Reporting Data Base
  • For C.F. released or Reserves.
Area Counsellor Assessment
  • Rehabilitation
  • Loss of Earnings
  • Health Insurance
  • Reporting Data Base
  • Dependent upon needs.
  • Health Insurance for Rehabilitation only.
Third Party Assessments
  • Rehabilitation
  • Loss of Earnings
  • Reporting Data Base
  • Expert or outside advice
Application Form
  • All NVC
  • Client Service Delivery Network
  • Earnings Loss would have income data
Record Of Decision
  • Rehabilitation
  • Health Insurance
  • Job Placement
  • Client Service Delivery Network
 
Re-establishment Survey
  • Rehabilitation
  • Loss of Earnings
  • Job Placement
  • Veterans Services Subject Records
  • Reporting Data Base
  • Voluntary survey
  • Includes Disability Awards only if in Job Placement or Rehabilitation
  • Excludes TB administered Health Insurance
  • Restricted access to data
Job Placement Contractor Reporting
  • Job Placement
  • Reporting Data Base
  • Program statistics
  • Employment achievement
Vocational Rehabilitation Contractor Reporting
  • Vocational Rehabilitation
  • Reporting Data Base
  • Program statistics
  • Assessments, training, plans
VAC Client Satisfaction Survey
  • Not NVC specific but would include NVC Rehabilitation, Job Placement, and Earnings Loss
  • Hard copy of analysis
  • Measure service quality and if "met needs"
  • Not linked to individuals
Reporting Data Base
  • All NVC except Health Insurance
  • Client Service Delivery Network
  • Program statistics and repository for client data
  • Refreshed nightly
FreeBalance
  • Loss of Earnings
  • Job Placement
  • Finance Program
  • Program expenditures
  • Excluded is Health Insurance

8.0 List of Figures

Date modified: