Language selection

Halifax Rehab Case Management Pilot Evaluation Framework - December 2008

ACKNOWLEDGEMENT

The Audit and Evaluation division would like to acknowledge the efforts of those who helped with this evaluation framework, with a particular thanks to the staff of the Halifax District Office, Program and Service Policy Division and Finance Division.

This Evaluation Framework was prepared by:
Kevin Edgecombe, A/Audit and Evaluation Director
Karen Walsh, Audit and Evaluation Manager

Executive Summary

This report provides an evaluation framework for the Halifax District Office Rehab Case Management Pilot. The evaluation framework provides guidance on evaluation issues, methodologies, data requirements and data sources. An evaluation would determine the impacts of the pilot, the success of the pilot, and areas for improvements. The results could be used by management in the district office to determine if the pilot should continue and to make any desired improvements.

Veterans Affairs Canada has conducted reviews of case management and the Rehabilitation Program. As a result, it was determined that case plans were not conforming to the principles of case management and several recommendations were made, which included specialization of case loads based on the type of client.

Management in the Halifax District Office (DO) anticipated that the delivery of the rehabilitation program could be improved by making changes to the workload distribution. On April 7, 2008, they initiated a pilot. The objectives1 of the pilot are to improve the quality of client service and to ensure efficiencies are gained.

There are specific risk factors associated with the pilot approach, including changes in operational methods and workload demands. The Halifax DO has responded somewhat to these risks by developing a business process for the new method of delivery. Management needs to identify and ensure proper consideration of all risks.

As part of performance measurement, the evaluation framework provides a list of key indicators based on the following success factors: engagement of experts, timely case management, and a consistent approach to rehabilitation. In order to provide program management with an indication of progress toward the pilot objectives, measurable targets must be set for the key indicators.

It is recommended that an assessment of the rehab case management pilot be undertaken in 2009. The assessment should include best practices and lessons learned and be done in the broader context of the work underway with the case management review.

  • R1 It is recommended that the District Director, Halifax District Office, together with the Atlantic Regional Director General, identify and assess the risks associated with the pilot and develop a risk management plan to address the key risk areas.
  • R2 It is recommended that the District Director, Halifax District Office, together with the Regional Director Client Services, set measurable targets for key indicators and ensure ongoing monitoring of the indicators occurs.
  • R3 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration, in consultation with the Atlantic Regional Director General, ensure an assessment of the rehab case management pilot is undertaken. The assessment is to include lessons learned and be done in the broader context of the work underway with the case management review being conducted by the Senior Assistant Deputy Minister, Policy, Programs and Partnerships.

1 As verbalized by the Halifax District Director and Client Service Team Manager

2.0 INTRODUCTION

This report is subdivided into two main sections. The first section provides a profile of the Halifax Rehab Case Management Pilot Project, including the background to the pilot, the objectives, rationale and structure of the pilot, and a discussion of associated risks. The second part of the report forms the evaluation framework. The purpose of the evaluation framework is to provide guidance on evaluation issues, methodologies, data requirements and data sources.

The objectives of the evaluation framework are to document the new case management approach and the objectives of the pilot project, to establish potential evaluation issues, to establish appropriate evaluation methodology and options for ongoing data collection.

The evaluation framework was developed through consultations with key informants from the program area, along with a review of documentation. The list of people consulted and documents reviewed is attached as Appendix A.

3.0 PILOT PROJECT PROFILE

3.1 Background

On April 1, 2006, VAC began offering a new suite of programs under the Canadian Forces Members and Veterans Reestablishment and Compensation Act. The programs were developed to assist Canadian Forces (CF) Members, Veterans, and their families make the transition to civilian life. Case management is the basis upon which the programs are delivered. The purpose of case management is to provide simplified and coordinated client access to a range of federal and provincial services and benefits designed to facilitate reintegration.

At the time of release from military service, each CF member is offered a transition interview. The interview assesses the member’s readiness for transition and identifies potential barriers to civilian reestablishment. This process serves as an intake process to more intensive levels of case management. Case management involves a comprehensive assessment of the barriers, the development of a case plan to identify the client’s goals, the coordination of resources, ongoing monitoring and outcome evaluation to determine if the goals have been achieved.

The Department has outsourced certain components of the new programs; however, the area counselor (AC) is the primary case manager for the coordination of rehabilitation services. The AC takes the lead role in coordinating service providers and benefits that are essential to the rehabilitation program.

The Department has studied the resources required to deliver programs to clients and has determined that the type of client has an impact on resources. A Resource Allocation Model (RAM) has been developed which divides the client mix into rehabilitation clients, Veterans, and survivors. The model allocates more resources for rehabilitation services, as they are deemed to be more labour intensive to coordinate relative to other benefits and services.

A review of case management was undertaken by the Department in April 2008. The purpose was to review the current vision of case management, to examine the extent to which districts are adhering to the standards, to identify gaps and impediments and to identify opportunities for improvement and innovation. The review identified several concerns, including case load management, professional development and leadership. Several recommendations were presented in the report; most notably, that case loads be specialized based on the type of client, and that case loads be reduced. The Department will be addressing the recommendations in the report and an evaluation of case management will occur in 2009.

In addition to case management, VAC undertook a study of the Rehabilitation Program. The purpose of the study was to assess the capacity of VAC to successfully engage CF Veterans in rehabilitation. As a result, it was determined that case plans were not conforming to the principles of case management and several recommendations were made, which included support for specialized case loads.

Traditionally, VAC has used a workload distribution tool to assign clients to ACs based on the postal code in which the client resides. Each AC is responsible for the clients that fall within an allocated postal code and the type of client is not the primary determining factor. Having ACs and CSAs focus on a specific type of client has not been implemented in the past.

3.2 Rationale

It was anticipated by management in the Halifax DO that the delivery of the rehabilitation program could be improved by designating resources that would focus primarily on rehabilitation clients. The pilot began on April 7, 2008.

3.3 Expected Results

The objectives of the pilot project2 are to improve the quality of client service and to ensure efficiencies are gained.

It is anticipated that the new workload structure, as described in the following section, will improve case management for rehabilitation clients and will ensure that clients are served efficiently. Ideally, there will be greater use of regional experts, more timely case management, follow up and monitoring of clients and a more consistent approach to case management. In addition, it is important that service to non rehabilitation clients is not negatively affected.

2 As verbalized by the Halifax District Director and Client Service Team Manager

3.4 Delivery Structure

Effective April 7, 2008, the Halifax DO has divided its geographical boundaries into northern, western and central regional health care teams. Each team has an AC and a CSA focusing primarily on rehabilitation clients. The central region has an additional rehabilitation-focussed AC and CSA; however, they have half as many rehabilitation clients as the other AC/CSA teams. Staff who are not specializing in rehabilitation will continue to be assigned cases based on postal codes.

Area counselors who are currently working with rehabilitation clients will continue to be responsible for their existing clients. New clients will be screened by the Client Service Team Manager (CSTM) and the rehabilitation-focussed AC and are brought forward to the Interdisciplinary Team for discussion and assignment to either a non rehabilitation-focussed AC or a rehabilitation-focussed AC. The assignment of cases is dependent on the counselor’s case load, complexity of the rehabilitation case and other factors.

Although staff were asked to provide input, the determination of the new workload distribution was ultimately based on a management decision. A full complement for the rehabilitation focussed AC/CSA teams consists of 30 rehabilitation clients and 300 non rehabilitation clients. The second rehabilitation team in the central region handles 15 rehabilitation clients and 580 non rehabilitation clients. The CSA in the second rehabilitation team also helps the transition services ACs3. Other ACs and CSAs carry a case load of approximately 850 non rehabilitation clients and five rehabilitation clients.

The table below depicts the breakdown of the workload by AC/CSA Team and number of clients assigned.

Table 1: Workload Breakdown
Region number of AC/CSA Teams number of Clients
for rehabilitation focused ACs
Rehab Non rehab Rehab Non rehab
Central 1.5 5.5 45 850
Western 1 5 30 300
Northern 1 6 30 300

NOTE: numbers in the above table are approximate

As the pilot progresses, it is anticipated that there will be changes in the pilot delivery structure. Management should document any changes to ensure the evaluators can consider any impacts of these changes.

3 The transition services AC interviews clients upon release from the Canadian Forces. The interview serves as an intake into more comprehensive levels of case management. Therefore, there are the following three types of area counselors in the Halifax DO: transition services ACs, rehabilitation focussed ACs and non rehabilitation focussed ACs.

3.5 Resources

The RAM has allocated the number of ACs and CSAs required to serve clients in a particular district office. It was determined that the Halifax DO requires approximately four additional ACs and 1.5 additional CSAs4 for the 2008-2009 fiscal year. Therefore, based on the RAM, the Halifax DO was understaffed at the time the pilot began. The impact of any possible under staffing on the pilot will need to be explored in the evaluation.

The new workload allocation model could have an impact on expenditures for AC travel. For example, the rehabilitation-focussed ACs might have to travel farther distances to meet with clients because their clients will fall within a larger geographical boundary. Secondly, there may be overlap in areas visited by ACs if different ACs are assigned clients in the same geographic boundary. The costs of travel balanced against the efficiencies gained and any service improvements will be of interest when determining if the pilot merits continuation.

4 Specifically, the RAM calculated that the Halifax DO requires 4.16 additional AC FTEs and 1.61 additional CSA FTEs.

3.6 Roles and Responsibilities

The roles and responsibilities for the rehabilitation-focussed ACs and CSAs are the following:

  • consulting informally with ACs and CSAs regarding rehabilitation clients
  • consulting with the Regional Mental Health Officer and Regional Rehabilitation Officer
  • consulting with the CSTM for triage and intake of new rehabilitation cases in their region
  • meeting with all rehabilitation-focussed ACs and CSAs to discuss issues
  • being back-up for other rehabilitation-focussed ACs and CSAs.

In addition to the above, rehabilitation-focussed ACs are also responsible for the following:

  • meeting on a monthly basis with Regional Rehabilitation Officer and Regional Mental Health Officer to review rehabilitation cases
  • meeting on a weekly basis with the Health Care Team Interdisciplinary Team to review rehabilitation cases
  • chairing the health care team and other team meetings, as required.

3.7 Issues/Risks

Recent studies conducted in Veterans Services Branch have identified several issues with respect to case management and the rehabilitation program. These include the following: fragmented directional guidance, unclear boundaries for case management, confusion surrounding roles and responsibilities, unclear business processes, inappropriate approach to case management (i.e., generalist approach), and a focus on benefit delivery rather than case management. Measuring the success of the pilot could prove to be challenging and the evaluators will have to determine what aspects are attributable to the pilot structure, and which are attributable to other factors.

In addition to the issues listed above, there are risks specific to the implementation of a pilot and these are explained in greater detail below.

Changes in roles and responsibilities: When roles and responsibilities change, there is a level of risk introduced in that all staff may not fully understand their new roles and responsibilities. Roles and responsibilities for case managers, mental health specialists and rehabilitation specialists will need to be clearly defined. Recent studies indicated that, at a national level, certain key players were not being appropriately engaged.

Changes in operational methods: New operational methods increase the likelihood of error as the employees learn the new methods. New processes introduce new risks to the organization.

There are other risks that should be considered, which include the following:

  • Complexity of Rehabilitation Program: The more complex an activity, the more likely it is not to be carried out properly or not to meet its objectives. Staff that are not focussed on rehabilitation clients may not be as efficient in the coordination of the Rehabilitation Program as the rehabilitation-focussed ACs. Recent studies within the department have indicated that functional direction has been inconsistent.
  • Resourcing: The latest RAM calculated that the Halifax DO is under resourced by 5.77 FTEs. This introduces a level of risk because an under resourced workforce is not likely to meet its objectives and may fail to notice errors or omissions. The Atlantic Management Team has taken steps to mitigate this risk by providing additional funding to the Halifax DO.

The risks identified could have a considerable impact on the pilot. An assessment of the likelihood and impact of these risks should be conducted and a plan should be developed to ensure the risks are adequately managed.

4.0 EVALUATION FRAMEWORK

This section of the report identifies how the Halifax Rehab Case Management Pilot could be evaluated. It outlines the purpose of the evaluation, the evaluation issues, proposed methodologies and provides guidance for the evaluation.

4.1 Purpose of the Evaluation

An evaluation of the Halifax Rehab Case Management Pilot Project will determine the impacts of the pilot, the success of the pilot, and areas for improvements. The results of the evaluation could be used by management in the district office to determine if the pilot should continue and to make any desired improvements. As part of determining areas for improvements, the key drivers of success would be identified. These drivers would enable other managers to determine the applicability of this approach in their districts.

4.2 Evaluation Issues

The evaluation issues are developed, in part, based on the opinions and concerns expressed by managers responsible for the delivery of the pilot and programs. The key issues are the following:

  • Success: Has the pilot been successful in meeting its objectives?
  • Impacts and Effects: What impacts, both intended and unintended, has the pilot had?
  • Alternatives: Are there opportunities for improvements to the piloted approach?

As indicated, the evaluation issues follow the basic issues of success, impacts and effects and alternatives. Because of the nature of the pilot project, the relevance of the pilot was not identified as an issue; however, the impact on clients will be determined based on the success indicators. Each issue has several indicators which help define the issue and direct the evaluation question. Data requirements are identified for each indicator and the data source is identified in an evaluation matrix, which is provided in Appendix B.

The associated indicators for the identified evaluation issues are the following:

  • Success
    • Issue 1: Has the pilot been successful in meeting its objectives?
      1. improved client service
      2. efficiencies have been gained
      3. pilot implemented as intended
  • Impacts and Effects
    • Issue 2: What impacts, both intended and unintended, has the pilot had?
      1. impact on quality of client service (for non rehabilitation and rehab clients)
      2. impact on district office workforce
      3. unrealistic expectations of the pilot objectives and expected results
      4. other positive/negative unintended impacts
  • Alternatives
    • Issue 3: Are there other ways within the existing framework to improve efficiency?
      1. drivers for success
      2. areas for improvements
      3. value added and costs of pilot delivery

4.3 Evaluation Methodologies

The evaluation will use baseline performance information on the Halifax DO prior to the implementation of the pilot and compare it to performance information after implementation of the pilot. To strengthen the attribution of the findings to the piloted approach, a comparison of a similar district office’s performance is required. The similar district office, as yet unknown, will be chosen based on client volume, client mix, client counselor ratios, proximity to a CF Base, and other relevant factors.

To determine the success of the pilot, the evaluation must assess the extent to which client service improved, the extent to which efficiencies were gained, and whether the pilot was implemented as intended. The evaluation will compare Halifax DO baseline performance data to ongoing performance data, and determine if improvements have been made and/or efficiencies gained. In addition, the performance of the comparison DO will be analysed to help attribute the success to the pilot.

The Quality Management Unit has developed a tool for reviewing case files to determine compliance with the standards and criteria for case management. In particular, the tool is used to gather information regarding eligibility of clients for case planning, evidence and appropriateness of interdisciplinary consultation, development of case plans, and coordination of services for the client. This tool would be useful in the evaluation to determine the quality of client service and case management.

The impacts of the pilot, both intended and unintended, will be assessed based on information gathered through focus groups and interviews with staff. The impact on staff and client service will be determined by analyzing the workload distribution. The workload distribution formulas used in the pilot will be compared with those determined by the RAM. In addition, the evaluation team will spend some time observing the operations of the office to obtain an understanding of its culture and how well the pilot is functioning.

Focus groups with staff will help identify areas for improvements and the key drivers for success. An analysis of the strengths, weaknesses, opportunities and threats of the piloted approach will help identify constraints and areas for improvement. This will be supplemented by findings from the success and impact issues. An analysis of the costs and benefits of the piloted approach will determine the value added of the pilot and the impact on workload.

The data collection activities are divided into primary data collection and secondary data collection. Primary data is collected by the evaluation team directly at the source whereas secondary data is collected and recorded by another person or organization.

4.3.1 Primary Data Collection

Focus groups

Focus groups with employees will help determine the constraints, drivers of success, and strengths and weaknesses of the pilot.

Case Studies

The purpose of the case study is to assess the quality of service to clients, to determine if the Rehabilitation Program is being delivered in a consistent manner, to determine if the pilot was implemented as intended, and to assess the quality of client service. A group of non rehabilitation and rehabilitation clients will be selected from the CSDN and the client data will be reviewed, including case plans.

Observation

An observation of the operations of the office will determine how well the pilot is functioning.

4.3.2 Secondary Data Collection

File Review

A file review is a data collection method which gathers pre-existing information about the program being evaluated. In this particular evaluation, the data of interest is contained primarily in client files and in existing databases and will include the following:

  • a review of program, client and workload data from VAC’s Reporting Database (RDB);
  • a review of business processes for the pilot and associated programs;
  • a review of financial information contained in the FREEBALANCE financial information system;
  • a review of minutes of district office meetings;
  • a review of formal/informal complaints.

4.4 Expected Results and Data Considerations

The following table identifies key indicators based on the expected results of the pilot. The District Director of the Halifax DO, in conjunction with the Regional Director Client Services, should set targets and monitor performance on a monthly basis. Most of the data referenced is available in the existing computer systems.

Table 2: Key Indicators by Expected Result
Expected Result Indicators Target Source
Engagement of Regional Experts
  • number of contacts with regional rehabilitation specialist
  • number of contacts with regional mental health specialist
to be determined by District Director in conjunction with the Regional Director Client Services Not currently captured
Timely case management
  • Turn-around-time (TAT) for assessments
to be determined by District Director in conjunction with the Regional Director Client Services RDB
Follow-up
  • TAT for active case plan issues
to be determined by District Director in conjunction with the Regional Director Client Services RDB
  Monitoring
  • TAT for case plans being monitored
to be determined by District Director in conjunction with the Regional Director Client Services RDB
Consistent approach to rehab
  • TAT for applications to rehab program
  • number of assessments completed
  • number of case plans created
  • number of reassessments completed
  • number of contacts with RO specialists
to be determined by District Director in conjunction with the Regional Director Client Services RDB

not currently captured

The evaluation will examine trends in the quality of client service. To help facilitate this, data will be collected on an ongoing basis. A baseline will be established for comparison. To determine trends, the baseline data should include monthly data from April 2007 to March 2008. In addition to the baseline data, ongoing monthly data should be collected from April 2008 onwards. Data should be gathered for the Halifax DO and the comparison DO.

The following table suggests the type and frequency of baseline and ongoing data for consideration. This data is consistent with established service standards for the Client Centred Service Approach.

Table 3: Suggested Baseline and Ongoing Data Requirements
Data Required Data Source Frequency Groups of Primary Interest
Number of new assessments completed* RDB monthly Client Service Team

Area Counselor
Number of reassessments completed* RDB monthly Client Service Team

Area Counselor
Active case plan issues* RDB bi-weekly Area Counselor
Case Plan Status RDB monthly Area Counselor
Overdue case plans according to target Date RDB monthly Area Counselor
Overdue case plans according to Service Standard RDB monthly Area Counselor
TAT for eligibility decisions** RDB monthly Client Service
Number of applications pending > 28 Days** RDB monthly Area Counselor
Length of pending applications** RDB monthly Area Counselor
VIP DO Client Screens RDB monthly District Office
VIP Reassessments RDB monthly District Office

* separated by rehabilitation and non rehabilitation clients
** for the Rehabilitation Program

The RDB provides much of the information listed in Tables 2 and 3. However, the information is reported at the district office level. Performance information is not readily available by AC and CSA. Some ad hoc reporting will be required from the RDB. Discussions need to be held with staff from the RDB to ensure data integrity at the AC and client service team level.

4.5 Evaluation Planning

In order for the results of the evaluation to be meaningful and useful to management, the timing of the evaluation must be optimal. The timing is dependent on several factors. First, there must be enough time allowed for staff in the Halifax DO to adjust to the new method of client service and become familiar with the processes. Second, time is required for the rehabilitation-focussed ACs and CSAs to gain experience in coordinating the rehabilitation program and to become specialists, not the generalists that they had been. Thirdly, a certification exercise is being undertaken in November 2008, which is designed to evaluate Client Service Teams to ensure that organizational practices are aligned with the philosophy and principles of the Client-centred Service Approach and the Integrated Service Delivery Framework. Finally, in order to identify trends, data must be gathered over an adequate period of time.

In the meantime, performance should be monitored regularly by the district office, regional office, and head office. Management should begin to collect and analyse data on key performance indicators, as outlined in Table 2. This will assist in identifying issues and determining trends.

Head Office will be addressing some of the issues identified in the recent report on case management. Further, there will be an evaluation of case management conducted by VAC’s Audit and Evaluation Division in 2009. In addition, the results of the certification of the Halifax DO will be available for review. The evaluator of the Halifax Rehab Case Management Pilot Project will need to be aware of these initiatives when planning the evaluation of the pilot.

5.0 RECOMMENDATIONS

R1 Management Response

R1 It is recommended that the District Director, Halifax District Office, together with the Atlantic Regional Director General, identify and assess the risks associated with the pilot and develop a risk management plan to address the key risk areas.

Management agrees on the importance of identifying any risks associated with the pilot and to develop a risk management plan to address the key risk areas. At the same time, management wants to present a balanced scorecard in relation to the pilot, so it will be equally important to identify the benefits associated with the pilot.

Several risks have already been identified, including:

  • possible increase in travel costs as rehab focussed staff overlap boundaries of other staff focussed on traditional clients;
  • unbalanced workload burden caused by the wrong mix of rehab clients versus traditional clients in workload allocation;
  • those Client Service Team (CST) members not part of the rehab focussed approach feeling their knowledge base is eroding thus putting them at a competitive disadvantage for career advancement (conversely, risk that being a rehab focussed staff member will cause management to deny them other developmental opportunities given the difficulty in replacing that specialized knowledge);
  • perception that rehab focussed staff should operate under a different job description than those handling traditional clients; high training costs and harder to backfill if there is attrition among rehab focussed staff; and,
  • creating solitudes within the CST whereby some staff focussed on rehab clients whereas others focussed on traditional clients.
Management Action Plan
Corrective Actions to be taken OPI (Office of Primary Interest) Target Date
Finalize list of all significant risks DDNS January 31, 2009
Conduct risk assessment rating of all identified risks. DDNS February 28, 2009
Develop risk management action plan to mitigate risks. DDNS March 31, 2009

R2 Management Response

R2 It is recommended that the District Director, Halifax District Office, together with the Regional Director Client Services, set measurable targets for key indicators and ensure ongoing monitoring of the indicators occurs.

Management agrees on the importance of setting measurable targets for key indicators, and to conduct ongoing monitoring of results against these indicators. Targets should be linked to existing RDB reports and data sources and should not involve establishing ad hoc reporting.

Management Action Plan
Corrective Actions to be taken OPI (Office of Primary Interest) Target Date
Review existing performance standards and reports for case management/client service as they apply to traditional clients and/or rehab clients. RDCS January 31, 2009
Where information gaps exist, identify those to VAC Head Office. RDCS January 31, 2009
Set measurable targets and monitoring schedule for rehab focussed approach based on existing data sources. RDCS February 28, 2009

R3 Management Response

R3 It is recommended that the Assistant Deputy Minister, Service Delivery and Commemoration, in consultation with the Atlantic Regional Director General, ensure an assessment of the rehab case management pilot is undertaken. The assessment is to include lessons learned and be done in the broader context of the work underway with the case management review being conducted by the Senior Assistant Deputy Minister, Policy, Programs and Partnerships.

Management agrees that such an assessment include lessons learned and be undertaken in the broader context of the work underway with the case management review.

Management Action Plan
Corrective Actions to be taken OPI (Office of Primary Interest) Target Date
Prepare a report on best practices and lessons learned from the rehab pilot. ADM SD and C/RDG February 28, 2009
Carry out an assessment of the rehab pilot in the broader context of the work underway with the case management review, including identification of lessons learned. RDG/DDNS June 30, 2009

6.0 DISTRIBUTION

  • Deputy Minister
  • Chief of Staff to the Minister
  • Chair, Veterans Review and Appeal Board
  • Senior Assistant Deputy Minister, Policy, Programs and Partnerships
  • Assistant Deputy Minister, Service Delivery and Commemoration
  • Assistant Deputy Minister, Corporate Services
  • Regional Director General, Atlantic Region
  • District Director, Halifax District Office
  • Director General, Communications Division
  • Director General, Departmental Secretariat and Policy Coordination
  • A/Deputy Coordinator, Access to Information and Privacy
  • Comptrollership Branch (TBS)
  • Office of the Auditor General
  • Program Analyst, Treasury Board of Canada, Secretariat

APPENDIX A

CONSULTATION AND DOCUMENT REVIEW

This appendix outlines the staff consulted and the documents reviewed that informed the evaluation framework.

List of staff consulted
Title Method
District Director, Halifax District Office Telephone
Director, Client Services and Quality Management Directorate In-person
Senior Policy Analyst, Strategic Program Initiatives In-person
National Manager, Client Services Telephone
Client Service Team Manager, Halifax District Office Email
Rehabilitation Consultant In Person
NVC Learning Team Member Email
A/National Manager, Quality Management Program In Person


List of Documents Reviewed
Doc
Number
Title
1 Draft Treasury Board Submission dated 2005
2 DRAFT Service and Program Modernization Task Force Reestablishment Program Integrated RMAF/RBAF, October 18, 2005
3 Case Management Report: Results of Quality Management File Analysis, April 17, 2008
4 Case Planning Practice Manual, January 2006
5 VPPM Vol 1, Chapter 6: CCSA Practice Manuals
6 Rehabilitation Review Report, 2008/2009 Fiscal Year. Prepared by: Adrienne Alfor-Burt, Regional Project Manager, Rehabilitation Review and Knowledge Management
7 Case Management Presentation to National Managers Conference, Montreal, Quebec, March 3-7, 2008
8 Rehabilitation Review Report Executive Summary, March 12, 2008
9 Case Management Discussion Paper: Where are we, How can we improve, Atlantic Region, February 2008, author: Anne Marie Smith
10 Rehabilitation Program Eligibility Decision Making Guide, Release 2, August 31, 2006
11 Rehabilitation Program Statistical Update, April 18, 2008
12 Integrated Service Delivery Framework Certification Standards and Criteria, 2006-03-27
13 New Veterans Charter Program Activity as of March 31, 2008
14 Rehabilitation-focussed AC and CSA, Halifax District Office Pilot Project: Processes and Guidelines, April 7, 2008
15 VPPM Vol. 5, Chapter 1, Subsection 1.2: Authorization of Services and Benefits (2007-08)
16 VPPM Vol. 5, Chapter 3: Rehabilitation Program
17 Rehabilitation Fact Sheet
18 Resource Allocation Model: Presentation to Deputy Minister, January 25, 2008
19 Area Counselor Work Description
20 The Client-centred Service Approach Standards, VPPM Vol. 1 Section 4, dated 08-07

APPENDIX B

EVALUATION ISSUES MATRIX

Issue 1: Has the pilot been successful in achieving its objectives?
Doc
Criteria/Indicator
Information Required Collection Methodology Data Source
a. improved client service
  • timeliness of service to clients
  • quality of case management
  • achieving service standards
  • consistent client service (by client group)
  • File review
  • Case study
  • Interviews
  • Staff
  • CSDN
  • Client files
  • Reporting Database
  • CCSA Service Standards
  • Baseline Data
b. efficiencies have been gained
  • coordination activities among delivery staff
  • productivity indicators (TATs, volume of cases pending, etc.)
  • problems and opportunities for improvement
  • Interviews
  • File review
  • Case Study
  • Reporting Database
  • Staff
  • Client files
  • Performance Targets
  • Baseline Data
c. pilot implemented as intended
  • identification of variances from the original pilot plan
  • extent/reasons for variances
  • impact of variances
  • Observation
  • Interview
  • Case Study
  • Staff
  • CSDN
  • Business processes
Issue 2: What impacts, both intended and unintended, has the pilot had?
Doc
Criteria/Indicator
Information Required Collection Methodology Data Source
a. impact on district office workforce
  • perceptions/opinions of staff
  • identification of problem
  • impact on workforce
  • consequences for client/staff
  • Interviews
  • Document Review
  • Observation
  • Focus Groups
  • Staff
  • Meeting minutes
  • Formal/informal complaints
  • RAM
b. unrealistic expectations of pilot objectives and expected results
  • perceptions/opinions of staff
  • workload distribution
  • performance targets
  • resourcing
  • impact on staff and client service
  • Interviews
  • Document Review
  • Workload Analysis
  • Staff
  • Client files
  • Workload Statistics
  • RAM
  • Performance Targets
c. impact on quality of client service (rehabilitation clients and non rehabilitation clients)
  • quality of client service
  • changes in client service
  • impact of changes on client service
  • Interviews
  • File Review
  • Case Study
  • Service Standards
  • Client files
  • Staff
  • CSDN
d. other positive/negative unintended impacts
  • identification of impact
  • extent of impact
  • consequences for client/staff
  • Interviews
  • Document review
  • Staff
  • Client files
Issue 3: Are there opportunities for improvements to the piloted approach?
Criteria/Indicator Information Required Collection Methodology Data Source
a. areas for improvements
  • perception/opinions of staff
  • constraints identified
  • opportunities identified
  • workload distribution and workload management
  • areas for streamlining the process
  • Focus group
  • Interviews
  • Document review
  • SWOT Analysis
  • Staff
  • RDB
  • CSDN
  • Process Chart
  • Staff
  • Financial Information
  • Performance Targets
b. drivers for success
  • benefits of the pilot
  • opinions/perceptions of staff
  • strengths and weaknesses identified
  • success of pilot (Issue 1)
  • impacts of pilot (Issue 2)
  • Focus group
  • Statistical Analysis
  • Interview
  • Document Review
  • Staff
  • RDB
  • Preliminary observations from other evaluation issues
c. value added and costs of pilot delivery
  • startup costs and continuing costs
  • cost and time impact of case load approach
  • File Review
  • Interview
  • Staff
  • Financial Information
  • Reporting Database

APPENDIX C

REPORTS AVAILABLE

The following is a list of reports that are generated by the Department, some of which would provide a useful baseline for comparison. This is by no means an exhaustive list of available data.

  1. Title: New Veterans Charter Program Activity

    Description: Number of applications received, withdrawn, pending and completed for individual NVC programs. It provides a status of completed decisions for individual NVC programs and TAT’s.

    Source: Statistics Directorate
  2. Title: Rehabilitation Program: Statistical Update

    Description: Provides a statistical update of the rehabilitation program, including TATs, active case plans, number of clients, and other performance indicators

    Source: National Rehabilitation Unit
  3. Title: 05.42.04.03e Rehabilitation -- All Pending Applications with Tracking (if applicable)

    Description: Identifies all clients with a rehabilitation application pending with associated tracking if applicable. This report is organized by Region and sorted by District. The report updates weekly.

    Source: Reporting Database
  4. Title: 42.04.04e Rehabilitation -- Applications Pending for more than 28 days with Tracking (if applicable)

    Description: Identifies clients with a rehabilitation application pending for more than 28 days with associated tracking if applicable. This report is organized by Region and sorted by District. The report updates weekly.

    Source: Reporting Database
  5. Title: 05.42.04.05e Rehabilitation Clients

    Description: To assist with case load management and performance monitoring by providing detailed summary information on all clients applying for or currently participating in the Rehabilitation Program. This report is updated monthly.

    Source: Reporting Database
  6. Title: 05.42.04.09e Eligible Rehabilitation Clients with a Date of Death

    Description: This report identifies clients who have a date of death recorded in CSDN and are showing as active in the Rehabilitation Program. The report is sorted by District and then by Area Counselor Code. The report updates monthly.

    Source: Reporting Database
  7. Title: 05.42.04.11e Rehabilitation In Progress / Vocational Rehabilitation Work Item Information Requests, from CPC to Region

    Description: This report identifies the files where the CPC has requested information from the District Office in order to be able to process the payment of an invoice for Vocational Rehabilitation. The report is used for follow-up, to ensure that District Offices are submitting missing information so that outstanding Vocational Rehabilitation invoices can be paid. The report is sorted by Region and then by District. The report updates weekly.

    Source: Reporting Database
  8. Title: 05.21.04.13e Case Plans with Active Issues

    Description: Assist with case load management and performance monitoring by providing detailed summary information on all clients receiving active case management services (defined as those clients having a case plan with active issues). The report updates weekly and monthly. These reports are organized by region, then by district, and then by area counselor. There are separate pages for each TAT. Summary totals are provided for national, regional, district, and individual TAT numbers. This report includes all clients who are alive and who have active issues in their case plan.

    Source: Reporting Database
  9. Title: 05.01.04.09e Area Counselor Case load Report

    Description: Identifies the area counselor case load. The report is presented by region and sorted by area counselor and municipality.

    Source: Reporting Database
  10. Title: Quality Management Unit Report on District Office Client Service Activities

    Description: Provides information on quality assurance indicators for districts and regions.

    Source: Client Services and Quality Management Directorate
Date modified: