Language selection

Evaluation of the Operational Stress Injury (OSI) Clinic Network - October 2008

Acknowledgement

The Audit and Evaluation Division would like to acknowledge the efforts of those who helped with this evaluation, with a particular thanks to staff of the National Centre for Operational Stress Injury. We would also like to acknowledge the assistance and support provided by the staff at the Operational Stress Injury Clinics.

Executive Summary

In April 2007, an evaluation of the Operational Stress Injury (OSI) clinic network was approved by Veterans Affairs Canada's (VAC) Audit and Evaluation Committee. The intent of this project was to support the expansion of the OSI clinic network. Fieldwork for this evaluation was conducted from November 2007 to June 2008. During this time, the National Centre for Operational Stress Injuries (NCOSI) was developing additional OSI clinics with planned implementation for fall 2008.

One of the objectives of this evaluation was to assess the relevance of the OSI clinic network. In 2002, VAC and the Department of National Defence (DND) jointly announced a mental health strategy to enhance the services and supports provided to the growing number of clients with an OSI. VAC had determined that there was not sufficient capacity within the community to provide the necessary services for clients as the demand for these services was increasing rapidly. As a result, VAC developed a network of five OSI clinics and in 2007 received Treasury Board approval to expand this network to ten OSI clinics.

A second objective was to assess the governance and accountability structure relating to the OSI clinic network. The network started with an initial pilot at Ste. Anne's Hospital in Montreal and is expanding to ten OSI clinics. However, in an effort to respond quickly to the identified client need, VAC began developing new OSI clinics without sufficient resources assigned and before the initial pilot had sufficient time to mature or before the pilot was adequately evaluated. The effect of this was that the original OSI clinics did not have defined business processes, program guidelines, or reporting requirements which created a number of challenges.

Initially the responsibility for the development and management of the OSI clinic network was at the national level with Ste. Anne's Hospital responsible for functional direction of clinical activities within the OSI clinics. As the network evolved, responsibilities shifted, and in 2007, the NCOSI became responsible for the development and management of the OSI clinic network with Mental Health Policy Directorate providing functional guidance on VAC policies and business processes. However, roles and responsibilities with respect to mental health policy and operations within VAC need to be clarified further. As the network has evolved, these areas have improved but still require further development and better communication.

The third objective was to obtain information on the relative costs and outputs of the OSI clinics. Once fully implemented the total forecasted operational expenditures for the OSI clinic network will be approximately $20 million. The NCOSI had recently implemented some significant improvements with the development of a new budgeting process and the creation of a new financial position dedicated to the OSI clinic network. However, there is a need to further improve the management of expenditures as historical expenditure information was not readily available, year-end financial reviews were not occurring and a cost sharing arrangement had not been developed with partner departments. Regarding outputs, VAC was collecting quarterly operational information and some recommended improvements are identified in this report.

The final objective was to assess the success of the OSI clinic network in meeting its objectives. Overall the direct client service activities were well developed and 88% of clients indicated satisfaction with the services provided. However, VAC had not clearly defined its expectations for outreach or research activities. In addition, VAC did not have a performance measurement system in place which was collecting the information necessary to assess the achievement of objectives or desired outcomes. At the completion of fieldwork, program managers had already initiated action to further define expectations and begin developing a performance measurement system.

Recommendations:

  • R1 It is recommended that the Director, National Centre for Operational Stress Injuries, update the "Program Guidelines for Operational Stress Injury Clinics."
  • R2 It is recommended that the Director, National Centre for Operational Stress Injuries, and the Director, Mental Health Policy Directorate, in coordination with the Director General, National Operations Division, revise the orientation program to ensure that it better meets the needs of new staff at the OSI clinics.
  • R3 It is recommended that the Senior Assistant Deputy Minister, Policy, Programs and Partnerships, and the Assistant Deputy Minister, Service Delivery and Commemoration, clarify the roles and responsibilities with respect to the OSI clinics.
  • R4 It is recommended that the Director, National Centre of Operational Stress Injuries, identify opportunities to improve the effectiveness of the network committees.
  • R5 It is recommended that the Director, National Centre for Operational Stress Injuries, ensure that year-end financial reviews are completed annually for all OSI clinics.
  • R6 It is recommended that the Senior Assistant Deputy Minister, Policy, Programs and Partnerships, develop a cost sharing arrangement with the Department of National Defence and the Royal Canadian Mounted Police for clients receiving service from the OSI clinics.
  • R7 It is recommended that the Director, National Centre for Operational Stress Injuries, revise the reporting requirements to provide a more detailed picture of operational activities.
  • R8 It is recommended that the Director, Mental Health Policy Directorate, and the Director, National Centre for Operational Stress Injuries, develop standardized reporting requirements for all external health service providers, including the OSI clinics.
  • R9 It is recommended that the Director, Program Policy Directorate, reassess the Department's requirement for clients with an OSI to periodically complete a re-assessment for the purpose of re-evaluating a client's level of compensation.
  • R10 It is recommended that the Director, National Centre for Operational Stress Injuries, determine achievability and identify supports in place for the objective to ensure continuity of service from the OSI clinic to service provided in the community.
  • R11 It is recommended that the Director, National Centre for Operational Stress Injuries, clearly define expectations for the direct and indirect service activities and provide direction in the allocation of resources to perform these activities.
  • R12 It is recommended that the Director, National Centre for Operational Stress Injuries, develop and implement a performance measurement system.

1.0 Introduction

VAC has defined an OSI Footnote 1 as "any persistent psychological difficulty resulting from operational duties performed while serving with the Canadian Forces". These psychological difficulties can include post traumatic stress disorder (PTSD), depression, anxiety, and addictions. An OSI can occur as a result of a variety of stresses including exposure to a traumatic incident, cumulative exposure to human atrocities, or simply the sustained exposure to intense military operation. Footnote 2

Since the end of the Second World War, the Canadian Forces have been engaged in military operations in, for example the Former Republic of Yugoslavia, Rwanda, and the Middle East. These intense military operations continue to expose Canadian Forces members to situations that may result in OSI and may require treatment and support.

To address this need, the Department of National Defence developed five Operational and Trauma Stress Support Centres (OTSSC) to provide assistance to members of the Canadian Forces and their families who are dealing with an OSI resulting from military operations. These OTSSCs consist of interdisciplinary teams of specialized health professionals who provide specialized assessment, treatment, and educational services for Canadian Forces members.

Additionally, many released Canadian Forces members have OSI which require care and treatment from VAC. In fact, over the past five years, the number of VAC clients identified as having an OSI has increased from 3,500 to more than 11,000. PTSD is the most common mental health condition; almost 60% of all clients who have received a favourable decision for disability benefits for a psychiatric condition have PTSD. Many clients with PTSD are challenged by other disorders as well, such as depression, substance misuse or anxiety; this is referred to as co-morbidity. Any of these disorders can compromise relationships, financial stability or a client's overall quality of life.

In 2002, VAC and the DND jointly announced a mental health strategy to enhance the services and supports provided to Veterans, Canadian Forces members, and eligible RCMP who suffer from OSI. This mental health strategy is described in Appendix A. One of the key elements of this strategy was to build capacity across the country to provide specialized care to clients with mental health needs associated with psychological trauma related to military service. This strategy led to VAC's decision to develop a network of OSI clinics across the country and the expansion of this network continues.

An OSI clinic takes an approach to treatment that relies on an interdisciplinary team of specialized health professionals who provide comprehensive and standardized assessment, treatment, education, prevention, and support services for clients. This interdisciplinary clinical model is similar to the OTSSC model developed by the DND. The DND's OTSSCs provide service to Canadian Forces members while the OSI clinics provide service to VAC clients as well as some DND and eligible RCMP clients.

2.0 Objectives

The objectives of this evaluation were to:

  • assess the relevance of the OSI clinic network;
  • assess the governance and accountability structure relating to the OSI clinic network;
  • obtain information on the relative costs and outputs of the OSI clinics; and
  • assess the success of the OSI clinic network in meeting its objectives.

3.0 Scope

The scope of this project was an evaluation of the OSI clinic network which included operations at VAC head office, the NCOSI, regional offices, district offices, and the OSI clinics. While aspects of how the OSI clinics operate within the network were examined, this evaluation was VAC-focussed. Detailed clinical operations related to the delivery of mental health services provided at the OSI clinics were not evaluated as part of this project.

4.0 Methodology

This evaluation was conducted in accordance with VAC standard practices and procedures for evaluation and followed the general evaluation guidelines issued by the Treasury Board Secretariat. This was an implementation evaluation, with fieldwork conducted from November 2007 until June 2008. Fieldwork for this evaluation included a comprehensive interview program consisting of 117 interviews with staff in head office, the NCOSI, regional offices, district offices, and at the OSI clinics in Montreal, Quebec City, London, Winnipeg, and Calgary. This interview program was supplemented with a detailed document and literature review as well as direct observation in VAC offices and the OSI clinics.

In addition, a stratified random sample of 150 client files was reviewed in order to assess VAC's management of client information. This sample, stratified by district, consisted of 75 client files drawn from the total population of clients actively receiving service at the OSI clinics as of December 31, 2007. The purpose of this stratified sample was to determine what information is received after the completion of an assessment as well as during the course of treatment. A second sample of 75 client files, stratified by district, was drawn from the total population of clients discharged from the clinic during 2007. The purpose of this sample was to determine what information is received after the completion of an assessment and treatment as well as whether VAC was informed of the clients' discharge.

5.0 Findings and Recommendations

The following sections present the findings and recommendations for the evaluation of the OSI clinic network.

5.1 Mental Health Services in the Community

VAC clients with an OSI have the option of using provincial and private health services available in the community. These services can include sessions with psychiatrists, psychologists, and social workers working in the community who can help reduce the mental health impact of trauma on people and assist in their recovery process. The cost incurred by eligible clients for mental health services provided in the community can be covered by VAC if the health services provider meets VAC's requirements and is registered with the Department.

Clients have been receiving treatment for an OSI since before the establishment of the OSI clinics. Many of these clients continue to use services in the community as they have already developed a treatment relationship with their health services provider. A benefit of using these services in the local communities is the close proximity to the client's home. However, many parts of the country are experiencing shortages of health professionals, especially psychiatrists, and the waiting period for mental health services can be long in some areas. This limits VAC's ability to ensure continuity of service from the OSI clinic to the community. A further explanation of this challenge is presented in section 5.8.2.

Respondents identified that clients with a more complex OSI require the support of an interdisciplinary team approach, in order to properly address the multiple aspects of treating an OSI. Generally, services in the community do not offer this interdisciplinary team approach which is required to support clients with more complex OSI.

Furthermore, respondents indicated that an understanding of the military culture and the client's experience is an important component of the treatment relationship which many health service providers in the community are not familiar with.

5.2 Operational Stress Injury Support Service

The Operational Stress Injury Support Service (OSISS) is a VAC-DND partnership program established in May 2001 to address the social, educational, and leadership aspects associated with OSI such as PTSD, anxiety, or depression. The mission of the OSISS program is to establish, develop and improve social support programs for current and former members of the Canadian Forces and their families affected by operational stress; bereaved families of military members and Veterans, as well as to create an atmosphere and environment which leads to a better understanding and acceptance of OSI.

The OSISS program consists of three components. The peer support component for Canadian Forces members and Veterans was established in 2002. Through this program, individuals who have had an OSI provide support to current and former Canadian Forces members who also have an OSI. It is important to note that these peer support coordinators are neither replacing nor acting as health professionals. Instead, they offer non-clinical assistance and support based on shared experience and the training that they have received. The family peer support component of the OSISS program was established in 2005. This component consists of family members who have had the experience of living with a Canadian Forces member or Veteran with an OSI who provide support to other families going through a similar experience. The bereavement component was established in 2006 and is comprised of volunteers who have experienced what it is like to lose a loved one due to military service and offer support to those who have recently lost a loved one.

The health professionals in the OSI clinics and staff within VAC strongly supported the important role that the OSISS program has in contributing to a client's progress. In fact, it was referenced that in many cases the peer support coordinators are often the first point of contact for VAC clients who may have an OSI.

5.3 Development of the Network of OSI Clinics

As part of VAC's mental health strategy, the first OSI clinic opened in 2001 as a pilot project at Ste. Anne's Hospital in Montreal. Based on this model, four additional OSI clinics were established in Quebec City, London, Winnipeg, and Calgary. The selection of these sites was based on identified client population needs and influenced by the location of the DND's OTSSCs in Halifax, Valcartier, Ottawa, Edmonton, and Esquimalt. The reason for this was VAC and the DND had formed a partnership to provide shared access for clients. However, given the significant demand for OSI services among active Canadian Forces members, the OTSSCs were limited in their ability to provide access to VAC clients. This limited access to the OTSSCs contributed to VAC's need to establish new OSI clinics in Fredericton, Ottawa, Edmonton, and Vancouver as announced in the Budget 2007. The following table identifies each of the OSI clinics and provides a timeline of when it was or is expected to be established.

Management Action Plan
City Name Date Opened Staff Size*
Montréal Ste. Anne OSI Clinic July 2001 13.2
Québec City CHUQ OSI Clinic** February 2004 16.0
London Parkwood OSI Clinic February 2004 10.2
Winnipeg Deer Lodge OSI Clinic September 2004 9.5
Calgary Carewest OSI Clinic April 2006 12.0
Fredericton Fredericton OSI Clinic January 2008 8.4
Vancouver Coastal Health OSI Clinic Fall 2008*** 9.8
Ottawa Royal Health OSI Clinic Fall 2008*** 11.3
Edmonton Capital Health OSI Clinic Fall 2008*** 13.0

* Source: Full-time equivalents identified in the 2008 - 2009 Budgets.
** Centre Hospitalier Université du Québec (CHUQ)
*** Target dates for implementation of new OSI clinics identified by the NCOSI

VAC has agreements with regional health authorities to provide exclusive service for VAC, CF, and eligible RCMP clients with an OSI. These service relationships are formalized in memoranda of understanding describing the obligations of each party and the conditions for payment.

To provide some added direction regarding expectations, VAC developed program guidelines. These program guidelines outline the operations of an OSI clinic describing areas such as the clinic accreditation process, program structure, assessment standards, and required treatment information. The guidelines also describe VAC's expectations regarding outreach, research, staff support, and development.

As the new OSI clinics opened, clinic staff received orientation to VAC, the network of OSI clinics, and were informed of operational requirements. Within VAC, the NCOSI and district offices were primarily responsible for delivering this orientation.

After an OSI clinic has been fully implemented, there is a need for ongoing communication and liaison between VAC and each OSI clinic. This ongoing communication occurs primarily with the NCOSI and the local district office in order to ensure that business processes work, roles are understood, referral criteria are clear, information needs are being met, and reporting expectations are understood.

Findings

  • VAC did not receive proper authority from Treasury Board Secretariat to expand the Ste. Anne's OSI clinic pilot and establish additional OSI clinics in Quebec City, London, Winnipeg, and Calgary. However, this issue was corrected in 2007, through a Treasury Board submission which provided authority to expand the network with additional OSI clinics.
  • VAC began developing the new OSI clinics before the initial pilot had sufficient time to mature and before the pilot was adequately evaluated. In addition, when the network was first developing, sufficient resources were not in place to be able to exercise appropriate leadership and direction. As a result, the earliest OSI clinics did not benefit from fully developed program guidelines nor did they have access to key template documents. Respondents from both the district offices and OSI clinics indicated that this created a significant challenge in establishing the OSI clinics and that development efforts were instead based more on verbal discussions than defined expectations and business processes.
  • VAC did not clearly define business processes nor requirements for the OSI clinics. In the absence of these requirements, OSI clinics and local district offices negotiated their own business processes and information sharing protocols.
  • Various sites developed different approaches to managing communication and day-to-day operations. The different approaches identified relied on direct communication between the district director and OSI clinic manager, scheduling regular meetings between VAC and the OSI clinic, or the formation of a working committee comprised of district office staff and OSI clinic staff.
  • In some sites, a senior staff position within VAC was assigned dedicated responsibility for managing the development of the OSI clinic. This commitment of a dedicated resource in some sites was reported to have improved communication, negotiation, and management of the development of new OSI clinics.
  • OSI clinics that opened more recently used the "Program Guidelines for Operational Stress Injury Clinics" and expressed appreciation for the resource. However, this document is out-of-date and makes several references to guidelines and business processes that are no longer applicable.
  • Health professionals in the OSI clinics indicated a need for improved orientation in areas such as the governance structure of the network, VAC business processes, and information relating to military culture in order to better understand clients' experience.

R1 Management Response

R1 It is recommended that the Director, National Centre for Operational Stress Injuries, update the "Program Guidelines for Operational Stress Injury Clinics."

Management agrees with the need to review the content of the "Program Guidelines for Operational Stress Injury Clinics" to adapt it to the current environment. Some of the guidelines are being reviewed. An action plan for the revision of the remaining ones will be completed. For greater clarity, the guidelines will be restructured and will comprise two categories: the administrative guidelines for the business processes within an OSI clinic, and the clinical guidelines for the delivery of care services. Development of the guidelines requires an exhaustive process of consultation and discussion to obtain buy-in from the OSI clinics.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
1.1 Develop action plan to complete the review of existing guidelines. NCOSI October 2008
1.2 Complete review of existing administrative and clinical guidelines in consultation with OSI clinics. NCOSI September 2009

R2 Management Response

R2 It is recommended that the Director, National Centre for Operational Stress Injuries, and the Director, Mental Health Policy Directorate, in coordination with the Director General, National Operations Division, revise the orientation program to ensure that it better meets the needs of new staff at the OSI clinics.

Management agrees with the need to improve the orientation program. The NCOSI shares with the regions the responsibility for development and delivery of the many subjects covered in the orientation sessions. The NCOSI is responsible for segments on:

  • The OSI clinic network.
  • An overview of the VAC structure and administrative functions, as well as the roles of Head Office, including the Mental Health Policy Directorate, the VAC regional and district offices, NCOSI, and key partners.
  • The interdisciplinary intervention model and the responsibilities in the management of an OSI clinic.
  • The administrative and clinical processes within an OSI clinic.

Since the establishment of the first OSI clinics, the context of the OSI clinics has evolved. To meet the network's training requirements, new three-day orientation programs were developed after consultation with OSI clinic managers: one for new OSI clinic managers and one for clinic staff. Feedback from recent sessions indicate a 90.6% satisfaction rate, and the sessions are being adapted to cover recent network changes.

The regions are responsible for development and delivery of orientation sessions in coordination with Mental Health Policy Directorate on:

  • VAC policies, programs, and services.
  • VAC business processes.
  • The regional structure, characteristics, and key roles.

A list of the subjects to be covered in the regional orientation program has been established and is being reviewed by the Mental Health Policy Directorate. To meet the needs of OSI clinic staff, a component on awareness of Canadian Forces and RCMP cultures is needed and consistency must be ensured across all regions. The Director General, National Operations Division, in coordination with the Mental Health Policy Directorate, will ensure the development and delivery of orientation sessions on VAC policies and business processes, and on military policing culture awareness.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
2.1 Revise the orientation sessions delivered by NCOSI and develop two structured programs for managers and members of the interdisciplinary team. NCOSI October 2008
2.2 Revise the list of subjects to be covered in the orientation session provided by regions including awareness of CF and RCMP cultures. MHPD November 2008
2.3 Develop orientation sessions to be delivered by regions to new OSI clinic staff in coordination with SDC. MHPD January 2009

5.4 Governance

The responsibilities for mental health policy and operations are shared among several different entities.

In 2006, the NCOSI, located at Ste. Anne's Hospital in Montreal, was given clinical program responsibility for the OSI clinic network and in 2007 it became responsible for the negotiation and management of MOU's with regional health authorities for the establishment of OSI clinics. The NCOSI consists of three sectors. The first is the Network Development and Coordination Sector which is responsible for managing MOUs with the OSI clinics as well as coordinating the development of business processes. The second is the Clinical Services Sector, consisting of the Ste. Anne OSI Clinic, which provides outpatient service. The third is the Clinical Expertise Sector which is responsible for the advancement of mental health clinical care through clinical program and policy development, consultation, education, research, and outreach activities.

The Program and Service Policy Division (PSPD), located in head office, has overall responsibility for departmental program and service policies and provides functional direction for the delivery of services. Within PSPD, the Mental Health Policy Directorate has been established to manage all policy related to mental health services. This includes the development of policy and business processes governing VAC's relationships with external health service providers. Also within PSPD is the Research Directorate which is responsible for providing leadership in the development of and performance of mental health research in order to support VAC's mental health program and policy design. The Client Service Quality Management Directorate is responsible for providing national guidance on case management.

The National Operations Division, located in head office, has the responsibility to develop and implement national initiatives, as well as to provide overall leadership in the delivery of VAC programs. This includes providing operational guidance and direction to the regional and district offices.

The regional offices are responsible for participating in the development and implementation of national initiatives, as well as for providing functional direction to the district offices. In 2006, a regional mental health officer (RMHO) position was created to support district office staff in dealing with OSI issues. Some of the RMHO's functions include providing training relating to mental health services, and participating on case consultations with staff in the district offices and health professionals in the OSI clinics.

The district offices are responsible for coordinating the delivery of benefits and services directly to VAC clients. This includes the responsibility for case management of clients with an OSI, as well as working directly with the OSI clinics and local health professionals in the community.

The Mental Health Steering Committee was established in order to provide integrated direction and oversight for the development and implementation of mental health programs, services, and policies. This committee consists of representatives from the NCOSI, PSPD, NOD, and the regions.

The Mental Health Research Committee was established in order to coordinate and support the development of all mental health research projects. This committee consists of a representative from the research directorate, mental health policy directorate, the NCOSI, and Ste Anne's Hospital.

External health service providers are paid to deliver mental health services for clients with an OSI. Health service providers in the community who are registered with VAC are paid a fee for service provided to VAC clients. The OSI clinics are a unique type of external health service provider as the operations of each OSI clinic is fully funded by VAC to provide exclusive service for VAC clients.

Several OSI clinic network committees were established in order to support the ongoing management of the network. These committees include: a clinic manager committee, clinical coordinators committee, a psychology committee, a nursing committee, and a social work committee.

Findings

  • As described above, both the NCOSI and the Mental Health Policy Directorate are responsible for developing policy and business processes for external health service providers which includes the OSI clinics. Respondents from both the district offices and the OSI clinics indicated that this creates inconsistency in the direction provided regarding the application of policies and procedures such as referral processes and reporting requirements for external health service providers.
  • Staff in both the district offices and the OSI clinics generally lacked a clear understanding of the role of the individual sectors within the NCOSI and the level of support that could be provided.
  • The NCOSI developed a draft governance document which provided an overview of the various entities involving mental health within VAC. However, respondents from both VAC and the OSI clinics indicated that this document required further development and broader communication.
  • There was a significant variance in the responsibilities being performed by the RMHOs, specifically with respect to their role in relation to the OSI clinics. Management was aware of this inconsistency and at the completion of fieldwork, VAC was in the process of reviewing the role and responsibilities of this position.
  • There were network committees for all health professions represented in the OSI clinics except psychiatry. Respondents from the OSI clinics, who participate on these committees, indicated that the network committees were most effective when the focus of the committee was on promoting collaboration, support, and knowledge exchange. Most respondents indicated that the committees often focussed too much time on supporting VAC processes rather than on clinic processes.
  • Respondents from the OSI clinics indicated that the monthly network committee teleconferences were not an effective approach for developing and editing technical documents. In addition, varying levels of proficiency in English and French among committee members created further challenges.

R3 Management Response

R3 It is recommended that the Senior Assistant Deputy Minister, Policy, Programs and Partnerships, and the Assistant Deputy Minister, Service Delivery and Commemoration, clarify the roles and responsibilities with respect to the OSI clinics.

Management recognizes the necessity of clarifying the roles and responsibilities, including those of the RMHO, to better define the requirements of the network. It has been agreed in the past that the Mental Health Policy Directorate is responsible for the development of functional direction on the overall OSI clinic program and that the NCOSI continue to act in an advisory capacity with a lead on the clinical content. A document defining the roles and responsibilities for VAC stakeholders in the OSI clinic network has been drafted and needs to be completed and communicated. A communication strategy will be developed to ensure that all OSI clinics and VAC personnel are adequately informed.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
3.1 MHPD in collaboration with the NCOSI and the ADM SDC will finalize, for approval, and communicate a document detailing the roles and responsibilities of VAC stakeholders in relation with OSI clinics. MHPD December 2008

R4 Management Response

R4 It is recommended that the Director, National Centre of Operational Stress Injuries, identify opportunities to improve the effectiveness of the network committees.

Management agrees that close attention to the various network committees is needed to improve their effectiveness. Consultation on the scope and functions of the network committees, and the development of terms of reference for each committee are underway. As always, endorsement by the OSI clinic managers is requested prior to the finalization of the network committee terms of reference. OSI clinic managers will approve an annual action plan of all of the network committees. NCOSI is currently studying Internet-based communication tools to support national network consultation and information exchange.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
4.1 Evaluate Web-based solutions for network collaboration. NCOSI March 2009
4.2 Consult with the OSI clinics on the terms of reference and scope of the network committees. NCOSI March 2009
4.3 Complete all terms of reference for network committees NCOSI June 2009

5.5 Management of Expenditures

In accordance with the MOUs, the OSI clinics are required to submit annual budget requests and VAC provides funding for the full operational costs for all OSI clinics. Originally, the OSI clinics annual budget requests were reviewed by the district and regional staff who were responsible for approving the annual budgets in consultation with the NCOSI. However, in 2008 a new budgeting process was implemented where the NCOSI is now responsible for approving all OSI clinic budget requests in consultation with district and regional staff.

The table below presents a summary of historical operating expenditures for the OSI clinic network.

OSI Clinic Network Historical Operating Expenditures
OSI Clinic 2008/09* 2007/08 2006/07 2005/06 2004/05 2003/04 2002/03
Ste. Anne 1,632,300 1,298,402 1,829,356 2,409,319 1,422,919 1,319,724 942,509
CHUQ 2,466,485 587,399 374,943 293,279 310,762 107,443  -
Parkwood 1,381,088 864,108 655,076 404,133 535,576 274,385  -
Deer Lodge 1,181,528 680,346 598,860 541,580 319,141  -  -
Carewest 1,920,945 1,208,568 1,090,570 495,635  -  -  -
River Valley 1,295,790 250,000  -  -  -  -  -
OSI Clinic Expenditures 9,878,136 4,888,823 4,548,805 4,143,946 2,588,398 1,701,552 942,509
NCOSI 3,227,000 2,183,805 1,412,386 217,650  -  -  -
Total Expenditures 13,105,136 7,072,628 5,961,191 4,361,596 2,588,398 1,701,552 942,509

Source: VAC regional finance and district offices

* Identified expenditures are the approved budgets for the upcoming year and do not represent actual operating expenditures.

Once fully implemented, the total forecasted operational cost for the new OSI clinics in Ottawa, Edmonton, Vancouver will be $6.3 million which will increase the OSI clinic network's total operating expenditures to approximately $20 million.

Findings

  • Prior to the implementation of the new centralized approach, there were some concerns identified regarding the consistency of budget decisions and the timeliness of budget approval. The new centralized approach contains a structured approval process which is expected to provide greater consistency in assessing funding requests. In addition, the 2008 - 2009 budgets were approved in a timely manner.
  • Payments made to the OSI clinics were not coded in a manner which would allow VAC's payment system to identify the expenditures for the OSI clinic network. In addition, no one within VAC was responsible for tracking the expenditures of the OSI clinic network. As a result, with the exception of Ste. Anne's, historical expenditure information for payments made to the OSI clinics was not readily accessible and the information presented in the above table was collected by the evaluation team from various sources in the regions and districts. Management was aware of this issue and at the completion of fieldwork, the NCOSI and VAC's Finance Division were in the process of developing unique financial coding within VAC's payment system which will allow for expenditures for the OSI clinic network to be readily accessible.
  • For the fiscal years 2004 - 2005 to 2006 - 2007, some costs were inconsistently charged either to the NCOSI or to the Ste. Anne's OSI clinic. As illustrated in the above table, this inconsistency of allocating costs created some variance in the reported expenditures for the two units. However, this inconsistency was corrected in 2007 - 2008.
  • The OSI clinics are advanced funds monthly based on approved budgets. As a result, VAC defined a process whereby regional finance was responsible for the year-end reviews of actual expenditures at the OSI clinics. However, only one regional unit was annually performing these year-end reviews.
  • In 2008, the NCOSI created a financial position responsible for tracking OSI clinic network expenditures, coordinating the budget approval process, as well as supporting other financial functions. The addition of this position should contribute to improving VAC's management of expenditures for the OSI clinic network.
  • As of December 31, 2007, there were 87 CF clients and 19 RCMP clients accessing the OSI clinics which represented approximately 16% of the total population across the network. Notably; in one site, 60% of clients receiving service from the OSI clinics were DND or RCMP clients. However, VAC was funding the full operational costs for all OSI clinics without any cost sharing arrangement developed with the DND or the RCMP.

R5 Management Response

R5 It is recommended that the Director, National Centre for Operational Stress Injuries, ensure that year-end financial reviews are completed annually for all OSI clinics.

Management agrees that annual year-end reviews must be completed for all OSI clinics. In 2008, the responsibility to establish and manage agreements was transferred from the regions to the NCOSI. In addition, the NCOSI created a finance position responsible for all financial aspects related to the NCOSI and its network of OSI clinics.

Further to a recommendation made by Audit Services Canada, it was decided that a standard review process be established and that the same team of people perform the review of each OSI clinic to ensure accurate and comparable results. This team will likely be composed of VAC staff primarily from the NCOSI and Finance. The review process used for the Quebec OSI clinic by the Quebec Finance Directorate's Internal Control sector was deemed appropriate. Therefore, they will be consulted in order to establish the new standards, and their review template will be used as a basis for future reviews. The review process will be submitted to the Chief Finance Officer for approval as well as reports on year-end reviews.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
5.1 Establish members of the review team. NCOSI September 2008
5.2 Establish the standard review process in consultation with the Internal Control sector of the Quebec Finance Directorate and seek Chief Financial Officer approval. NCOSI November 2008
5.3 Schedule review dates with the OSI Clinics. NCOSI February 2009
5.4 Complete year-end reviews for 2008-2009 and submit report to the Chief Financial Officer. NCOSI September 2009

R6 Management Response

R6 It is recommended that the Senior Assistant Deputy Minister, Policy, Programs and Partnerships, develop a cost sharing arrangement with the Department of National Defence and the Royal Canadian Mounted Police for clients receiving service from the OSI clinics.

Management agrees that a cost-sharing arrangement must be developed between VAC, DND, and the RCMP for clients receiving service from the OSI clinics. The Joint Network for Operational Stress Injuries Management Committee was established by VAC, DND, and the RCMP. The committee established working groups for the cost-sharing arrangement. Composed of representatives from each department, the working groups are establishing the list of core services and related fees, and discussing options for the methods of cost sharing between the departments. In addition, the NCOSI will consult the OSI clinics in the national network to get their feedback and help determine the preferred method of cost sharing. A Program Management Schedule will be signed under the Memorandum of Understanding and implemented by the parties in the cost-sharing framework.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
6.1 Agree on list of core services and the related fees. NCOSI September 2008
6.2 Consult the OSI Clinics and get their feedback. NCOSI November 2008
6.3 Determine the preferred method of cost sharing. NCOSI January 2009
6.4 Establish Program Management Schedule under the Memorandum of Understanding between VAC, DND, and the RCMP. Senior ADM PPP February 2009
6.5 Implement the cost-sharing framework. NCOSI April 2009

5.6 Operational Reporting

In September 2006, all OSI clinics began providing VAC with quarterly administrative reports which summarize operational activities performed. These reports listed activities such as the number of new referrals completed, number of files closed, outreach activities performed and research conducted. The table below presents a roll-up of some of the information included in these reports.

OSI Clinic Network (January - December, 2007)
OSI Clinic Average Number of Active Files Number of Clients Referred to the OSI Clinic Number of Files Closed
Ste. Anne's 267 62 41
CHUQ 113 47 45
Parkwood 122 79 20
Deer Lodge 94 88 58
CareWest 82 66 44
Total 678 342 208

Source: Quarterly Administrative Reports provided by the OSI clinics
Note: See page 6 for OSI clinic locations

Findings

  • While the quarterly administrative reports present a lot of information, the information reported does not provide a detailed picture of operational activities. For example, an active file can include clients referred for an assessment only, clients currently receiving treatment, clients waiting to access the OSI clinic and clients who are no longer at the OSI clinic, but whose file has not yet been closed.
  • These quarterly reports did not include other information that would be useful, such as the number of clients receiving treatment, number of hours of treatment provided, number of assessments completed, and reason for file closure. This information is already being collected by the OSI clinics; however, VAC was not requesting it because, when the quarterly administrative reports were some OSI clinics were not in a position to provide the data in a uniform manner across the network.

R7 Management Response

R7 It is recommended that the Director, National Centre for Operational Stress Injuries, revise the reporting requirements to provide a more detailed picture of operational activities.

Management agrees with the necessity of improving the reporting requirements to include pertinent data on demographic profiles, operational volumetrics, and quality and performance indicators. The current quarterly reports and the client satisfaction survey are seen as complementary tools to the network balanced scorecard. As such, they are an integral part of the OSI clinic network performance management framework. The current reporting mechanism was developed in collaboration with the OSI clinics and was limited to what common information could be provided by all OSI clinics. The content of these quarterly reports is under review as part of the development of a performance measurement system and further consultation with the OSI clinics will be required. Mentalh Health Policy Directorate is working on the client identifier in the Client Service Delivery Network (CSDN) which will help the process of referral to an OSI clinic.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
7.1 Study feasibility of including client identifier in CSDN for OSI clinic referral. MHPD November 2008
7.2 Identify what new information OSI clinics will be asked to report on. NCOSI January 2009
7.3 Consult with the OSI clinics on the requirements for approval. NCOSI June 2009
7.4 Develop an action plan in collaboration with OSI clinics which includes target dates for the implementation of new reporting requirements NCOSI December 2009

5.7 Management of Client Information

VAC is responsible for case managing all of its clients. These clients can have many different health and support needs; however, the core case management functions remain the same. These core case management functions are assessment, case planning, monitoring/evaluation, advocacy, and disengagement.

Effective case management ensures that clients have access to the resources they require. A key aspect of effective case management is case planning which involves the use of a working document to organize the activities required to achieve the desired outcomes for a client. A well developed case plan provides a detailed account of what actions have been completed and identifies further actions that are required.

The information management system designed to support case management at VAC is the Client Service Delivery Network which consists of a large database used to store and manage all client information, record client interaction with the department, and support the delivery of programs and services to VAC clients. In addition, VAC maintains a hard copy client file of documents recording the client's history.

The OSI clinics maintain their own client files containing confidential medical information regarding clinical treatment. These OSI clinics are responsible for sharing the client information that is necessary for VAC's case management. This information includes reports on assessed mental health needs, treatment plans, and progress updates.

In order to assess VAC's management of client information, the evaluation team reviewed a random sample of 150 files of clients receiving service from the OSI clinics.

Findings

  • VAC did not have a system in place to identify which clients were receiving mental health services from the OSI clinics or in the community. As a result, VAC must rely upon the OSI clinics to identify the population of VAC's clients receiving services.
  • Forty-two percent of the client files reviewed at VAC's district offices had neither the OSI clinic's assessment nor a treatment plan on file. In addition, of those files which did contain an assessment or treatment plan, many did not contain progress updates. In total, only twenty-seven percent of VAC's client files reviewed contained an assessment and treatment plan with progress updates since January 1, 2007.
  • Results varied significantly by district but in total VAC did not have a case plan developed for forty-five percent of the client files reviewed. Case planning ensures that the client's health needs are actively being case managed by VAC. However, in one site, the district office completed a review of all clients with an OSI to ensure that each client was being appropriately case managed. For this site, almost all client files reviewed had an up-to-date case plan on file.
  • At the completion of fieldwork, the Quebec region was conducting a national study of the operationalization of the Mental Health Strategy. This study included VAC's case management of clients with an OSI.
  • The reporting format and type of information contained in the assessment reports, treatment plans, and progress updates were not standard among the OSI clinics. This is also an issue for the client information provided by health service providers in the community.
  • At the completion of fieldwork, VAC was in the process of conducting a review of business processes and reporting requirements for third party service providers with the intent of identifying gaps and developing standardized requirements.

R8 Management Response

R8 It is recommended that the Director, Mental Health Policy Directorate, and the Director, National Centre for Operational Stress Injuries, develop standardized reporting requirements for all external health service providers, including the OSI clinics.

Management agrees that the reporting requirements need to be standardized. The Service Providers Enhancement Project began in January 2008 with the objective of standardizing the reporting process. Policies and business processes are pending approval. Assessment and progress report templates are being finalized. Once approved, a communication plan to inform service providers, OSI clinic staff, and VAC personnel will be developed.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
8.1 Approve policies and business processes. MHPD October 2008
8.2 Finalize assessment and program report for service providers. MHPD December 2008
8.3 Implement communication plan. MHPD January 2009

5.8 Objectives of the OSI Clinic Network

The OSI clinics perform four key activities: assessment, treatment, outreach, and research. Based on these activities, VAC had defined the following objectives for the OSI clinic network:

  • Develop and offer clinical programs of assessment and treatment to respond to the clients' needs.
  • Ensure the development and maintenance of clinical skills and knowledge through continuing education and training opportunities in mental health, and specifically trauma and associated disorders.
  • Provide information and education to regional and local VAC employees and community care providers to improve knowledge, identification, treatment, and recovery from OSI and associated conditions.
  • Collaborate on research related to the prevention, assessment, and treatment of operational stress injuries and associated conditions.

Sections 5.8.1 to 5.8.4 present detailed findings and recommendations for each of the key OSI clinic activities. In addition, Section 5.9 presents the findings and recommendations for VAC's performance measurement.

5.8.1 Assessments

There are four reasons why a client might be referred to an OSI clinic for an assessment. The primary reason is to assess a client's mental health needs and to develop a treatment plan. This assessment and treatment plan can then be used at the OSI clinic or shared with a health service provider in the community. A secondary reason for assessment is to determine a client's eligibility for a VAC disability award. In these cases, some clients may not be receiving or interested in receiving treatment for their OSI but the assessment is required to support VAC's administrative processes. Other secondary reasons for assessment are to re-evaluate the client's level of compensation or to determine if a client can participate in vocational rehabilitation.

Given the complexity of OSI, the assessment process generally requires several hours and is performed by various health professionals over a period of days or weeks which can be very intense for a client. The first step in the assessment process at the OSI clinic is an initial assessment which is performed by a nurse in order to collect the client's background information, identify the main problems, and appraise the urgency of the client's health needs. The results of this initial assessment are then presented during the OSI clinic's interdisciplinary team meeting where the client's potential health needs are discussed and decisions are made regarding which additional assessments are required.

Findings
  • The OSI clinics were generally able to perform the assessments within the 15 day turnaround time established by VAC. However, in some sites, there were occasional waiting periods. For these sites, the OSI clinics managed the waiting lists by coordinating with VAC to ensure that any clients requiring immediate service received it. Respondents indicated that even during these occasional waiting periods, the wait time for service at the OSI clinic was significantly less than in the community.
  • Health professionals in the OSI clinics indicated that the quantity and quality of information VAC provided with referral packages varied. This created challenges for the OSI clinic staff in completing the assessment process. The purpose of the assessment, the client's military history, and any available medical information was not always provided. However, at the completion of fieldwork VAC was in the process of finalizing new business processes for district staff referring clients to the OSI clinics. These new business processes identify the information required in the referral package.
  • In most districts, VAC referrals to the OSI clinic were sent by the area counselors managing the client's file. However, in one site, the district office in collaboration with the OSI clinic developed a referral process where a single client service team manager was responsible for the coordination of all referrals to the OSI clinic. For this site, having a single individual coordinate all referrals ensured consistency in the quality and quantity of information provided by VAC to the OSI clinics.
  • Some respondents raised a concern regarding the appropriateness of VAC's requirement to periodically complete a full re-assessment for the purpose of re-evaluating a client's level of compensation. The concern raised was that this administrative requirement can have a negative effect on the well-being of the client.
  • Another challenge with this process for completing a re-assessment is that the health professional providing treatment to the client cannot perform the re-assessment as it creates a conflict of interest. In order to manage these requests, some sites used other health professionals in the OSI clinic and some sites used service providers in the community.
R9 Management Response

R9 It is recommended that the Director, Program Policy Directorate, reassess the Department's requirement for clients with an OSI to periodically complete a re-assessment for the purpose of re-evaluating a client's level of compensation.

Management agrees that psychological injuries are unique, in that the assessment process itself may have an effect on the condition. Management recognizes that the nature of occupational stress injuries must be taken into consideration when balancing the statutory requirement to assess in accordance with the extent of the disability with the potential effects of completing the assessment process too soon or, on the contrary, the potentially negative effects of withholding payment until the condition has stabilized. It is for this reason that Program Policy leaves latitude in decision-making with respect to when assessments and reassessment occur. This reflects the importance that Policy places on operational flexibility when working with mental health clients with differing needs.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
9.1 Undertake consultations with Operations to determine if there is a policy gap present and if so provide updated functional guidance on this issue. PPD November 2008

5.8.2 Treatment

The OSI clinics are comprised of health professionals who operate as an interdisciplinary team to provide specialized treatment which stabilizes a client's OSI and supports the maintenance of therapeutic gains over time. Treatment plans are tailored to each client's individual needs and the intensity and type of treatment offered can vary from short term intensive sessions to longer-term periodic sessions. These treatment sessions can be for individuals, groups, or couples. In addition, OSI clinics offer education sessions and short-term counselling for family members.

Another objective of the OSI clinics is to ensure continuity of service from the OSI clinic to service provided in the community. This is based on the expectation that, as a client's treatment progresses, the client's mental health needs may stabilize to the point where he or she no longer requires an interdisciplinary approach to treatment. At this stage, it may be appropriate for the client to be referred to services in the community.

Findings
  • The profile of the client population differed significantly across the country. OSI clinics in close proximity to a Canadian Forces base served a client population that generally had more acute OSI treatment needs. In contrast, the client population at other OSI clinics generally had more chronic OSI treatment needs. This variance in the client population's general treatment needs had an impact on how OSI clinics delivered clinical programs.
  • All OSI clinics had developed the interdisciplinary treatment model as was described in the program guidelines and were offering specialized OSI treatment tailored to the client's needs. In addition, OSI clinics had developed group therapy programs for clients and psycho-education sessions for clients and family members.
  • The most common concern identified by respondents was the need to enhance services offered to family members of clients with an OSI. Currently, family members of a client are not eligible to receive service at an OSI clinic unless the client is receiving treatment at the clinic. Then even if family members are eligible, only limited services are available.
  • At the completion of fieldwork, VAC was in the process of conducting a review of services available for family members in order to develop an integrated framework of mental health services for families.
  • For some clients living in isolated areas, travelling distance may be a barrier to treatment access. In order to address this need, some OSI clinics, in coordination with regional health authorities, had developed pilot projects to explore the effectiveness of tele-mental health for the delivery of OSI clinic services. Preliminary results were favourable and it is expected that, if implemented, telemental health will contribute to expanding access to the services offered by the OSI clinics.
  • Generally, the objective of continuity of service from the OSI clinics to the community was not being achieved as most clients were remaining at the OSI clinics for an indefinite period or until treatment was completed. Some health professionals in the OSI clinics did not believe that referring clients to a new health professional in the community was practical as it could have a negative impact on treatment progression. Some health professionals in the OSI clinics expressed concern that in some cases adequate services were not available in the community to meet client needs. Outreach helps develop capacity in the community which, as described in Section 5.8.3, requires further development.
  • Given the long-term support needs of many clients with an OSI, if continuity of service from the OSI clinic to the community must be promoted so that the OSI clinics can be able to manage client demand over the long term without continually expanding resources.
R10 Management Response

R10 It is recommended that the Director, National Centre for Operational Stress Injuries, determine wheather it is achievable and identify supports in place for the objective to ensure continuity of service from the OSI clinic to service provided in the community.

Management agrees that continuity of service can be improved. It is generally recognized that few services specific to the VAC client base are available in the community; the interdisciplinary approach is unique to the OSI clinics and qualified specialists are rare. Studies are needed to identify the resources that will best serve clients in the community to achieve continuity of services with OSI clinics. Currently, as part of the development of performance measurement (see Recommendation #12) indicators are being implemented in the OSI clinics. This information is needed to properly assess the achievability of this objective and to help identify what additional supports are required. As a first step, this objective will be discussed during the June 2009 meeting with the OSI clinic managers. A strategy will be developed that includes identifying providers in the community who can offer services to clients with an OSI, and encouraging OSI clinics to offer clinical internships. Guidelines on referral to community service providers are to be developed in collaboration with Mental Health Policy Directorate. Training and support for other providers in the community are also needed, and an action plan will be developed.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
10.1 Monitor wait list times to ensure that newly released members of the Canadian Forces are not waiting unacceptable times for assessment. NCOSI October 2008
10.2 Begin collecting performance information data. NCOSI April 2009
10.3 Provide guidance and share best practices on outreach and networking strategies and activities aimed at building capacity in the community. NCOSI June 2009
10.4 Develop action plan for training and professional support for service providers to be delivered by the OSI clinics. NCOSI September 2009
10.5 Develop guidelines on client referral to community or other service providers in coordination with MHPD. NCOSI December 2009
10.6 Analyze information and identify additional outreach and network strategies required. NCOSI April 2010

5.8.3 Outreach

The primary purpose of outreach activities is to familiarize the public with OSI and to provide support to health service providers in the community in their clinical service delivery efforts. This is generally accomplished by meeting with health service providers to promote the OSI clinic as well as to provide support, information, and training regarding services for an OSI.

In addition, outreach to the community can also include building relationships with various associations and related groups such as mental health associations or local universities. These outreach activities are an integral part of VAC's mental health strategy which includes building capacity across the country to provide specialized care to clients with mental health conditions. In addition, as described in Section 5.8.2, the development of capacity within the community is a necessary component to ensure continuity of service from the OSI clinics to service in the community.

Another aspect of outreach is providing support and information to VAC and the DND staff. This generally consists of providing education sessions to staff at the local VAC district office or Canadian Forces base as well as conducting meetings with local management to improve business processes. The time invested in these types of government outreach activities helps strengthen relationships and improve overall service to clients.

Findings
  • VAC was not providing coordinated direction to the OSI clinics with regard to requests for outreach activities. Generally, outreach activities were only occurring in response to requests from various sources such as the NCOSI, VAC's district office, VAC 's head office or the local Canadian Forces base.
  • VAC had not clearly defined which activities would be classified as outreach. This resulted in some variance in the reporting of outreach activities. For example, some OSI clinics were reporting participation on the monthly network committee teleconferences as outreach activities while other OSI clinics were not. This reporting inconsistency can misrepresent the amount of actual outreach being performed and limits the comparability among OSI clinics.
  • In total, only 35% of reported outreach activities related to building capacity within the community, which is considered the primary purpose of outreach. Respondents generally agreed that the OSI clinics need to focus more effort on performing outreach activities in the community.
  • While most OSI clinics had developed a strong relationship with the local district office, these OSI clinics also provide service to clients in other districts. It was identified that further outreach activities are required with these other districts to educate VAC staff on what services an OSI clinic can provide in order to promote referrals for clients in those districts.
  • In one site, the OSI clinic worked with VAC to develop an annual outreach plan. The outreach plan identified priorities for the year with a balance between government and community outreach activities while still allowing the flexibility for ad hoc requests as necessary.
  • In one site, the OSI clinic was working with the local district office to contact all of VAC's registered providers of mental health services in order to offer support and training. This type of outreach activity builds the capacity of health service providers in the community and supports providing continuity of service from the OSI clinics to services in the community.
  • In one site, the OSI clinic had developed a website to promote outreach to the community. This website defined the OSI clinic's outreach program, identified presentations and information available and provided contact information. This was an effective method of communicating with external health service providers and builds relationships between the OSI clinics and external health service providers.
  • In some sites, the OSI clinic was using psychology residents from the local university. This practice promotes recruitment as some residents may obtain full employment after graduation. In addition, those residents who go on to work in the community have gained experience providing service to VAC clients with an OSI.
  • In one site, the telemental health infrastructure was being used to provide long distance training with health professionals. This is a cost effective method of expanding the reach of outreach services.

5.8.4 Research

A key element of VAC's mental health strategy is conducting research, which contributes to building knowledge to better understand and treat OSI and to enhance VAC's reputation as a centre of expertise. In addition to VAC's research committee described in section 5.4, a research sub-committee was established. Consisting of representatives from the OSI clinics, the research sub-committee is responsible for developing and annually updating a three-year research plan. As part of this process, potential research projects are discussed and, once developed, a formal research proposal is submitted to the VAC research committee for review and approval to conduct research with VAC clients. If funding is required, it is provided from other sources such as research grants or partnerships with universities.

Findings
  • Research within the OSI clinic network was still in development. In 2007 - 2008, the focus of the research sub-committee was on implementing an organizational structure, processes, resources, and developing research projects.
  • In 2007 - 2008, VAC provided funding for a research and program evaluation officer in each OSI clinic. One of the roles of this new position will be to facilitate the development of research projects and to build research partnerships with external organizations such as universities.
  • Staff within the OSI clinics indicated that the opportunity to conduct research was an effective recruitment incentive. However, some health professionals who were interested in conducting research indicated that they weren't able to because of high caseloads or because their proposals had been rejected by VAC.
  • At the completion of fieldwork, there were no research projects in progress. However, members of the research committee indicated that the focus of research will be to develop partnerships with universities and a number of projects are in the early stages of development.

5.9 Performance Measurement

Performance measurement is the process of collecting information that identifies progress toward the attainment of objectives and desired outcomes. The purpose of performance measurement is to support informed decision making and to facilitate action to further improve the delivery of a particular program or service.

Effective performance measurement can have several benefits. Performance measurement provides regular feedback regarding the quality, quantity, cost, and timeliness of service delivery. It also supports management planning and can help management prepare for future conditions that are likely to effect service delivery. In addition, performance measurement provides early warning indicators of problems and feedback on the effectiveness of corrective actions taken.

Findings

  • VAC had clearly defined its expectations for the service activities relating to client interventions, such as assessment and treatment. However, VAC had not clearly defined its expectations for outreach or research activities which performance could be measured against. Furthermore, VAC was not providing direction to the OSI clinics regarding the allocation of resources to perform these client service and outreach or research activities.
  • VAC did not have a performance measurement system in place which was collecting the information necessary to assess achievement of objectives or desired outcomes.
  • In the absence of defined expectations and a comprehensive performance measurement system, VAC was not able to assess the network's success in meeting objectives or monitor service delivery to clients.
  • VAC had developed client satisfaction surveys which could be used to measure certain aspects of performance. These client satisfaction surveys were distributed to clients receiving treatment at all of the OSI clinics. For 2007, 88% of clients indicated satisfaction with services provided.
  • VAC had developed a "Continuous Quality Improvement Framework" which included the development of quality domains that performance would be measured against. These quality domains are presented in Appendix B. At the completion of fieldwork, VAC was in the process of developing quality indicators based on the recently developed quality domains.
  • The program guidelines developed by VAC outlined a requirement for all OSI clinics to develop an assessment process to examine their activities with regard to planning, delivering, and assessing the clinical services and programs they offer to their clients. The information collected from this process could support VAC's measurement of OSI clinic performance; however, VAC was not collecting this information.
  • There were no other external organizations performing accreditation or assessing the performance of the OSI clinics. In the absence of a performance measurement system, these external reviews of the OSI clinics could have provided some assurance to VAC that the OSI were providing quality service to clients.

R11 Management Response

R11 It is recommended that the Director, National Centre for Operational Stress Injuries, clearly define expectations for the direct and indirect service activities and provide direction in the allocation of resources to perform these activities.

Management agrees with the recommendation. A performance indicator has been developed to track the effort spent on clinical intervention-related activities (treatment, assessment, reporting) and on other activities (outreach, networking, research, training). It identifies the ratio of clinical intervention time to total paid hours. Targets of 75% for the time spent on clinical intervention and 25% for other actvities were set. Once this indicator is fully implemented in the clinics, there will be a basis for analysis and future planning for resource allocation. There is already a three-year plan which defines various research activities, and one for outreach and networking activities will be developed and implemented.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
11.1 Initiate collection of data for this performance indicator. NCOSI April 2009
11.2 Develop three-year plan for outreach and networking activities. NCOSI June 2009
11.3 Perform study on the data collected to trend the amount of time spent on clinical intervention. NCOSI April 2010

R12 Management Response

R12 It is recommended that the Director, National Centre for Operational Stress Injuries, develop and implement a performance measurement system.

Management agrees that a performance measurement system is required. The Performance Management Framework has been developed and approved by the OSI clinic managers who identified its 13 key performance indicators—these form the basis of the OSI clinic balanced scorecard. Seven have been articulated, service standards have been determined, and they are ready for implementation. OSI clinic managers are in the process of performing an operational readiness exercise to determine their information management system's capacity for collecting the data required to support these indicators and develop an action plan. The remaining six, which relate to the effectiveness and efficiency of network performance, will require extensive analysis and trending of research evidence derived from clinical effectiveness studies of current programming. To determine a costing structure, trending of this data is needed along with the development of a client profile model and the review of other reporting requirements.

Management Action Plan
Corrective Actions OPI (Office of Primary Interest) Target Date
12.1 Perform operational readiness exercise for action plan and implementation of the first seven indicators by OSI clinics. NCOSI January 2009
12.2 Start collection of preliminary data for the first seven indicators. NCOSI April 2009
12.3 Develop protocol for standardization of evaluation of client clinical outcomes across the country. NCOSI September 2009
12.4 Trend and analyze information on clinic activities, client profile activities and budget for purposes of development of efficiency indicators. NCOSI September 2010
12.5 Trend and analyze clinical research outcomes for purposes of development of effectiveness indicators. NCOSI November 2010
12.6 Implement remaining indicators. NCOSI April 2010

6.0 Distribution

  • Deputy Minister
  • Senior Assistant Deputy Minister, Policy, Programs and Partnerships
  • Assistant Deputy Minister, Service Delivery and Commemoration
  • Assistant Deputy Minister, Corporate Services
  • Executive Director, Ste. Anne's Hospital
  • Director General, Program and Service Policy Division
  • Director General, Human Resources Division
  • Director General, Strategic Policy Division
  • Director, National Centre for Operational Stress Injuries
  • Director, Mental Health Policy Directorate

Appendix A

Mental Health Strategy

The VAC Mental Health Strategy is a commitment by the Department of Veterans Affairs to improve the quality of life of its clients who live with mental health conditions and their families-Veterans, still-serving members of the Canadian Forces and eligible members of the Royal Canadian Mounted Police. The thrust of the strategy is to develop a comprehensive continuum of mental health services that can support clients and their families with the services that they require as close to their local community as possible. The strategy consists of four elements.

  1. Implementation of a comprehensive continuum of mental health services and policies which includes promotion, early intervention, treatment, rehabilitation and ongoing care.
  2. Building capacity across the country that provides specialized care to clients with mental health conditions associated with psychological trauma related to military service. This includes the development of an integrated network of Operational Stress Injury clinics, peer support coordinators through the Operational Stress Injury Social Support program, as well as a network of service providers at the local community level.
  3. Strengthening the role of Veterans Affairs Canada as a leader in the field of mental health, including the ongoing development of the National Centre for Operational Stress Injuries in clinical matters related to mental health.
  4. Development of strong, collaborative partnerships with other organizations who share the goal of responding effectively to the needs of clients living with mental health conditions.

Appendix B

Quality Domains

  • Accessibility: Ability of the client to obtain clinical care services at the right place and right time based on identified needs.
  • Acceptability: Clinical care and services meet the expectations of the client.
  • Availability: Services and resources (financial, human, information and equipment) are available to meet the needs of the clients.
  • Appropriateness: Clinical care and services provided are relevant to the client's needs and based on established standards.
  • Competence: Knowledge, skills and actions of individuals providing clinical care and services are appropriate.
  • Effectiveness: Clinical care and services achieved desired results.
  • Efficiency: Organization and clinical programs achieved desired results with the most cost-effective use of resources.
Date modified: