Language selection


3.0 Methodology

The methodology focussed on the use of various lines of evidence, both qualitative and quantitative in nature, to assess the relevance of the RTC and its program reach.

3.1 Qualitative

3.1.1 Interviews

Interview guides were developed and semi-formal, structured interviews conducted in support of gathering qualitative information. Interviews were conducted with a wide range of individuals with knowledge and expertise in the field of mental health. This included individuals with work duties related to the RTC and independent experts providing a balanced perspective in terms of observations and opinions. Moreover, some interviewees were able to provide input from more than one perspective. Additionally, bias was mitigated by conducting multiple interviews within key functions/areas:

  • Staff directly engaged with the Residential Treatment Clinic including its professional staff (8),
  • Subject Matter Experts in Canada, USA, and Australia and the DND (8),
  • Ste. Anne's Hospital senior management staff who provide oversight to the RTC (4),
  • VAC staff within the Mental Health Directorate responsible for VAC's overall mental health strategy (6),
  • "Front-line" staff in VAC's district and regional offices (4),
  • Peer support staff (6), and
  • Individuals who are receiving/have received OSI treatment in the RTC (4).

In total, forty interviews were conducted and the resulting comments, information and opinions were reviewed and summarized by evaluation question and then further analyzed in conjunction with other evidence to develop the evaluation findings.

3.1.2 Literature review/research

The literature review has included research studies and related articles, Treasury Board submissions, the Budget 2007 as well as the 2007 and 2010 Speech from the Throne, past evaluations and related websites. In addition, the NCOSI provided presentation decks from Ste. Anne's Hospital dating back to 2001 which outlined the early vision for a residential treatment clinic. For a detailed listing of internal and external sources used to inform this evaluation (refer to Annex C).

3.1.3 Site Visit

The evaluation team visited Ste. Anne's Hospital to observe the RTC facility, conduct face-to-face interviews with staff directly engaged with the Residential Treatment Clinic, as well as senior management of Ste. Anne's Hospital and the NCOSI. In addition, the site visit provided the opportunity to speak with client-patients of the RTC. It is noteworthy that the opportunity to interview RTC program participants originated with the client-patients themselves when they learned that the evaluation team was on site.

3.1.4 Comparison with Australia and the United States

Both Australia and the United States have experienced a longer recent history in armed conflict than has Canada. Both countries sent troops to the Vietnam War and dealt with the physical and mental health issues their soldiers experienced in that conflict. It was important that subject matter experts in both countries be interviewed to determine their respective approach to the care and treatment of their Veterans and their lessons learned. The comparison was enhanced by specific research studies and documents shared by the Australian and U.S. experts with the evaluation team. The comparison is intended to provide further insight and value-added information for VAC decision-makers.

3.2 Quantitative

VAC's Mental Health Directorate produces a quarterly statistical report which provides data on VAC's clientele, specifically, those with a favourable Disability Benefit decision for a psychiatric condition, as well as a profile of these clients - Veterans, CF still-serving, CF and RCMP members. This report provided useful contextual data on the size of the eligible population and historical trends. Other data references of note include the following:

New Veterans Charter (NVC) Evaluation File Review

A file review completed by an external occupational therapist provided an overall assessment of progress for participants in the NVC Rehabilitation Program. The sample population for this file review consisted of individuals who entered the NVC Rehabilitation Program between April 1, 2006 and October 31, 2009, and had participated in the program for at least six months. A stratified random sample of 350 participants was drawn. This descriptive sample allowed for the analysis of the sample's characteristics and a comparison among various groups, but did not allow for the results to be generalized for the entire participant population. However, the findings from the file review remarkably mirrored the results and percentages gathered when the entire population was studied using departmental statistical information. This evaluation utilized the information from the NVC Evaluation file review as additional evidence on numbers presenting with mental health issues and pain.

Life After Service Study (LASS)

This research study, a joint effort between VAC the DND and Statistics Canada is comprised of four parts. The first part of the LASS is an Income study of CF Members who released over a ten-year period from 1998 - 2007 while the second part is a Survey on Transition to Civilian Life (STCL). The STCL information drew from a nationally representative sample of 4,721 of 32,015 Regular Force Veterans released from 1997 to 2008 and who were subsequently contacted by Statistics Canada during February and March 2010. The majority of the sample responded to the survey (71 percent), and the great majority of those (94 percent) agreed to share their responses with VAC and the DND. The information from the STCL report was used to inform this evaluation in terms of the co-morbidity of mental and physical health conditions.

Parts three and four of the LASS will be the subject of a future report on the CF Cancer and Mortality Study and are not intended to inform this analysis.

Client Satisfaction Survey

The RTC professional staff has provided data with respect to the client-patients in both program tracks of the RTC, including client patient demographics and client satisfaction surveys for the stabilization program component of the RTC. They have also tracked the number of client readmissions and gathered data on discharge/exit surveys.

Additional Data Sources

A DND OSI study, due to be released in 2011, will use a stratified sample of 2,500 to determine how many still-serving CF members have PTSD directly attributable to the military mission. The prediction of the target population for inpatient care and treatment is a vital piece of information as VAC needs to predict the future demand for its services. Data was also gleaned from an Australian research study and from interviews with subject matter experts in Australia and the United States citing the estimated percentage of severe PTSD cases which require inpatient care.

3.3 Limitations and Challenges

This evaluation was challenged by the length of operation of the Residential Treatment Clinic. VAC's Multi-Year Risk-Based Evaluation Plan for 2011 - 2012, approved in March 2011, limited the evaluation to an assessment of the RTC's relevance and performance. With respect to the latter, performance outcomes as a determinant of the Clinic's effectiveness, were limited to the immediate outcome of program reach. Scope limitations relative to performance also included any assessment of economy and efficiency. The evaluation was spurred by the potential transfer of the Ste. Anne's Hospital to the Province of Quebec and it will serve to inform VAC's Senior Management on the RTC's continuing need and program reach.

The clinical outcomes of the RTC are also a challenge to obtain. The already existing four-bed, inpatient stabilization program which commenced in 2002 was co-located with the new six-bed residential rehabilitation program to form the Residential Treatment Clinic for Operational Stress Injuries in mid-February 2010 to offer two program streams: stabilization treatment and rehabilitation therapy services. The latter is comprised of structured, intensive group therapy sessions for the target population. Thus, while clinical outcomes are available for the longer running stabilization program, clinical outcomes for the residential rehabilitation program and the overall Residential Treatment Clinic program are not readily available. The current action plan for the RTC management team provides for an internal evaluation of clinical outcomes in the near future. The National Centre for OSI has recently issued a public tender for a Client Reported Outcome Monitoring System (CROMIS) which is a web-based data collection system that will generate real time reports both for the client and the clinician to monitor progress, or lack thereof, for the purpose of treatment adjustment. It will be used for the entire NCOSI network. The DND has also adopted the same instrument as VAC to measure client outcomes for similar monitoring purposes.

The breadth of the subject matter has exposed the evaluation team to many areas of need as well as areas where improvements can be made which are outside the parameters of the evaluation. The challenge will be to organize and effectively communicate these additional findings/observations to the appropriate VAC personnel.

Date modified: