2.0 Evaluation Purpose, Issues and Methodology

2.1 Purpose and Scope

The purpose of the evaluation was to determine the effectiveness, efficiency and relevance of OVO operations. This was the first evaluation conducted of the OVO since its creation in 2007.^{Footnote 4} This evaluation followed an Evaluability Assessment completed April 25, 2019.

The scope of the evaluation was all activity areas of the OVO and on the last 5 years of OVO operations (FY 2014/2015 to 2019/2020). However, for the relevancy/mandate component, the evaluation looked back to establishment of the OVO in 2007, to understand why the office was established, its original mandate, roles and responsibilities; and then, examine whether they remain relevant in meeting the current needs of Veterans/clients.

2.2 Evaluation Issues and Questions

The issues and questions used to guide the evaluation included:

Effectiveness

1. Are Veterans complaints being resolved, and what are the key barriers in the OVO’s ability to resolve complaints?
2. What impact/change is resulting from systemic research report recommendations?
3. What additional or unexpected outcomes (positive or negative) have resulted due to the work of the OVO?

Efficiency

4. Is the OVO Performance Strategy adequate to measure its impact?
5. Is the governance structure in the OVO appropriate and efficient for achieving expected outcomes?
6. Considering other ombudsman offices, are there alternative structures or delivery options (e.g., powers, tools) that would enhance the effectiveness and efficiency of the OVO?
7. Have the activities of the OVO been delivered in an efficient and economical manner?

Relevance / Mandate

8. What are the needs and expectations of Veterans/clients for an Ombudsman?
9. Is the OVO’s mandate still relevant considering the current context and Veterans’/clients’ needs?
10. Is there an appropriate level of independence for the OVO?

2.3 Methodology

To address the evaluation issues/questions and triangulate the data collected, a mixed-method approach was used. The methodology included: key informant interviews, a review of documents and databases, two case studies of individual complaints and one systemic review, as well as a comparative assessment with other ombudsman offices (federal and Ontario). A matrix that summarizes the evaluation questions, indicators and methods used to conduct the evaluation can be found in Annex 2.

The focus of the document review was on mandate documents, annual plans and reports, performance data from the OVO’s database (Ombudsman File Tracking System-OFTS), OVO Client Surveys (2018-19 and first half of 2019-20) and the results of client surveys conducted by VAC between 2014/2015 and 2019/2020. The document review covered the whole system that Veterans can access, including: the distinct mandates of all relevant organizations and levels of appeal (e.g., OVO, VAC, VRAB, Bureau of Pensions Advocate (BPA), National Defence and Canadian Forces Ombudsman); their relevance to Veterans; and, areas of gaps and complementarity. The case studies of two individual cases and one of a recent systemic report can be found in Annex 3. The documents and databases reviewed are listed in Annex 4.

Semi-structured key informant interviews were conducted with 22 individuals from the following categories (the interview guides are located in Annex 5):

• Management and staff in the OVO (6 interviews with 8 individuals);
• Key senior officials in VAC who are engaged with the OVO (4);
• Veterans Ombudsman Advisory Council members, Veterans, and organizations helping Veterans (9);
• Parliamentary Veterans Sub-Committee Chair (1).

Responses from key informants are reported in the findings as follows:

• A few: less than 25 percent
• Some: between 25-49 percent
• Majority: between 50-79 percent
• Most: over 80 percent
• All: 100 percent

The evaluation included two case studies of individual claims to provide the opportunity to examine particular effectiveness or efficiency issues in a more in-depth manner, to learn lessons and highlight best practices. It also examined a systemic review “Meeting Expectations: Timely and Transparent Decisions for Canada’s Ill and Injured Veterans” to show how the recommendations influence change.

The comparative assessment^{Footnote 5} with other ombudsman offices was included to answer questions around relevance, particularly relating to alternatives in mandate, independence and powers of the office, governance and management structure, human and financial resources, etc. The assessment also examined how other ombudsman offices measure their performance, as well as the international standards for Ombuds offices. The comparison reviewed the practices of the Federal Taxpayers’ and Ontario’s Ombudsmen in detail, and compiled insights from four other Federal Ombuds offices (National Defence, Victims of Crime, Procurement, and Responsible Enterprise). As part of the comparative assessment, the evaluation team reviewed the Principles on the Protection and Promotion of the Ombudsman Institution (“Venice Principles”), adopted by the Venice Commission on March 15, 2019.^{Footnote 6} The 25 Venice Principles represent the first, independent, international set of standards for the Ombudsman institution. They provide guidelines for the improvement of existing and new Ombudsman offices.^{Footnote 7}

2.4 Limitations

The evaluation relied on documents and performance data provided by the OVO, and did not independently verify this information. This limitation had minimal impact on the evaluation process as all stakeholders agreed that the OVO’s data was accurate and reliable, and most was publicly reported.

The evaluation interviewed a limited number of key informants (n=22) due to timeframe and resources available. This limitation had minimal impact on the evaluation as representatives from all key stakeholder groups were interviewed, and interviews with Veterans and Veteran stakeholder groups accounted for half of all interviews conducted allowing for the triangulation of the data.