Annex 2: Evaluation Matrix
| Questions | Sub-questions | Indicators | Data Collection Method | Source of DataFootnote 21 |
|---|---|---|---|---|
| Issue: Relevance | ||||
| Ongoing need for the OVO | ||||
| 1. What are the needs and expectations of Veterans/clients for an Ombudsman? | 1.1 What unique needs are met by the OVO within the context of its current mandate? | 1.a Mandate stated needs addressed by OVO 1.b Types of complaints received 1.c Stakeholders views of OVO mandate |
Key Informant Interviews (KII) Document Review (DR)Data base review (DBR) Case studies (SC) |
KII: VAC and OVO management & staff; members of Minister Advisory Group (MAG); members of Veterans Ombudsman Advisory Council (VOAC) DR: OVO Mandate; annual reports DBR: CRM (Ombudsman File Tracking System) CS: file review; systemic review report; OVO KII |
| 1.2 Considering other bodies/organizations that address Veterans’/clients’ concerns/complaints, is there overlap or duplication with the OVO; and, are there any gaps where Veterans’/clients’ concerns are unmet? | 1.d Degree of overlap with other bodies and organizations addressing Veterans’ concerns / complaints 1.e % of Veterans / clients complaints received by the OVO that cannot be addressed within mandate of OVO |
Key Informant Interviews Document Review Data base review |
KII: VAC and OVO staff, MAG, VOAC DR: Order in Council (OIC) P.C. 2007-530; annual reports; five year strategic and integrated business plan 2017; Service Level Agreements (SLAs) with VAC, 2016-2019 DBR: CRM (Ombudsman File Tracking System) |
|
| Relevance of the OVO over time | ||||
| 2. Is the OVO’s mandate still relevant considering the current context and Veterans’/clients’ needs? | 2.1 Has the OVO adhered to its mandate (why or why not)? | 2.a Degree of change since 2007 2b. Evidence of shift in mandate |
Key Informant interviews Document Review |
KII: OVO, VAC DR: Order in Council (OIC) P.C. 2007-530; Report of the Standing Committee 2007; 2007 Planning documentation; OVO five year strategic and integrated business plan 2017 |
| 2.2 Have the needs of Veterans’ changed over time? | 2c. Systemic Veterans’ issues over time | Key Informant interviews Document Review |
KII: OVO, VOAC DR: Systemic reports of the OVO over time |
|
| 3. Is there an appropriate level of independence for the OVO? | 3.1 Is there an appropriate level of independence for the OVO? | 3.a Adequacy of the OVO level of independence to meet Veterans needs | Key Informant interviews Document Review Comparative Assessment Case Studies |
KII: OVO, VAC, MAG, VOAC DR: OVO Mandate CA: Procurement / Corrections Ombudsman Offices; By-Monthly Ombudsmen’s meeting CS: file review; systemic review report; OVO KII |
| 3.2 How does the OVO’s mandate and powers compare to other ombudsman offices? | 3.b Adequacy of OVO mandate and powers compared to others | Key Informant interviews Document Review Comparative Assessment Case Studies |
KII: OVO, VAC DR: OVO Mandate CA: Procurement / Corrections Ombudsman Offices; By-Monthly Ombudsmen’s meeting CS: file review; systemic review report; OVO KII |
|
| Issue: Effectiveness | ||||
| Achievement of outcomes | ||||
| 4. Are Veterans complaints being resolved, and what are the key barriers in the OVO’s ability to resolve complaints? | 4.1 Are Veterans/clients aware of and accessing the OVO? | 4.a % of Veterans and other clients surveyed who indicate they are aware there of the Office of the Veterans Ombudsman and the services and benefits it provides 4.b % of official stakeholders (e.g., veterans’ groups) engaged annually 4.c # of Town Halls conducted annually across Canada |
Key Informant Interviews Document Review Case studies |
KII: OVO, VAC, MAG, VOAC DR: VAC National Surveys/POR, OVO / VAC Annual Reports, Engagement reports/dashboard CS: File review; systemic review report; report card; KII: OVO |
| 4.2 Are Veterans treated respectfully and fairly by the OVO, and are their concerns addressed by VAC? | 4.d % of OVO clients that indicate their complaints were addressed respectfully 4.e % of complaints that were assessed as unfair by the Office of the Veterans Ombudsman that were resolved by Veterans Affairs Canada |
Key Informant interviews Document Review Database Review |
KII: OVO, VAC, MAG, VOAC DR: Client feedback questionnaire DBR: CRM (Ombudsman File Tracking System) |
|
| 4.3 Do Veterans have timely responses to their complaints? | 4.f % of complaints closed within 60 working days 4.g % of OVO clients who indicate they received a response to their complaint when indicated |
Document Review Database Review |
DR: Client feedback questionnaire DBR: CRM (Ombudsman File Tracking System) |
|
| 4.4 Do systemic investigations address issues of concern to Veterans/clients? | 4.h Effectiveness of mechanisms to identify research topics 4.i Degree of alignment of research with Veteran’s concerns over time 4.j Are procedures for conducting and reporting systemic investigations effective? |
Key Informant interviews Document Review Database Review Case study |
KII: OVO, VAC, MAG, VOAC DR: OVO research reports; Report Cards DBR: CRM (Ombudsman File Tracking System) CS: systemic review report; report card; OVO KII |
|
| 4.5 What internal or external barriers / factors prevent the OVO from achieving its outcomes? | 4.k Degree to which barriers prevent achievement of outcomes | Key Informant interviews Document Review |
KII: OVO, VAC, MAG, VOAC DR: OVO / VAC annual reports; OVO research reports |
|
| 5. What impact/change is resulting from systemic report recommendations? | 5.1 What impact is resulting from the advice provided to the Minister of Veterans Affairs and Parliamentary Committees? | 5.a % of OVO recommendations on systemic issues implemented by the Veterans Affairs Portfolio 5.b % of OVO recommendations on systemic issues accepted by the Veterans Affairs Portfolio |
Key Informant Interviews Document Review Case studies (CS) |
KII: OVO, VAC, MAG, VOAC DR: OVO / VAC annual reports; Report Cards, 2017 & 2018 CS: File review, KII OVO |
| Unexpected outcomes and external factors | ||||
| 6. What additional or unexpected outcomes (positive or negative) have resulted due to the work of the OVO? | 6.a Evidence of additional or unexpected outcomes | Key Informant Interviews Document Review |
KII: OVO, VAC, MAG, VOAC DR: VAC / OVO annual reports |
|
| Issue: Efficiency | ||||
| Performance Measurement | ||||
| 7. Is the OVO Performance Strategy adequate to measure its impact? | 7.1 Are expected outcomes well-articulated to indicate the impact of the OVO | 7.a Clarity of outcome statements 7.b Strength of causal link between outcomes and ultimate outcome |
Document Review | DR: Evaluability Assessment, Performance Information Profile (PIP) |
| 7.2 Has a performance measurement strategy been developed and implemented? | 7.c Evidence of a measurement strategy 7.d Evidence that the OVO is collecting data to measure its relevance, effectiveness and efficiency |
Key Informant Interviews Document Review |
KII: OVO, VAC (AED) DR: Performance Information Profile (PIP); OVO/ VAC annual reports; CRM (Ombudsman File Tracking System) |
|
| 7.3 Is the OVO performance data reliable, timely, and valid? | 7.e Assessment of performance data | Key Informant Interviews Document Review |
KII: OVO, VAC DR: Evaluability Assessment |
|
| 7.4 Is performance information reported and used in decision-making? | 7.f Evidence of reporting using OVO performance data 7.g Evidence of use in decision-making |
Key Informant Interviews Document Review |
KII: OVO, VAC DR: OVO / VAC annual reports |
|
| Governance and Management | ||||
| 8. Is the governance structure in the OVO appropriate and efficient for achieving expected outcomes? | 8.1 How does the OVO’s structure facilitate or impede its success (e.g., powers of the OVO, funding model, reporting relationships, succession, etc.)? | 8.a Degree to which governance and management structures and processes allow the OVO to achieve expected outcomes | Key Informant Interviews Document Review |
KII: OVO, VAC DR: Audit of VAC’s Governance, December 2017 |
| 8.2 Has the management approach (direction and processes in the OVO) been strategic and adapted appropriately over time? | 8.b Stakeholder perceptions on degree to which changes in direction and processes in the OVO improved its efficiency | Key Informant Interviews | KII: OVO, VAC (current and former) | |
| 9. Considering other ombudsman offices, are there alternative structures or delivery options (e.g., powers, tools) that would enhance the effectiveness and efficiency of the OVO? | 9.a Strengths and weaknesses of alternative structures or delivery options to achieve OVO’s outcomes and improve efficiency | Key Informant Interviews Document Review Comparative Assessment |
KII: OVO, VAC DR: OVO Organizational Chart, February 2019 CA: Procurement and Correctional Ombudsman Offices; By-monthly Ombudsmen meeting |
|
| Efficiency of delivery | ||||
| 10. Have the activities of the OVO been delivered in an efficient and economical manner? | 10.1 Does the OVO have the right tools and processes for efficient delivery? | 10.a Average time and resources required to resolve complaints 10.b Veterans / clients level of satisfaction with complaint remedy 10.c % of complaints in the backlog 10.d % of systemic investigations submitted to the Minister within 90 days for micro investigations and within 1 year for systemic investigations 10.e % of engagement plan activities completed annually |
Key Informant Interviews Document Review Database review Case studies |
KII: OVO DR: Front line service efficiency study, client feedback surveys, Report Cards, review of engagement dashboard DBR: CRM (Ombudsman File Tracking System) |
| 10.2 Do staff have the required capacities, and are they being retained, to promote efficient delivery (e.g., communication and frontline staff)? | 10.f Views on staff capacities and retention 10.g Evidence that staff training is offered and accessed 10.h Level of staff turnover |
Key Informant Interviews Document Review Case studies |
KII: OVO DR: OVO HR and Financial Reports |
|