Veterans Affairs Canada's website is undergoing maintenance. If you are experiencing any issues, please contact us. We apologize for the inconvenience this may cause.

3.0 Audit Results

3.0 Audit Results

3.1 Implementation and Effectiveness of Telephony System

An external caller’s first interaction with the Department is through Veterans Affairs Canada (VAC)’s Interactive Voice Response (IVR). IVR is telephony technology which allows an external caller to interact with a pre-programmed database of selections using a touch-tone telephone. An effective IVR enhances caller experience by directing the call efficiently. A review of the IVR identified some areas of possible improvement, including:

  • Shortening the length of the recorded selection options (English/French)
  • Simplifying the message
  • Improving the quality of the recording (voice, music)
  • Adding an option to repeat previous message

Based on callers’ IVR selections, the majority of calls are directed to the National Contact Centre Network (NCCN), which is the central point of contact to obtain general information and request services.

If an NCCN analyst is unable to resolve the caller’s concern, the call is transferred to another area of the Department. According to the first contact resolution (FCR) model, the call will be transferred to the first available Veteran service agent (VSA) in an area office, when the issue is related to VSA responsibilities and authorities. Field observations confirmed that the FCR model was working as intended for most callers. However, depending on the volume of calls transferred to the area offices, the FCR model may, in some cases, directly impact the area office’s ability to address other workload. Staff noted that, in most cases, callers transferred from the NCCN take priority over those who submitted applications or written requests for follow-up. There was no evidence of risk mitigation in regards to this concern.

Staff reported significant delays in actioning service requests for telephony equipment. The documentation review identified several instances when requested essential equipment, such as telephone headsets, took more than one month to receive. Such delays negatively impact service levels as employees are not equipped to provide telephony services effectively and create difficulties when training new employees.

3.2 Compliance

3.2.1 Compliance with Processes

The audit team assessed NCCN-related business processes, with documentation found to be relevant, mostly current, and accessible online. Also, staff generally reported receiving sufficient training. A one-on-one WebEx training session was provided to most users by Intelligent Call Exchange (ICE) subject matter experts with an ICE training guide distributed for reference. In general, staff noted that ongoing training was adequate.

Field observation and file reviews revealed that calls received by the NCCN were answered following procedures. These procedures include: greeting the caller, verifying the caller’s identity, resolving the issue, closing and documenting the call. Records and information received from callers were maintained in accordance with policies and processes. As shown in Table 1, the audit team identified a high degree of compliance with the above noted procedures and staff carried out their duties effectively.

Table 1: Observed NCCN Call Compliance Result
Criteria Result Percentage
Greeting 182/182 100%
Verifying 152/154 99%
Resolving 121/155 78%
Closing 177/183 97%
Documenting 164/170 96%

Note: Denominator varies due to type of call; e.g., not all calls required verification.

The audit team noted that the existing caller verification process is lengthy and has limited security. To confirm their identity, callers are required to provide their name, date of birth, file number, full address, postal code, and phone number. This information is verified against information stored in the Client Service Delivery Network Footnote1 (CSDN) . With minimal effort, others could obtain this information via publicly available means. In addition, there was no evidence of a Threat and Risk AssessmentFootnote2 (TRA) regarding the current caller verification process.

Password verification, available in CSDN, is currently implemented upon the Veteran’s request. This verification method could significantly decrease the verification process time and increase the level of security.

3.3 Management Control Framework

As part of the assessment of the management control framework, key monitoring activities and quality assurance (QA) processes for the telephony system were reviewed by the audit team.

3.3.1 Quality assurance and performance monitoring

Performance standards for NCCN Analysts provide specific targets for telephony services. These standards define VAC’s expectations of its employees and should assist the Department in delivering efficient, effective, and consistent service when handling inbound calls from anywhere in Canada. The current standards for NCCN Analysts are:

  • average handling timeFootnote3 : 5 minutes, 30 seconds
  • average documentation time: 4 minutes, 50 seconds
  • bumped callsFootnote4 : maximum of 5 per month
  • logged in timeFootnote5
  • average not-ready timeFootnote6 : 90 minutes per day

To monitor performance, NCCN Team Leaders:

  • monitor analyst performance and identify training needs;
  • review 10 calls four times per year per analyst; and
  • provide feedback to analysts.

The audit team found that NCCN Team Leaders have access to various ICE reports and perform regular quality assurance reviews in relation to the above standards. However, the data/review is limited to individual performance compared to individual indicators. There is no evidence of performance monitoring at a site level.

The audit team completed a detailed data analysisFootnote7 to determine the overall site results against the performance indicators, and found significant variances in performance by site and results generally below established standards. For example, at two sites, 50% of staff was meeting the performance standard for average handling time in contrast to 81% at another site. With respect to average documentation time, one site had 40% of staff meeting the performance standard in contrast to 91% at another site.

Improved performance results can lead to increased site productivity. Further, answering a greater number of calls should improve service standard Footnote8 results for accessing VAC’s telephone service and reduce wait times for callers.

3.3.2 Service Standard

VAC service standards must be relevant to the recipient, measurable and linked to monitoring activities Footnote9 and must address the aspects of VAC’s services that are most valued and meaningful to the recipients of the service.

The current service standard for accessing VAC’s telephone services is defined as "Your call to our National Contact Centre Network (NCCN) will be answered within 2 minutes". Footnote10 The service standard is deemed to be met when 80% of calls are answered within this timeframe. The TBS Guideline on Service Standards states "publish results for clients". VAC has not published the service standard results since 2012-13 and, therefore, callers are not aware of current service levels. Table 2 below presents the Department’s service standard results from 2012-13 to July 2015.

Table 2: Service Standard Results for Accessing VAC’s Telephone Service
Criteria Result
2012–13 81%
2013–14 84%
2014–15 75%
2015–16 (as of July 31) 81%

Source: Strategic and Enabling Initiatives Directorate

3.4 Audit Recommendations

The following recommendations and management responses address the identified issues and will enhance service to recipients:

Recommendation 1 (Essential)

It is recommended that the Director General, Field Operations, improve the Department’s telephony tools by:

  1. reviewing and restructuring the IVR process flow, scripting and quality of vocal prompts and music; and
  2. implementing a more secure and rapid method of caller verification.

Management Response to Recommendation 1

Management agrees with this recommendation.

  1. Management implemented the following changes to the IVR to address the recommendation:
    • New IVR scripts, based on best practices
    • Simplified messaging and process flows
    • Scripts recorded in a professional manner in both official languages
    • New pick added for disability calls to better manage call routing
    • Ability to replay messages added
  2. A project is underway to develop password functionality in CSDN, which will be provided to Veterans as the main method for verifying the caller’s identity. An evaluation will be undertaken to identify other non-tombstone information available to NCCN analysts that they can ask the caller.

Recommendation 2 (Essential)

It is recommended that the Director General, Field Operations, improve workload management by:

  1. mitigating the impact of the FCR model on non-telephony related workload in area offices; and
  2. addressing the gap between NCCN Analyst performance expectations and results.

Management Response to Recommendation 2

Management agrees with this recommendation.

  1. New Veterans Service Agents are being hired in area offices. The telephony and non-telephony workload will be monitored by area directors and reported through to the Director General on a regular basis. A contract to review the First Contact Resolution model is about to be awarded, with an expected completion date of December 2016. Recommendations regarding VSA workload management, including indicators, will be considered at that time and implemented by spring 2017.
  2. NCCN analysts’ performance expectations are being revised to reflect the current workflows via phone and secure messaging. New expectations will be communicated to staff during the first quarter of 2017. Results will be constantly measured by Management over the next fiscal year to ensure any unit or individual gaps in performance are addressed through the performance management process.

Recommendation 3 (Essential)

It is recommended that the ADM, Chief Financial Officer and Corporate Services Branch, annually publish the current service standard for accessing VAC’s telephone service.

Management Response to Recommendation 3

ADM, Chief Financial Officer and Corporate Services Branch, agrees that the Branch will annually publish the current service standard for accessing VAC’s telephone service.

Recommendation 4 (Essential)

It is recommended that the Director General, Information Technology, Information Management and Administration, purchase an appropriate number of headset units to be stored and managed centrally in order to quickly respond to replacement requests.

Management Response to Recommendation 4

Management agrees with this recommendation. Since November 19, 2015, VAC has been purchasing and storing headsets centrally. Between November 2015 and March 2016, 248 headsets have been distributed from this inventory for use by VAC staff across the country.

3.5 Audit Opinion

The audit team observed that the relevant procedures and business processes were generally clear and up-to-date. Through field observations and file reviews, the audit team identified a high degree of compliance with procedures and business processes with staff carrying out their duties efficiently and exercising appropriate judgement to process requests. It was noted that improvements are required in key monitoring activities for Area Offices and quality assurance reporting. The audit team also noted gaps in coordination and monitoring of tools and equipment.

Overall, the audit team determined the results to be "Generally Acceptable". The opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that meet the auditing standards for the Government of Canada and were agreed on with management. The opinion is applicable only to the entity examined.

Definitions of recommendation rankings and audit opinions are available in Appendix B.

 
Date modified: