3.1 Application assessment
Call for applications
The Veteran and Family Well-Being Fund invites eligible organizations to submit an online application for grants and contributions. This invitation to apply is referred to as a call for applications. The Fund is a relatively new VAC program and, at the time of the audit, VAC had processed two rounds of funding. Per VACs website, eligible organizations may contact VAC to get notification of future calls for applications. Once a call for applications is made, organizations have several weeks to submit the online application.
For this stage in the process, the audit scope included only the second round of funding. We expected that VAC would have taken steps to ensure that potential recipients had access to information about the Program including application eligibility requirements and the criteria against which applications would be assessed.
We found that the application form itself was not available publicly (unless the applicant was registered to the online application portal); however, the Program objectives, assessment criteria, and other pertinent details were readily available on VACs website. Further, the team noted that VAC followed its standard communication processes at the time of call-out, including news releases, social media outreach, information for the National Contact Centre Network to answer questions, website promotion, and stakeholder outreach.
Assessment of applications
The second call for applications was held from February 28, 2019 to March 29, 2019 and resulted in 114 applications. Once the call for application closed, VFWBF staff reviewed and assessed each one. Two staff conducted the assessments, and they worked together to ensure consistency in assessments. After assessing projects for eligibility, VFWBF staff used a scoring tool which included supplemental guidance to assist in its application.
The scoring tool included a ranking scale of 1-5, from poor to excellent, on eight criteria: efficacy and safety, fit, resource availability, regulatory considerations, readiness, impact, cost, and need. The applicant’s score was the sum of the scores of each criteria. Program staff indicated that they had two weeks to assess the applications.
We expected VAC to be consistent in its application of the scoring mechanism, and we found that it was. However, we noted that the scoring tool allowed for much subjectivity and the guidance was too vague. Given the subjectivity of the scoring tool, coupled with minimal time to review applications, many projects received high marks. For example, one third of the applicants scored 30 or higher out of a potential of 40 points This did not allow for a large enough differentiation between projects when deciding on where to allocate limited funds.
Recommendation 1:
It is recommended that the Director General, Policy and Research Division improves the scoring tool and related guidance such that it results in greater distinction between projects.
Management response
The Director General, Policy and Research Division, agrees with this recommendation. The Policy and Research Division will either keep and update the current scoring tool or develop a new tool. Any changes made will include clear scoring definitions as well as consistent and objective measurement content which will ensure greater distinction between projects. The scoring tool and related guidance will be reviewed to ensure the criteria used in the evaluation links to the Fund’s stated objectives. Training will be provided to affected Program staff.
Target Date: August 1, 2021.
Once the initial assessment and scoring was completed, the Fund’s staff briefly consulted with VACs Research Directorate on the applicants that were more research in Audit of the Veteran and Family Well-Being Fund 5 January 2021 nature. The Research staff described the consultation as a high-level, cursory review because of the limited time available. A package of material was then prepared for the Directors General Steering Committee. This committee was tasked with assessing the information and providing recommendations on which projects to fund. The information package highlighted 32 projects recommended by the Program staff for consideration by the committee. The committee members were also provided with a brief description and scoring of all applicant projects. The committee met for approximately two hours to go over the information and identify which projects it recommended move forward for funding.
Interviews with staff and committee members revealed a common theme: the assessment process was rushed and there was little time for back and forth with applicants or for consultation with subject matter experts. Further, committee members indicated they were not aware of the final funding recommendations being put forward by the committee or of the ultimate funding decisions.
Recommendation 2:
It is recommended that the Director General, Policy and Research Division adjusts its processes to allow for adequate time for assessment and consultation. Management response The Director General, Policy and Research Division, agrees with this recommendation. The Policy and Research Division will develop a standard critical path with updated steps and timeline for each Call for Applications, taking into consideration the time required for assessment and consultation with VAC staff which are considered Subject Matter Experts.
Management response
The Director General, Policy and Research Division, agrees with this recommendation. The Policy and Research Division will develop a standard critical path with updated steps and timeline for each Call for Applications, taking into consideration the time required for assessment and consultation with VAC staff which are considered Subject Matter Experts.
Target Date: November 1, 2021
Funding decisions
The ADM of Strategic Policy and Commemoration Branch has been delegated the authority to approve projects for funding. We expected the department to maintain a record of decisions on funding projects in line with the application assessment results and in keeping with the Fund’s terms and conditions.
The audit found that the department maintained a record of decisions on approved funding projects up to the DG Steering Committee level:
- a master spreadsheet with the assessment and scoring of each application;Audit of the Veteran and Family Well-Being Fund 6 January 2021
- a detailed information package with highlighted projects for the steering committee;
- meeting minutes from the steering committee; and
- a listing of the final recommendations made by the steering committee.
In addition, there were documented ADM approvals for each successful project. However, the Program area was not able to provide us with records supporting funding decisions between the DG steering committee recommendation and the ADM’s final approval and seven of the 22 approved projects were not recommended by the steering committee. The lack of record keeping diminishes the transparency & credibility of the process and documentation to support decisions is lost for both future reference and to provide feedback to denied applicants.
Recommendation 3:
It is recommended that the Director General, Policy and Research Division ensures that documentation exists to support the rationale for final funding decisions.
Management response
The Director General, Policy and Research Division, agrees with this recommendation. The Policy and Research Division will review and assess what additional documents are required to improve record keeping.
Target Date: August 31, 2021
In general, approved projects met the Program’s terms and conditions. The audit did note a few instances where projects did not completely meet all requirements. Further, almost all of the applications did not demonstrate self-sufficiency post VAC funding, which is a requirement of the Program. The audit team questioned whether that requirement was still relevant given that essentially none met it.
One final note on documentation. The audit team expected that the department would document its rationale for decisions on the type of transfer payment, a grant versus a contribution, taking into account the application ask, the Program budget, and the stacking limits established under the Program’s terms and conditions. The audit found that the Program area does not document its rationale for grants versus contributions to support their decision and many organizations that applied for a contribution received a grant and vice-versa. The decision between a contribution or a grant is important and should be based on a risk assessment of the project. Projects deemed higher in risk Audit of the Veteran and Family Well-Being Fund 7 January 2021 would be awarded a contribution which would hold the organization to certain contractual obligations such as the submission of financial claims for reimbursement and progress reports. Projects awarded a grant would be of lower risk and, as such, have few requirements of the organization post funding. Documenting these decisions results in transparency in this step of the process.
Recommendation 4:
It is recommended that the Director General, Policy and Research Division ensures that documentation exists to support the rationale for decisions on transfer payment type.
Management response
The Director General, Policy and Research Division, agrees with this recommendation. The Policy and Research Division will develop a risk-based decision tool to be used to facilitate determining whether a grant or contribution will be used.
Target Date: February 28, 2021
3.2 Payments and ongoing monitoring
Once a project receives approval, a funding agreement is created and signed by both parties. Separate funding agreements exist for grants and for contributions. Many clauses are the same, but there are distinctions. The biggest difference between a grant and a contribution centres around the disbursement of funds.
Grants are paid up front and the recipient is not required to fulfill any future obligation. For the VFWBF, if a grant was a multi-year arrangement, the recipient did have to provide a progress report before any further disbursement of funds.
Contributions are typically paid after the fact, and the recipient submits a financial claim for reimbursement. Contribution agreements typically require certain milestones be met before funds are disbursed. For the VFWBF, typical milestones revolved around the submission of interim reports and financial claims. Many of the contribution projects were multi-year initiatives. Treasury Board policy does allow for advance contribution payments in certain situations, and there were three projects that received advanced contribution payments.
When projects are multi-year initiatives, the ongoing monitoring and the payments go hand-in-hand. Monitoring involves ensuring the project is going forward as intended and is on track with its plans, timelines, and commitments. It also includes reviewing Audit of the Veteran and Family Well-Being Fund 8 January 2021 submitted progress and final reports, reviewing financial claims for reimbursement, and communicating with recipients to validate and clarify information.
Payments
Grants and contribution payments have some internal controls in common. For example, regardless of whether a funding arrangement was a grant or a contribution, we expected that the department would have a signed funding agreement in place prior to any disbursements. We also expected payments to be made in a timely manner to avoid any unnecessary burden on the recipients. In all cases, we found that these expectations were met.
With contribution agreements, there are additional controls over disbursements of funds. Specifically, prior to any disbursement (or additional disbursement) of funds, recipients were required to submit progress reports, financial claims for expenditure reimbursement, and financial forecasts. We found that financial claims were always submitted; however, the department did little in the way of validating the expenditure claims, often taking information at face value. Further, stacking limits were not reassessed post-application as no revenue data is disclosed in financial claims. Even though financial forecasts were an explicit requirement of the funding agreement, Program staff only requested them for advance payment arrangements.
The Program area recognized the need to have greater scrutiny over expenditures and projects and had put forward tools / templates to guide these processes. We were advised that these changes were not implemented due to the Program’s lack of resources, staff turnover, and competing priorities. Without any validation of the project’s financial claims and progress reports, there is a risk that Program funds are not being used as intended and that VAC is funding Federal dollars in excess of Program expenditures. The Program area needs to balance its stewardship responsibilities with the administrative costs of increased monitoring.
Advanced contribution agreements are the riskiest form of transfer payment, and as such, the TBS directive outlines additional controls over these types of payments. The audit found that the Program area managed advance contributions in line with the requirements of the directive. However, as with all contribution payments, little was done to validate of the accuracy of the financial requests.
Recommendation 5:
It is recommended that the Director General, Policy and Research Division develops a risk-based approach to assess the accuracy and validity of information and claims submitted by recipients.
Management response
The Director General, Policy and Research Division, agrees with this recommendation. The Policy and Research Division will develop and implement a standard monitoring process for high risk projects pursuant to the Treasury Board Directive on Transfer Payments.
Target Date: March 1, 2022
On-going monitoring
On-going monitoring of projects is an important part of any grant and contribution program. It ensures that funding recipients are using the funds as intended, that they are on track with their projects, and fosters relationship building.
We expected VAC to have monitoring practices to ensure the funding recipients complied with the obligations and performance objectives in their funding agreements. The audit found that VAC obtained progress reports and monitored the progress of the projects. Audit team members noted evidence of correspondence with recipients to request clarification when needed. In general, all terms of funding agreements were met prior to disbursement of funds. The audit did note that there was a lack of guidance on appropriate monitoring procedures. Without documented procedures, there is no communication of approved and consistent processes and practices. In addition, corporate knowledge is lost when staff turnover occurs.
Given the number of multi-year projects, each call for application results in more and more projects to monitor at any one time. The Fund is maturing in its practices and now is the time to put more rigor around the monitoring processes.
Recommendation 6:
It is recommended that the Director General, Policy and Research Division documents the approved monitoring processes for grants and contribution agreements.
Management response
The Director General, Policy and Research Division, agrees with this recommendation. The Policy and Research Division will develop and document a standard monitoring process for projects.
Target Date: November 1, 2021
A complementary monitoring practice referred to in the TB guidance is recipient audits/ on-site visits. We expected that the Fund would have determined when recipient audits were necessary to complement other departmental monitoring activities and to have developed/executed a risk-based plan for these recipient audits. The audit found that the Program area had not incorporated audits into its monitoring activities. Further, no criteria had been developed to identify when an audit/on-site visit would be warranted. Having criteria for site visits/ audits allows for stronger stewardship of taxpayer dollars. Further, the department is missing the opportunity for outreach and relationship building with stakeholders/ Veteran organizations.
Final recipient reports were to be published to VACs external website. The intention was to share knowledge within the Veteran community and to foster relationships and networking. The audit noted that the recipient final reports were not published to the website, but there was a brief description of the project.
Recommendation 7:
It is recommended that the Director General, Policy and Research Division develops guidelines/criteria for recipient audits/ site-visits for higher risk contribution agreements.
Management response
The Director General, Policy and Research Division, agrees with this recommendation. In conjunction with the actions taken to address Recommendation 6, Policy and Research Division will develop guidelines for monitoring projects, including outlining guidelines for recipient audits / site-visits (in-person or virtual).
Target Date: November 1, 2021
Recommendation 8:
It is recommended that the Director General, Policy and Research Division determines whether publishing of final reports is necessary to the Veteran and Family Well-Being Fund’s objectives and updates the external website accordingly.
Management response
The Director General, Policy and Research Division, agrees with this recommendation. Policy and Research Division will be publishing one page summaries on VAC’s external. Work is currently underway to develop these products.
Target Date: April 1, 2021
3.3 Audit opinion
The Program Policy and Litigation Coordination division has controls in place to manage the day-to-day operations of the Veteran and Family Well-Being Fund. Opportunities exist to strengthen controls, specifically as they relate to documenting decisions and implementing more robust monitoring processes. The Veteran and Family Well-Being Fund is a relatively new program, and strengthening the controls that promote transparency and stewardship is important as the Program continues to develop.