The recent Lac Carling workshop on measuring performance in government ESD reiterated that the challenge of evaluating public sector initiatives goes beyond the financial calculation of return on investment; rather, it includes more complicated issues related to the preservation of the public interest and the definition of, and movement toward, longer term strategic outcomes.
The workshop, led by Ann Gabriel of the Government of Ontario, included a report from a working group that had been formed at Lac Carling VI. The group had been asked to define the term ‘return on investment (ROI)’ as it applies in a public sector context, and to consider common tools and methodologies that might be considered by governments. The committee report defined the term and assessed its application to the public sector through an examination of current practices. Significantly, the report then raised concerns over the adequacy of ROI as a measurement of public sector ESD initiatives. The workshop discussion proposed further work in the area of performance measurement in the public sector, indicating that considerations of the broader public interest need to be included in a value-based framework for public sector ESD evaluation.
The working group report defined ROI as a financial calculation of quantifiable benefits, putting forth the following formula for consideration: ROI = average annual operating cash inflow (benefits) / net investment. Other financial metrics that can be considered include net present value, total cost of ownership, benefit cost ratio, discounted cash flow, internal rate of return and payback period.
The working group had looked at case studies from the three levels of government in an effort to assess how ROI was currently being used in measuring the effectiveness of ESD investments. The Government of Canada case studies measure efficiency on a case-by-case basis as ‘total return on investment, cost savings/avoidance opportunities and operational efficiencies from on line delivery.’ Examples of project assessment included cost savings as well as non-quantifiable benefits. Ontario’s Ministry of Community, Family and Children’s Services Business Transformation Project was able to use value-based measurement approaches that demonstrated concrete and measurable ROI. The third case study described the City of Edmonton’s e-business strategy: it requires a business case for each solution that includes ROI as well as other qualitative and quantitative benefits such as reduced operating costs, increased citizen satisfaction and improved quality of life.
The last two points indicate the challenges that governments have in measuring success in ESD investments. The blunt fact is that financial gain is rarely the sole measure of performance for governments. For one thing, the nature of government clientele is complicated: governments must serve all citizens, not simply those that it wishes to in order to reduce costs, and must sometimes treat its users as clients (for discretionary services) and sometimes as citizens (in the case of entitlements). Second, governments now realize, thanks to research such as Citizens First and its follow-up surveys, that citizens have expectations regarding service that go beyond cost: for example, they expect to be able to choose between the e-channel, telephone, walk-in and mail channels when accessing government services. Third, governments increasingly are concerned with what can be called the public interest, the meeting of longer-term social policy objectives such as ‘improved education’, ‘better health care’, and ‘relevance’ as part of their investment. These outcomes are difficult to measure. As the workshop report noted: ‘E-government initiatives deliver on both tangible and intangible policy goals’, a concept (it was noted in reference to the keynote speech given by Frank McKenna earlier) that is often difficult for Auditors-General and central agencies to grasp.
The working group then outlined three examples of emerging approaches to dealing with the need for governments to address both qualitative and quantitative concerns when demonstrating the benefits of investing in ESD. In the State of Iowa, benefits of IT projects are defined as those that include benefits to citizens, to government or to both; ROI accounts for ten out of a possible 100 points. Alberta’s Cost Benefit Assessment Framework for IT Projects includes ROI as a consideration in a quantitative analysis that focuses on the benefits of IT investments to the government. The third example, from Gartner Research, proposed the concept of value on investment (VOI), which Gartner defines as ‘the total measure of benefits derived from soft initiatives; ROI is a component of VOI.’ Finally, the workshop report referred to a recent document entitled ‘High Payoff in Electronic Government’ from the U.S. General Services Administration which concluded that the best way to measure the success of ESD initiatives is to assess each one relative to its specific objectives, including qualitative indicators such as user satisfaction.
Discussion confirmed that participants in the room believed that it was important to include quantitative as well as difficult-to-measure qualitative indicators related to issues such as outcomes and the public interest as part of the performance of government ESD initiatives. It was noted that, for governments, issues such as service quality and relevance (the importance that citizens attach to government) may in fact go beyond the need to save money and create efficiencies, and that a broader framework for public sector ESD assessment is required that includes these measures. Another participant said it was unfortunate that benefits are usually assessed and measured project-by-project, rather than through connected, horizontal and longer-term outcomes defined by government and citizens.
The working group indicated that it would continue investigation in this area for next year, and recommended that a repository of the research and best practices on this topic be established.
