Reporting out

In 2001, the provincial Liberal's New Era campaign platform promised a Campbell government would "establish service plans that include measurable performance standards and targets." But, in 2005 and 2007, legislators reported low usage of those service plan reports - which are released as part of the government's February budget cycle. This, according to a recently-released five year study by University of Victoria public administration professor Jim McDavid.

The reason: among New Democrats surveyed in 2007, the service plan reports were seen by some as giving "government an opportunity to gloss over risk issues." And, because the material used to prepare those reports isn't independently audited, opposition members felt they could "be inaccurate and verge of being market puff pieces." As a result, they expressed a desire for the reports to be "more detailed and broader in scope."

By comparison, Liberals wanted to make them "simpler - to make them easier to read, shorter, more like a report card so they are more accessible to the public." Indeed, according to backbenchers, service plans reports constitute "too much reading" for private members. And there was a concern, in 2005, that the bureaucrats preparing those plans "learn the game so well that they give you what they think you want which may not be reality."

But one view New Democrats and Liberal do share is this: the public is the least likely group to use the government's performance measures. That being said, though, opposition and government members also found value in those reports. New Democrats acknowledged they included "comprehensive reporting that isn't available elsewhere," serving as a "reference when needed." And, for their part, the Liberals said the reports make government more "business-like." The following is a complete copy of the executive summary of Prof. McDavid's report.

***

Findings from a five year study of legislator uses of performance reports in British Columbia

Executive Summary

This report tracks legislator uses of the annual Service Plan Reports in the BC government from 2003 through 2007. Three anonymous surveys (2003, 2005 and 2007) asked MLAs to what extent they found the Service Plan Reports useful for 15 different purposes. These 15 uses cluster into five themes: accountability uses; information uses; improving efficiency and effectiveness uses, policy decision-making uses; and budget decision-making uses.

Key Finding: The Gap Between Expected Uses and Actual Uses

Overall, a key finding from the three surveys is the pronounced gap between initial expectations and MLA assessments of the actual usefulness of the Reports. For Liberal Government MLAs overall, the declines from the 2003 survey of expected uses to the 2007 surveys of actual uses, ranged from 35.9% for communication uses to 56.0% for budgeting uses. When results for backbenchers and Cabinet members are separated, only one exception to the significant declines is evident: for communications uses of the annual Service Plan Reports, the decline for backbenchers was only 24.5% (from a mean of 3.65 to a mean of 3.0).

Among the five clusters of uses, accountability uses dropped the least for Cabinet ministers, and communication uses dropped least for non-Cabinet MLAs. What dropped the most were uses focused on improving efficiency and effectiveness, making policy decisions, and making budgeting decisions.

In 2007, the Members of the Opposition reported that they generally used the annual Service Plan Reports less that do Government MLAs. The once exception was the accountability cluster of uses: Opposition MLAs reported somewhat higher use levels.

Explaining the Gap: Were Expectations Realistic?

One way to address the gap is to ask whether expectations were realistic. Performance reporting is part of broader efforts to reform governments to make them more accountable, efficient and effective. Advocates for performance measurement and reporting tend to underestimate the challenges in designing and implementing performance measurement and reporting systems. Resolving the technical challenges of getting the right measures, collecting reliable and valid information, and writing good reports do not address the people problems - getting buy-in and ongoing commitment in situations where reporting performance results is risky for the bureaucracies and for the Government. As well, research points out that legislators are so pressed for time that reading lengthy reports is a low priority. Further, the incentives they have in their roles and responsibilities tend not to reward performance measurement and accountability work.

In sum, expectations may not have been realistic in the BC Government, when the service planning and service plan reporting process was implemented in 2002 and 2003. But aside from this issue, there is the question of how much actual use MLAs are reporting.

MLAs Report Low Levels of Uses in 2005 and 2007

In the figures and tables presented in this report, the actual use levels in 2005 and 2007 are low. On a five point scale describing the 15 possible uses, possible values range from "have not used at all" (a value of 1 on the scale) to "have used to a great extent" (a value of 5 on the scale). Many of the averages in 2005 and 2007 are closest to "have used a little bit" (a value of 2 on the scale). For Cabinet Members, four clusters of uses (information uses, efficiency and effectiveness uses, policy-making uses, and budgeting uses) are closest to "have used a little bit". The only exception is "accountability uses" which tends to be somewhat higher in 2005 and 2007. For non-Cabinet MLAs, the only exception to this pattern is "communication uses" which tend to be higher. For the Opposition in 2007, four of the five clusters of uses are closest to "have used a little" but two are actually below that level.

Liberal Government MLAs and Opposition MLAs See Legislators as the Top Users

In the 2007 survey, MLAs were asked to pick the top four user groups for the annual Service Plan Reports. The key similarities between Government and Opposition MLAs are:

* both Liberal Government and Opposition MLAs picked legislators as the group most likely to use the Reports;
* both sides picked the public as least likely to use the Reports; and
* both sides of the Legislature picked government executives as important users (perhaps because their involvement in preparing the Reports).

The biggest difference was the importance of the media: 55 percent of Government MLAs said the media would be among the top users compared to only 27 percent of the Opposition.

Service Plans are not Used More than Service Plan Reports

Also included in the 2007 survey was a question about uses of the Service Plans that are published each February as part of the budget cycle. When uses of the annual Service Plan Reports are compared to uses of the Service Plans, there are no big differences among Liberal Government MLAs or among Opposition MLAs. There are significant differences between Liberal Government and Opposition MLAs in their uses of the Service Plans - Opposition MLAs use them more for accountability purposes and Government Members use them more for communication and for improving service quality.

Legislator Suggestions for Improving the Reports

The surveys included several open-ended questions that provide suggestions for improving the usefulness of the reports.

One theme that emerged from Liberal MLA comments was to make them simpler - to make them easier to read, shorter, more like a report card so that they are more accessible to the public. This theme suggests that the reports need to be succinct and compelling. On the other hand, NDP Members of the Legislature wanted the reports to be more detailed and broader in scope. Clearly the two sides of the House had different views of how much detail was desirable.

Another theme related to making the reports more useful was to layer the reports - having a short, simple version that is widely available, backed up by longer and more detailed versions for different audiences, perhaps even with appendices. These themes are echoed in the literature that makes the point that different audiences require different reports. The information needs that legislators have will differ even among themselves, and those needs will be different from the public, and other stakeholders.

Given recent (2008) changes in the format and content of the annual Service Plan Reports - the reduction in the maximum number of measures from twelve to six for each ministry, the reports will be pitched at a higher level than was true prior to this change. These changes tend to support the view, consistent with the findings from Liberal MLAs, that the reports are a primarily a communications tool and a high level commitment to being accountable, and less an aid to improving efficiency and effectiveness, or making policy and budget decisions.

What Does All This Mean for MLAs and for the Government?

Assuming finite resources for performance planning and reporting, simplifying the service planning and reporting cycle could make it possible to better address the internal performance management needs of agencies. The literature points out that when performance results are reported externally, and particularly when they are subjected to external audits, public organizations tend to decouple the external performance measurement and reporting process from the rest of the work that they do. Concerns with reporting results that somehow might reflect badly on the agency mean that external performance reports become quite bland and sanitized and tend not to be used for internal performance management

If the goal of measuring results is to get them used, it may be that the key users are internal to the organizations themselves. To facilitate internal performance measurement, and managing using performance results, there may be a case for formally acknowledging that external reporting will continue to have high level accountability and communication uses, particular for Government MLAs, and devoting resources to building internal performance management capacity.

Two Suggestions for Changes

First, the limited uses of the annual Service Plan Reports suggest that it may be time to focus the reporting process so that they are more useful for communicating with constituents and other stakeholders. Government MLAs suggested that short, simple reports are important to them and should be the centerpiece of the process.

Second, the literature suggests that public performance reports have very limited utility as a decision-making resource for managing performance within public organizations. The process of reporting externally is risky, and in our Canadian federal and provincial governance system, there are inherent pressures to avoid reporting shortcomings. Managers need performance information they can use and trust to adjust the work they do - and as need be, learn from their mistakes. In the 2007 survey, there were several suggestions that the resources that go into the current system be re-assigned. One way to do that would be to support internal performance management more, and recognize that public reporting is different from managing performance.

Leave a comment

Copyright © 2004 - Public Eye Mediaworks. Reproductions of any portion of this Website are permitted only with the expressed permission of Public Eye Mediaworks.
Canadian Web Hosting graciously provided by dotcanuck Web Services. Layout and graphics courtesy of Art Department Design.