Non-Performing Performance Indicators
It is shocking that, in quite a few countries which embarked on performance budgeting reforms years ago, performance indicators remain awful. Performance indicators are, after all, a basic building block of any performance budgeting system. Yet OECD surveys – most recently in 2018 – have repeatedly shown that in half of member countries, available performance indicators are largely irrelevant for budgetary decision-making. My experience in many developing countries tells me that the situation there is often worse. What is the problem?
The explanation is not that it can’t be done. There are quite a few countries which have, as part of their performance budgeting systems, developed excellent suites of performance indicators. France, Australia, South Africa and New Zealand are examples.
There is an old joke about a drunken man intently searching for his lost car keys on the footpath under a street lamppost. A police officer comes along and, after helping him search for a while, asks him whether he is sure that that is where he lost his keys. The man replies that, no, he lost them somewhere on the other side of the street, but that he is searching under the lamppost because the light is so much better there.
In quite a few countries, it has been exactly like that with the development of performance indicators. Ministries chose performance indicators based on the ready availability of the data rather than the relevance of the indicators concerned. The result is that the budget documents are stuffed with indicators such as the average time taken to fill vacant staff positions, average class sizes, and the number of business licenses issued – because these are things that are typically measured and managed as a routine part of day-to-day administration. (This means, to put it in technical language, measures of inputs, activities (i.e. work processes), and output quantity (i.e. volume of services delivered).) Virtually absent are indicators which are crucially important performance budgeting purposes – namely outcome (effectiveness), quality, and efficiency indicators.
I have seen this mechanism at work in many countries which are in the early days of developing performance budgeting. When undertaking the process of selecting performance indicators for programs, the focus is entirely upon indicators which be developed most easily. Because there is little data on outcomes, quality, and efficiency, few if any of these types of indicators find their way onto paper.
To avoid this, three principles should be applied in the development of indicators for performance budgeting.
The first is that the aim should be to develop, for each program, four types of indicator – effectiveness (outcome), output quantity, quality, and efficiency.*
The second is that important types of indicator should not be omitted simply because no data is available. If, for example, there is no outcome data, the government should nevertheless define the outcome indicators which it needs to develop – be they school student literacy levels, population vaccination rates, or air quality indexes. These indicators should then be included in the list of program indicators in the budget documents with an asterisk indicating that they are under development and will be reported in future.
The third and final principle is that indicators of inputs and activities are, generally speaking, inappropriate for performance budgeting purposes. Knowing how long it takes to fill vacant staff positions is important for internal management purposes, but has no place in a performance budget.
To ensure that these principles are adhered to, the ministry of finance should set clear guidelines for indicator selection and should enforce them, including by reviewing and approving all program indicators proposed by spending ministries.
Developing performance indicators that are truly helpful for budget decision-making is something which takes time and effort. There are no shortcuts here. Using only indicators for which the data happens to be on hand simply guarantees failure.
————————————————————————————————————
* Approximately speaking. There are a few “wrinkles” to this rule – for example, that it does not apply to “administration” programs. For more detail, see my 2011 performance budgeting manual.
Tweet
