Accuracy
Last updated
Last updated
##Category Four: Accuracy
Ensure accuracy by carefully calculating targets to drive precise achievement. This usually occurs when there is a wealth of historical performance data and when leaders expect program managers to leverage descriptive and inferential statistics to set achievable targets that continuously improve performance.
###Signs of accuracy:
Baseline data exists.
There is internal consensus about the validity of the measure.
There is internal consensus about the validity of the target.
The measure and/or target can be shared internally and with the public.
The target is set at least annually and revised on predetermined intervals.
The target is always set beyond past performance in order to drive improvement.
The target is within reasonable range based on the evidence of past performance.
The target is forecasted for the entire performance period.
The target is constructed using a combination of descriptive and inferential statistics.
The target accounts for contextual variables (e.g., current budget environment, industry trends, policy changes, etc.).
People routinely discuss methods to improve target accuracy.
There is evidence of a relationship between the target and actual performance
Actual performance continues to improve.
#####Figure 5
###How to practice accuracy?
Leverage multiple datasets (variables). An accurate target is one that accounts for as many variables as possible that impact the performance measure. For example, when forecasting the number of permits a building department can issue, think carefully about the influence of time, productivity, staffing levels, and permit type on achieving those targets. They likely all contribute to performance, so leveraging data on as many variables as possible will increase the accuracy of target predictions.
Use proven data science practices. Predicting the expected performance is the holy grail of setting great performance targets, but doing it well requires more advanced data science techniques. After gathering all of the relevant datasets and conducting data hygiene (i.e., cleaning it up), initiate an exploratory analysis of the data (e.g., identifying measures of central tendency, variability, and dispersion). From there, consider using random sampling to conduct data modeling and statistical inference on the data to build a predictive model.
Run multiple what-if scenarios and sensitivity analysis. With an underlying data model, variables can be changed to run multiple scenarios. For example, how sensitive is the target to changes in staff size, case loads, or operating hours. This is a great time to engage senior leadership in defining the inputs to the data model, because it helps secure their buy-in for the final target if they understand the methodology behind its creation.
Pick an acceptable error range. No predictive model is perfect, so decide in advance how wrong it is acceptable to be. Error ranges can be statistically calculated and applied to the final target the same way a political polls have “margins of error.”
###How not to practice accuracy?
Do not get intimidated. Target setting can get advanced quickly, and the data science practices involved can intimidate even the best data enthusiasts. But the barriers to entry have never been lower for those trying to advance their data science and predictive modeling practices. The tools and training are cheaper than ever (many are free and online), and the community of practitioners is rapidly expanding.
Do not bend your target toward the actuals. As results come in, it is tempting to refine your targets up or down to match the actuals. This can prevent important learning about the accuracy and inaccuracies in the data model, and ultimately does nothing to inspire performance improvement.
Do not refine the targets too frequently. One way to prevent target “bending” is to set them and leave them unchanged, perhaps updating them quarterly (at most). Tinkering with the targets too frequently, in an attempt to be more accurate, opens the door for target manipulation.
Setting Targets vs. Forecasting
There is a subtle but important difference between setting targets and forecasting. Setting a target is defining where to aim. It implies aspiration based on capabilities. It is also intended to inspire improvement - always aiming higher and higher or closer and closer to the bulls eye. Forecasting is different. It is the science of predicting the exact mark most likely to be hit (e.g., where the hurricane will make landfall). As new data comes in about a program or service, revising the forecast is exactly the right thing to do; but resist the urge to revise the target. Instead, focus on understanding the variance among your targets, forecasts, and actuals so you can improve them moving forward.
