This quote from Seth Godin (Linchpin, 2010), sparked my thinking about metrics and outcomes. It parallels what I’ve quipped in presentations, “I’d rather have a fuzzy measure of something important than a precise measure of something unimportant.” In Measuring and Managing Performance in Organizations (1996) Rob Austin discusses the difference between desired outcomes and performance measures—and that those are likely to be in conflict with one another. Because of the difficulties in finding good measures for desired outcomes, we often substitute things that are easy to measure, but that cause long-term damage. Assessing the desired outcome of programmer productivity by measuring lines of code is a classic misuse of a metric.
Historically we assess project success, or failure, by measuring scope, schedule, and cost—especially cost and schedule because they are easy to measure (think earned value analysis). However, the desired outcome, from a customer perspective is to receive a releasable product, of acceptable quality, within a set of constraints (schedule or cost for example). For most projects, rather than asking, “Did we implement all the requirements (a scope question)?” the question should be “Can we release this product now?” Accessing release-ability (a value-oriented question) and quality may be “fuzzier” than measuring cost and schedule, but just maybe they are closer to representing the customer’s true desired outcome.