We all make decisions every day, yet seldom do we think about how we make decisions. Doug Hubbard in How to Measure Anything: Finding the Value of Intangibles in Business outlines common misconceptions pertaining to gathering information for quantitative decision making. Hubbard provides guidance on how to prioritize measurements based on the value of various uncertainties.
Misconceptions of Measurement
As suggested in the title, Hubbard claims that everything can be measured, provided one uses the correct definition of measurement. Hubbard suggests that people believe various things are immeasurable based on three misconceptions: the concept of measurement, the object of measurement, or the method of measurement.
So, which concept of measurement is the most misunderstood?
That a measurement yields a single value
A measurement should also include some amount of uncertainty. Mathematically speaking, a measurement is not a point in some Euclidean space, but instead a distribution. This distribution is a probability density function, which gives a notion of instantaneous probability. In this way, we can consider each measurement as a random variable.
The next common misconception is the object of measurement.
To overcome this misconception, you must take the time and effort to objectively define that which you are trying to measure. You may think that this is not always possible, but consider: the thing you are trying to measure must be perceivable. If this were not the case, you would not care to try to measure it in the first place. There must be some tangible outcome whenever there is more or less of the former thing.
One example of this is collaboration. In practice, we see everyone working more efficiently in a collaborative environment. So, we can get a reasonable estimate of the amount of collaboration based on the amount of work being done together as opposed to everyone working independently.
The final misconception of measurement is the method of measurement.
Most often this objection arises for obscure measurements, such as the number of undetected intrusions on a computer network. This seems impossible at first glance, but there are techniques to estimate this.
Let’s say you install two independent intrusion detection devices (hardware or software), then compare the number of overlapping detections to the number of detections total. In this way, we can estimate the number of undetected intrusions to the network. Having overcome these three misconceptions, Hubbard can conclude that everything, when properly defined, can be measured.
As noted with the concept of measurement, each measurement is a statement of uncertainty. We can also apply this to estimates of uncertainty. In general, people are not very good at estimating their own uncertainty in estimates, most of the time overestimating his or her confidence.
The good news is that Hubbard has a way to calibrate yourself to become more accurate.
Using Measurement to Improve your Decision Making
The method is simple: ask yourself to give your 90% confidence interval for various trivia questions, such as the average mass of a jelly bean. If you ask yourself many such questions, then about 90% of the correct answers should land in the interval you gave. If this is not the case, then you can repeat the process with another set of trivia questions and eventually you will get better at estimating your own confidence.
With this background in hand, Hubbard describes how one can use measurements to improve decisions. To start, you make a model of the monetary result of the decision consisting of several facets as random variables. This may require monetizing some aspects, which can be difficult, but with proper definitions is doable.
Once variables are monetized, you can run a Monte Carlo simulation to view many different possible scenarios. The simulation would yield an approximation of the probability distribution of the random variable which represents the model as a whole. From there, one can make a loss exceedance curve, which is a graph of dollar amounts and the probability that the random variable will be less than or equal to that dollar amount. Then it is up to the individual to decide what level of risk is acceptable. Moreover, should the probability distribution be too wide, you could compute which of the input random variables are contributing to the uncertainty of the total and then make more detailed measurements to try to narrow down that uncertainty.
Final Thoughts
On the whole, I think How to Measure Anything: Finding the Value of Intangibles in Business does a good job of explaining the shortcomings of many less quantitative methods as well as describe better alternatives. Hubbard breaks down bigger problems into component, bite-sized chunks. I do not consider the material to be inherently mathematically-challenging, but the logic could be applied to the most difficult problems and decisions.
