I have been in solid meetings the past two days and yesterday watched a presentation that reminded me of the story of the crash of Eastern Airlines Flight 401 in 1972.
The pilots were coming in for a landing but the “gear down” light didn’t illuminate in the cockpit. They tapped the light. They flipped switches. The co-pilot opened a hatch and climbed down to see what the problem was. They continued to obsess about the light but they didn’t notice when one of them bumped into the steering column and turned off the autopilot, putting the jet into a slow descent.
No one looked out the windshield. They were looking at the dead nose gear light.
Splat. 99 dead. 77 survived.
Metrics — the act of collecting data about systems and processes — and then reporting them in dashboards can lead to the type of tunnel vision those pilots displayed 36 years ago. The obsession with gathering status reports for the sake of gathering status reports can divert the organization and its people from the task at hand. If you’re trying to smelt gold but you spend so much time tracking ingot development that you fail to notice that you’re in fact smelting lead — then you’re going to be really good at ingot development, but oblivious to the quality of the final output. This is why formerly good things get ruined when big companies acquire them and start to obsess about the efficiencies. “We’ll just swap out the good stuff for the okay stuff and no one will notice.”
Subjectivity — the measurement of quality — is it good? is it bad? Do we suck or do we rock? Those unmeasurable intangibles are dismissed by technocrats as “feeling” behavior prized by people to sloppy to appreciate precision. Or, they attempt to quantify the subjective with surveys and stupid metrics like “sentiment.”
Objectivity — the measurement of facts — has become de rigeur ever since Neutron Jack Welch of GE set forth the commandment that you have to measure it to manage it. And so commenced the age of the tyranny of metrics. The Excel tyrants are really really good at demanding status reports and updates, but the reality is no one looks at their work and is terrified to say: “Go away. Here’s a beach. Start counting grains of sand and give me a TPS report by tomorrow.”
Metrics people — turn yourself into analysts by looking out the window and telling your boss the swamp is getting really close.
David…I couldn’t agree more. It amazes me how a hyperfocus on metrics gets in the way of what could be meaningful marketing activity. It is critical that we look at the output of our campaigns, but to do so with the blind faith that they are the entire story is incredibly dangerous and shortsighted.
Since you used an aviation metaphor…
Sometimes it makes sense to look out the window; other times there’s more useful information inside. Any flight through clouds involves careful attention to the instruments, because the view outside is useless. Instruments provide the information required to stay on course and right-side up when the windshield view is inadequate. More recent CFIT (controlled flight into terrain) accidents led to the adoption of ground-proximity and obstacle warning systems, which alert the crew to hazards such as cumulo-granite formations (mountains) in the clouds.
So, it depends. There’s good information out the window, but the instruments provide insights that the eye can’t always see. OK, the metaphor’s still holding up.
Flight 401 is routinely mentioned in discussions of Crew Resource Management (the other CRM), because the big failure was that the crew forgot every flight instructor’s constant refrain: Fly the plane. They all fixated on the light, instead (which was a burned-out bulb–the gear was fine). CRM teaches division of labor and coordination of efforts, and one outcome is that someone should be flying the plane while others are troubleshooting.
Applying CRM to the metrics/observation question suggests there’s room for both approaches, and companies shouldn’t fixate on either unstructured observation or metrics to the detriment of the other. It’s not either/or–it’s both, but it’s not necessarily one person’s job to do both.
Attended a breakfast discussion this morning full of venture capitalists and startups. On the way home it struck me that these folks face a (high stakes) version of the same challenge facing media companies today. There are lots of ideas. It’s difficult to know in advance which ones will work. So it’s difficult to know which ones to invest in.
Some people respond by only investing in ideas somebody else has already proven (i.e., metrics obsession).
That may be a way to make a few shekels, but it certainly doesn’t look like the way to strike it rich. The way to strike it rich is to have people with great judgment and great creativity, and collectively figure out which of the creative but unproven ideas deserve some chips.
I agree completely. To paraphrase Avinash Kaushik’s job description for the ideal metrics analyst on must investigate whether the candidate is just a sophisticated “numbers person” or has business acumen. The ideal mix would be 70% biz acumen and 30% numbers.
Correlation and Causation don’t always go hand in hand. More here: http://www.kaushik.net/avinash/2008/02/excellent-analytics-tip-12-unsuspected-correlations-are-sweet.html
Social Media Metrics beware: KPIs such as “Tone Index” are already sprouting up – but we have to start somewhere.
Tone Index — I heard that one today for the first time. Give me a break.
dc
I love this post / rant. I wish that more people in Senior positions were this enlightened and, more importantly, held their staff accountable for providing analysis and not just “reporting”.
The problem, well one of the problems, is that organizations expect predictability and “regular updates”. There is nothing predictable about analysis. Then how do you hold people accountable, and do you trust them enough to say “find me insights I can action each week”.
People are astounded when I say 30% of your “dashboard metrics” should die each year, if they don’t you are not working hard enough to understand your business and reevaluate whats most relevant now.
Am off to look outside the cockpit for a little while now. : )
-Avinash.
PS: Full disclosure: Am such a unabashed ThinkPad fan that I just bought my wife a fully loaded X61 just so we could have matching laptops! And of course I’ll do anything, *anything*, to get a X300 – even hold it up at every conference I speak at! : )
What I find frightening is that it is so easy to turn off the auto-pilot in a plane. I hope they have some sort of cover mechanism for the switch now thanks to this accident. Also, you’d think that during pre-flight check there would be a way to check for burnt out lightbulbs like a button that you press to make all lights come on (I suppose and hope every bulb is now replaced by LEDs).
Design is very important. In anesthesia, it used to be that in certain machines you turn knobs clockwise to increase the gas and in others counterclockwise. This has all been standardized to be in one direction, but you can see the problems this orginal lack of forethought caused as anesthesiologists moved from machine to machine.
Regarding metrics, most people prefer not to think and to just follow set procedures which just leaves room open for the competition:
“Most people would rather die than think; in fact, they do so”: Bertrand Russell
Regarding the X300, I think it is possibly underpowered with a 1.2 GHz processor and the Apple Air with 1.6 GHz was wiser in that regard. If Lenovo would only put a 1400×1050 12″ display in the X61(s) (as there is an an option in the X61 Tablet) that would be good enough for many.
For those of you not in the “Analytic Kool Kid Klub” – Avinash K. literally wrote the book on web analytics:
Web Analytics: An Hour a Day
http://www.amazon.com/Web-Analytics-Hour-Avinash-Kaushik/dp/0470130652/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1204925844&sr=8-1
Buy it. Read it. Live it.
I’m not worthy.
X300s for all!
David F: x301 …. I am waiting, even after spending two solid days next to David Hill, the X300 designer and having a serious case of new notebook envy. The thing is sweet.
Measurements should have a lifetime. Early in their creation there is a time where the measurement needs to be optimized so that a business decision can be made from the facts. As soon as possible it should transform from a report to a programed action where the business response is initiated.
The first cars require a mechanic to sit in the car an constantly make adjustment depending upon the situation and how long the car was on the road. We don’t need that now because all those activities are now part of the cars system. It allows us to focus on what is really important and as a result we can go faster, safer, and with more reliability.
The same goes for business measurements. When they are dealing with a new business environment or situation we need the measurements to understand what choices to make and directions to take, but as soon as we learn from then we should be able to “program†our business activities and get back to the imaginative and creative aspects of business.
I fear we are about to tip into TPS hell.
A minor quibble:
Subjectivity is not the measurement of quality. It is the measurement of beauty, sometimes referred to as aesthetics. Quality encompasses both the subjective and the objective. Objective qualities are facts; Subjective qualities are impressions. To properly distinguish between good and bad one must take into account both the objective and the subjective.
See Zen and the Art of Motorcycle Maintenance by Robert Pirsig
Spelling flames are lame, but the irony of this sentence is deep:
“Those unmeasurable intangibles are dismissed by technocrats as “feeling†behavior prized by people to sloppy to appreciate precision.”
Oh god preserve me from copyeditors and proofreaders ….