Lots more noodling in Blogistan about the “lies, damn lies, and statistics” of emerging media. As I draft my first column for a major business magazine’s online version on this very topic, I am gathering string.
First from Steve Safran at Lost Remote, via Scoble, is this commentary sparked by the zeFrank/Rocketboom nerd fight.
Lost Remote TV Blog
“There’s simply no way for us to measure viewership of podcasts. But we keep reporting numbers from the networks, big sites and podcasters without questioning them (guilty as charged) and we need to step back for a moment and ask: “How do we figure out who is really watching or listening to our podcasts?” Then we have to admit “We don’t know.”
Let’s back up a second and understand why this stuff is even measured in the first place (above and beyond bragging rights along the lines of mine-is-bigger-than-yours, which I call the “Time Warner PathFinder Effect” back in the day when Time Warner’s execs boasted about getting “millions” of hits the way McDonald’s quotes the nebulous statistic of “Billions Served). Why do we care about accuracy in media measurement? Especially since the old media measured crap like “reach” and “audience” based on the numbers of cars projected to crawl past a billboard on Highway 101 during rushhour and the number of Nielsen households who pressed the right button at the right time during the Beverly Hillbillies? Or magazines that claimed precision on completely freaky statistics like “pass-along” (which would seem to count a moldy copy in a dentist’s office about a gazillion times) or “recall?”
The reason that numbers matter, aside from the tyrannical rise of the “measure to manage” actuarials in the CFO’s office who worship at the altar known as “ROI”, is that marketers are still buying at the head of the long tail –where things like “mass” and “reach” seem to matter.
Now we find ourselves in the wonderfully mechanical world of web logs, when every hit, download, and interaction is logged by our Apache servers, and suddenly the Web world has been held up as the most accurately measured media in history.
Hah!
I’ve gamed web logs. Everyone has. I can pull some pearl out of a web log and say, “Aha, Left-handed Latvians prefer my site on Sundays!” Now, as we exit the era of Page Views and enter the era of Engagement, things get even squishier and gamier. Downloads versus views? Good luck.
My confrere, Jim Hazen, asks the simple question today about third party verification.
“Should there be some sort of official, universally accepted standards that all companies adhere to in determining true traffic? Like SEC accounting rules for Web Metrics? How bought a federally mandated web metrics tag for all sites?! Maybe some crazy alogrithm that can be based on Google searches or something, since they essentially run the web now. Not sure what the answer would be, but it’ll probably end up as another highly questionable reach calculation like the ones they’ve used forever with tv and radio.”
Jim, welcome to the world of ComScore, and Nielsen, and that total farce, Alexa.
I’ve said it before and I’ll say it again, the burden of proof and measurement can’t be abdicated to a third party measurement system. The Internet Advertising Bureau has fallen down in not presenting a solid set of standards for metrics reporting. The equivalent of the BPA or the ABC hasn’t emerged for online, and in the end, it comes down to the buyer has to beware. The only statistic that matters for a person renting eyeballs is this: did it work for me? Did the traffic to my site, the click through on that search term, the download of that funny viral yield anything of value to me.
In other words, stop pointing a radar gun at the pitchers. Worry about whether the catcher is on the ball.
The fact of the matter is that everything can be spoofed (ahem! “interpreted”) to serve some purpose. If some metric does not “show good” you can measure something else, say “this ‘visits’ look really low… only my mom went to the site, but she did download/purchase/view, so I have 100% conversions!!, now THAT looks cool”.
I guess there’s not much to do about it, it has happened ever since media exists, and it will keep the same way.
For those of us working on web-analytics, metrics, marketing, effectiveness, user satiscationess, convertioness, 2.0ness and weirdness; I think the challenge is to set up a diferent perspective; look things from the “half empty” department. Instead of tweaking figures to make us look good, tweaking them to make us look awful and take action upon it. Might be more honest.
The challenge for “catchers” is that most gloves don’t fit. Most web analytics tools are woefully inadequate regardless of slick marketing.
Web analytics tools evolved from reporting tools. Added later were data warehouses with cost-effective schemas, not so much “client-flexible” schemas built on open architectures. Summarized or sample data is often persisted over time (and reported). That just doesn’t work for the “New Web Order” and tracking this somewhat now annoying notion of Web 2.0…whatever that is…
First, data models and schemas must be augmented, especially the notion of page view, to include granular, subordinate “events.” Then when we have a structure and a way to describe these “events” like “click,” “roll,” “move,” “play,” “view,” “stop” and so on. Extensions beyond the js page tag, advanced hybrid models, and scripting languages for working natively with technologies like Flash/FLEX (which exist today in the marketplace) all help too.
Judah —
Totally agree. Current debate over metrics is based on the gross-tonnage/page view model. Tools, algorithms … all of it is based on static HTML models and has doodly to do with interaction and engagement. Watch a blog monitoring service try to subjectively indicate “sentiment” of buzz and you’ll see what I mean.
The new KPI, I believe, is comment and connection based. Not clicks.
Oh, come on, David. One of the hugest advantages of this medium at BOTH ends of the tail is measurability, and to abdicate that is silly when you have the most measurable form of media ever invented.
It’s just defeatist to argue that these things can’t be measured, or should only be measured on the other end. Not all advertising is transaction-oriented. Brand building still occurs, and not just through blogs.
If you sit and think about the nerd fight, and have a basic grasp of the technical underpinnings, it’s really not that complicated. But it takes a lot of time and energy to sit and think about the issues even BusinessWeek doesn’t seem to really understand what’s going on when the wrote about the issue on their blog, not does TechCrunch.
There probably will never be a perfect technical solution, but there will be something “good enough,” and it makes sense to always strive to improve it.
Ryan — my point isn’t that metrics are bullshit, on a relative basis they give some degree of precision. My point is that marketers who use existing page view/CPM/CTR metrics to base their buys on are missing the boat and need to meet those buys with, as you say, their internal metrics to map conversions against inbound clicks. The conversion that is not caught in the rapidly dying page view model is the concepts of attention and engagement. As zeFrank points out. He counts full views. Rocketboom counts downloads. Big difference.
I think interactive media is overweening about some paradise of realtime metrics driving campaign optimization and not looking at more substantive KPIs.
Don’t get me going on brand metrics. That is truly grenade/horseshoe measurement.
Interesting to note that ‘click fraud’ is quite common in the music industry. It is pretty well known that record label execs*cough*Diddy*cough* have paid people for years to “vote” online or via 800# for their music videos to be played on MTV, BET, etc. to raise awareness and drive sales. True we are talking about paid advertising here, but the ‘fraud’ concept is basically the same.
As I said on Hazen’s site, I think the somewhat old-fashioned practice of exit surveys “Where’d you hear about us?” is the most accurate measuerment of success.
Hey David, I see your point. Makes sense.