Truth and Numbers

A very smart person remarked that he liked numbers because they didn't lie. People lie about numbers.

Over the next 30 minutes, I demonstrated how two honest people can have two valid interpretations of the numbers, and have their models supported by the same facts.

An hour later, during our measurement science biweekly meeting, I invited the team to analyze a 5x5 RM table, and asked a fairly loaded question about it. Diversity in opinion eventually gave way to consensus around a mean. Several honest people had feedback and conflicting models about the way the world really worked. Each version perhaps more probably true than the last.

'Truth' is one of those really strange words in analytics. It's something we all claim to have ownership to. Some believe that they have exclusive ownership to it. You're entitled to your version of the truth. What you decide to do, however, with information that is contradictory to your position is the real differentiating behavior.

Successful people tend to have some sort of sustained competitive advantage. It involves the updating of current knowledge with new knowledge - much like sharpening a knife. If you're willing to accept that your version of truth could be updated with new knowledge, and it improves accuracy of your decisions, then real progress is made. And if you defend your right to be curious and challenge other versions of truth, it should be done in a way where knowledge expands.

I believe that some people really do try to pull fast ones with numbers. People lie. Which is why if the evidence you know to be true doesn't gel, then you're free to challenge and discard. And that's one of those more philosophical issues in analytics.

Proving something not to be true doesn't automatically make it false. It just might as well be not proven. Psychologically - it makes engaging with others around data sets to be that much more rewarding.

Truth is out there. Each more predictably accurate than the last.

The Onion's Satire on Web Analytics

"L.A. Law Wikipedia Page Viewed 874 Times Today", an article from the satirical media giant The Onion, is funny because it's painful.

The article starts off telling a story about irrelevant content. In this case, web analytics about a really old TV show on Wikipedia:

"Our L.A. Law page typically gets 915 views on weekdays and 670 on weekends, so we're about 40 off the pace," Wikipedia web moderator Ben Stern said of the entry for the Steven Bochco series, which hasn't aired a new episode since 1994. "Then again, the day isn't over, and if our metrics are correct, Corbin Bernsen's IMDB page should be viewed at least 15 more times before midnight. We generally get some runoff from that."

How often have you read something to the effect of:

"Of the 874 unique visitors today, 762 stayed on the site for less than 80 seconds, with 203 navigating to YouTube to view the L.A. Law opening-credit sequence. However, 366 remained on Wikipedia and clicked on the various hyperlinks within the entry, with 156 accessing the page for "in the closet," 42 clicking on the link for "Bentley," 328 viewing the entries for both "AIDS" and "mentally retarded," and 12 people consulting the article for "running gag," which has been viewed 64 times today."

The benefit of selecting Key Performance Indicators that are relevant to the organization is, in part, the mitigation of such paragraphs.

Evident Utility

One of my favourite sites is KillerStartups.com. It's everything I love about startup culture and innovation.

There are hundreds of independent variables that goes into explaining why some of these startups are going to thrive, and why most won't.

(It's more complex than biology because people are involved!)

My favourite variable is evident utility.

Each startup has two paragraphs to convince me to even click to learn more.

Do I see an actual use? Does it do something that somebody else already does in a better way? Cheaper way? Is it generalizable.

It's not the most predictive variable of success though.

Twitter is a good example of something I could see no evident utility for. Eventually I saw utility, at which point I joined. (After staying away from it for so long because I was suspicious of those who were raving about it).

Not perfect, but a lot of fun nevertheless to watch so many go at it.

Enjoy the site.

Web Analytics Wednesday Toronto Roundup: The One Sheeter Experiment

We did something very different for last night's Web Analytics Wednesday Toronto. Out with the invite was a strongly worded request to produce three bullet points on one sheet.

The hypothesis was that if you give analysts a platform for sharing some work with others, they will take it.

The expected outcome was lower turnout with a higher intensity of participation, and a higher perception of value.

Six sheets were presented by: Martin Ostrovsky (Repustate), Brian Cugelman (Alterspark), Kevin Richard, Heather Roxby, Greg Araujo, and myself (Syncapse). They were excellent and sparked very active debate.

Fifteen people in total came out, including web analysts (Mark Vernon, @web_analyst), creative (@mimc03), data miners (Gar et al), developers (@chrismendis et al), managers, directors (Linton), and measurement scientists (a portion of the team).

Specific feedback so far has included:

• "This is by far the best Web Analytics Wednesday I've ever been to"
• "It would have been nice if more sheets were on Web Analytic data"
• "You should ask for a list of 10 people to pull together one sheeters. You'll get more people out."
• "The sheets should have focused more on financial measures"
• "I don't think most people want to come out and talk about analysis after analyzing all day"
• "I learned something"
  

By the numbers, turnout was down 25% from what is normal in November. I know that some people stayed away because they didn't feel comfortable putting something together and talking to it. Or procrastinated on it until it was too late. Others simply couldn't make it which is normal and expected.

If we were going to do a straight trade-off analysis, the quality of the night was high. Conversation focused on outputs and even bled out into a challenge to survey everybody in the bar downstairs. (Until everything degenerated into a debate about survey methodology!) That was the flavor of the night, and, at least, it took on a much more real flavor. And it should be noted that people walked away with something in hand.

The quality of the discussion, which is usually very high, was excellent. There were concrete points made based on concrete data. On this criterion, it's a success.

Some advice to other organizers of Web Analytics Wednesday in other cities:

• Do a call for papers and check how many people will commit to putting a one-sheeter together. You can run a very successful event on 5 papers.
  
• Announce Web Analytics Wednesday and specify the time period that the one-sheeters will be presented. Include a call for additional one-sheeters so that it's democratic.
• Allow time for people to roll in and grab a beer and clump into their normal groupings. (Don't fight it.)
  
• If attendance exceeds 25, get everybody up on their feet and get people to circulate into their normal groups. Even at 15 people, groups of 6 will form. Keep it organic (ie. disorganized).
• As the organizer, you will have to moderate and grease things along.

A few variants that might be worth trying:

• Including a link to a given dataset and suggesting that a few people might want to take slants on that.
• Enabling multiple people to author a single sheet (teamwork).
• Publishing a communal data table and asking everybody to write three bullet points for the event.
  

In sum, it's a design pattern worth repeating, with modification.

Thank you to those who took the plunge and contributed and those that came out and added to the conversation.

The next WAWTO will be at some date in January, following the holiday season. There may be a call for papers forthcoming from that.