How Communities Learn

Most communities have jargon. Buried within that jargon are all the biases, beliefs and worldview that are held by that community. (This can be referred to as a paradigm.)

The web analytics community is no exception.

The terms 'analytics', 'optimization', 'engagement', 'unique', 'pageviews', 'funnel', 'A/B Test Split Test', 'personalization', and 'filter' all have their own baggage that anybody outside the community might not fully understand.

Sometimes people get into disagreements over definitions in an effort to gain specificity. These activities are really quite important. An outsider might be mystified by why such disagreements become so heated. That's because sometimes the real fight is over the paradigm or some feature of the paradigm.

(For instance, the fighting over the term 'unique' was much more about the tension between accuracy and understandability.)

These shifts are indicative learning. I'm finally backed up on this whole hypothesis that language and learning are inextricably linked by Bickerton, 1995.

For instance: Google and Web Analytics. Let's set aside the disruptiveness of FREE for now (besides, it isn't free, because time has a cost), and turn to some of the new words.

"Filter" is one of them. It's about three years old now and is really just a proxy for the word 'dimension', which is a data warehousing / data modeling term. Now we use the term 'custom segment' instead.

Two new terms: "possible causal factor" and "statistically significant", have been recently introduced. I welcome the additions to the Google Analytics product.

Usually I need to export a large amount of data out of tools like Google, reformat them, and then load them into SPSS to look for 'possible causal factors' that are 'statistically significant'.

Now Google promises to democratize that process for all those who don't have SPSS, or know how to use such tools.

There's going to be a learning cycle where somebody will have to explain the difference between sampling error and Types I and II error and between confidence interval and confidence level. If we're indeed merging with the Business Intelligence and Data Mining communities - we'll need to learn a harmonized language. It might as well be the right language.

The process of how this community will learn will be contentious and heavily based in definitions.

We can be fairly certain that some vendor will try relabel a word like 'confidence' in an effort to get first mover status. Somebody will mislabel the word 'correlation' for 'causality'. And I'm fairly certain that we're going to spend two or three years undoing the damage.

This is basically how communities learn. Through jargon and discussion about what the underlining terms mean.

Enjoy!

Last Day At Critical Mass

Friday will be my last day at Critical Mass.

There are implications for what is written in this space.

For one, there was a large body of material that I simply self-censored. This will change.

For two, I'm anticipating that post volume will go up, at least in the short-run until I'm gradually consumed by this next role.

Much doesn't change.

You can expect the same length of posts and the occasional rants and use of images.

The subject matter will probably continue to focus on the meta and larger social issues around analytics.

Most of the relentless plugs for the Web Analytics Association, Web Analytics Wednesday, and TDMF will persist.

As for the next challenge:






I'm pretty excited.

My twitter is of course @cjpberry and if you need to get in touch, you know how to at me.

Survey Methods and On-Line and Off-Line Thinking

I'm on the final chapter of what has been a very difficult read: "Language and Human Behavior" by Bickerton.

He tackles some very difficult concepts in a clear cut way, with frequent deep dives into certain pockets of goodness. It's hard read because it's very dense, and perhaps I'm not horribly familiar with the subject matter.

The material in there about consciousness and the notions of On-Line thinking and Off-Line thinking are driving this post. I haven't figured out a way of expressing the differences in one paragraph or less without Bickerton finding out and reaming me out for getting it not quite right.

Into the meat of the post:

I frequently draw the line between observed behavior and reported behavior. One of the reasons for my caution with online satisfaction surveys is because it's reported by the user and frequently involves some form of prospection.

In an obscure reference, the Canadian Election Study, if taken at face value, would predict voter turnout several percentage points higher than it actually is at the ballot box.

That is to say, the survey predicts, based on the questions "Will you vote" and the post-survey "Did you vote" - a much higher rate of turnout than what really happened.

So, is the opt-in sample skewed (A person who is likely to fill out a massive survey about politics is naturally more inclined to vote anyway) or are people just very bad about prospection? (I told what I believed was the truth: I will vote. But the odds of me actually going to vote on voting day will be low.),

Or - did the survey actually raised some form of awareness in the person and made it more likely that they would actually vote: and the self-reported voting rate actually happens to agree with what actually happened to them at the ballot box. (Ie. they're telling the truth about their turnout).

I've frequently argued, quite unsuccessfully I might add, that a survey is unto itself a form of user experience that impacts perception. An on-site survey is one of the few ways that people can actually communicate with a company. After years of combing through comments and applying longtail analysis it becomes readily clear that a comments box is some sort of a cross between a help-desk box and an invitation to engage in 4Chan anonymous behavior.

Customers frequently see companies as being monolithic. Why wouldn't they? And why shouldn't expect a survey to be some form of vital communication instead of a research tool to make things better. Customers don't care. And I happen to agree with them.

It's for this reason that 'voice of the customer' online survey software is to be treated as a proxy for the truth and not as gospel. It has uses, to be sure, but it should be handled with care. The feedback contained within the survey is valid, and if the survey is constant over time - it can be used as a KPI. It has 'internal validity', but I'd become really uncomfortable about taking a sample size of 1000 and asking them "will you buy this product" and applying that rate against all visitation to the website. At least you're not guessing. (And we don't guess). But it is very dirty.

It's something.

I wouldn't bet the farm on a survey though.

The best feedback is observed. If you want to know what people really think and how they really feel - one should focus on watching them.

So - to tie this on back to Bickerton:

I prefer recording observed behavior because the user remains in a state of On-Line thinking. To borrow from physics: I'm not changing the position of an electron by measuring its speed.

Surveys have their place, to be sure, but they're inferior compared to other methodologies.

Creativity and Web Analytics

There's a review up on the Web Analytics Association's website on modeling the determinants of creativity in advertising. I think Smith and MacKenzie et al did a good job on the paper.

The term 'creative' is completely loaded. After all, isn't it all subjective?

In our defense, even as web analysts, we often try to quantify the subjective all the time. The feeling thermometer and the probability map are two ways that we've tried to quantify feelings and prospection. Even the concept of satisfaction, when operationalized through a survey methodology, is subjective.

Just because a concept is subjective doesn't mean that we throw up our hands and walk away. Rather, we should be always trying to improve how we ask and derive methods for linking perception with observed behavior. The denial (or ignorance) of this link between reported and observed behavior continues to generally plague the #measure community. (There's a bully in the community that I won't call out. Yet.)

The impact of creative on conversion is typically only spoken about in the context of an A/B test, and very frequently, only within a kind of spitting criticism. The oft-verbalized criticism of Google and their "testing of 140 shades of blue" methodology is one example.

Isn't creative more than just the color of a button or text though?

I think so. And so do Smith and MacKenzie. They break creativity out into 'divergence' and 'relevance'.

Let's tackle 'divergence' first.

Divergence means 'standing out'. What makes an ad stand out from the clutter?

I'd argue that it's the same thing that causes a punchline of a joke to be funny. Something unexpected. It's something that is at least two, maybe three standard deviations from the mean. Something that stands out from the crowd is spiky in nature. I'd argue that making something spiky is a creative process.

Relevance means 'of interest to me at this point in time'. It's another way of saying "right message to the right customer at the right time". We might well have successful delivery of said message, but if there's no divergence, that message will totally get lost in the clutter.

I'm arguing that there's an opportunity here to use web analytics as a force for good in the creative world. It wouldn't be a pursuit of sucking the 'fun' and 'creative license' out of the creative process. Quite to the contrary.

Rather, maybe we could incorporate creativity into our predictive and explanatory models - or at least consider and properly value proper creative.

NeuroCognitivePsychoLingualAnthropology

I read the first 120 pages of Joseph Carrabis’ new book “Reading Virtual Minds Volume 1” last night and polished it off this morning while sitting at the airport.

The book certainly forced me to think about being really aware of being aware of how hard I was thinking. I was engaged the whole way though, and in the end, I asked “wholly shit, what just happened there?"

I spent the better part of the night dreaming about it (always a sign that something upstairs is getting restructured).

I’ll write about the experience without spoiling it for you.

Joseph tells the story about how NeuroCognitivePsychoLingualAnthropology came to be. In spite of how long that word is, the book is very accessible, readable, useful, and intensely personal. The love leaps off many pages. (And one page where the middle finger literally leaps off the page. It’s not directed at the reader and it’s refreshingly honest.)

I’m taking away more than a few things that’ll become part of my every day speech.

The first is how NextStage’s machine runs. Joseph explains the principles of how it works specifically and uses accessible metaphors to expand. Those with an appreciation for collective intelligence and algorithm design will want to pay attention to how he explains it: it’s superior.

The second relates to political science and some of the social ills (suppressed political participation) that a good colleague has been trying to understand for the better part of a decade. There are applications of the technology that could explain what we think we’re seeing in the Canadian Election Study (CES). While I hope that Elections Canada and SSHIRC continue to fund the CES, NextStage offers a method of predicting a breakout election and perhaps a compelling explanation for turnout suppression. I haven’t been more inspired since reading “How Institutions Evolve”.

The third goes to marketing. It’s generally accepted that people think differently. But how differently? And do those differences matter? And if so in which contexts? The book gives a concrete example of how much and how it matters to marketers. The notion of intensity channels is a useful and accessible schema for quantifying those differences and acting upon them.

The fourth goes to changes to how we define experience design. On this point, you really need to read the book for yourself.

The next three takeaways are far more personal.

The first deals with a preference of mediums. One NextStage dimension is ‘visual’, and it explains a lot about me. I’d sooner go over to somebody’s desk and talk before writing an email before sending a text message before picking up the phone. In that preference order. And this includes literally hunting somebody own in a large office to find them in person. If the person is remote, I’d much rather use email. I’m that visual. So whether that means looking a digital signal, composed entirely of words with no tone: at least I can see the shapes of the words and the patterns. Thankfully the world is coming around with video chat.

The second deals with being intuitive and filters. Thankfully, Joseph uses as much common vocabulary as possible. We all know what we’re talking about when it comes to filters. There’s a reason why it’s acceptable to fart in certain social situations and it’s utterly unacceptable to so happen so much as speak a run-on sentence in another: even though they're both forms of passing gas. There’s a certain degree of self-awareness that goes with it: that a big part of understanding how others are reacting also involves the kinds of signals that you’re giving off.

The third will be the subject of future blog posts.

Just go get the book. It's a very good read and most of the people I know who read this space will find it valuable.