Five Questions and Directional Answers

I asked five questions flowing out of eMetrics Toronto, posted on May 1, 2011.

Some editorial to contribute. No hard evidence by way of a survey yet. (You're all surveyed out anyway.)

Does the a culture of testing, if sustainable and feasible, drive incremental improvements in usability simple because the organization becomes more aware of usability?

• If the dependent variable is clear, uncontested, and lends itself to a direct attribution model, a culture of testing is likely to drive incremental improvements.
• There exists a 'bathtub' point where the marginal returns on optimization are consumed entirely by headcount and technology attempting that optimization.

Why don't/won't designers and analysts work together more often?

• The Excel communication medium hampers the relationship.
• The Photoshop communication medium hampers the relationship.
• The distance on the org chart between creatives and analytics/intelligence folk is the greatest barrier.
  

How should intra-video navigation (scrubbing) be isolated, treated and considered?

• Time spent scrubbing and waiting to load should be recorded and non-additive to total time spent. The entire purpose of the Time Spent Watching metric is to make traditional broadcasters feel good and smart (it's a cooptation metric). Since time spent scrubbing (changing the channels) is of marginal value to TV broadcasters (pre-Tivo, though some executives absolutely love this kind of analytics), it should be of marginal concern in digital, especially since you can't scrub through commercials online (a horrible design pattern since I've already seen the Ontario Trillium Fund commercial 45+ times in two weeks, and guaranteed, the Government of Ontario is ill serviced by such an idiotic saturation model.) The insight that may be harvested from such behavior is of analytical interest, which is why it ought to be, ideally, recorded as such.
• Great shows are cancelled based on another metric, not on scrubbing.

Will accountants and people in finance reach a point where they'll be comfortable with the ambiguity in marketing analytics?

• Likely. I've met one person who has made the finance-analytics transition. He's okay with it.
  

Does that mean we ought to sacrifice descriptive accuracy (accounts, http cookies) for descriptive simplicity (lead, customer, person, people)?

• Likely. If you accept commercial success as a leading indicator, simple descriptive wins every time.

I'll test this editorial on you, the reader, before I'll seriously start putting together a survey or some experiment.

Toronto Open Data, 311, and Google Refine

I grabbed this data from the Toronto Open data site. I loaded it into Google Refine. I used SPSS to understand just what was going on. I've stripped this post of political editorial, so if you're here for that, this post will dissapoint.

The story:

Always read the data dictionary and description. In this instance, I have a file containing a sample of a sample of all the service calls to 311. Toronto has a single call center routing system called 311. It's pretty efficient, in that it's a single department, and that any citizen can dial and report something, and get routed through to the right place. It's an example of very good policy learning.

The disclaimer is that only 25% of the calls to 311 are service in orientation. The data only represents about 25% of those service calls, and, it's not comprehensive. It's a pseudo-random sampling though, but the hand of manipulation appears to be quite heavy in this one. And it's for one month, between October 7 to November 7, 2010.

My experience with Google Refine was positive. I have a list of all the postal code prefixes in Toronto, so I was able to distinguish neighborhoods. The effort was much less error prone than what I'm accustomed to with SPSS. I was able to augment and clean at the same time. It's a quality utility, and I thank the product team behind it. Thank you.

I loaded it into SPSS to test a few hypotheses.

Toronto's residents like to call. A lot. The sample contains 21,000 calls.

They like to complain about garbage. A lot.

And that's pretty much what I will say.

The whole experience was fun, and I recommend others to do the same.

Making the Data Actionable

Nearly 3/4 of the respondents to the WAA Outlook Survey cited 'making the data actionable' a top concern and priority. Just under 1/3 of respondents reported using web analytics as an input into budget and planning.

Great. So what are we going to do about it?

To make something actionable, you have to understand what people are trying to action. And there's huge industry variance. KPI identification is at the core of what web analytics consultants and leaders do. So, that's all known, and not a research question. We know it.

And sometimes, the website just isn't that core to how a firm makes money. It could be. And relevance can be found through compelling business cases. That's all known and effective.

What is it that we don't understand?

We don't understand several general laws about the impact of analytical evidence on decision making and judgment.

Do we really understand the impact of previous evidence on current evidence (anchor and adjust)? Do we really understand why variation in web analytics communication exists? Do we really understand why some analysts scream 'that's not an insight' while others say 'that is an insight', in response to the exact same information?

So, I've raised a lot of questions. What of solutions?

Research Methods

Since joining the research committee years ago, I've personally tried to shy away from hitting the membership with surveys. We already have the periodic studies that rely on surveys. We have membership satisfaction, outlook, and the compensation studies - that are important, direct, and relevant. One other project, on the horizon, has collective intelligence written all of it, and I'm cheering that on. So we're good there. That instrument is in use.

We also have secondary research happening, by way of the Peer Review Journals, and that line of research is rocking.

So what else is available?

Simulation.

A reactor of sorts.

Let's expose different groups of people to different data structures and ask them to make decisions. Let's watch people make decisions. Let's put into place a reward for the best performance, and then measure the meta.

It's a question paired with a research method.

What do you think?

WAA Webcast Series, Industry Outlook

Amanda Wood and I will be presenting/paneling findings from the WAA Industry Outlook Study 2011 this coming Wednesday. Members of the WAA are welcome to dial in for it at noon.

Quite a few surprises and a few trends on what's happening out there. A few shifting priorities which are important to note. It's good timing for this webcast too, as many of us start to refocus for September and the usual Q3/Q4 madness.

Toronto Open Data Application: Wellbeing Map

The previous post in this space questioned if open data would make for better public policy. There are causes for optimism. Further to that, the City of Toronto just launched its wellbeing map, based on that open data. You can find it here.

Explore.

Will open government data cause better public policy?

There's a pretty big movement afoot in Canada. It's called the open data movement and several levels of government are getting on board.

It's the movement for governments to make large datasets freely available to the public.

It's pretty rough going right now. I'm reading reports that the sets frequently lack a data dictionary and suffer from some pretty bad accessibility issues. The early efforts are to be commended. I've spoken to several government statisticians who are both excited and frustrated by what they're able to share with the public, and where they're totally blocked. They're bullish on this movement.

These pains in the public sector mirror those in the private sector.

Will open data cause better public policy, and by extension, a better society?

I'm optimistic that it will in Canada.

For one, the evidence borne from the scientific method, as applied to datasets that everybody can see, are likely to be accepted for what they are, as evidence. We have a fairly well educated urban society that believe in pragmatic, incremental testing and an incredibly well developed rural society that uses science every day to build better livestock, soils, and crops. Nearly every sector of Canadian society is educated.

For two, evidence still matters in making many public policy decisions, at least, in many municipalities and provinces. Politicians will seek evidence, regardless if it's really about convenient reasoning, to back up decisions.

For three, more eyes on the same data, subjected to more analytical scrutiny, ought to generate more evidence and better insight. A generation of companies, emergent in my neighborhood, are motivated to generate new algorithms and services to interpret that data to generate profit first, and social goods second. More evidence, disseminated through to the public, ought to create a more informed public and better decision making.

Skepticism

If there's a desire for better public policy on the part of constituents, a desire for better evidence by government officials and representatives, and interest from analytics practitioners and entrepreneurs from the private sector crossing into this space, what could go wrong?

Many things.

For one, selling to the public sector is extremely risky. If practitioners can't imagine alternative niche markets to buttress their risk profile and ensure sufficient margin, then there will be less investment in that sector.

For two, spending on making open data more open may screech to halt owing to the deep spending cuts that will follow in an effort to correct nasty structural deficits that have developed.

For three, the chain of causality between open data, new value creation, better public policy, and a better society, is extremely long. Chains that exceed one link are hard enough to follow, little though prove. There's an awful lot of opportunity for unseen factors to disprove the theory.

On balance

The origin of many of the methods common in analytics goes back to the public policy debate around temperance. Evidence was important then. Evidence is important now. There are good indicators that open data can really work in Canada.

It's a great time to be in analytics and for public policy.