Monday, April 26, 2010
Social Marketing Analytics, a response to John Lovett
This response is divided into three parts. It starts with a 'I see where you're coming from', then 'a few questions and inquiries' and then 'a few caveats and ways I'd improve it'.
First, I see where John is coming from.
John states, clearly, that "The objectives and metrics defined....in this report are a starting point for the infrastructure of social media measurement." (p. 6). The whole document then goes into a very transparent goal alignment strategy - where four business objectives are lined out based on a goal, then KPI's are identified out. He uses a circle-bulls-eye sort of device to describe that process (which I like, because it's a lot more accessible than a formal hierarchy, while retaining the relationships you have in a goal architecture framework). He then defines 12 KPI's and how they align back.
I'll say that I see where he's coming from. I've been practicing goal alignment strategy for the better part of my career - and this is a very disciplined one. The approach is excellent, and as analytics strategists, we'd all be better off if you used this general methodology.
Next, this quote: "By making learning and continuous improvement a primary goal, your social marketing activity will develop in a positive direction" (p. 6). It's one of the nicer ways of saying 'evidence based marketing should result in sustainable competitive advantage'. I enjoyed that passage too. It's a less jargony way of saying it.
John outlines four broad business objectives associated with social media marketing: foster dialog, promote advocacy, facilitate support, and spur innovation.
These are not the only objectives possible in social media marketing. He never said that they were. In fact, "Not all objectives and metrics will resonate with each audience nor will our foundational framework give you all the elements necessary for success." (p. 7).
It's at this point, again, that I accept where he's coming from. We set that aside, and we start to dig in.
Secondly, a few questions and inquiries:
The initial list of questions was around 100 deep, at which point I realized that there was little utility in going that far. Instead, I'll focus on just three points of inquiry:
The third metric, conversation reach, is defined as total people participating divided by total audience exposure. Are unique visitors people? Are unique visitor figures traceable? "Conversation reach can be evaluated in both volume and location across social media channels". (p. 13). Is this indeed the case? Can they? How are creators accounted for - in terms of actual conversing, as opposed to lurkers - or people who are observing the conversation (the exposure)? Is conversation reach better understood as the total number of people who have actually been exposed to the conversation, as opposed to the ratio between the participation and the exposure?
The fourth metric, active advocates, is a marketing one. I applaud John for using the word advocate over influencer (which I think blurs a fundamental marketing line). Could somebody be considered an advocate if they are constructively critical of the product and yet refer people to the product? Indeed, this is very common among innovators at the beginning of the product lifecycle. The devil remains in the term 'positive' and 'negative', and what an advocate is. The recency aspect is particularly excellent.
Which leads to the eleventh metric: sentiment ratio. First, is the positive/neutral/negative paradigm really indicative of innovation? Ie. Does it measure 'innovation'? As applied to a topic area, indeed, raw general sentiment scores have been used - but it's only done well if rigid topic-object hierarchy is identified. NextStage Sentiment Analysis (NSSA) is the closest that I've seen that takes into account additional dimensions over and above the straight positive/neutral/negative paradigm.
Finally - 'a few caveats and how I would improve it'.
A confluence of three thoughts. The first is Claude C. Hopkins who, eighty years removed, implored me to think of analytics and scientific advertising as a profit center, not a cost center. The second is Jim Novo (of course) who has been imploring us to link up with the CFO. The third is a baptism at Syncapse - which is the closest thing to a phd in management science that I could hope for and is responsible for reinforcing an underlining bias about innovation.
There should be three central goals with social media: to make money, to offset cost, and to realize sustainable competitive advantage.
I would improve the framework by calling that out: to make money.
There are many products that are high consideration and where word of mouth / social influence play a huge role. Try ordering a cheap malt liquor at Bier Markt on a Thursday night and watch the reaction from your developer friends. (What? No Delierium?). There is real money to be made in social marketing because the consumption of certain products is indeed a social exercise. Always has been. It's now, increasingly, in a medium where we can observe and quantify it (The actioning of that intelligence continues to be a sore point). I think that's what has really changed: the observable WOM.
Some of these metrics can be worked into a cause-effect model of that. Earned Media Value (EMV) might very well be an excellent metric as part of that cause-effect model. There will be no one-size-fits-all attribution model for sales driven by social. (At least, not within the next 2 years).
To offset cost is another one. And that's attractive with the current state of the economy. Cost offsets may very well be realized through the 'facilitate support' business objective.
Sustainable competitive advantage can be realized through learning and spurring innovation. The accumulation and actioning of intelligence and real insight is a huge key. To John's credit, he uses the term 'spur' innovation, not 'do innovation' or 'action innovative ideas', which is an organizational KPI best left to the mythical balanced scorecard.
There are other dimensions from a different paradigm, for a different time.
In general -
I applaud and thank John Lovett and Jeremiah Owyang for coming out with this. The approach is solid. You can make it your own. It's an excellent document for what it does.
While this is termed a 'response to John Lovett', I'd like to carry on this discussion through cross-blogs, in comments, and at eMetrics London in May with anybody who is interested in this area. There is so much to discuss.
Sunday, April 11, 2010
eMetrics Toronto Summary, Segmentation and Future
This post is long, and divided into three parts: a summary, a response to Glinski, and then a few thoughts about the next eMetrics summit to come to Toronto.
The first presentation was Theresa Locklear of the National Hockey League. She demonstrated just how far that team had come in just two years. She presented real data - how it's really presented - across multiple parts of the organization. I applaud that degree of transparency and I applaud her in particular for bringing her entire team. And her analytics team is simply brilliant. They're well inspired and well informed. Solid.
I'll start with the Quant/Qual mix panel on Wednesday night. There was no real controversy generated. I chalk this up to the panel being composed of five excellent, well balanced marketing professionals. Jim Novo expressed a strong preference of beginning with an insight derived from a quantitative method, exploring it with a qualitative method, and then communicating it to decision makers by way of a quantitatively derived story. If an ethnographer was around - well - we'd have a much less polite discourse. Issues with social media datasets was suppressed until the evening discussion.
Syncapse and Unica sponsored the elbow rub reception. It was well attended and I saw a lot of smiles. I met Jacques Warren in the flesh for the first time and instantly got along. Jacques is, essentially, the Avinash of the Francophone web analytics world. John Lovett, who I've talked to for years on the WAA Research Committee and Forrester calls/interviews, was there in the flesh. Major discussions centered on the Certification Exam, the impending INFORMS collaboration / collision, and how our economic ecosystem is changing.
The entire measurement science team from Syncapse was there too: Paul Cowan, Nadia Kim, Greg Araujo, Zoe Siskos, Hemash Bhatti, and Kevin Richard. A very entertained Meg Burns, my right brain, was pulled from circle of talkers to circle of talkers as we worked the room. Andrea Hadley, the tireless organizer of eMetrics Toronto, hosted a dinner, and, in an aspect that continues to make me really happy to be at Syncapse, our lead developer on Science attended.
The next morning we had the Avinash of the Anglophone world. That is - Avinash Kaushik himself. He was excellent, and will, someday, come with Patrick and I to the top floor of Bar Wellington. Avinash directly referenced that in sentiment analysis, positive/neutral/negative doesn't work - and that more dimensions were needed. Again we'll echo that point.
I spent time Thursday talking to Ned Kumar, Stephane Hamel, Jean-Paul Isson, Marko Hurst, Graeme McLaughlin, Mike Sukmanowsky, Simon Austin, Michael Helbling, and Breanna Wigle - among many others. Much of the discussion, impressively, centered on communication of insights, institutional models, education, strategic analytics and a few methodological concerns. In effect, talking about a single tool or a single vendor just didn't happen in any of those conversations. However, it was great to talk to prospective customers and the talent in our industry.
On Friday, Sionne Roberts asked me to join a panel at the SMX conference next door. He was nice enough to show me his notes in advance, and it went well. My POV on mobile search marketing is pretty simple - we're reaching an inflection in the adoption S-curve. When it hits, the cost-curve is going to shift quickly as we march up that hype-cycle. Same cycle is same.
Patrick Glinski then presented an excellent Aviva case study. There was real transparency there and beautifully communicated. You could see how the strategy was interwoven with the results. The imagery was beautiful. It was easily one of the best presentations of the entire conference. I'd love to see more of that. I sure wouldn't want to have followed Glinski. ;)
I presented real data from the WAA and Syncapse. Jacques was kind and called for more presentations like that. Novo sat in the back of the room and did his best not to shout or troll. His mouth was gushing with blood because he bit his tongue so hard.
The final panel was well attended. We had been joking that only five people would be there at the tail end. There were many, many more. And the discussion was open and lively - with the audience actively participating. And it was a power panel - Hamel, Warren, Lovett, Novo. It was a real honor to join them up there. And it was a really great to have dataminers: Emma Warrillow (the regent of weak-ties), Wigle, and June Li contributed heavily to the discussion. I think we expressed frustration that excel just wasn't cutting it and there are real issues with workflow.
Name dropping and summarization aside - I'll indulge in some cross-blog talk and some deeper discussion.
Patrick Glinski wrote on the IC blog:
On the “dread” side, I can’t help but feel this conference is going to be the end of my formal association with the digital measurement community.I’ve been a practicing Web Analytics analyst for about 6 years now – an old guy in digital terms. But even in that space, I always felt a bit like an outsider as a user experience strategist (researcher) first and an analytics practitioner second – a divide that is becoming an argument of validation versus prediction.
Meanwhile, many of the dominant conversations are distant and foreign. As Web Analysts, we focus our efforts on what will derivate the largest impact. Usually that means optimizing the user experience around the highest revenue-generating opportunities. Rather than focus our efforts on any one individual, we analyze patterns because it’s a better use of our time. But it seems as though we’ve lost our own way – now focusing in on technologies and methods designed to derive actionable results around strategies that most clients can’t afford to build in the first place.
I want to unpack that.

Glinski (bowtie pictured above) has left in his wake a thread of practitioners that I still work with to this day. I've inherited some of them. The digital measurement community is severely bifurcated. There's now (finally) an impressive cohort of people with 5 years+ of digital measurement and ux strategy experience. In almost every way - you can't be in the digital measurement industry for that long without picking up a lot of UX experience. I've always thought of this cohort as the 'second wave' of analysts - the post dot-bust generation that have been heavily influenced by the pre-busters that survived and by the new design thinkers. Glinski is part of that cohort. He's a big part of that cohort. What he's expressing is a frustration that nobody else really understands that cohort.
I'm relating to his experience. He's now a social innovation strategist. I'm now a strategic social marketing scientist. I can relate. However, I don't think that most people will.
There's now the post-Davenport wave that's coming into their own. They have a moderately different slant. I refer to people who have been heavily influenced by the book 'Competing on Analytics'. They tend to be people from a marketing or finance background (as opposed to a research background), who are beyond the tool and the people. They just want to get to the good stuff. They just want the good insights. They don't want to read a paragraph of text - unless of course they disagree with the insight - at which point they want material to disprove it. They want the Harrah's. They want the Fedex. And, they've been led to believe, rightly or wrongly, that it's a fairly efficient and easy operation. Analytics is easy. Davenport says it's so.
You also have people who are very new and are looking for tools. A big function of an industry that grows in the double digits year on year is that there are new people.
There's a big difference between analytics for the sake of analytics, analytics for the sake of convenient justification or perception control, and analytics as a strategic source of competitive advantage. That is to say, a difference between analytics as a means to a means, a means to a political end, and a means to a strategic end. The purpose of analytics depends on who you ask. Validation versus prediction? Oh yes. It's there. And it's common in DW and BI circles too.
Glinski writes on:"But it seems as though we’ve lost our own way – now focusing in on technologies and methods designed to derive actionable results around strategies that most clients can’t afford to build in the first place...
For the rich few, yes, these new methods and technologies can (and will) create a competitive advantage. But for the poor greater, this only further creates barriers to entry – introducing alienating languages, foreign skills, and high costs. I’ve heard many people in this industry talk about how wonderful Web analytics is because it’s accessible to marketers and (with a little help), it can be understood all the way to the top. Because of this, there is power in our recommendations."Glinski is right. The analytically rich are becoming richer. Most companies do not have a Goal Alignment Strategy in place or an analytics department. They do not have a unified understanding that spells out goals and business objectives and alignment with KPI's and a cause-effect architecture. Sushi menu style analytics is the order of the day, and many analysts, both on the BI side and the WA side, have taken to retreating behind a wall of no. The strategic - tactical misalignment persists well into 2010. The lack of a solid GAS remains a major barrier between reporting and analytics success.
And the creation of such alignment remains a foreign skill.
It continues to have very real consequences.
As for methodological innovation: it's necessary. The problems I've faced down in social analytics mandated the adoption of new methodologies and new ways of bridging the qual/quant divide. It isn't innovation for the sake of innovation. It's innovation for the sake of solving an actual business problem. And - innovation doesn't always have to be expensive. But it usually is because anything impacting the status quo always consumes a lot of energy. A big part of presenting the initial part of EMV was answering a very real business problem.
Glinski goes on:While I can’t be certain yet, I get the sense that digital measurement is going the way of the great black box – a fiefdom for a chosen few to understand. It’s also focusing on things most clients simply can’t resource against. While, yes, the decision support may push us towards greater accuracy, we can’t forget the politics behind all of this. I’ve been in enough Marketing Business Intelligence briefings to know (whether right or wrong) who makes the final decisions. In the world of mixed quantitative and qualitative data, whoever tells the best story usually gets the most attention. Black boxes don’t make for good stories.
Three points to tackle: black boxes, story telling and chosen few.
The push for black boxes is driven by profit and a fairly weak patent protection system. It's also distrust. I'm seeing more of it. Expect more black boxes because of weak patents and weak trust.
We don't do a good job at telling stories. The medium of powerpoints are designed to tell pretty good stories if we can make them sing. I've screamed repeatedly in this space that dashboards don't tell stories, and yet, they're our principle output. There's some innovation in terms of visualization to tell stories too. To Patrick's direct point, we're not doing a good job. I've asked for help for 5 years and spent much of that time helping myself while detractors boo and hiss. I certainly hope my detractors continue to enjoy clucking themselves and I continue to be thankful to those who help me tell better stories - creatives and the marketers on my team in particular. We don't have a systematic way of doing it, and what is common sense to great storytellers is a nightmare of ill-defined rules of thumb for those who are not.
If those who tell great stories are frustrated with those of us who don't - it's a two way street.
Finally - the chosen few and eMetrics.

eMetrics was a success for Syncapse because we worked really hard at it. I'm really quite introverted and I really don't like to meet unfamiliar people. So it's always a stretch for me to go out and work a room. To just walk up to a group of people standing in a circle, butt in, and immediately beginning to talk and shake hands is a stretch. I think it's a stretch for most others there too. It's a stretch.
It was also a stretch for much of my team. Except Kevin. I continue to be amazed by Kevin Richard's ability to work a room. Many people knew Syncapse before I got to them because Kevin already knew them and gave them the story.
I got some of the best value through these interactions with people. I get the best value when I generally stretch myself. Some of the presentations were very good. Some of the presentations were not relevant to my interests. I got to spend 45 minutes with one of the best minds in database analytics on the esoteric problem of behavioral WOM. (Yeah. For real!). I got to spend a solid hour with Wigle as we shared tips on social segmentation. I also got to spend an evening with two of the most knowledgeable minds in analytics. I hope that I've begun really solid professional associations and friendships. It was an incredible experience that was facilitated by the conference.
I also enjoyed talking to many of the newcommers. I loved hearing about their specific challenges, their questions, and helping them out. I also look forward in telling specific people more about what we got planned at Syncapse.
80% of the most valuable experiences happened outside the main conference.
Andrea Hadley and I have had many conversations about this effect while sitting outside in the lobby of another conference. (See??!?!?!?)
You have a group of people who don't want to talk about tools or black boxes. They want to focus on communication, institutional integration with UX and Qual, and strategy. These are your 5 year+ analysts. There aren't many of them. They have real problems and concerns, and there's real value in getting together with them, face to face, and talking about these types of managerial and strategic concerns.
So here's one of the paradoxes of what Glinski is alluding to:
You have a group of people who are grappling with issues like "how should UX and Evidence Based Marketers work together?", and another group that grapples with issues like "why doesn't comscore match up with Omniture and how do I explain that?". One subset of question isn't better than the other. They're just different. Because we're at different points in the lifecycle.
There are already fiefdoms because there are segments.
So how does eMetrics, as a web optimization summit, retain its pull?
I think it would be an incredible experience for 20 experts to gather into a room and talk.
The downside is that it would be a ghetto.
I would hate for the perception of the newcommers to be that we're all crusty, exclusive, and inaccessible. After all, I was a newcommer once too. And I benefited incredibly from having access to those experts.
From a marketer of Syncapse's point of view - it's twofold. We want good reach. We want to scout out talent and prospective customers. We also value getting perspectives. I also value having the team there so they can learn what others are doing and saying.
I don't know what to suggest to Andrea aside from some sort of hybrid approach to keep the Glinski's, Hamel's, and Dykeman's coming back and getting value. Both segments are worthy, equal, and should be facilitated and supported.
Would an advanced track, open to everybody, be satisfactory - even if many who came on in were ultimately alienated by the content?
Glinski's frustration doesn't appear to be just with his eMetrics experience. It's with digital measurement writ large. I don't think there are enough of us within the WAA, within INFORMS, or within the broader SiG community to achieve any sort of reach or impact. There might be all of 400 people on Earth who stand at the analytics/ux/social/strategy/business juncture: and 200 of them are in India. Not talking to each other. ;)
I don't begrudge the 6 billion others who aren't there with us, and who don't get it. And I don't judge the 50,000 who will start to get it really soon. It's a young industry.
That said, it's a young industry from which many move on.
The image macro below is about the best representation I can find to express my thoughts.

Thank you to Andrea Hadley, June Li, and Alex Langshur for an excellent eMetrics Toronto. If any of you reading this would like to contribute thoughts - feel free to post a common here or cross-blog-link it.