0 DIGIDAY:TARGET, or, Yogi Berra Meets Data in the Online World.

I was scheduled to attend the DIGIDAY:TARGET conference on May 4 but wasn't able to be there. (Download the conference agenda and presentations.) Happily, my colleague and big-data guru Matt Doering was able to take my place. Here are Matt's thoughts:

Yogi Berra meets data in the online world.

At the recent Digiday:Target conference (Park Central Hotel, NYC, May 4 2011) a moderator posed the question “Which is better: More Data, Consistent Data or Data Expertise”. Not surprisingly there was a wide variety of opinions both from the panel as well as from many attendees I talked to later in the day. Many I listened to were really intrigued and conflicted by this question. To understand the real answer let us first review the pros and cons of the three possible answers.

Background

More Data – Large volumes of data from varied sources.
Pros:
• Richer data content from any given data source.
• Data sources tend to enrich each other, if properly managed.
• More likely to find the outliers that many times can be the real profit makers.

Cons:
• Many companies don’t have the resources to handle very large volumes of data.
• Lack of Metadata about data sources.
• No real experience with merging multiple data sources with different element codes and timeframes.
• Data hygiene can be an issue if you are working with a data set that is new to your organization.

Consistent Data – All data conforms to some industry standard. Any data not conforming to the model is discarded or reduced.
Pros:
• All data is easily understood and documented in a Metadata stack.
• Data hygiene is easy to define and enforce.
• Data processing performance profiles are well understood. This makes it very easy to scope a system or project.

Cons:
• Let’s admit it; all homogenized milk tastes the same. Where is the differentiation potential?
• In the process of conforming to a standard more detailed data is lost. For example if the industry standard requires that age elements be bucketed into 10 year breaks what happens if for your product offering you need 6.5 year breaks?

Data Expertise – Deep experience with very large data sets.
Pros:
• Small data, large data, inconsistent data are not a problem. Expertise can handle all these issues.
• These resources understand the role that standardized data plays in data analysis (like a good coat of primer on a wall) but also know that the real value is in what is different.
• Most data experts love to teach so the entire data IQ of you organization increases.
• Able to distinguish between dirty data and gold nuggets.

Cons:
• These resources can be hard to find. It’s not a matter of having the right degree its more of who they are. Just as simply having a degree in fine arts doesn’t make you an artist a degree in stats doesn’t make you a good data scientist. In fact one of the best data scientists I know never took a stats course.

“Its déjà vu all over again”

Yogi had it right. If, as I strongly believe, data expertise is of critical importance for the media world it’s not the first industry where this is true. A number of industries over the past 25 years have had to deal with the “big data” problem. Early examples of this are the classic CPG scanner data, pharmaceutical detailing data and financial services direct marketing data sets. All these industries faced large and diverse data issues and they all succeeded in overcoming the problem with technique not CPU.

Now it might be tempting to claim that our space generates significantly higher volumes of data or more diverse data, but is that really true? At first this appears to be true, but when you factor in the computing power available at the times it is not that far fetched to say the adjusted data volumes are actually very similar. Keep in mind that the data scientists of those days were working with computers with less horse power and memory then the average iPad used by the majority of attendees at Digiday:Target.

So where do you find this expertise? Look to the industries named above. Membership of the Direct Marketing Association and those who attended the NCDM (National Center for Database Marketing) is a good place to start. Look for people from the telecommunications industry who helped build systems to analyze Call Detail Records (CDRs). Experience in genome sequencing and pairing should grab your attention. Do these people know clicks from conversions? Probably not, but on the other hand for them more data is the breath of life. We need to recruit the talent that is out there into the industry and avoid having to reinvent it “all over again”.

0 Privacy: Does Anybody Care?

To paraphrase HL Mencken, no one ever went broke underestimating the American public's commitment to privacy. "Quit Facebook Day" reportedly generated 31,000 account closings, compared with the roughly 500,000 new accounts that Facebook adds each day.

This lack of interest in privacy is a tremendous pity, because privacy violations can cause many types of real harm:

- identity theft
- physical violations including stalking and burglary when people are known to be out
- unjustified commercial treatment (e.g. denial of credit or employment) based on irrelevant or incorrect information
- unjustified government activity (e.g., placement on a No Fly list) based on irrelevant or incorrect information

Ironically, such problems seem to generate less public concern than techniques such as "behavioral targeting", even though the consequence of that is...um...receiving a relevant advertisement. I fully understand the real issue is people feel creepy to know that someone is sort-of watching them. But it's probably a good thing to remind them because the watching will continue whether behavioral targeting is regulated or not.

As the Facebook example shows, most people really don't care enough about privacy to protect it at the cost of other benefits, even minor ones like participating in Facebook. Similarly, many Americans seem downright eager to sacrifice their privacy from government surveillance in the name of national security.

The pity is that it's not an either/or choice. In many cases, technology can be designed to preserve privacy and still give the desired benefits. As a good example of what privacy-consciousness looks like when someone really cares, consider how the gun buyers are protected: gun dealers must check buyers' names against a database of felons, but the buyers' names are erased after a few days. (Of course, loopholes apply to "gun shows" but that's another discussion.) Another example -- never implemented so far as I know -- is that instead of reading drivers license information to prove patrons are old enough to drink, bars could have devices that simply scan the license and flash a green or red light depending on whether the person is old enough.

The point in both cases is that systems can be designed to access and retain the minimum amount of information necessary to fulfill their function. Many behavioral targeting systems already work this way -- capturing relevant data but not the actual identity of an individual. These principles could be applied more broadly and more systematically, but only if the people designing and regulating these systems made them a priority.

In practice it seems that other, less rational approaches are being adopted because they are more popular. To quote Mencken again, “For every problem there is a solution which is simple, clean and wrong.”

Without being excessively cynical, I think it's relevant to point out that privacy doesn't have much of a lobby, at least compared with, say, the National Rifle Association. Businesses want to collect data for marketing purposes. Consumer-friendly government officials are the natural opponents of this collection, but are constrained because many government agencies want the data for their own social and security purposes. The only organized opposition comes from a small set of privacy activists who themselves vary considerably in their priorities and capabilities. This means that, as a marketer, I don't spend much energy worrying about seriously restrictive privacy regulations -- even though I'd actually like to see some intelligent restrictions on data gathering by both business and government.