0 AgilOne Combines Marketing Database, Analytics and Execution: Yep, That's a Customer Data Platform

Well, this is embarrassing.

Here I am, all excited about discovering a new category of Customer Data Platform systems, which combine marketing database management, predictive modeling, and decision engines. Then I bump into Omer Artun, CEO of AgilOne , which he founded seven years ago to combine marketing database management, predictive modeling, and decision engines. It makes me feel much less clever.

But I guess I can’t hold that against AgilOne. As Artun tells the story, the company was created to provide marketers with a packaged, cloud-based version of the advanced data management, analytics, and execution capabilities that are usually available only to the largest and richest firms. The key is a set of 400 standard metrics, which AgilOne derives by mapping each client’s unique data into a standard structure. This, combined with advanced machine learning techniques, lets AgilOne build ten standard predictive models (engagement, next product, lifetime value, etc.) and three standard cluster models (products, behaviors, and brands) with minimal effort. The system builds on these to deliver packages of standard alerts, reports, guided analytics, individual customer profiles, and campaign lists. It also makes its data and predictions accessible to external systems such as call centers and Web sites via real time API calls, so those systems can use them to guide their own customer treatments.

This quick summary doesn’t do justice to the cleverness or sophistication of AgilOne’s approach. Clever, because the standardization allows it to quickly and cheaply deliver a full stack of capabilities, starting with database building and ending with advanced analytics, recommendations, and execution. Sophisticated, because it tailors the standard structures to each client’s business, so what it delivers isn’t some simple, cookie-cutter output.

Some of the tailoring is unavoidably manual, such as mapping client data sources to the standard data model. But much is highly automated, such as predictive models, clusters, and recommendations. I was particularly intrigued by the standard alerts, which look for significant changes in key performance indicators such as churn, margin, or average order value.  That sort of alerting is exactly what I've long felt marketers really wanted from their analytics tools.  AgilOne takes this a step further by automatically listing the data attributes with the greatest statistical impact on each item. The company refers to these items as goals to prioritize, which is a bit of a stretch – the most powerful variable isn’t necessarily the one marketers should focus on the most. But, as Damon Runyon said*, that’s the way to bet.

The system also recommends actions related to each alert, such as certain types of marketing campaigns. Again, there’s a bit less here than meets the eye, since the recommendations are drawn from a knowledgebase that’s the same for all clients. But that’s still better than nothing, and clients can customize their copy of the knowledgebase if they want.

The other especially noteworthy strength of AgilOne is data preparation. My original concept of the Customer Data Platform included customer data integration, which involves standardizing and matching customer records from different systems. I’ve pulled back from that because almost none of the vendors actually do such processing. Most assume it will be done elsewhere, or not at all, and only associate records with an exact match on a key such as a customer ID.  AgilOne does the hard stuff: quality checks, outlier detection, name parsing, address standardization, geocoding, phonetic matching, persistent ID management, and more. This is also highly automated and uses the company’s own technology. The lack of these capabilities prevents many companies from building a truly integrated customer database at many companies, so it’s extremely valuable for AgilOne to provide it.

If AgilOne has a weakness, it's at the execution end of the process.  Users can set up campaigns that generate lists on demand or on a regular schedule.  But I didn't see multi-step campaign flows or sophisticated decision management, such as arbitration across multiple eligible offers.  Some of that can probably be managed through advanced filters and custom models, which the system does provide.  However, making it truly accessible to non-technical users requires a specialized interface that the system apparently lacks.

While AgilOne just recently appeared on my personal radar, plenty of other people had already noticed: the company says nearly 100 brands are using the system. Sales efforts have been concentrated among mid-size B2C organizations, typically with at least 200,000 customers and $15 to $20 million in revenue. Pricing is published on the company Web site and is based on the features used and number of active customers. Entry price for the complete set of features starts around $9,000 per month.



_________________________________________________________________________________
*“The race is not always to the swift nor the battle to the strong, but that's the way to bet.” Runyon himself credited Chicago journalist Hugh Keough.

0 Customer Data Platforms: My New Whitepaper Explains the Excitement

You may have noticed that I've been uncharacteristically aggressive in promoting the Customer Data Platform concept.  Sorry, but I just can't help it: a new system category is even more rare than a new B2C marketing automation system (which, as yesterday's post pointed out, is much rarer than a new beetle).  More important, I think the category itself is a very important development that could really help marketers solve some big problems.  So it's worth several shoves to get the ball rolling.

Along those lines, I am embarrassingly excited to report my formal whitepaper explaining the CDP in depth has just been published.  It was sponsored by ReachForce but they had no influence on the actual content. (In fact, they probably would have preferred something a bit more on-message for their own marketing, so let me thank them for their indulgence.)  You can download it here.  Comments are welcome!

0 Provenir Adds Social Listening to Customer Decisions: Another Customer Data Platform

I’m still collecting examples to illustrate my new category of Customer Data Platform (CDP) systems. The latest is Provenir, a company founded in 1992 that has long sold a system to make credit risk and fraud decisions in real time. Over the past year, the company has added “social listening” capabilities and begun offering itself to marketing agencies as a customer interaction manager. It has met with good success and is now offering its “social listening platform” more broadly. *


It’s a slight stretch to call Provenir a CDP, because it doesn’t manage a permanent customer database.  Rather, like most interaction managers, it calls data from external sources during each decision.  But Provenir does have some customer matching capabilities and stores at least some information internally. Moreover, it completely meets the other three CDP criteria: predictive modeling, real-time decisions/recommendations executed through external systems, and a non-technical user interface. It’s also sold as the “glue” connecting data sources, modeling, and execution systems, which is exactly the role played by a CDP.  So, what the heck…welcome to the club!

Provenir is organized around process flows, which cover a particular task such as reacting to a Web site visit. Users define each process by building a flow chart, or, as the cool kids call them today, a graph.** These, um, graphs***, can contain branches, loops, and other advanced structures.  The nodes can also contain other graphs that define a subprocess in more detail. Nodes can perform a wide range of operations including data gathering, calculations, updates, decisions, and messages to external systems. Although setting these up is inevitably rigorous, Provenir makes it as painless as possible by providing help such as letting users draw lines to map fields from one system to another; building rules through score cards, tables and decision trees; and warning if a flow is incomplete.

Provenir relies on external systems to assemble, integrate, and store customer data.  Users can build matching processes with system graphs, although the vendor recommends connecting to other products to load reference data or do advanced "fuzzy" matching.  Provenir can monitor source systems for selected events and issue queries to assemble data as needed. The social listening features can monitor Twitter for keywords and Tweets by specified individuals.  These can trigger process flows that can retweet a message, send a direct Twitter message to the poster, or respond through another channel. The system can also monitor and post messages on Facebook. Other channels will be added over time.

Predictive modeling in Provenir is also done in external systems. The system can import PMML code or call models in SAS, R, or even Excel. Data mapping functions can automatically extract the list of required variables from PMML, do basic transformations and calculations when loading model inputs, and manage parameters, constants, and local variables.

Decisioning is Provenir’s greatest strength. The process flow…I mean graph…is inherently very flexible, and the ability to define rules as tables, trees, score cards, and other formats adds even more power. Users can set up champion/challenger tests as splits within a process flow; results are stored in a database for analysis and reporting. Users can also build simulated data sets, containing specified distributions of particular variables, and use these to forecast results of their flow designs. Such simulation is one mark of a mature decision system.

Provenir has some built-in messaging capabilities, but most decisions are executed externally.  The system has been connected with email, Web content management, call centers, campaign management, text messaging, and other execution platforms.

Pricing for Provenir’s social listening product is based on the size of the customer database. Starting price can be as a low as several thousand dollars per month. The system is usually sold on a Software-as-a-Service (SaaS) basis, but on-premise licenses are also available.


_______________________________________________________________
* For extra credit, compare and contrast Provenir’s primary Web site  with the site for their listening division.

** Defined in Wikipedia as “mathematical structures used to model pairwise relations between objects”.

*** Would it be even cooler to call them grafs or, better still, grafz?

0 I've Discovered a New Class of System: the Customer Data Platform. Causata Is An Example.

It has taken me a while to connect the dots, but I’m now pretty sure I see a new type of software emerging. These systems that gather customer data from multiple sources, combine information related to the same individuals, perform predictive analytics on the resulting database, and use the results to guide marketing treatments across multiple channels. This differs quite radically from standard marketing automation systems, which use databases built elsewhere, rarely include integrated predictive modeling, and are focused primarily on moving customers through multi-step campaigns. In fact, the new systems complement rather than compete with marketing automation, which they treat as just one of several execution platforms. The new systems can also feed sales, customer service, online advertising, point of sale, and any other customer-facing systems.

Given how much vendors and analysts love to create new categories, I’m genuinely perplexed that no one has yet named this one. I’ll step in myself, and hereby christen the concept as “Customer Data Platform”.  Aside from having a relatively available three letter abbreviation (see Acronym Finder for other uses of CDP), the merits of this name include:

- “Customer” shows the scope extends to all customer-related functions, not just marketing;
- “Data” shows the primary focus is on data, not execution; and
- “Platform” shows it does more than data management while supporting other systems

But, you may ask, is this really new? Certainly systems for Customer Data Integration (CDI) have been around for decades: these include specialized products like Harte-Hanks Trillium and SAS DataFlux, CDI features within general data management suites like Informatica and Pentaho, and integration within cloud-based business intelligence products like GoodData and Birst. Many of those products have limited capabilities for working with newer data sources like Web sites and social networks, but the real distinction between them and CDPs is that the older systems are mainly designed to assemble data.  Some also provide analytics, but they don't extend to real-time decisions based on predictive models.

Similarly, there have long been specialized systems for real-time interaction management (such as Infor Interaction Advisor and Oracle Real Time Decisions) and for predictive modeling (SAS, IBM SPSS, KXEN). Some interaction managers do create predictive models, and the really big vendors (IBM, SAS, Oracle) have all three key components (CDI, real-time decisions, and predictive models) somewhere in their stables. But systems that closely couple just those features with the goal of feeding data as well as recommendations to execution systems? Those are something new.

By now, you’re probably wondering if I’ll ever get around to actually naming the vendors I have in mind. I’ve recently written about some of them, including Reachforce/SetLogik and Lattice Engines.  I also include RedPoint in the mix, because it has all the key capabilities (database development, predictive models, and real time decisions) even though it also offers conventional campaign management. Others I haven’t yet written about include Mintigo and Gainsight. Of course, each has a different mix of features and its own market position.  Indeed, several have specifically told me they do not compete with the others. Fair enough, but I still see enough similarity to group them together.

All this is a very long-winded introduction to Causata, yet another member of this new class. By now, you can probably guess Causata’s main functions: assemble customer data from multiple sources, consolidate it by customer, place it in an analytics-friendly format, run predictive models against it, and respond in real time to recommendation requests from other systems including Web sites, email, banner ads, and call centers. And you’d be right.

But that’s not the end of the story. With any product, it’s the details that matter. Causata is particularly strong in the data management department, accepting both batch and real-time data feeds and storing data as different types of events (email sent, Web site visit, call center interaction, etc.), each having predefined attributes. The system also has a particularly sophisticated “identity association” service, which looks for simultaneous events involving different identifiers as a way to link them, and can chain identifiers that were linked at different times. When I spoke with Causata about two months ago, the association rules were pretty much the same for all clients, but they promised users would get more control in the future. Users could already choose which types of associations to use in specific queries.

Causata stores the assembled data in HBase, a Hadoop-based database management system that is particularly well suited to large data volumes, many different data types, and ad hoc queries. In addition to the raw data, the system can store derived values such as aggregations (e.g., number of Web page view in past 24 hours) and model scores. Users can run SQL queries to extract data for analysis and predictive modeling in third-party software including QlikView, Tableau, SAS, and R. Prebuilt QlikView reports show the predictive power of different variables for user-specified events. The lack of native analysis and modeling tools creates some friction for users, but also lets them stick with familiar products. So the pros and cons probably cancel each other out.

The system’s decision tools are straightforward. For each situation, users define a “decision engine” that can select among multiple options, such as campaigns, products, or marketing content. These options can have qualification rules. To make a decision, the system can test the options in sequence and pick the first one for which a customer is qualified, or pick the option with the highest predictive model score. Users can also specify a percentage of customers to receive a random option, to gather data for future decisions. An engine can return multiple decisions for situations that require more than one option, such as a Web page with several offers. Causata has some machine learning algorithms to help with the decision process. It plans to expand these to automatically select the best option in a given situation.

Decision engines are called by external systems through a Web services API that can respond in under 50 milliseconds. This is fast enough to manage Web banner ads – something not all interaction managers can achieve. Model scores and other data are updated in real time during an interaction.

Causata can be deployed on-premise by a client or as a cloud-based service. The vendor says a typical implementation starts with three or four data sources and is deployed in about 30 days – very fast for this type of system. In February, Causata introduced prebuilt applications for cross-sell, acquisition, and return programs in financial services, communications, and digital media. These will further speed deployment.

Pricing is based on the number of data sources and touchpoints, with additional charges based on data storage. Cost begins around $150,000 per year.

0 RedPoint Offers Broad, Deep B2C Marketing Automation

On days when I have nothing else to be cranky about, I sometimes fuss at how business-to-business software vendors hijacked the term “marketing automation” despite its long and relatively honorable history describing systems for consumer marketing. Oracle’s recent agreement to purchase Eloqua reinflamed that wound, since much of the commentary ignored Oracle’s extensive existing suite of consumer marketing systems.

More productively, the deal also shifted discussion from components within a marketing suite to where marketing systems fit within the larger world of unified customer management. This perspective has always been part of consumer marketing, where the classic description of Customer Relationship Management (CRM) was “marketing, sales and service”. That formula fell out of favor when the most prominent CRM system became B2B-oriented Salesforce.com, whose very name reflects its origins in B2B sales automation.

One reason that B2B perspectives have dominated the recent discussion of marketing automation is that the major B2C marketing automation products have receded from view after being absorbed by larger corporate platforms: Unica is part of IBM, Aprimo is owned by Teradata, Portrait Software is owned by Pitney Bowes, Epiphany is owned by Infor (which last month acquired marketing resource management leader Orbis Global, a bit of news I'd missed), Alterian is owned by SDL, and SAS is owned by, well, SAS. This diminishes most products’ independent profiles (Aprimo is the exception), although they are still selling nicely. Even some of the less prominent B2C products have been acquired recently: Conversen by Experian and Entiera by FICO, although neither of these are enterprise software vendors.

The net result of all this harvesting by big buyers has been to clear the ground for the next crop of B2C marketing automation systems. These fill a demand for powerful but moderately-priced options by both marketing services providers and mid-sized companies. Those buyers often find that systems from big enterprise software vendors become too expensive or require too many ancillary components from their corporate parents.

One vendor taking advantage of this opening is RedPoint. Founded in 2006 by veteran CRM consultants and technologists from Accenture, the firm has created an exceptionally extensive marketing automation product including not just campaigns and content management, but also database maintenance, which isn't usually part of marketing automation.  Although IBM, SAS, and Pitney Bowes provide a similar scope, RedPoint is unique in having built all those components itself and keeping them tightly integrated.

The data features are especially impressive. RedPoint offers a rich set of standard data management features including loads from files, databases, and semi-structured formats like XML and EDI; transformations including regular expressions, file splits, and table joins; batch and on-demand process flows.  There's also a novel option to use Web service calls for on-demand data appending, a particularly noteworthy concept.  Beyond those, the system provides specialized functions to manage customer data including name/address parsing, standardization, and matching. Users can apply the system's built-in rules for these or modify the rules to meet their own preferences. All data functions are managed with a sophisticated workflow engine that allows fast development of new marketing databases -- often in a matter of weeks, not months.  This removes (or at least shrinks) the single greatest obstacle facing many new marketing systems.

The system was also designed to scale.  It can work with standard files and SQL databases, with native connectors for SQL Server, Netezza, Oracle, and Teradata. Support for Hadoop, Hbase, PIG, Hive, Cassandra, GreenPlum and other “big data” technologies is due early this year. Clustering is available at all application service layers and the system supports true multi-tenancy (multiple instances running on the same installation).

Campaign management features are equally deep although less unusual.  A drag-and-drop flow builder allows complex, rule-based branches and can react to behaviors during the campaign. Users can define audiences with batch selections, scheduled processes, and “subscription groups” of anonymous individuals (such as Web site visitors) who match specified behavior profiles. Splits can be based on logical conditions, random selections, or auto-generated cells with all possible combinations of specified variables. The auto-generation feature, which can easily produce segmentations with hundreds or thousands of cells, is a hallmark of sophisticated B2C marketing automation systems. It is used for segmentations such as RFM (recency, frequency, monetary value) cells, age/gender/cluster cells, branch or dealer assignments, and product splits.

RedPoint also provides self-training predictive models.  These are currently used within the data management functions and matching algorithms. They will be soon applied to select offers for Web personalization.

Content management includes an editor to create outputs for email, Web, SMS, Twitter, Facebook, FourSquare, LinkedIn and other formats. Users can build shared templates that are later modified for individual projects. Objects can contain Web forms and dynamic content blocks driven by selection rules or mapped to audience segments. The system can generate Web tags to capture user behaviors and can react to those behaviors in real time. It manages approval workflows, version tracking, and precise control over which users can access different objects and functions.

Pricing for RedPoint is based on a combination of deployment services, software licenses, and hosting fees. A minimum system starts around $60,000 per year. There are more than 100 installations, about half sold directly and half through partners such as marketing service providers. The system can be hosted by RedPoint, by a service partner, or by the client.

0 SetLogik Offers B2B Marketers a Real Marketing Database

I’ve now done more detailed research into the SetLogik B2B data management system I mentioned in my Dreamforce post.  If anything, I’m even more impressed.

I originally saw SetLogik as a tool to associate marketing leads with sales opportunities, even when they are not connected directly within Salesforce.com. That’s important in itself, since those missing links are the greatest obstacle to showing the value of B2B marketing efforts through revenue attribution.

But the bigger story, which SetLogik itself recognizes clearly, is that they’re creating a real marketing database. This has been sadly lacking in most B2B marketing automation systems, which supplement the Salesforce.com database with barely-extensible lead profiles and contact histories. In fact, I’ve recently taken to citing the B2B systems' fixed, built-in database as the fundamental difference distinguishing them from B2C systems, which connect to externally-managed databases with any structure.

SetLogik doesn’t replace the database built into the B2B systems.  Rather, it creates a separate database that merges data from marketing automation, Salesforce.com (or, potentially, any other CRM system), and whatever other sources a company has available. The matching capabilities that initially caught my eye are just one part of a larger suite of functions to load, clean, standardize,  merge, and enhance B2B data, ultimately storing it within a database where it can be used with SetLogik tools for segmentation, selection, reporting (including attribution), and predictive model-based lead scoring. Cleansed data and results such as lead scores can be fed back into CRM and marketing automation systems for direct access by their users. SetLogik’s own diagram expresses this separation reasonably well, although I would have suggested they clarify that there’s an independent, persistent database within their cloud.
 

As consumer marketers learned long ago, building a serious marketing database is a big project. The challenge is even greater in B2B, which manages two data levels, companies and contacts, instead of just one level of consumers.  It’s no wonder that B2B marketing automation vendors avoided the issue by piggybacking on the Salesforce.com structure: otherwise, the cost and complexity of building a separate database would have severely limited their growth.

SetLogik’s addresses the problem directly, by creating a nearly-automated system to build the database. The company promises to deliver a completely functional database within 60 days, and to deliver the database plus predictive lead scoring models in 90 days. Compared with the many months or years needed to deploy a traditional marketing database, this is lightning quick.

I call the system “nearly-automated” because a SetLogik analyst works with each client to set up the data preparation steps, tweaking the standard rules and processes as necessary, and because the predictive models are also built by human analysts. These are advantages, not flaws, since a skilled user adds substantial value to both processes. The system still does most of the work, so the initial data quality set-up takes just a few hours of labor – although the full process typically takes several days because clients need time to make decisions. Similarly, modeling takes about two weeks – again, more wait time than work time.  In fact, the model building is so efficient that the company includes it for free in its Enterprise edition, which starts at $1,400 per month for up to 25,000 records.

None of this would matter if the quality of SetLogik’s results were poor. But, while I haven’t run a test, the company certainly describes the features I'd want. Standard inputs include leads, contacts, campaign members, accounts, and opportunities from Salesforce.com, plus leads and activities from marketing automation. Data preparation includes standardization and verification of addresses in the U.S., Canada, United Kingdom, Australia; phone numbers verification for North America; email format verification (but not sending test emails); table-based transformations and coding for elements like titles and sales territories; and enhancement with client-licensed external data such as D&B listings.

The matching engine uses multiple rule sets, supports both similar and exact matches, and can compare data across several fields (such as mobile vs. home vs office phone number). The system will match at individual and company levels and can link individuals to companies. It will choose the best value for each field and return a consistent best record to all source systems. Predictive modeling can include derived variables, such as number of emails received, as well as raw inputs. The system’s database stores snapshots of old data values so it can track changes and trends. New and changed records run through the system at user-determined intervals that can be frequent as hourly.

The system doesn’t provide an interface for end-users to set their own data processing rules, although one is planned. As SetLogik correctly argues, very few B2B marketers have the interest or skills to do this. In fact, the company’s larger problem is that so few marketers even recognize they need better data cleansing, let alone a separate marketing database. This will likely limit SetLogik's initial clients to the upper tier of sophisticated marketers who do see the problem.  We can hope that the importance of a serious marketing database will eventually become clear to everyone.

SetLogik is a Software-as-a-Service application, available directly from the company or through the Salesforce.com AppExchange. The system has an Eloqua connector today and a Marketo connector in the works. The company promises basic implementation in 60 days, although it is usually much less, and full implementation including predictive modeling in 90 days.

Pricing is based on the system edition and number of records (unique individuals and companies). The Express Edition, starting at $500 per month for 25,000 records, builds the database and feeds the cleansed, enhanced records back to Salesforce.com and marketing automation. Professional Edition, starting at $1,000 per month, adds segmentation, list building, attribution, and other reporting. Enterprise Edition, starting at $1,400 per month, offers all the other features plus predictive lead scores. The price tag gets more serious for large systems – Enterprise costs about $11,000 per month for one million-records – but is still much less than a conventional marketing database. In fact, SetLogik points out that some services built into the price, such as address and phone verification or access to lead profiles within Salesforce.com, would ordinarily cost nearly as much as the entire SetLogik fee if purchased separately.

SetLogik officially released its system in October 2011 and now has several large enterprise clients.

0 More Surveys Agree: Web and Non-Web Data Must Be Integrated

It’s not that I’m obsessive, but just to gnaw a bit more on last week’s bone about the coming unification of Web and other marketing data...

- a recent survey sponsored by enterprise marketing automation vendor Unica found that “integration with other marketing solutions” was the most commonly cited web analytics challenge (46%).

This was followed by “verifying accuracy of data (inflation/deflation)” (41%) and “not comprehensive/missing types of data” (32%). It’s interesting that the question was about Web analytics in particular – even analyzing Web results by itself requires non-Web data.

- another survey by another marketing automation vendor, Alterian, found the most-commonly cited top obstacle in online marketing (25% of respondents) was “integration of online with database marketing and offline channels”.

Now, this isn’t exactly the same answer as the Unica survey, since the question isn’t limited to data integration. But Alterian also reported that “lack of ability to assess or manage internal infrastructure and culture challenges (25%) and the integration of all the technology to power the cycle (20%) were identified as the biggest factors in implementing the customer engagement cycle.” So clearly data and system integration are indeed primary concerns for multi-channel marketing.

For still more on this topic, see today’s post on the MPM Toolkit blog, describing a very detailed report in eMarketer about Online Brand Measurement. This provides still more evidence for the need for integration of Web and non-Web data, in addition to other issues.

Case closed.

0 Denodo Helps Mesh Enterprise Data

Every now and then you read something along the lines of “more words will be published in the next 30 minutes than in the whole of human history up to the battle of Lepanto”. Whether this is literally true or not (and who could know?), there is certainly more data sloshing around than ever. This is partly because there are so many more transactions, partly because each transaction generates so much more detail, and partly because so much previously hidden data is now exposed on the Internet. The frustration for marketers is that they know this data could be immensely useful, if only they could employ it effectively.

This is the fundamental reason I spend so much time on database technology (to manage all that data), analytics (to make sense of it) and marketing automation (to do something useful with it). Gaining value from data is the greatest new challenge facing marketers today—as opposed to old but still important challenges like managing your brand and understanding your customers. Since the answers are still being discovered, it’s worth lots of attention.

One subject I haven’t explored very often is mining data from the public Internet (as opposed to data captured on your own Web site—the “private” Internet, if you will). Marketers don’t seem terribly concerned with this, apart from specialized efforts to track comments in the press, blogs, social networks and similar forums. Technologists do find the subject quite fascinating, since it offers the intriguing challenge of converting unstructured data into something more useful, preferably using cool semantic technology. It doesn’t hurt that tons of funding is available from government agencies that want to spy on us (for our own protection, of course). The main marketing application of this work has been building business and consumer profiles with information from public Web sources. Zoominfo may be the best known vendor in this field, although there are many others.

But plenty of other interesting work has been going on. I recently spoke with Denodo, which specializes in what are called “enterprise data mashups”. This turns out to be a whole industry (maybe you already knew this—I admit that I need to get out more). See blog posts by Dion Hinchcliffe here and here for more than I’ll ever know about the topic. What seems to distinguish enterprise mashups from the more familiar widget-based Web mashups is that the enterprise versions let developers take data from sources they choose, rather than sources that have already been formatted for use.

Since Denodo is the only mashup software I’ve examined, I can’t compare it with its competitors. But I was quite impressed with what Denodo showed me. Basically their approach is to build specialized connectors, called “wrappers,” that (a) extract specified information from databases, Web sites and unstructured text sources, (b) put it into queryable structure, and (c) publish it to other applications in whatever format is needed. Each of these is easier said than done.

Denodo showed me how it would build a wrapper to access competitive data exposed on a Web site—in this case, mobile phone rate plans. This was a matter of manually accessing the competitor’s site, entering the necessary parameter (a Zip code), and highlighting the result. Denodo recorded this process, read the source code of the underlying Web page, and developed appropriate code to repeat the steps automatically. This code was embedded in a process template that included the rest of the process (restructuring the data and exposing it). According to Denodo, the wrapper can automatically adjust itself if the target Web page changes: this is a major advantage since links might otherwise break constantly. If the Web page changes more than Denodo can handle, it will alert the user.

As I mentioned, Denodo will place the data it retrieves into a queryable format—essentially, an in-memory database table. It could also copy the data into a physical database if desired, although this is an exception. The data can be sorted and otherwise manipulated, and joined with data from other wrappers using normal database queries. Results can be posted back to the original sources or be presented to external systems in pretty much any format or interface: HTML, XML, CSV, ODBC, JDBC, HTTP, Web service, and the rest of the usual alphabet soup. Denodo can join data using inexact as well as exact matches, allowing it to overcome common differences in spelling and format.

The technicians among you may find this terribly exciting, but to most marketers it is pure gobbledygook. What really matters to them is the applications Denodo makes possible. The company cites several major areas, including gathering business and competitive intelligence; merging customer data across systems; and integrating business processes with Web sites.

Some of these applications resemble the integration-enabled interaction management offered by eglue (click here for my post). The difference is Denodo’s greater ability to access external data sources, and what I believe is a significantly more sophisticated approach to data extraction. On the other hand, eglue offers richer features for presenting information to call center agents. It does appear that Denodo significantly lowers the barriers to many kinds of data integration, which should open up all sorts of new possibilities.

The price seems reasonable, given the productivity benefits that Denodo should provide: $27,000 to $150,000 per CPU based on the number of data sources and other application details. An initial application can usually be developed in about two weeks.

Denodo was founded in Spain in 1999. The company has recently expanded outside of Europe and now has nearly 100 customers worldwide.

0 What's New at DataFlux? I Thought You'd Never Ask.

What with it being Valentine’s Day and all, you probably didn’t wake up this morning asking yourself, “I wonder what’s new with DataFlux?” That, my friend, is where you and I differ. Except that I actually asked myself that question a couple of weeks ago, and by now have had time to get an answer. Which turns out to be rather interesting.

DataFlux, as anyone still reading this probably knew already, is a developer of data quality software and is owned by SAS. DataFlux’s original core technology was a statistical matching engine that automatically analyzes input files and generates sophisticated keys which are similar for similar records. This has now been supplemented by a variety of capabilities for data profiling, analysis, standardization and verification, using reference data and rules in addition to statistical methods. The original matching engine is now just one component within a much larger set of solutions.

In fact, and this is what I find interesting, much of DataFlux’s focus is now on the larger issue of data governance. This has more to do with monitoring data quality than simple matching. DataFlux tells me the change has been driven by organizations that face increasing pressures to prove they are doing a good job with managing their data, for reasons such as financial reporting and compliance with government regulations.

The new developments also encompass product information and other types of non-name and address data, usually labeled as “master data management”. DataFlux reports that non-customer data is is the fastest growing portion of its business. DataFlux is well suited for non-traditional matching applications because the statistical approach does not rely on topic-specific rules and reference bases. Of course, DataFlux does use rules and reference information when appropriate.

The other recent development at DataFlux has been creation of “accelerators”, which are prepackaged rules, processes and reports for specific tasks. DataFlux started offering these in 2007 and now lists one each for customer data quality, product data quality, and watchlist compliance. More are apparently on the way. Applications like this are a very common development in a maturing industry, as companies that started by providing tools gain enough experience to understand how the applications commonly built with those tools. The next step—which DataFlux hasn’t reached yet—is to become even more specific by developing packages for particular industries. The benefit of these applications is that they save clients work and allow quicker deployments.

Back to governance. DataFlux’s movement in that direction is an interesting strategy because it offers a possible escape from the commoditization of its core data quality functions. Major data quality vendors including Firstlogic and Group 1 Software, plus several of the smaller ones, have been acquired in recent years and matching functions are now embedded within many enterprise software products. Even though there have been some intriguing new technical approaches from vendors like Netrics and Zoomix, this is a hard market to penetrate based on better technology alone. It seems that DataFlux moved into governance more in response to customer requests than due to proactive strategic planning. But even so, they have done well to recognize and seize the opportunity when it presented itself. Not everyone is quite so responsive. The question now is whether other data quality vendors will take a similar approach or this will be a long-term point of differentiation for DataFlux.

0 Low Cost CDI from Infosolve, Pentaho and StrikeIron

As I’ve mentioned in a couple of previous posts, QlikView doesn’t have the built-in matching functions needed for customer data integration (CDI). This has left me looking for other ways to provide that service, preferably at a low cost. The problem is that the major CDI products like Harte-Hanks Trillium, DataMentors DataFuse and SAS DataFlux are fairly expensive.

One intriguing alternative is Infosolve Technologies. Looking at the Infosolve Web site, it’s clear they offer something relevant, since two flagship products are ‘OpenDQ’ and ‘OpenCDI’ and their tag line is ‘The Power of Zero Based Data Solutions’. But I couldn't figure out exactly what they were selling since they stress that there are ‘never any licenses, hardware requirements or term contracts’. So I broke down and asked them.

It turns out that Infosolve is a consulting firm that uses free open source technology, specifically the Pentaho platform for data integration and business intelligence. A Certified Development partner of Pentaho, Infosolve has developed its own data quality and CDI components on the platform and simply sells the consulting needed to deploy it. Interesting.

Infosolve Vice President Subbu Manchiraju and Director of Alliances Richard Romanik spent some time going over the details and gave me a brief demonstration of the platform. Basically, Pentaho lets users build graphical workflows that link components for data extracts, transformation, profiling, matching, enhancement, and reporting. It looked every bit as good as similar commercial products.

Two particular points were worth noting:

- the actual matching approach itself seems acceptable. Users build rules that specify which fields to compare, the methods used to measure similarity, and similarity scores required for each field. This is less sophisticated than the best commercial products, but field comparisons are probably adequate for most situations. Although setting up and tuning such rules can be time-consuming, Infosolve told me they can build a typical set of match routines in about half a day. More experienced or adventurous users could even do it for themselves; the user interface makes the mechanics very simple. A half-day of consulting might cost $1,000, which is not bad at all when you consider that the software itself is free. The price for a full implementation would be higher since it would involve additional consulting to set up data extracts, standardization, enhancement and other processes, but cost should still be very reasonable. You’d probably need as much consulting with other CDI systems where you'd pay for the software too.

- data verification and enhancement is done by calls to StrikeIron, which provides a slew of on-demand data services. StrikeIron is worth knowing about in its own right: it lets users access Web services including global address verification and corrections; consumer and business data lookups using D&B and Gale Group data; telephone verification, appends and reverse appends; geocoding and distance calculations; Mapquest mapping and directions; name/address parsing; sales tax lookups; local weather forecasts; securities prices; real-time fraud detection; and message delivery for text (SMS) and voice (IVR). Everything is priced on a per use basis. This opens up all sorts of interesting possibilities.

The Infosolve software can be installed on any platform that can run Java, which is just about everything. Users can also run it within the Sun Grid utility network, which has a pay-as-you-go business model of $1 per CPU hour.

I’m a bit concerned about speed with Infosolve: the company said it takes 8 to 12 hours to run a million record match on a typical PC. But that assumes you compare every record against every other record, which usually isn’t necessary. Of course, where smaller volumes are concerned, this is not an issue.

Bottom line: Infosolve and Pentaho may not meet the most extreme CDI requirements, but they could be a very attractive option when low cost and quick deployment are essential. I’ll certainly keep them in mind for my own clients.