0 PivotLink: Flexible On-Demand Business Intelligence

I did a Webinar recently (click here for slides) about on-demand business intelligence systems, sponsored by Birst. It boiled to two key points:

- most of the work in business intelligence is in assembling the underlying database, even though the term “BI systems” often refers to the query and reporting tools (a.k.a. the “presentation layer”).

- vendor strategies to simplify BI include: using simpler interfaces, automation or pre-built solutions to make conventional technology easier; or using inherently more efficient alternative technologies such as in-memory and columnar databases.

Naturally, the full Webinar was jammed with much other wisdom (you missed Time-Traveling Elvis and the Reese’s Peanut Butter Cup Fallacy). But those two points alone provide a useful framework for considering business intelligence systems in general.

I bring this up because I’m finally writing about PivotLink, which I looked at more than a month ago. It turns out that my framework helps to put PivotLink into perspective.

Here’s the thing. PivotLink is an on-demand business intelligence product. Its most interesting technical feature is an in-memory columnar database. If you follow the industry, you know that type of database is about the geek-sexiest thing out there right now. I myself find it totally fascinating and had a grand time digging into the details.

But the rest of the world doesn’t care if it’s geek-ilicious:* they want to know how PivotLink can help them. Here’s where the framework comes in, since it clarifies what PivotLink does and doesn’t do. Or, to put that a bit more formally, it shows which elements of a complete business intelligence system PivotLink provides.

The answer to that being, PivotLink works largely at the presentation layer. It can import data from multiple sources and join tables on common keys. But it won’t do the complicated transformations and fuzzy matching needed for serious data integration. This means that PivotLink must either work with data that's already been processed into a conventional data warehouse or can be usefully analyzed in its original state.

There’s actually more of this naturally-analyzable data than you might think. Purchase transactions, a common source for PivotLink, are a good example. The obstacle to working with these has often been the size of the data sets, which meant lots of expensive hardware and lots of (how to put this delicately?) deliberately-paced IT support. These are exactly the barriers that on-demand systems overcome.

This brings us back to PivotLink's technology. In-memory, columnar databases are especially well suited for on-demand business intelligence because they compress data tightly (10% of the original volume. according to PivotLink), read only the columns required for a particular query (providing faster response) and don’t require need special schemas or preaggregated data cubes (requiring less skill to set up and modify).

But even columnar systems vary in their details. PivotLink sits towards the flexible end of the spectrum, with support for incremental updates, many-to-many table relationships, and abilities to add new columns and merge data along different query paths without reloading it. The system also allows calculations during the data load and within queries, caches data in memory and further compresses it after an initial query, and supports user- and group-level security at the row, column or individual cell levels. Not all column-based systems can say the same.

On the other hand, PivotLink does not support standard SQL queries and doesn’t run on a massively parallel (“shared nothing”) architecture. Both limits are typical of older columnar databases, a reminder that PivotLink began life in 1998 as SeaTab Software. Although shared-nothing architectures are generally more scalable, PivotLink is already running data sets with more than 10 billion rows in its current configuration. Response is very fast: according to the company, one client with several billion rows of point-of-sale data runs a nightly update and then executes a summary report in under one minute. Still, PivotLink recognizes the benefits of shared-nothing systems and plans to move to that architecture by the end of 2009.

Lack of SQL compatibility means users must rely on PivotLink’s tools for queries and reports. These let administrators import data from CSV, TXT and Excel files and map them to PivotLink tables. (The actual storage is columnar, with different compression techniques applied to different data types. But to the user, the data looks like it’s organized in tables.) The system reads the source data and makes suggestions about field types and names, which the user can accept or revise.

Users then define queries against the tables. Each query contains a selected set of columns, which are identified either as a header (i.e., dimension) or metric. When queries involve multiple tables, the user also specifies the columns to join on. Each report is written against one query.

The distinction between tables and queries is important in PivotLink, because it provides much of the system’s flexibility. The same column can be a dimension in one query and a metric in another, and the same tables can be related on different keys in different queries. All this happens without reloading the data or running any aggregations. The metadata used to build reports is defined by the associated query, not the underlying tables.

Reports are built by dragging headers and metrics into place on a grid, defining drill paths, and selecting tabular or chart formats. Reports can also rank and sort results and select the top or bottom rows for each rank. For example, a report could rank the top ten stores per region by revenue. Users can combine several reports into a dashboard.

End-users can apply filters and drill into the selected elements within a report. However, PivotLink does not apply filters for one report to the rest of the dashboard, in the way of QlikView or ADVIZOR. This feature is also on the agenda for 2009.

PivotLink clients can import data, define queries and build reports without help from the company. PivotLink said it takes a couple of days to train an administrator to load the data and build queries, a day or two to train a user to build reports and dashboards, and minutes to hours to learn to use the reports. Based on what they showed me, that sounds about right. You can also find out for yourself by signing up for a free trial.

Pricing of PivotLink is based on the size of the deployment, taking into account data volume and types of users. It starts around $3,000 per month for 50 million rows. When I spoke with the company in early March, they had about 60 clients supporting over 6,000 users, and had doubled their volume in the past year.

-----------------------------------------------------------------------------------------
* geek-alicious? is it possible to misspell a word that doesn’t exist?

0 Good Look at QlikView from a Microstrategy Consultant's Viewpoint

I noticed some visitors this morning from the blog of Microstrategy consultancy Aellament, and found they have published a nice look at QlikView on their blog. It's worth a read, and quite interesting in their appreciation of the advantages that QlikView offers over the product they know best. They fairly point out some disadvantages too, of which I think lack of a unified metadata view is probably most significant.

What really resonated was Aellament's sense (in an earlier blog post, which contains the link back to this blog) that QlikView is empowering departmental users to do work that otherwise takes support from technical specialists. That's exactly what I've seen as QlikView's advantage and I think it's fundamentally reshaping the industry.

As I see the future, BI specialists will still be needed to assemble source data into usable forms (i.e., build the data warehouses). This has always been the heavy lifting. But the huge army of people who then essentially reformat that data into cubes, reports, dashboards, etc. will dwindle as business analysts do that for themselves using tools like QlikView. Bad news for Microstrategy consultants (presumably why Aellament is hedging its bets with QlikView training) but good news for business users.

0 Pedowitz Group Offers Free Support for New Eloqua Clients

The Pedowitz Group announced this morning that it was offering $15,000 in free consulting services and guaranteeing a five-day implementation to new clients who purchase Eloqua demand generation software . (Click here for the announcement.)

(If you’re not familiar with The Pedowitz Group, President and CEO Jeff Pedowitz ran the professional services group at Eloqua for several years before starting the company. The firm also works with Marketo, Silverpop Engage B2B and MarketingGenius depending on client needs. )

My initial reaction to the announcement was to wonder if it would be interpreted as evidence that Eloqua requires a lot of consulting to implement. Certainly that’s how I’d spin it if I were a competitor. But then again, maybe I wouldn't, because it focuses attention on how much support is really needed to deploy other systems.

On the surface, this is a strength of vendors who promise free implementation and deployment within a few days. But the reality is that most marketers need outside help to design their email campaigns, nurture programs, scoring rules, and CRM integration. This has less to do with learning the software than with knowing what works and how to apply it to their own business.

Yes, some products really are easier to use than others, especially for simple programs. That's one reason Pedowitz works with several vendors. (Download the Raab Guide report on Vendor Usability Scores for more on this.) And yes, some marketers will get their system running with no more than telephone support.

But it’s just plain silly to think that most marketers can quickly deploy sophisticated demand generation programs without some expert help. This is what’s highlighted by the Pedowitz Group offer – and it’s a discussion that vendors selling the dream of an instant deployment should probably avoid.

0 True Influence Opens a Window into Future Demand Generation

People develop new products because they feel they can offer something existing products do not. In the early stages of an industry, the new products are often similar because several people have independently spotted the same opportunity and built something to tap it. As the industry matures, second-generation products are designed to improve on the original products, either by adding new capabilities or by delivering the same capabilities faster, easier or cheaper. This leads to more variety as vendors experiment with different approaches to a now-defined problem. In a third stage, variety diminishes as widely successful approaches become templates for standard configurations.

Demand generation systems are in that second stage. This means new products reflect the lessons each vendor has drawn from the industry’s history to date.

True Influence illustrates this nicely. CEO and co-founder Brian Giese had extensive experience in business sales and marketing and with existing demand generation systems when he began developing True Influence two years ago. So it’s possible to see True Influence, which was released at the end of last year, as his well-educated guess at what future demand generation systems will look like.

Giese’s conclusion was that marketers’ overwhelming need is simplicity. In fact, he said he has actually removed features from the system because customers weren’t using them. But he also decided that marketers want Webinar integration, digital asset management, APIs to capture data from external Web forms, and a dedicated IP address for email. These are not yet standard features on most demand generation products. But if Giese is right, they will be.

Like all demand generation systems, True Influence can import lists, send emails, create Web forms and surveys, score leads, set up multi-step campaigns, and integrate with CRM systems. Capabilities in these areas tend to be adequate but minimal. For example:

- emails and Web forms can be personalized with lead data, but don’t incorporate rule-selected content blocks.

- the list selection interface uses a form that lets users apply values to a list of all data elements. This is simple but doesn’t easily support complex conditions. Nor does True Influence support random splits for tests.

- Answers to Web surveys are limited to list box or radio button formats.

- The system allows an unlimited number of survey questions, but it will overwrite previous replies if a question is answered more than once.

- The lead record allows only four user-defined fields.

- Lead scores can be based on just a few attributes and activities: industry, job title, company size, location, lead source, email status, activity indicator, most recent activity date, and visits to specific Web pages. Giese said that other elements could be exposed but clients haven’t requested them.

On the brighter side:

- The system API lets users easily adopt externally-built and -hosted Web forms to post into the True Influence database. This saves clients the trouble of replacing existing Web forms when they deploy the system. Clients can also build and host their forms within True Influence if they prefer.

- The base price includes a separate IP address. This protects the client if any other True Influence customers run afoul of the anti-spam police. Most other vendors charge extra for a dedicated IP address if they make it available at all.

- The system includes a resource library for both internal assets (templates, emails, Web forms, etc.) and downloadable collateral such as white papers and brochures. This might replace a separate digital asset management system for clients who don't need approval workflows or fine-grained user rights management. The system does support some version control.

Campaign management reflects a particularly interesting set of design choices. Users define campaigns by building a flow chart with icons for steps and delays. Most simplicity-oriented systems avoid flow charts, so I was surprised to see them in True Influence. The system does simplify the diagrams a bit by embedding the decision rules within the lines that link the icons instead of creating separate decision icons. I was also surprised to find that each icon can have its own schedule – another feature typically reserved for advanced systems.

Giese assured me that his clients like the flow charts and successfully use them for "very complex" campaigns. But we didn't discuss the meaning of "very complex" and I suspect my definition is more demanding than his.

More in line with what I’d expect from this system, decision rules are limited to a few essential functions (opened email, clicked link, registered for Webinar, joined Webinar, submitted Web form, completed step, action completed, action failed). Campaign actions are also constrained: for example, the system can send emails but has no particular support for other media such as direct mail or call centers.

Campaigns can also add a lead to a list (which might in turn trigger another campaign), update the lead score, convert a prospect to a lead and send it to the CRM system, send an email alert to a salesperson or sales manager, and publish a landing page or Web form. Treating lead scoring, conversion to a CRM lead and publication of Web pages as campaign functions, rather than executing these outside of individual campaigns, is typical of simplicity-oriented demand generation systems. So is moving leads among campaigns by adding them to lists rather than assigning them directly.

Wait steps can either delay the campaign for a specified period (e.g., wait seven days for a reply), or be keyed to a specific date (e.g., send a reminder three days before a Webinar). This is more flexible than some other products. However, event-based triggers are limited to submission of a Web form. Otherwise, users can achieve near-real-time triggers by scheduling a campaign step to check for specified conditions at regular intervals defined in hours or minutes.

Webinar support includes prebuilt campaigns with registration and confirmation forms; emails for invitations, reminders and thank-you messagess; and salesperson alerts. These are all part of one campaign flow. More important, the vendor has prebuilt integration with Webinar vendor DimDim (a pretty interesting product in its own right). This lets True Influence capture actual attendance and automatically load it into the contact history. Less extensive integration is also available with GoToWebinar.

True Influence has existing integrations with Salesforce.com and SugarCRM. These provide bi-directional synchronization at five minute intervals. The system can exchange data with other CRM systems through batch updates as needed.

Campaign reporting in True Influence includes campaign activities (emails sent and received, page visits, etc.) and lets users drill down to the list of individuals in each campaign. Users can also view the profile, list memberships and activity details for an individual. The system provides its own Web analytics, based on page tags.

Pricing of True Influence is also designed for on simplicity. Fees are based on the number of “actionable” names in the client database, which basically means valid email addresses. This accommodates the large volume of bad data often imported from mailing lists. There are no separate charges for deployment, training, individual modules, extra seats or dedicated IP addresses. Prices are being revised at this writing but are generally intended to be competitive in the middle tier of the demand generation market.

True Influence was released in late 2008 and currently has about fifteen clients.

0 DemandBase Creeps Up the Value Chain

I had a nice little chat with DemandBase two weeks ago. I’d been aware of them since they were founded in 2006 but in their original incarnation as a data provider. That is, they take business data from sources including Hoovers, D&B, LexisNexis, AccuData, BusinessWatch Network, Jigsaw) and merge it into one big contact list that people can use for outbound promotions or to enhance their own files. It’s a perfectly reasonable business, but not one I find especially exciting.

But it turns out that DemandBase has been inching its way up the value chain. Some time ago they released a free widget, DemandBase Stream, that shows the companies visiting your Web site in a news ticker on your computer desktop. That’s somewhat entertaining and can be useful to salespeople who might notice and reach out to current clients or prospects. But the technology is nothing special: a page tag sends the visitor’s IP address to DemandBase, which looks up its owner in standard Internet registries. In any case, being free, it's really just a promotion for DemandBase.

Their new product, DemandBase Professional, is another matter. It also captures the visitor stream to a company Web site and uses the IP address to identify the company. But now it matches that company against the main DemandBase database to actually apply details such as location, company size, industry, and further finds contact names that marketing or salespeople can reach out to. That’s not much more entertaining, but a lot more salable.

DemandBase Professional actually goes further than simply looking up the information, by letting clients specify the industries, company sizes, geographic regions, search terms and number of page views they care about, and only returning information on visitors who fit those parameters. It can also import a client’s list of target accounts from Salesforce.com and issuing alerts when those companies visit. The good news is that clients only pay for alerts that meet their specified parameters, keeping both the cost and volume within reason. The company only charges for incremental prospect names, so you pay don't again if someone visits your Web site twice.

I was impressed by the sheer cleverness of all this, precisely because the underlying technology (page tags, IP owner lookup, name matching, simple filtering) is so straightforward. There’s no identification of actual visitors and no cookies.

Ok, some of the underlying database preparation and matching is actually pretty demanding. In fact, DemandBase told me that they can use only about 20% of the data they get from their partners because quality problems with the rest. But Rodney Dangerfield on a bad day got more respect than data hygiene experts.

One obvious question is how many useful new names can DemandBase actually provide. According to the company, it typically finds that 50-60% of the Web site visitors come through a generic Internet Service Provider connection and thus can’t be traced to a particular company. Of the remainder, 10-20% fall within the target company sizes, industries or regions. Thus, DemandBase might send information on 5-10% of total visitors.

This is still twice the 2-4% of visitors who DemandBase says typically identify themselves directly. DemandBase said they hadn’t made a formal study of the quality of those leads, but felt confident they were more than worth the cost.

DemandBase plans to continue adding value, possibly by pushing automated messages to selected visitors. But they don’t intend to become a lead management system. We discussed social media briefly; they aren’t using social data (e.g. blog, Facebook, Twitter posts) but are considering trying to find degrees of separation in social networks like Linked In. They didn’t mention it, but I could also see them providing input to Web site personalization systems, so the site itself would tailor its content to what DemandBase infers about the visitor.

The company said that about one thousand companies have set up its free widget and a couple hundred are testing DemandBase Professional. Pricing for Professional depends on volume and starts around $200-$300 per month.

I suppose I could just say this is a useful-sounding product and let it go at that. But if you’re looking for Some Larger Significance: it’s also an example of how the traditional distinction between sales and marketing is dissolving. DemandBase is mostly a salesperson’s tool, but it is identifying names much earlier than sales typically gets involved. Even more to the point, a particular name might go to either sales or marketing depending on the situation. This means that sales and marketing have to agree on rules for who does what—requiring considerably more cooperation than just throwing a lead over the wall once it's deemed “qualified”. I'd say that's significant.