Just kidding.
0 Will Twitter teach us to be concise and save the world from information overload? See my 1000 word blog post
0 Blog Posts I'll Never Write (With Apologies to Borges)
Jorge Luis Borges wrote surreal stories that dealt largely with the nature of literature and authorship. His most popular work is Ficciones, a collection that includes a series of reviews of books that were never written. This was a brilliantly efficient conceit, allowing Borges to analyze the themes and issues of these stories without going to the trouble of actually creating them.
In that same spirit, here are summaries of three blog posts I doubt I’ll ever have time to actually write.
1. The Future of Technology Analysts. This is a reaction to a blog post Do CRM Analysts Provide Value for Money? by veteran consultant Graham Hill. Basically he concludes that the ‘big-name’ analysts don’t tell him much that he he can’t pick up from his own experience or other sources like blogs, books and academic papers. He does feel that ‘niche’ analysts provide more value.
As a ‘niche’ analyst myself, I tend to agree. But the more important issue is the movement away from traditional analysts to a community or peer-to-peer model. Rather than listen to expensive experts, marketers want to hear from other marketers, whom I think they perceive as more practical and less self-interested. This implies a change in the business model of analysts and consultants, who must deliver concrete business solutions rather than simply offering knowledge that clients can’t get elsewhere.
This is probably a good thing in the long run, although it does raise the question of who will do original research when no one will pay the analysts for the results. (This is a special case of the larger issue raised by the Internet, where professional journalists are squeezed out by volunteers who work for free but may not do a thorough or accurate job.) People will need to evolve new mechanisms to judge the value of different information sources—a problem that is widely recognized but nowhere near being solved.
2. A Cookbook for Demand Generation. Probably the biggest obstacle to selling demand generation systems is that many marketers don’t know what to do with them. This reminds me of the little cookbooks that come with kitchen appliances, which give a few simple recipes that make use of the appliances’ capabilities. How about a little demand generation cookbook that would describe a dozen or so starter projects, each with a list of ingredients (emails, landing pages, etc.), step-by-step instructions for setting them up, and maybe a snapshot of the expected results? I guess we could call it “The Joy of Demand Gen.” The goal is to make this as non-intimidating as possible.
3. Taxonomy for Twitter Analytics. I think there is a hierarchy of analytical methods for Twitter and other social media. The simplest layer would be to count mentions of a company or product. The next would be to list the people who are making those mentions, ranked by frequency. Then you’d measure the influence of those people based on number of followers, links, reposts/retweets, etc. Next look at the content of the mentions: are they positive or negative? Finally, really understand the content, taking into account things like sarcasm and emoticons, and maybe convert this into some sort of long-term attitude measure (enthusiastic, skeptical, objective, etc.). Most or all of these measures are available to some degree, but I don’t think I’ve seen them all linked together in a single solution.
In that same spirit, here are summaries of three blog posts I doubt I’ll ever have time to actually write.
1. The Future of Technology Analysts. This is a reaction to a blog post Do CRM Analysts Provide Value for Money? by veteran consultant Graham Hill. Basically he concludes that the ‘big-name’ analysts don’t tell him much that he he can’t pick up from his own experience or other sources like blogs, books and academic papers. He does feel that ‘niche’ analysts provide more value.
As a ‘niche’ analyst myself, I tend to agree. But the more important issue is the movement away from traditional analysts to a community or peer-to-peer model. Rather than listen to expensive experts, marketers want to hear from other marketers, whom I think they perceive as more practical and less self-interested. This implies a change in the business model of analysts and consultants, who must deliver concrete business solutions rather than simply offering knowledge that clients can’t get elsewhere.
This is probably a good thing in the long run, although it does raise the question of who will do original research when no one will pay the analysts for the results. (This is a special case of the larger issue raised by the Internet, where professional journalists are squeezed out by volunteers who work for free but may not do a thorough or accurate job.) People will need to evolve new mechanisms to judge the value of different information sources—a problem that is widely recognized but nowhere near being solved.
2. A Cookbook for Demand Generation. Probably the biggest obstacle to selling demand generation systems is that many marketers don’t know what to do with them. This reminds me of the little cookbooks that come with kitchen appliances, which give a few simple recipes that make use of the appliances’ capabilities. How about a little demand generation cookbook that would describe a dozen or so starter projects, each with a list of ingredients (emails, landing pages, etc.), step-by-step instructions for setting them up, and maybe a snapshot of the expected results? I guess we could call it “The Joy of Demand Gen.” The goal is to make this as non-intimidating as possible.
3. Taxonomy for Twitter Analytics. I think there is a hierarchy of analytical methods for Twitter and other social media. The simplest layer would be to count mentions of a company or product. The next would be to list the people who are making those mentions, ranked by frequency. Then you’d measure the influence of those people based on number of followers, links, reposts/retweets, etc. Next look at the content of the mentions: are they positive or negative? Finally, really understand the content, taking into account things like sarcasm and emoticons, and maybe convert this into some sort of long-term attitude measure (enthusiastic, skeptical, objective, etc.). Most or all of these measures are available to some degree, but I don’t think I’ve seen them all linked together in a single solution.
0 Low Cost Systems for Demand Generation
My new obsession with Twitter (follow me as @draab) has led to several messages from people who seemed to have trouble choosing between Eloqua and Marketo . This is a bit perplexing, since those products are at the opposite ends of the spectrum: Marketo being relatively low cost / limited functionality / easier to learn, and Eloqua being higher cost / richer functionality / takes more training. It shouldn't be hard to figure out which one suits you better.
But what really concerns me is that these people are apparently limiting their consideration to just those two products. I do recognize that they are the best known vendors in the space (with apologies to Vtrenz, whose identity is somewhat blurred since its purchase by Silverpop). But there are plenty of other options, particularly for marketers with limited budgets. Marketo is certainly a fine product, but marketers should still look around before picking it by default. Here are some alternatives that will come in at or below Marketo’s published starting price of $2,400 per month (or $1,500 for their “Lite” version). (To be fair, many of the vendors below charge an installation of that can be several thousand dollars or more, while Marketo doesn't. But even including that, the first year cost for most of these will be less.) :
Manticore Technology: a full-featured demand generation product. See my 2007 blog entry for some information or buy the Raab Guide to Demand Generation Systems for a detailed review. (Just kidding...or am I?) Pricing on their Web site is quite close to Marketo’s: $1,000 a month for a limited edition and $2,400 per month for all the bells and whistles, and no extra charge for installation.
Pardot: another pretty powerful product; see my blog review from December 2008. But much better pricing at the low end $750 per month for the smallest complete system, and $1,250 per month for something that should be adequate for larger firms.
OfficeAutoPilot: this is the current incarnation of what used to be Moonray. I took a detailed look at a beta of the next release a couple of weeks ago and liked both the interface and breadth of functionality. I’m just waiting for the official release (due in for late February) to publish a detailed review. [Click here to read the review, published in April.] Pricing for a full-featured system was $597 per month—quite a bargain. Even cheaper options are available if you need fewer functions.
Treehouse Interactive: another company I spoke with recently, and another one I’m not writing about yet because they showed me some features that won’t be released for a while (early March). [Click read my March 18 review.] The company has a low profile but has been selling its demand generation product for nearly ten years. It offers a pretty complete set of features, with shortfalls in some areas balanced by strengths in others. More important to people who need it, the company offers partner management and channel sales management products that integrate with its demand generation offering. Pricing starts at $599 per month.
Infusionsoft: I spoke with them just yesterday, and there’s nothing preventing me from writing about them in detail except that I do like to sleep occasionally. Maybe next week. [I got to it on February 12. Read it here.] They aim to be a complete business operating system for very small companies (under 25 employees). So they offer not just demand generation but everything from contact management to e-commerce. However, marketing is their core function and they provide a decent set of features, although certainly not as polished as some other products I’ve mentioned. On the other hand, pricing starts at $199 per month for a 2 user system with pretty much all of their marketing features, and you’d be hard-pressed to spend more than $500 per month unless you want lots of users for non-marketing functions such as sales automation or order entry. Although you may not be familiar with them, the company is five years old and has more than 12,000 (yes, that's twelve thousand) clients.
Active Conversion: I had a chat with president Fred Yee last August although I didn’t publish a detailed review. [I did publish a real review in July 2009.] They focus on email nurturing campaigns and marketing measurement. The system didn’t do landing pages or forms when I spoke with Fred, although he tells me it does now. Pricing starts at $250 per month and averages around $500.
Act-On Software: a slightly different take, with strong Webinar support and an option to use its own low-cost sales automation system as an alternative to Salesforce.com Price starts at $499 per month. I reviewed it here in March.
Before I get any complaints from other vendors in the industry, let me stress that the systems I've listed above are ones that I know have low price points. My own consideration set also includes Marketbright, Market2Lead, MarketingGenius, LeadLife, LoopFuse, LeadGenesys, , eTrigue and SalesFusion360, although I haven't looked at all of those in detail.
Obviously you need to evaluate these products in depth before deciding which is right for you. There are free papers on the Raab Guide site that can help you organize this process and of course I do consult in this area for a living. But the point of today's post isn't that you should run a thorough selection project. It's simply that should recognize that you do have choices, and take advantage of them.
But what really concerns me is that these people are apparently limiting their consideration to just those two products. I do recognize that they are the best known vendors in the space (with apologies to Vtrenz, whose identity is somewhat blurred since its purchase by Silverpop). But there are plenty of other options, particularly for marketers with limited budgets. Marketo is certainly a fine product, but marketers should still look around before picking it by default. Here are some alternatives that will come in at or below Marketo’s published starting price of $2,400 per month (or $1,500 for their “Lite” version). (To be fair, many of the vendors below charge an installation of that can be several thousand dollars or more, while Marketo doesn't. But even including that, the first year cost for most of these will be less.) :
Manticore Technology: a full-featured demand generation product. See my 2007 blog entry for some information or buy the Raab Guide to Demand Generation Systems for a detailed review. (Just kidding...or am I?) Pricing on their Web site is quite close to Marketo’s: $1,000 a month for a limited edition and $2,400 per month for all the bells and whistles, and no extra charge for installation.
Pardot: another pretty powerful product; see my blog review from December 2008. But much better pricing at the low end $750 per month for the smallest complete system, and $1,250 per month for something that should be adequate for larger firms.
OfficeAutoPilot: this is the current incarnation of what used to be Moonray. I took a detailed look at a beta of the next release a couple of weeks ago and liked both the interface and breadth of functionality. I’m just waiting for the official release (due in for late February) to publish a detailed review. [Click here to read the review, published in April.] Pricing for a full-featured system was $597 per month—quite a bargain. Even cheaper options are available if you need fewer functions.
Treehouse Interactive: another company I spoke with recently, and another one I’m not writing about yet because they showed me some features that won’t be released for a while (early March). [Click read my March 18 review.] The company has a low profile but has been selling its demand generation product for nearly ten years. It offers a pretty complete set of features, with shortfalls in some areas balanced by strengths in others. More important to people who need it, the company offers partner management and channel sales management products that integrate with its demand generation offering. Pricing starts at $599 per month.
Infusionsoft: I spoke with them just yesterday, and there’s nothing preventing me from writing about them in detail except that I do like to sleep occasionally. Maybe next week. [I got to it on February 12. Read it here.] They aim to be a complete business operating system for very small companies (under 25 employees). So they offer not just demand generation but everything from contact management to e-commerce. However, marketing is their core function and they provide a decent set of features, although certainly not as polished as some other products I’ve mentioned. On the other hand, pricing starts at $199 per month for a 2 user system with pretty much all of their marketing features, and you’d be hard-pressed to spend more than $500 per month unless you want lots of users for non-marketing functions such as sales automation or order entry. Although you may not be familiar with them, the company is five years old and has more than 12,000 (yes, that's twelve thousand) clients.
Active Conversion: I had a chat with president Fred Yee last August although I didn’t publish a detailed review. [I did publish a real review in July 2009.] They focus on email nurturing campaigns and marketing measurement. The system didn’t do landing pages or forms when I spoke with Fred, although he tells me it does now. Pricing starts at $250 per month and averages around $500.
Act-On Software: a slightly different take, with strong Webinar support and an option to use its own low-cost sales automation system as an alternative to Salesforce.com Price starts at $499 per month. I reviewed it here in March.
Before I get any complaints from other vendors in the industry, let me stress that the systems I've listed above are ones that I know have low price points. My own consideration set also includes Marketbright, Market2Lead, MarketingGenius, LeadLife, LoopFuse, LeadGenesys, , eTrigue and SalesFusion360, although I haven't looked at all of those in detail.
Obviously you need to evaluate these products in depth before deciding which is right for you. There are free papers on the Raab Guide site that can help you organize this process and of course I do consult in this area for a living. But the point of today's post isn't that you should run a thorough selection project. It's simply that should recognize that you do have choices, and take advantage of them.
0 QlikView Is Champion In Aberdeen AXIS (But Is This Graph Necessary?)
I occasionally do some development work in QlikView and have made no secret that I think it's a great product. So part of me was pleased to learn that it had been listed as a—indeed, the only—“champion” in an Aberdeen Group AXIS report on BI/Performance Management software. Although QlikView has received plenty of recognition recently, it deserves every bit of it. (If you’ve been living in a cave, or, much worse, not reading this blog: QlikView is business intelligence software that combines an in-memory database, special “associative” data model, data transformation scripts, and a powerful presentation layer. My own experience has shown it lets business analysts build in days what would take months of work by heavyweight IT experts with conventional BI tools. For details, see What Makes QlikTech So Good?--by far the most popular post I've ever written.)
Here is the AXIS itself, with QlikView alone at the top:
And therein lies the problem. Fond as I am of QlikView, I can’t imagine any measure by which it would be a dominant business intelligence tool, let alone rank so far above its competitors. So I was more than a little perplexed to see the Aberdeen ranking, especially because I wasn’t aware that Aberdeen produced something similar to the Forrester “waves” and Gartner “magic quadrants”.
Indeed, it turns out that the AXIS program is brand new. The BI report is the second to be issued but will be followed by several per quarter. Aberdeen’s materials describe its reports as “unlike other comparative products that primarily focus on feature and functionality”, but to my eye it looks pretty similar. Apparently what makes it “the market’s first technology solution software provider assessment tool that is truly customer-centric” is that one of its two dimensions, “value delivered”, is based on Aberdeen’s surveys that identify best-in-class companies.
Exactly how this relates to the vendor rankings is unclear, although I could find out if I paid $895 for the report. Maybe a higher “value delivered” ranking means the product is used by more best-in-class companies, although I rather doubt it given QlikView's still-limited market penetration. More likely, the higher ranked products are viewed by users as delivering greater value, perhaps with some weighting towards the relative performance of the companies doing the ranking. I’ve no doubt that QlikView users are more enthusiastic than any other vendors’, both because it truly is a great product but also because it’s still in the relatively early adapter stage where users tend to be highly motivated and vocal. Perhaps Aberdeen looks at the features desired by best-in-class companies and compares those to the features delivered by the different products, although this too seems unlikely.
The second dimension of “market readiness” is described by Aberdeen as based on “evaluation of responses to a standardized vendor questionnaire, analyst briefings, public records and customer interviews.” This sure sounds like a conventional feature-and-function assessment to me.
As someone who has been evaluating software for many years, I fully appreciate the appeal of these sorts of matrices. Vendors love them because, if they’re ranked near the top, it gives them something to crow about. Buyers love them because they can save work by considering only the top-ranked alternatives (even though vendors piously warn this is inappropriate). The analyst firms love them because they get lots of publicity, both directly because the press loves a horse race and from the winning vendors who promote them.
I have always avoided producing such rankings, even when people ask for them, precisely because they make it too easy for buyers to avoid the essential work of assessing products against their own needs. Still, the commercial advantages of the rankings are so great that I may yet feel impelled to produce them. That being the case, I can’t really criticize Aberdeen for rolling out their own. But I do hope they make a serious effort at educating people on what the "AXIS" means and how it should and shouldn’t be used.
Here is the AXIS itself, with QlikView alone at the top:
And therein lies the problem. Fond as I am of QlikView, I can’t imagine any measure by which it would be a dominant business intelligence tool, let alone rank so far above its competitors. So I was more than a little perplexed to see the Aberdeen ranking, especially because I wasn’t aware that Aberdeen produced something similar to the Forrester “waves” and Gartner “magic quadrants”.
Indeed, it turns out that the AXIS program is brand new. The BI report is the second to be issued but will be followed by several per quarter. Aberdeen’s materials describe its reports as “unlike other comparative products that primarily focus on feature and functionality”, but to my eye it looks pretty similar. Apparently what makes it “the market’s first technology solution software provider assessment tool that is truly customer-centric” is that one of its two dimensions, “value delivered”, is based on Aberdeen’s surveys that identify best-in-class companies.
Exactly how this relates to the vendor rankings is unclear, although I could find out if I paid $895 for the report. Maybe a higher “value delivered” ranking means the product is used by more best-in-class companies, although I rather doubt it given QlikView's still-limited market penetration. More likely, the higher ranked products are viewed by users as delivering greater value, perhaps with some weighting towards the relative performance of the companies doing the ranking. I’ve no doubt that QlikView users are more enthusiastic than any other vendors’, both because it truly is a great product but also because it’s still in the relatively early adapter stage where users tend to be highly motivated and vocal. Perhaps Aberdeen looks at the features desired by best-in-class companies and compares those to the features delivered by the different products, although this too seems unlikely.
The second dimension of “market readiness” is described by Aberdeen as based on “evaluation of responses to a standardized vendor questionnaire, analyst briefings, public records and customer interviews.” This sure sounds like a conventional feature-and-function assessment to me.
As someone who has been evaluating software for many years, I fully appreciate the appeal of these sorts of matrices. Vendors love them because, if they’re ranked near the top, it gives them something to crow about. Buyers love them because they can save work by considering only the top-ranked alternatives (even though vendors piously warn this is inappropriate). The analyst firms love them because they get lots of publicity, both directly because the press loves a horse race and from the winning vendors who promote them.
I have always avoided producing such rankings, even when people ask for them, precisely because they make it too easy for buyers to avoid the essential work of assessing products against their own needs. Still, the commercial advantages of the rankings are so great that I may yet feel impelled to produce them. That being the case, I can’t really criticize Aberdeen for rolling out their own. But I do hope they make a serious effort at educating people on what the "AXIS" means and how it should and shouldn’t be used.
0 SQLStream Simplifies Event Stream Processing
I spoke earlier this week with SQLStream, which offers software to execute queries against data streams such as stock market prices, Web logs and credit card transactions. These queries can include on-the-fly calculations such as moving averages, as well as scans for patterns like a sequence of failed log-in attempts. Typical applications include security monitoring, fraud detection, and general business activity monitoring. Marketers can use the queries to identify new leads and select cross-sell and upsell offers. Although the connection is a little less obvious, the system can also be used as an alternative to conventional batch data preparation methods for tasks like customer data integration.
SQLStream’s particular claim to fame is that its queries are almost identical to garden-variety SQL. Other vendors in this space apparently use more proprietary approaches. I say “apparently” because I haven’t researched the competition in any depth. A quick bit of poking around was enough to scare me off: there are many vendors in the space and it is a highly technical topic. It turns out that stream processing is one type of “complex event processing,” a field which has attracted some very smart but contentious experts. To see what I mean, check out Event Processing Thinking (Opher Etzion) and Cyberstrategics Complex Event Processing Blog (Tim Bass). This is clearly not a group to mess with.
That said, SQLStream’s more or less direct competitors seem to include: Coral8, Truviso, Progress Apama, Oracle BAM, TIBCO BusinessEvents, KX Systems, StreamBase and Aleri . For a basic introduction to data stream processing, see this presentation from Truvisio.
Back to SQLStream. As I said, it lets users write what are essentially standard SQL queries that are directed against a data stream rather than a static table. The data stream can be any JDBC-accessible data source, which includes most types of databases and file structures. The system can also accept streams of XML data over HTTP, which includes RSS feeds, Twitter posts and other Web sources. Its queries can also incorporate conventional (non-streaming) relational database tables, which is very useful when you need to compare streamed inputs against more or less static reference information. For example, you might want to check current activity against a customer’s six-month average bank balance or transaction rate.
The advantages of using SQL queries are that there are lots of SQL programmers out there and that SQL is relatively easy to write and understand. The disadvantage (in my opinion; not surprisingly, SQLStream didn’t mention this) is that SQL is really bad at certain kinds of queries, such as queries comparing subsets within the query universe and queries based on record sequence. Lack of sequencing may sound like a pretty big drawback for a stream processing system, but SQLStream compensates by letting queries specify a time “window” of records to analyze. This makes queries such as “more than three transactions in the past minute” quite simple. (The notion of “windows” is common among stream processing systems.) To handle subsets within queries, SQLStream mimics a common SQL technique of converting one complex query into a sequence of simple queries. In SQLStream terms, this means the output of one query can be a stream that is read by another query. These streams can be cascaded indefinitely in what SQLStream calls a “data flow architecture”. Queries can also call external services, such as address verification, and incorporate the results. Query results can be posted as records to a regular database table.
SQLStream does its actual processing by holding all the necessary data in memory. It automatically examines all active queries to determine how long data must be retained: thus, if three different queries need a data element for one, two and three minutes, the system will keep that data in memory for three minutes. SQLStream can run on 64-bit servers, allowing effectively unlimited memory, at least in theory. In practice, it is bound by the physical memory available: if the stream feeds more data than the server can hold, some data will be lost. The vendor is working on strategies to solve this problem, probably by retaining the overflow data and processing it later. For now, the company simply recommends that users make sure they have plenty of extra memory available.
In addition to memory, system throughput depends on processing power. SQLStream currently runs on multi-core, single-server systems and is moving towards multi-node parallel processing. Existing systems process tens of thousands of records per second. By itself, this isn't a terribly meaningful figure, since capacity also depends on record size, query complexity, and data retention windows. In any case, the vendor is aiming to support one million records per second.
SQLStream was founded in 2002 and owns some basic stream processing patents. The product itself was launched only in 2008 and currently has about a dozen customers. Since the company is still seeking to establish itself, pricing is, in their words, “very aggressive”.
If you’re still reading this, you probably have a pretty specific reason for being interested in SQLStream or stream processing in general. But just in case you’re wondering “Why the heck is he writing about this in a marketing blog?” there are actually several reasons. The most obvious is that “real time analytics” and “real time interaction management” are increasingly prominent topics among marketers. Real time analytics provides insights into customer behaviors at either a group level (e.g., trends in keyword response) or for an individual (e.g., estimated lifetime value). Real time interaction management goes beyond insight to recommend individual treatments as the interaction takes place (e.g., which offer to make during a phone call). Both require the type of quick reaction to new data that stream processing can provide.
There is also increasing interest in behavior detection, sometimes called event driven marketing. This monitors customer behaviors for opportunities to initiate an interaction. The concept is not widely adopted, even thought it has proven successful again and again. (For example, Mark Holtom of Eventricity recently shared some very solid research that found event-based contacts were twice as productive as any other type. Unfortunately the details are confidential, but if you contact Mark via Eventriicty perhaps he can elaborate.) I don’t think lack of stream processing technology is the real obstacle to event-based marketing, but perhaps greater awareness of stream processing would stir up interest in behavior detection in general.
Finally, stream processing is important because so much attention has recently been focused on analytical databases that use special storage techniques such as columnar or in-memory structures. These require processing to put the data into the proper format. Some offer incremental updates, but in general the updates run as batch processes and the systems are not tuned for real-time or near-real-time reactions. So it’s worth considering stream processing systems as a complement to that lets companies employ these other technologies without giving up quick response to new data.
I suppose there's one more reason: I think this stuff is really neat. Am I allowed to say that?
SQLStream’s particular claim to fame is that its queries are almost identical to garden-variety SQL. Other vendors in this space apparently use more proprietary approaches. I say “apparently” because I haven’t researched the competition in any depth. A quick bit of poking around was enough to scare me off: there are many vendors in the space and it is a highly technical topic. It turns out that stream processing is one type of “complex event processing,” a field which has attracted some very smart but contentious experts. To see what I mean, check out Event Processing Thinking (Opher Etzion) and Cyberstrategics Complex Event Processing Blog (Tim Bass). This is clearly not a group to mess with.
That said, SQLStream’s more or less direct competitors seem to include: Coral8, Truviso, Progress Apama, Oracle BAM, TIBCO BusinessEvents, KX Systems, StreamBase and Aleri . For a basic introduction to data stream processing, see this presentation from Truvisio.
Back to SQLStream. As I said, it lets users write what are essentially standard SQL queries that are directed against a data stream rather than a static table. The data stream can be any JDBC-accessible data source, which includes most types of databases and file structures. The system can also accept streams of XML data over HTTP, which includes RSS feeds, Twitter posts and other Web sources. Its queries can also incorporate conventional (non-streaming) relational database tables, which is very useful when you need to compare streamed inputs against more or less static reference information. For example, you might want to check current activity against a customer’s six-month average bank balance or transaction rate.
The advantages of using SQL queries are that there are lots of SQL programmers out there and that SQL is relatively easy to write and understand. The disadvantage (in my opinion; not surprisingly, SQLStream didn’t mention this) is that SQL is really bad at certain kinds of queries, such as queries comparing subsets within the query universe and queries based on record sequence. Lack of sequencing may sound like a pretty big drawback for a stream processing system, but SQLStream compensates by letting queries specify a time “window” of records to analyze. This makes queries such as “more than three transactions in the past minute” quite simple. (The notion of “windows” is common among stream processing systems.) To handle subsets within queries, SQLStream mimics a common SQL technique of converting one complex query into a sequence of simple queries. In SQLStream terms, this means the output of one query can be a stream that is read by another query. These streams can be cascaded indefinitely in what SQLStream calls a “data flow architecture”. Queries can also call external services, such as address verification, and incorporate the results. Query results can be posted as records to a regular database table.
SQLStream does its actual processing by holding all the necessary data in memory. It automatically examines all active queries to determine how long data must be retained: thus, if three different queries need a data element for one, two and three minutes, the system will keep that data in memory for three minutes. SQLStream can run on 64-bit servers, allowing effectively unlimited memory, at least in theory. In practice, it is bound by the physical memory available: if the stream feeds more data than the server can hold, some data will be lost. The vendor is working on strategies to solve this problem, probably by retaining the overflow data and processing it later. For now, the company simply recommends that users make sure they have plenty of extra memory available.
In addition to memory, system throughput depends on processing power. SQLStream currently runs on multi-core, single-server systems and is moving towards multi-node parallel processing. Existing systems process tens of thousands of records per second. By itself, this isn't a terribly meaningful figure, since capacity also depends on record size, query complexity, and data retention windows. In any case, the vendor is aiming to support one million records per second.
SQLStream was founded in 2002 and owns some basic stream processing patents. The product itself was launched only in 2008 and currently has about a dozen customers. Since the company is still seeking to establish itself, pricing is, in their words, “very aggressive”.
If you’re still reading this, you probably have a pretty specific reason for being interested in SQLStream or stream processing in general. But just in case you’re wondering “Why the heck is he writing about this in a marketing blog?” there are actually several reasons. The most obvious is that “real time analytics” and “real time interaction management” are increasingly prominent topics among marketers. Real time analytics provides insights into customer behaviors at either a group level (e.g., trends in keyword response) or for an individual (e.g., estimated lifetime value). Real time interaction management goes beyond insight to recommend individual treatments as the interaction takes place (e.g., which offer to make during a phone call). Both require the type of quick reaction to new data that stream processing can provide.
There is also increasing interest in behavior detection, sometimes called event driven marketing. This monitors customer behaviors for opportunities to initiate an interaction. The concept is not widely adopted, even thought it has proven successful again and again. (For example, Mark Holtom of Eventricity recently shared some very solid research that found event-based contacts were twice as productive as any other type. Unfortunately the details are confidential, but if you contact Mark via Eventriicty perhaps he can elaborate.) I don’t think lack of stream processing technology is the real obstacle to event-based marketing, but perhaps greater awareness of stream processing would stir up interest in behavior detection in general.
Finally, stream processing is important because so much attention has recently been focused on analytical databases that use special storage techniques such as columnar or in-memory structures. These require processing to put the data into the proper format. Some offer incremental updates, but in general the updates run as batch processes and the systems are not tuned for real-time or near-real-time reactions. So it’s worth considering stream processing systems as a complement to that lets companies employ these other technologies without giving up quick response to new data.
I suppose there's one more reason: I think this stuff is really neat. Am I allowed to say that?
Subscribe to:
Posts (Atom)
