0 Picking Your Best Marketing Automation Vendor: One Size Won't Fit All

Summary: Vendor scores from our new B2B Marketing Automation Vendor Selection Tool offer new proof of an old truth: there's no one best system for everyone.

The one point I make every time I discuss software selection is that you have to find a vendor that matches your own business needs. No one ever denies this, of course, but the typical next question is still, Who are industry leaders? – with the unstated but obvious intention of limiting consideration to whomever gets named.

It’s not that these people didn’t listen: they certainly want a system to meet their needs. But I think they’re assuming that most systems are pretty much the same, and therefore the industry leaders are the most likely meet their requirements. The assumption is wrong but it’s hard to shake. My reluctance to contribute to this error is the main reason I’ve carefully avoided any formal ranking of vendors over the years.

But of course you know that I’ve now abandoned that position with the new B2B Marketing Automation Vendor Selection Tool (VEST) – which I’ll remind you is both totally awesome and available for sale on this very nice Web page. I’ll admit my change is partly about giving the market what it wants. But I also believe the new VEST can help to educate people about product differences, leading them to look more deeply than they would otherwise. Certainly the VEST gives them fingertip access to vastly more information about more products than they are likely to gather on their own. So, in that sense at least, it will surely help them to consider more options.

Back to the education part. Even someone as wise as you, a Regular Reader Of This Blog, may wonder whether those Important Differences really exist. After all, wouldn’t it be safe to assume that the industry leaders are in fact the strongest products across the board?

Nope.

In fact, the best thing about the new VEST may be that I finally have hard data to prove this point. The graphic below may not be very legible, but it’s really intended to illustrate patterns rather than show a lot of detailed information.

Before you squint too hard, here’s what you’re looking at:

- left to right, I’ve listed the 18 VEST vendors (nice alliteration) in order of their percentage of small business clients. So vendors with mostly small clients are at the left, and vendors with mostly large clients are at the right.

- reading down, there are three big blocks relating to vendor scores for small, mid-size and large businesses. (In case you missed a class, the VEST has different scoring schemes for those three client groups because their needs are different.)

- within the three big blocks, there are blocks for product categories (lead generation, campaign management, scoring and distribution, reporting, technology, usability and pricing) and for vendorcategories (company strength and sector presence [sectors are another term for the small, mid-size and large businesses]). Each category has its own row.

- the bright green cells represent the highest-ranked vendors for each category. Specifically, I took the vendor scores (based on the weighted sum of vendor scores on the individual items—as many as 60 items in some categories) and normalized on a scale of 0 to 1. In the product categories, green cells represent a normalized score of .9 or higher (that is, the vendor’s score was within 10% of the highest score). In the vendor categories, where the top vendor sometimes scores much higher than the rest, green cells represent a normalized score of .75 or better.

- the dark green cells show the highest combined scores across all product and vendor categories. The combined scores reflect the weights applied to the individual categories, as I explained in my earlier posts. Again, the scores are normalized and the green indicates scores higher than .9 for product fit and .75 for vendor fit.

Ok then. Now that you know what you’re looking at, here are a few observations:

- colored cells are concentrated at the left in the upper blocks, spread pretty widely in the middle, and to the right in the lower blocks. In concrete terms, this means that vendors with the most small business clients are rated most highly on small business features, vendors with a mix of clients dominate the middle, and vendors with large clients have the strongest big-client features. Not at all surprising but good validation that the scores are realistic.

- there are no solid columns of cells. That is, no single vendor is best at everything, even within a single buyer type. The nearest exception is at the bottom right, where Neolane has five green product cells out of seven for large clients. Good for them, of course, but note there are five dark green cells on the large-company product fit row: that is, several other vendors have combined product scores within 10% of Neolane’s.

- light green cells are spread widely across the rows. This means that most vendors are among the best at something. In fact, only Genius lacks at least one green cell somewhere on the diagram. (And this isn’t fair to Genius, which has some unique features that are very important to certain users.)

- dark green cells aren’t necessarily below the most light green cells. The most glaring example is in the center row, where True Influence has a dark green cell (among the best over-all) without any light green cells (not the best in any category). This reflects the range in scores within each vendor: that is, vendors are often very good at some things and not so good at others.

All these observations lead back to the same central point: different vendors are good at different things and no one vendor is best at everything. This is exactly what buyers need to recognize to understand why it isn’t safe to look only at the market leaders. Nor can they simply decide based on the category rankings: there’s plenty of variation among individual items within those rankings too. In other words, there’s truly no substitute for understanding your requirements and researching the vendors in detail. The new VEST will help, but whether you buy it or not, you still have to do the work to make a good choice.

0 B2B Marketing Automation Vendor Comparisons: New Report Next Week and The Coolest Sample Yet

I suspect you may be getting tired of reading about the features in my new report comparing B2B marketing automation vendors, and want some actual information. Soon, I promise: the final data is all ready and only some light editing stands between you and a completed report. Well, that and the fact that the e-commerce features of the www.raabguide.com Website need some work. Either way, the report will come out next week -- even if I have to take credit card orders by phone.

But I finished the final interactive component of the report yesterday and I think it's exciting enough to be worth sharing. It lets you do what I think most buyers really want, which is to compare selected vendors side by side on their specific features. You can also compare their scores, which, since you can change the weights applied to different inputs, means you can get your very own, custom comparative ranking. If that's not fun, what is?

But there's more: you get to see the results in colorful graphs. Here's a screenshot:


You can also download an interactive sample. (This is scrambled data and vendor names are replaced by sports teams. Beware that the document uses Adobe Flash; Mac users in particular may need to use Adobe Reader rather than their usual viewer. And, alas, it won't work on your iPad.)

As you can see, the screen lets you pick up three vendors, a weight set (small, mid-size or large), and the type of data to view: a summary or the individual items within each category. For each item, you see the actual input values (2, 1 or 0 depending on whether the vendor complies fully, partly, or not at all) and the scores calculated once the weights are applied. You can change the category weights (by adjusting the figures in the little gray boxes at the right) and watch the scores themselves change as the individual weights are adjusted proportionately. The graphs also adjust immediately as you make changes.

My purpose in all this is to help buyers look beneath the scores themselves to understand where the scores came from. This lets them judge whether they really care about the factors that are driving the relative rankings. Similarly, making it easy to change the weights raises the question of which weights really are appropriate. Thinking about this should lead buyers to a better decision.

The screenshot above illustrates the importance of the weights. Look at Technology: there are pretty big differences between the different "vendors", but the category as a whole has such a low weight that these make little difference in the final rankings. This reflects a judgment on my part that small business buyers don't really care much about technology and that their technology needs are pretty simple.

If you squint hard enough, you'll also notice that the middle vendor has the highest total input value for Technology, but the lowest weighted score. That's pretty common because the weights do vary substantially from one item to the next. In this instance, the main reason is that small business scores apply negative weights to many advanced features, on the theory that they detract from value by adding complexity. You'll recall that I wrote about that in an earlier post.

The downloadable sample only has descriptions under all the other tabs, but everything else is actually ready. I'll make a formal announcement next week about price and availability of the new report.

0 Ranking B2B Marketing Automation Vendors: Part 3

Summary: The first two posts in this series described my scoring for product fit. The third and final post describes scoring for vendor strength. And I'll give a little preview of the charts these scores produce...without product names attached.

Beyond assessing a vendor's current product, buyers also want to understand the current and future market position of the vendor itself. I had much less data to work with relating to vendor strength and there are many fewer conceptual issues. From a buyer’s perspective, the big questions about vendors are whether they’ll remain in business, whether they’ll continue to support and update the product, and whether they understand the needs of customers like me.

As with product fit, I used different weights for different types of buyers. As you'll see below, the bulk of the weight was assigned to concentration within each market. This reflects the fact that buyers really do want vendors who have experience with similar companies. Specific rationales are in the table. I converted the entries to the standard 0-2 scale and originally required the weights to add to 100. This changed when I added negative scoring to sharpen distinctions among vendor groups.


These weights produced a reasonable set of vendor group scores – small vendors scored best for small buyers, mixed and special vendors scored best for mid-size buyers, and big vendors scored best for big buyers. QED.


I should stress that all the score development I've described in these posts was done by looking at the vendor groups, not at individual vendors. (Well, maybe I peeked a little.) The acid test is when the individual vendors scores are plotted -- are different kinds of vendors pretty much where expected, without each category being so tightly clustered together that there's no meaningful differentiation?

The charts below show the results, without revealing specific vendor names. Instead, I've color-coded the points (each representing one vendor) using the same categories as before: green for small business vendors, black for mixed vendors, violet for specialists, and blue for big company vendors.






As you can see, the blue and green dots do dominate the upper right quadrants of their respective charts. The other colors are distributed in intriguing positions that will be very interesting indeed once names are attached. This should happen in early to mid January, once I finish packaging the data into a proper report. Stay tuned, and in the meantime have a Happy New Year.

0 Ranking B2B Marketing Automation Vendors: Part 2

Summary: Yesterday's post described the objectives of my product fit scores for B2B marketing automation vendors and how I set up the original weighting for individual elements. But the original set of scores seemed to favor more complex products, even for small business marketers. Here's how I addressed the problem.

Having decided that my weights needed adjusting, I wanted an independent assessment of which features were most appropriate for each type of buyer. I decided I could base this on the features each set of vendors provided. The only necessary assumption is that vendors offer the features that their target buyers need most. That seems like a reasonable premise -- or at least, more reliable than just applying my own opinions.

For this analysis, I first calculated the average score for each feature in each vendor group. Remember that I was working with a matrix of 150+ features for each vendor, each scored from 0 to 2 (0=not provided, 1=partly provided, 2=fully provided). A higher average means that more vendors provide the feature.

I then sorted the feature list based on average scores for the small business vendors. This put the least common small business features at the top and the most common at the bottom. I divided the list into six roughly-equal sized segments, representing feature groups that ranged from rare to very common. The final two segments both contained features shared by all small business vendors. One segment had features that were also shared by all big business vendors; the other had features that big business vendors didn't share. Finally, I calculated an average score for the big business vendors for each of the six groups.

What I found, not surprisingly, was that some features are more common in big-company systems, some are in all types of systems, and a few are concentrated among small-company systems. In each group, the intermediate vendors (mixed and special) had scores between the small and large vendor scores. This is additional confirmation that the groupings reflect a realistic ranking by buyer needs (or, at least, the vendors’ collective judgment of those needs).


The next step was to see whether my judgment matched the vendors’. Using the same feature groups, I calculated the aggregate weights I had already assigned to the those features for each buyer type. Sure enough, the big business features had the highest weights in the big business set, and the small business weights got relatively larger as you moved towards the small business features. The mid-size weights were somewhere in between, exactly where they should have been. Hooray for me!



Self-congratulation aside, we now have firmer ground for adjusting the weights to distinguish systems for different types of buyers. Remember, the small business scores in particular weren’t very different for the different vendor groups, and actually gave higher scores to big business vendors once you removed the adjustment for price. (As you may have guessed, most features in the “more small” group are price-related – proving, as if proof were necessary, that small businesses are very price sensitive.)

From here, the technical solution here is quite obvious: assign negative weights to big business features in the small business weight set. This recognizes that unnecessary features actually reduce the value of a system by making it harder to use. The caveat is that different users need different features. But that's why we have different weight sets in the first place.

(As an aside, it’s worth exploring why only assigning lower weights to the unnecessary features won’t suffice. Start with the fact that even a low weight increases rather than reduces a product score, so products with more features will always have a higher total. This is a fundamental problem with many feature-based scoring systems. In theory, assigning higher weights to other, more relevant factors might overcome this, but only if those features are more common among the simpler systems. In practice, most of the reassigned points will go to basic features which are present in all systems. This means the advanced systems get points for all the simple features plus the advanced features, while simple systems get points for the simple features only. So the advanced systems still win. That's just what happened with my original scores.)

Fortified with this evidence, I revisited my small business scoring and applied negative weights to items I felt were important only to large businesses. I applied similar but less severe adjustments to the mid-size weight set. The mid-size weights were in some ways a harder set of choices, since some big-company features do add value for mid-size firms. Although I worked without looking at the feature groups, the negative scores were indeed concentrated among the features in the large business groups:


I used the adjusted weights to create new product fit scores. These now show much more reasonable relationships across the vendor groups: that is, each vendor group has the highest scores for its primary buyer type and there’s a big difference between small and big business vendors. Hooray for me, again.


One caveat is that negative scores mean that weights in each set no longer add to 100%. This means that scores from different weight sets (i.e., reading down the chart) are no longer directly comparable. There are technical ways to solve this, but it's not worth the trouble for this particular project.

Tomorrow I'll describe the vendor fit scores. Mercifully, they are much simpler.

0 Ranking B2B Marketing Automation Vendors: How I Built My Scores (part 1)

Summary: The first of three posts describing my new scoring system for B2B marketing automation vendors.

I’ve finally had time to work up the vendor scores based on the 150+ RFP questions I distributed back in September. The result will be one of those industry landscape charts that analysts seem pretty much obliged to produce. I have never liked those charts because so many buyers consider only the handful of anointed “leaders”, even though one of the less popular vendors might actually be a better fit. This happens no matter how loudly analysts warn buyers not to make that mistake.

On the other hand, such charts are immensely popular. Recognizing that buyers will use the chart to select products no matter what I tell them, I settled on dimensions that are directly related to the purchase process:

- product fit, which assesses how well a product matches buyer needs. This is a combination of features, usability, technology, and price.

- vendor strength, which assesses a vendor’s current and future business position. This is a combination of company size, client base, and financial resources.

These are conceptually quite different from the dimensions used in the Gartner and Forrester reports* , which are designed to illustrate competitive position. But I’m perfectly aware that only readers of this blog will recognize the distinction. So I've also decided to create three versions of the chart, each tailored to the needs of different types of buyers.

In the interest of simplicity, my three charts will address marketers at small, medium and big companies. The labels are really short-hand for the relative sophistication and complexity of user requirements. But if I explicitly used a scale from simple to sophisticated, no one would ever admit that their needs were simple -- even to themselves. I've hoping the relatively neutral labels will encourage people to be more realistic. In practice, we all know that some small companies are very sophisticated marketers and some big companies are not. I can only hope that buyers will judge for themselves which category is most appropriate.

The trick to producing three different rankings from the same set of data is to produce three sets of weights for the different elements. Raab Associates’ primary business for the past two decades has been selecting systems, so we have a well-defined methodology for vendor scoring.

Our approach is to first set the weights for major categories and then allocate weights within those categories. The key is that the weights must add to 100%. This forces trade-offs first among the major categories and then among factors within each category. Without the 100% limit, two things happen:

- everything is listed as high priority. We consistently found that if you ask people to rate features as "must have" "desirable" and "not needed", 95% of requirements are rated as “must have”. From a prioritization standpoint, that's effectively useless.

- categories with many factors are overweighted. What happens is that each factor gets at least one point, giving the category a high aggregate total. For example, category with five factors has a weight of at least five, while a category with 20 factors has a weight of 20 or more.

The following table shows the major weights I assigned. The heaviest weight goes to lead generation and nurturing campaigns – a combined 40% across all buyer types. I weighted pricing much more heavily for small firms, and gabe technology, lead scoring and technology heavier weights at larger firms. You’ll notice that Vendor is weighted at zero in all cases: remember that these are weights for product fitness scores. Vendor strength will be scored on a separate dimension.


I think these weights are reasonable representations of how buyers think in the different categories. But they’re ultimately just my opinion. So I also created a reality check by looking at vendors who target the different buyer types.

This was possible because the matrix asked vendors to describe their percentage of clients in small, medium and large businesses. (The ranges were under $20 million, $20 million to $500 million, and over $500 million annual revenue.) Grouping vendors with similar percentages of small clients yielded the following sets:

- small business (60% or more small business clients): Infusionsoft, OfficeAutoPilot, TrueInfluence

- mixed (33-66% small business clients): Pardot, Marketo, Eloqua, Manticore Technology, Silverpop, Genius

- specialists (15%-33% small business): LeadFormix, TreeHouse Interactive, SalesFUSION

- big clients (fewer than 15% small business): Marketbright, Neolane, Aprimo On Demand

(I also have data from LoopFuse, Net Results, and HubSpot, but didn’t have the client distribution for the first two. I excluded HubSpot because it is a fundamentally different product.)

If my weights were reasonable, two things should happen:

- vendors specializing in each client type should have the highest scores for that client type (that is, small business vendors have higher scores than big business vendors using the small business weights.)

- vendors should have their highest scores for their primary client type (that is, small business vendors should have higher scores with small business weights than with big business weights).

As the table below shows, that is pretty much what happened:



So far so good. But how did I know I’d assigned the right weights to the right features?

I was particularly worried about the small business weights. These showed a relatively small difference in scores across the different vendor groups. In addition, I knew I had weighted price heavily. In fact, it turned out that if I took price out of consideration, the other vendor groups would actually have higher scores than the small business specialists. This couldn't be right: the other systems are really too complicated for small business users, regardless of price.


Clearly some adjustments were necessary. I'll describe how I handled this in tomorrow's post.

_______________________________________________________
* “ability to execute” and “completeness of vision” for Gartner, “current offering”, “market presence” and “strategy” for Forrester.

0 More on Web Traffic Rankings for Demand Generation Vendors

Last week’s post on Web traffic rankings of demand generation vendors generated a couple of private responses from vendors, pointing out that the Alexa statistics include traffic to operational subdomains for client landing pages and user log-in. In itself, this doesn’t bother me, since it provides data on actual use of the systems. But different products work differently*, so figures for some vendors include landing page traffic while figures for other vendors do not. This makes the rankings even less accurate than they seemed before (which wasn’t very accurate to begin with).

Despite these concerns, I still think the Alexa figures are a useful measure of vendors’ relative market position. My ultimate justification is that the rankings roughly correspond to the vendors self-reported client counts and growth rates. There's value in having an objective, non-self-reported measure, even a crude one, to put the different competitors in perspective.

Some of the vendors offered alternative measures, such as search counts from Google AdWords. Those are interesting too, although they are probably more driven by the scope of the vendor’s marketing program than anything else. My admittedly vague notion of “market presence” includes more than the number of inquiries a vendor is attracting. Basically the goal is to help people identify the “major players” in the industry, since most buyers want to focus on those products.

An interesting by-product of the post was to learn that some marketers were apparently questioned by their management about why they ranked where they did, with the implicit suggestion that a low rank suggested they were doing a poor job. Given that the Alexa figures are heavily influenced by existing client activities, this is not at all a fair inference.

That the question came up at all suggests these companies have not already established standard measures of marketing performance. If such measures were in place and reported regularly, then a random and irrelevant factiod like the Alexa rankings would not have raised any concerns or at least would have made it easy to respond. I guess it’s no news that many companies don't have proper marketing performance measures, but this is still more evidence of why they need them.

_____________________________
* Most vendors host landing pages for their clients, but sometimes the addresses are a subdomain of the client site rather than of the vendor. Client subdomains would presumably not be captured in the Alexa statistics. Many vendors offer both options, so their statistics would capture traffic for some client landing pages but not others.

0 Demand Generation Vendor Traffic Rankings

Summary: Based on Web traffic rankings, new demand generation vendors with low prices are gaining market presence. Pardot and (perhaps) Genius.com look particularly strong. But Eloqua, Silverpop and Marketo remain industry leaders.

Last November, after much consideration of alternatives, I settled on Alexa three-month Web traffic rankings as a reasonable way to measure the relative market presence of demand generation vendors. You can see that post here. I revisited that data today, adding a few new vendors and dropping some of the very minor ones. Results are in the following table.

(Note: after I posted this, it was pointed out to me that the bulk of traffic on several sites relates to customer log-ins rather than marketing prospects. For example, Alexa says that 89.4% of visitors to eloqua.com next go to now.eloqua.com, which is the domain for client landing pages and user log-in. I don't know whether this particular nuance makes the Alexa rankings a less useful indicator of market presence, but it probably means the figures relate more to existing customers than prospects. Alexa is a crude measure for many reasons -- although I do think the rankings correlate roughly with a vendor's volume of business and marketing actvitiy, I wouldn't go much further. - David)

There are no huge surprises. The leaders among demand generation systems are still Eloqua, Silverpop and Marketo. Infusionsoft and Genius.com also rank very highly, but they serve broader markets (small business and salespeople, respectively) so a direct comparison with pure-play demand generation vendors may not be appropriate. Silverpop's figures may also be inflated by its consumer email production business.

Vendors showing significant growth (highlighted in green) are mostly new entrants with below-average prices: Pardot, OfficeAutoPilot and LoopFuse. ActiveConversion is not new but also has a low price point. The outlier here is eTrigue, a long-established player that I've never looked at in depth. Their ranking is still very low, but has increased substantially. Judging from the press releases on their Web site, this may be due to a new release last October that added Salesforce.com integration. I'll explore further when time permits.

The only vendor with a really major drop in ranking was Lead Genesys, another long-time industry participant.

As the entries in the first column indicate, I've reviewed nearly all these vendors in either this blog or the Raab Guide to Demand Generation Systems. (The links all point to blog entries.) The only important exception is LoopFuse, which I have deferred at the company's request. (How about it guys? Ready yet?) NurtureHQ doesn't quite seem to be a full-scale demand generation system, insofar as it seems to lack landing pages. But its relatively high rank still surprised me, so I'll make it a priority to learn more.

My general interpretation of these numbers is that demand generation remains a dynamic market -- new participants can still enter successfully if their product and pricing are attractive enough. This is good news for marketers, since continued competition will result in continued product improvements. Major advances are still needed in usability, particularly for complex marketing programs, and in coordination with sales systems. Vendors who can deliver on these key requirements at reasonable price points should earn their place as tomorrow's industry leaders.

reviewed in:	vendor:	Alexa rank: November 2008	Alexa rank: July 2009
blog	Infusionsoft		4,993
guide	Eloqua	20,234	10,036
guide	Silverpop / Vtrenz	29,080	28,640
guide	Marketo	68,088	51,463
blog	Genius.com		70,007
blog	Pardot	211,309	92,530
blog	OfficeAutoPilot	509,868	153,232
guide	Marketbright	167,306	180,141
blog	ActiveConversion	257,058	192,634
guide	Manticore Technology	213,546	203,501
blog	Marqui Software	211,767	265,780
guide	Market2Lead	235,244	296,914
blog	Act-On Software		344,806
	LoopFuse	734,098	353,994
	NurtureHQ		367,152
blog	Treehouse Interactive		419,315
guide	Neolane	566,977	537,863
blog	LeadLife		677,156
	eTrigue	1,510,207	728,720
blog	SalesFusion360		846,961
	Lead Genesys	557,199	1,015,851
blog	True Influence		1,246,454
blog	RightOnInteractive 5Buckets		1,342,985