Screen Shot 2015-03-09 at 8.56.26 AMWhen data-driven marketers want to develop better strategies, they turn to A/B testing. We chatted with Sean Switzer, CMO at apparel brand White Plum, about how the company has used insights and learnings from A/B testing to help drive their marketing strategy. 

What advice would you provide a marketer just getting started with A/B testing? 

Sean: While data points are important, gut instinct goes a long way. Obviously, you want to formulate tests based on actual results you’ve seen in the data, but sometimes informal tests can give you information that’s just as important. Being bogged down with too much data can result in something my colleagues and I call “analysis paralysis.” Here are some mistakes to avoid: running too many concurrent tests, disregarding conflicting variables, and assuming causation where there is actually correlation. It’s helpful to be open-minded and loose with tests and their results. You never know what your findings might be or where the tests might lead.

What type of A/B tests have you run for SEM that you think have been particularly impactful?

Sean: Calculating the overlap across multiple channels is the most significant A/B test that I’ve discovered. This provides marketers the ability to better understand incremental revenue. The baseline of channel-specific revenue driven through SEM category-targeted campaigns and AdRoll high-intent retargeting campaigns are both positively impacted when they are run in conjunction. More succinctly, the revenue from these combined campaigns is higher when run together than when run separately.

What type of business results did you see from these specific tests?

Sean: As indicated by the A/B test of our combined SEM-category targeted campaign and our AdRoll FBX Newsfeed high-intent campaign versus the non-combined campaigns of the same types (controlled for all potential variables), we experienced a 38% increase in attributable, incremental revenue. The data further illustrates that the ROAS was positively impacted, resulting in a 22% increase in ROAS across total spend on both campaigns.

What KPIs are most important when monitoring the success of A/B tests?

Sean: The goal of any business is to make money. As a result, ROI and ROAS are paramount when testing any campaign types. All other things being equal, the most important metric is the one that drives incremental revenue. Click-through rates (CTRs), total impressions, total clicks, and cost per impression (CPM) are all significant in their own way, but they drive the one KPI that can be most closely attributed to higher ROI/ROAS and increased incremental revenue: CPA. The calculation of CPA, or how much a marketer spends on each acquisition, just by its inherent inclusiveness, reflects all of the aforementioned metrics. CPA is the end-game for all digital marketers.


Want more great insights on A/B testing? Check out our webinar on demand: Testing Success: Top tips for unlocking revenue.