Incrementality Testing vs. A/B Testing: What Is Each For?

May 9, 2025

Incrementality Testing vs. A/B Testing: What Is Each For?

It’s a Tuesday morning, and your marketing team is on Zoom for your weekly sync. But what started as a quick half-hour check-in soon turns into an hour-long argument over the subject lines in your latest email campaign. 

Finally, your boss pipes up: “We’ll settle it with an A/B test.” 

Whether you’re arguing about the syntax of a subject line or the color on your company homepage, teams frequently turn to A/B tests to optimize creative details. This dynamic comparison of two creative variants is one of the most trusted methods in a marketer’s toolkit. 

But too often, marketers discuss A/B testing interchangeably with another popular form of testing: incrementality testing. So, what’s the difference? 

While A/B testing compares variants within a campaign, incrementality testing helps you measure the campaign’s causal impact. Through randomized control trials, incrementality testing helps you figure out how many conversions were a direct result of your campaign. In short:

  • A/B testing measures which variant of a treatment is more effective
  • Incrementality testing measures whether the overall treatment drives desired business outcomes

Naturally, these two methodologies are complementary. You could run an incrementality test to see if upper-funnel Meta campaigns are an incremental channel for your brand. Once you’ve determined that it’s incremental, you could run an A/B test that helps you decide whether you should use Video Asset A or Video Asset B for your Meta campaign. 

As channels multiply — from social to search — and campaigns get complex, you need micro-level insights on the details and macro-level proof of overall impact. So let’s explore the differences between these two testing methods and how they can work together.

Overview: It’s a difference of scope and insight

You’re starting a garden, but… you don’t know what you’re doing. So you decide that this first go-round, you’re just going to experiment and learn. To determine if you need fertilizer, you sprinkle fertilizer over one plot, then just stick to water and sunlight for the adjacent plot.

Three months pass, and you learn that you do, in fact, need fertilizer. Those tomatoes that didn’t get fertilizer are looking awfully paltry. So the following season, you use fertilizer — but you test out two different fertilizers

This example underscores the fundamental difference between A/B testing and incrementality testing. A/B testing two different fertilizers doesn’t tell you if using fertilizer is more effective than not using fertilizer. And incrementality testing won’t tell you which fertilizer is most effective. 

It’s a difference of scope: A/B testing provides feedback on variants within a given treatment, while incrementality testing assesses the treatment’s overall effectiveness. By holding out a control, you measure causation, not just correlation.

Practical differences: Incrementality testing vs. A/B testing

Not all experiments are created equal. When you choose a test, consider these trade-offs:

Incrementality Testing A/B Testing
Primary purpose Measures causal lift Compares two creative variants to see which one resonates best
Key question “Would this conversion have happened without this treatment?” “Which variant of this treatment drives more conversions?”
Scope & structure Tests full campaigns or channels via treatment vs. holdout Tests specific elements — both groups see an experience, just with different versions
Complexity & cost Incrementality requires bigger samples, deeper analysis, and data science expertise A/B is more DIY — faster, cheaper, ideal for quick creative tweaks
Limitations & attribution Ignores flawed last-click models and measures pure lift Often relies on platform attribution, so you see relative wins, not net-new impact

Each method has its place — run A/B tests for rapid creative feedback, then layer in incrementality whenever you need causal proof to optimize budgets.

The fundamental differences: A/B vs. incrementality testing

We’ll put it bluntly: Incrementality testing is just a bit more involved than A/B testing. An incrementality test result might lead to a significant change in strategy or budget allocation. That’s why incrementality testing is generally recommended for brands advertising on multiple channels and spending millions of dollars yearly on marketing. 

For instance, FanDuel turned to incrementality testing with Haus because they needed to future-proof their business amid privacy changes. These were major changes that would affect the company for years to come. When they ran incrementality tests, they were able to adjust their investments and save millions of dollars.

An A/B test result can certainly affect strategy and budgeting, but rarely on that scale. A/B tests are more often used for fine-tuning after you’ve already settled on an overall strategy. An advantage? They’re quick and lean, while incrementality demands more data, more setup, and more expertise. That’s why many companies turn to an incrementality testing partner instead of handling it in-house.

A/B testing is great for optimizing creative details

A/B testing is your go-to when you want to optimize every element. When you swap a button’s shade or tweak a headline, you see which version resonates better — like auditioning actors on stage and tracking audience applause.

It’s perfect for testing one variable at a time (e.g., subject lines, button colors). Because you control exposure fully, you isolate which variation performs best — and get clear, actionable results fast.

Use A/B testing when you need rapid, low-cost insights on UI, copy, or layout tweaks. Most platforms handle the randomization, audience split, and reporting required for an A/B test — so you can focus on creative optimization.

But keep in mind: A/B testing can’t show what would’ve happened without any ad exposure — it doesn’t test the counterfactual. You'll need incrementality testing to prove true lift and net-new impact.

Ideal use cases for A/B testing

Here are a few scenarios where A/B testing really shines:

  • Email campaigns. Swap subject lines or test CTA copy. For example, maybe you run two versions of your newsletter — one question-based, one benefit-driven — and compare open rates.
  • Landing pages. Experiment with headlines or form placements. For instance, you might test a concise headline against a descriptive one, then track form submissions.
  • On-site elements. Tweak button placement, color, or product descriptions. Try a sticky “Add to cart” button vs. a static one, then measure clicks to checkout.

Ready to see Haus in action?

Discover how Haus can help you drive incremental returns on your investments

Get A Demo

Incrementality testing tracks how campaigns lead to sales

Incrementality testing measures the causal impact of your budget allocation across channels or markets, functioning as a randomized control trial for your marketing. These tests hold out a control group in order to measure how many extra conversions you drove — above what would’ve happened without an ad.

Haus’ Principal Economist Phil Erickson describes incrementality another way: “Incrementality measures how a change in strategy causes a change in business outcomes. For example, how would my revenue increase if I upped my ad budget by 10%?”

This approach scales to multi-channel campaigns, letting you see which channels are actually adding new customers, not just driving last-click conversions. 

Use cases for incrementality testing

Here are a few common situations where brands turn to incrementality testing:

  • Testing tactics: Maybe you’re trying to figure out the right mix of upper-funnel and lower-funnel creative. When Javvy tested this with Haus, they unlocked major growth
  • New channel pilots: Test before you go all-in on a shiny new platform. That’s what Ritual did before adding TikTok to their evergreen media mix
  • Optimal spend level: Perhaps your team wants to find the point of diminishing returns to allocate budget efficiently. When FanDuel tested to find their diminishing return curve, they optimized YouTube spend
  • Retargeting validation. Confirm you’re creating new conversions, not retargeting the same shoppers. For example, see Lalo’s upper-funnel testing strategy
  • Identify halo effects: If you’re an omnichannel brand, you might want to know if your ads are driving customers to your official site, social channels, Amazon, or some combination. For Newton Baby, unpacking this cross-channel journey gave them a major ROI boost.

When to use A/B testing vs. incrementality testing

Here’s your cheat sheet: start with the question you need to answer, then pick the experiment that fits.

  • Which headline converts best? Choose A/B testing.
  • Did this channel add net-new customers? Choose incrementality testing.
  • Which push-notification message maximizes click-through? Choose A/B testing.
  • Does our referral bonus program bring net-new users? Choose incrementality testing.
  • Which call-to-action button color yields more conversions? Choose A/B testing.

Keep these nuances top of mind as you plan experiments. Start by asking the right question, then apply the right method.

How A/B testing and incrementality work together

A/B testing and incrementality testing can be combined to produce even more powerful results. This involves randomly assigning users to different treatments while also holding out a control group that sees no treatment.

For instance, you could design a test where one-third of designated marketing areas (DMAs) see a PMax campaign with brand terms, one-third see a PMax campaign without brand terms, and one-third see no PMax campaign at all. 

This was Caraway’s approach when they used Haus incrementality tests to maximize PMax performance. This 3-cell test design helps you not only determine whether you should include brand terms but also whether PMax campaigns are incremental more broadly.

Why Haus leads with incrementality

These unique differentiators have led more and more brands to trust Haus as their incrementality testing partner. 

  • Proven track record: Haus has optimized billions in ad spend for customers like FanDuel, Caraway, Intuit, Sonos, Ritual, Pernod Ricard, Jones Road Beauty, and more.
  • Rigorous frontier methods: Haus uses frontier methodologies to maximize experiment accuracy and precision, like synthetic controls, which produce results that are 4X more precise than those produced by matched market tests.
  • Time to value: Haus gets you up and running quickly so that you can have your first results in a matter of weeks.
  • Dedicated team: Our teams of PhD economists, data scientists, and measurement strategists work with hundreds of large brands, running 3,000+ analyses each year.
  • Omnichannel: Haus ingests Amazon and retail data to report total lift across all sales channels for every experiment you run, helping you measure the true impact of your marketing efforts even when your customers convert on DTC, Amazon, or retail. 

Incrementality testing vs. A/B testing: FAQs

What is the fundamental difference between A/B testing and incrementality testing?

A/B testing compares two variants within an existing campaign to see which performs better, whereas incrementality testing measures the causal lift of an entire treatment by holding out a control group.

When should I use A/B testing vs. incrementality testing?

Use A/B testing for rapid, low-cost creative tweaks (e.g., subject lines, button colors) and incrementality testing when you need causal proof of net-new impact (e.g., validating channel effectiveness or budget allocation).

How can A/B testing and incrementality testing work together?

You can initially run an incrementality test to confirm a campaign’s overall lift, then follow up with A/B tests to optimize specific creative elements within that proven channel.

What practical trade-offs differentiate these testing methods?

A/B tests are quick, inexpensive, and DIY-friendly but only show relative wins, while incrementality tests require larger samples, deeper analysis, and expertise to deliver true causal insights.

Ready to see Haus in action?

Discover how Haus can help you drive incremental returns on your investments

Get A Demo

Subscribe to our newsletter

All blog articles

Marketing Measurement: What to Measure and Why

Education
May 5, 2025

This guide outlines the metrics, testing methods, and proven frameworks you can use to measure marketing effectiveness in 2025.

Why An Econometrics PhD Left Meta To Tackle Big Causal Questions at Haus

Inside Haus
May 2, 2025

Senior Applied Scientist Ittai Shacham walks us through life on the Haus Science team and the diverse expertise needed to build causal models.

What You’re Actually Measuring in a Platform A/B Test

Education
May 1, 2025

Platform creative tests may not meet the definition of a causal experiment, but they can be performance optimization tool within the bounds of the algorithm.

Beyond the Buzzwords: Why Transparency Matters in Incrementality Testing

From the Lab
Apr 29, 2025

Brands need to have complete information to make responsible decisions like their company depends on it.

Should I Build My Own MMM Software?

Education
Apr 11, 2025

Let's unpack the pros and cons of building your own in-house marketing mix model versus working with a dedicated measurement partner.

Why An Analytics Expert Left Agency Life to Become Haus' First Measurement Strategist

Inside Haus
Apr 10, 2025

Measurement Strategy Team Lead Alyssa Francis sat down with us to discuss how she pushes customers to challenge the testing status quo.

Why An Analytics Expert Left Agency Life to Become Haus' First Measurement Strategist

Why An Analytics Expert Left Agency Life to Become Haus' First Measurement Strategist
Inside Haus
Apr 10, 2025

Measurement Strategy Team Lead Alyssa Francis sat down with us to discuss how she pushes customers to challenge the testing status quo.

Understanding Incrementality Testing

Understanding Incrementality Testing
Education
Apr 2, 2025

Fuzzy on some of the nuances around incrementality testing? This guide goes deep, unpacking detailed examples and step-by-step processes.

MMM Software: What Should You Look For?

MMM Software: What Should You Look For?
Education
Mar 27, 2025

We discuss some of the key questions to ask a potential MMM provider — and the importance of prioritizing causality.

How to Know If An Incrementality Test Result Is ‘Good’ – And What to Do About It

How to Know If An Incrementality Test Result Is ‘Good’ – And What to Do About It
Education
Mar 21, 2025

Plus: What to do when a test result is incremental but not profitable, and a framework for next steps after a test.

Why A Leading Economist From Amazon Came to Haus to Democratize Causal Inference

Why A Leading Economist From Amazon Came to Haus to Democratize Causal Inference
Inside Haus
Mar 19, 2025

We sit down with Principal Economist Phil Erickson to talk about Haus’ “unhealthy obsession” with productizing causal inference.

Haus x Crisp: Measure What Matters in CPG Marketing

Haus x Crisp: Measure What Matters in CPG Marketing
Haus Announcements
Mar 13, 2025

When real-time retail data meets incrementality testing, CPG brands can finally measure what’s working and optimize ad spend with confidence.

Why Magic Spoon’s Former Head of Growth Embraces Incrementality at Haus

Why Magic Spoon’s Former Head of Growth Embraces Incrementality at Haus
Inside Haus
Mar 10, 2025

In our first episode of Haus Spotlight, we speak to Measurement Strategist Chandler Dutton about the in-the-weeds approach Haus takes with customers.

Do YouTube Ads Perform? Lessons From 190 Incrementality Tests

Do YouTube Ads Perform? Lessons From 190 Incrementality Tests
From the Lab
Mar 6, 2025

An exclusive Haus analysis shows YouTube often delivers powerful new customer acquisition and retail halo effects that traditional metrics miss.

Getting Started with Causal MMM

Getting Started with Causal MMM
Education
Feb 24, 2025

Causal MMM isn’t rooted in historical correlational data – it’s rooted in causal reality.

A First Look at Causal MMM

A First Look at Causal MMM
Haus Announcements
Feb 19, 2025

Causal MMM is a new product from Haus founded on incrementality experiments. Coming 2025.

Would You Bet Your Budget on That? The Case for Honest Marketing Measurement

Would You Bet Your Budget on That? The Case for Honest Marketing Measurement
From the Lab
Feb 14, 2025

Acknowledging uncertainty enables brands to make better, more profitable decisions.

Incrementality: The Fundamentals

Incrementality: The Fundamentals
Education
Feb 13, 2025

Let's explore incrementality from every angle — what it is, what you can test, and what you need to get started.

Getting Started with Incrementality Testing

Getting Started with Incrementality Testing
Education
Feb 7, 2025

As the customer journey grows more complex, incrementality testing helps you determine the true, causal impact of your marketing.

Matched Market Tests Don't Cut It: Why Haus Uses Synthetic Control in Incrementality Experiments

Matched Market Tests Don't Cut It: Why Haus Uses Synthetic Control in Incrementality Experiments
From the Lab
Jan 28, 2025

Haus’ synthetic control produces results that are 4x more precise than those produced by matched market tests.

Incrementality School, E6: How to Foster a Culture of Incrementality Experimentation

Incrementality School, E6: How to Foster a Culture of Incrementality Experimentation
Education
Jan 16, 2025

Having the right measurement toolkit for your business is only meaningful insofar as your team’s ability to use that tool.

Geo-Level Data Now Available for Amazon Vendor Central Brands

Geo-Level Data Now Available for Amazon Vendor Central Brands
Industry News
Jan 6, 2025

Vendor Central sellers – brands that sell *to* Amazon – can now use Haus to measure omnichannel incrementality.

How Does Traditional Marketing Mix Modeling (MMM) Work?

How Does Traditional Marketing Mix Modeling (MMM) Work?
Education
Jan 2, 2025

Traditional marketing mix modeling (MMM) often relies on linear regression to illustrate correlation, not causation.

2025: The Year of Privacy-Durable Marketing Measurement

2025: The Year of Privacy-Durable Marketing Measurement
From the Lab
Dec 28, 2024

Haus incrementality testing doesn’t rely on pixels, PII, or other data that may be vulnerable to privacy regulations.

Meta Shares New Conversion Restrictions for Health and Wellness Brands

Meta Shares New Conversion Restrictions for Health and Wellness Brands
Industry News
Nov 25, 2024

Developing story: Starting in January 2025, some health and wellness brands may not be able to measure lower-funnel conversion events on Meta.

Incrementality School, E5: Randomized Control Experiments, Conversion Lift Testing, and Natural Experiments

Incrementality School, E5: Randomized Control Experiments, Conversion Lift Testing, and Natural Experiments
Education
Nov 21, 2024

Sure, the title's a mouthful – but attributing changes in data (ex: ‘my KPI went up') to certain factors (ex: ‘we increased ad spend’) is hard to do well.

Incrementality Testing: How To Choose The Right Platform

Incrementality Testing: How To Choose The Right Platform
Education
Nov 19, 2024

Whether you’re actively evaluating incrementality platforms or simply curious to learn more, consider this checklist your first stop.

Incrementality School, E4: Who Needs Incrementality Testing?

Incrementality School, E4: Who Needs Incrementality Testing?
Education
Nov 14, 2024

As brands' marketing strategies grow in complexity, incrementality testing becomes increasingly consequential.

Incrementality School, E3: How Do Brands Measure Incrementality?

Incrementality School, E3: How Do Brands Measure Incrementality?
Education
Nov 7, 2024

Traditional MTAs and MMMs won't measure incrementality – but geo experiments reveal clear cause, effect, and value.

Incrementality School, E2: What Can You Incrementality Test?

Incrementality School, E2: What Can You Incrementality Test?
Education
Oct 31, 2024

Haus’ Customer Marketing Lead Maddie Dault and Success Team Lead Nick Doren dive into what you can incrementality test – and why now's the time.

Incrementality School, E1: What is Incrementality?

Incrementality School, E1: What is Incrementality?
Education
Oct 24, 2024

To kick off our new Incrementality School series, three Haus incrementality experts weigh in describing a very fundamental concept.

Inside the Offsite: Why Haus?

Inside the Offsite: Why Haus?
Inside Haus
Oct 17, 2024

At this year's offsite, we dove into why – of all the companies, options, and career paths out there – our growing team chose Haus.

Haus Named One of LinkedIn's Top Startups

Haus Named One of LinkedIn's Top Startups
Inside Haus
Sep 25, 2024

A note from Zach Epstein, Haus CEO.

Google Announces Plan to Migrate Video Action Campaigns to Demand Gen

Google Announces Plan to Migrate Video Action Campaigns to Demand Gen
Industry News
Sep 6, 2024

The news leaves advertisers swimming in uncertainty — which is why it’s so important to test before the change.

Conversion Lag Insights: How Haus Tests Can Help Optimize Q4 Budgets

Conversion Lag Insights: How Haus Tests Can Help Optimize Q4 Budgets
From the Lab
Sep 5, 2024

Post-treatment windows offer a unique glimpse into the lingering impacts of advertising campaigns after they’ve concluded.

PMAX Experiments Revealed: Including vs. Excluding Branded Search Terms

PMAX Experiments Revealed: Including vs. Excluding Branded Search Terms
From the Lab
Aug 20, 2024

We analyzed experiments from leading brands to understand the incremental impacts of including vs. excluding branded terms in PMAX campaigns.

CommerceNext Session Recap: How Newton Baby Uses Incrementality Experiments to Maximize ROI

CommerceNext Session Recap: How Newton Baby Uses Incrementality Experiments to Maximize ROI
From the Lab
Aug 9, 2024

“We ran the test of cutting spend pretty significantly and it turns out a lot of that spend was not incremental,” says Aaron Zagha, Newton Baby CMO.

Introducing Causal Attribution: Your New Daily Incrementality Solution

Introducing Causal Attribution: Your New Daily Incrementality Solution

Causal Attribution syncs your ad platform data with your experiment results to provide a daily read on which channels drive growth.

Haus Announces $20M Raise Led by 01 Advisors

Haus Announces $20M Raise Led by 01 Advisors
Haus Announcements
Jul 30, 2024

With this additional support, Haus is well-positioned to deepen our causal inference capabilities and announce a new product: Causal Attribution.

3 Ways to Perfect Your Prime Day Marketing Strategy

3 Ways to Perfect Your Prime Day Marketing Strategy
Education
Jun 26, 2024

Think Amazon ads are the only effective marketing channel for Prime Day? Think again.

Maximize Your Q4 Growth With 4 High-Impact, Low-Risk Tests

Maximize Your Q4 Growth With 4 High-Impact, Low-Risk Tests
Education
Nov 8, 2023

Not testing during your busy season may be costing you more than you think.

Why Maturing Direct to Consumer Brands Need to Run Incrementality Tests

Why Maturing Direct to Consumer Brands Need to Run Incrementality Tests
Education
Sep 15, 2023

The media strategy that gets DTC brands from zero to one does not get them from one to ten.

5 Signs It’s Time to Invest in Incrementality

5 Signs It’s Time to Invest in Incrementality
Education
Aug 9, 2023

5 common signs that indicate it is definitely time to start investing in incrementality.

$17M Series A, Led by Insight Partners

$17M Series A, Led by Insight Partners

Haus raises $17M Series A led by Insight Partners to build the future of growth intelligence.

Why Meta's “Engaged Views” Is a Distraction, Not a Solution

Why Meta's “Engaged Views” Is a Distraction, Not a Solution
Industry News
Jul 25, 2023

While additional data can be useful, we must question whether this new rollout is truly a solution or merely another diversion.

Why You Need a 3rd Party Incrementality Partner

Why You Need a 3rd Party Incrementality Partner
Education
Jul 6, 2023

Are you stuck wondering if you should be using 3rd party incrementality studies, ad platform lift studies, or trying to design your own? Find out here.

iOS 17 Feels Like iOS 14 All Over Again. What It Means for Growth Marketing…And Does It Matter Anymore?

iOS 17 Feels Like iOS 14 All Over Again. What It Means for Growth Marketing…And Does It Matter Anymore?
Industry News
Jun 12, 2023

A single press release vaguely confirmed that Apple will continue its assault on user level attribution. Here, I unpack what I think it means for growth marketing.

How Automation Is Transforming Growth Marketing

How Automation Is Transforming Growth Marketing
Education
May 30, 2023

As platforms force more automation, the role of the media buyer is evolving. Read on to learn what to expect and what levers are left to pull.

Statistical Significance Is Costing You Money

Statistical Significance Is Costing You Money
From the Lab
Apr 13, 2023

It is profitable to ignore statistical significance when making marketing investments.

The Secret to Comparing Marketing Performance Across Channels

The Secret to Comparing Marketing Performance Across Channels
Education
Mar 2, 2023

While incrementality is better than relying on attribution alone, comparing them as-is is challenging. Thankfully, there’s a better way to get an unbiased data point regardless of the channel.

Your Attribution Model Is Precise but Not Accurate - Here’s Why

Your Attribution Model Is Precise but Not Accurate - Here’s Why
Education
Feb 8, 2023

Learn which common marketing measurement tactics are accurate, precise, neither or both.

How to Use Causal Targeting to Save Money on Promotions

How to Use Causal Targeting to Save Money on Promotions
Education
Feb 1, 2023

Leverage causal targeting to execute promotions that are actually incremental for your business.

Are Promotions Growing Your Business or Losing You Money?

Are Promotions Growing Your Business or Losing You Money?
Education
Feb 1, 2023

Promotions, despite their potential power and ubiquity, are actually hard to execute well.

User-Level Attribution Is Out. Geo-Testing Is the Future.

User-Level Attribution Is Out. Geo-Testing Is the Future.
Education
Jan 27, 2023

Geotesting is a near-universal approach for measuring the incremental effects of marketing across both upper and lower funnel tactics.

The Haus Viewpoint

The Haus Viewpoint
Inside Haus
Jan 18, 2023

We are building Haus to democratize access to world-class decision science tools.