Minimum Detectable Effect
Understanding Minimum Detectable Effect is crucial for effective A/B testing in mobile app development.

You launch a test on Friday. By Monday, the dashboard says the variant is up a little, or down a little, or basically flat. Nobody knows whether to call it a win, leave it running, or kill it and move on.

That's where most mobile app teams get stuck. The creative might be better. The onboarding flow might be cleaner. The paywall might improve conversion. But if the test wasn't designed to detect the kind of change you care about, the result is just noise dressed up as analysis.

Minimum detectable effect fixes that. It gives your team a practical answer to one question that matters before launch: what size improvement is this test capable of detecting? For app growth teams, that's not academic. It affects how long you run tests, which metrics you trust, when you use proxy metrics instead of LTV, and whether your roadmap is driven by evidence or vibes.

Table of Contents

• Why Your A/B Test Results Are Inconclusive
  • The missing ruler
• What Is Minimum Detectable Effect Really
  • Think of MDE like a fishing net
  • The planning choice most teams skip
• The Four Levers That Control Your MDE
  • Sample size changes everything
  • Alpha and power are business settings
  • Metric noise matters more than people expect
• How to Choose the Right MDE for Your App
  • Start with the decision, then work backward
  • Choose differently by metric, not by habit
  • A practical way to set an MDE
  • Example MDEs for common app metrics
• Common MDE Pitfalls and How to Avoid Them
  • Mistaking MDE for a post-test hurdle
  • Design mistakes that quietly wreck sensitivity
• Making MDE Your Competitive Advantage

Why Your A/B Test Results Are Inconclusive

A new UA manager usually meets minimum detectable effect the hard way. It's Monday morning, spend is live, and the first readout looks useless. Variant B has a slight edge on install-to-trial rate, retention looks mixed, and revenue is bouncing around enough that everyone starts arguing from prior beliefs instead of data.

That's not always because the experiment failed. Often, the test was never built to answer the question the team thought it was asking. If you didn't define the size of effect worth detecting before launch, you don't know whether a flat result means “no impact” or “not enough sensitivity.”

!A professional analyzing an A/B test chart on a computer screen in a home office.

A lot of confusion starts when teams mix up test setup. Running a proper experiment isn't the same as loosely comparing performance between campaigns. If your team still blurs that line, this breakdown of A/B tests versus two ad sets is worth reading before you trust any result.

The missing ruler

Minimum detectable effect is your ruler. It tells you the smallest true effect your test is designed to have a good chance of detecting.

Without that ruler, teams make two expensive mistakes:

• They stop too early. A promising change gets labeled “no difference” because the test never had enough sensitivity.
• They run too long. A test drags on even though it was only realistic for large swings and was never going to resolve tiny movement.
• They choose the wrong metric. Revenue or long-window retention gets used when the data is too noisy for the traffic available.
• They confuse movement with meaning. Small changes on a dashboard look important until you ask whether the experiment could reliably detect them.

Practical rule: If you can't say what size lift would matter to the business, you're not ready to launch the test.

For mobile apps, this matters more than many teams admit. Install funnels move quickly, but useful decisions often depend on downstream behavior like activation, subscription, retention, and monetization. Those metrics don't all behave the same way. A clean onboarding conversion event is one thing. LTV is another.

The point isn't to become a statistician. The point is to stop treating inconclusive tests as bad luck when they're often just bad planning.

What Is Minimum Detectable Effect Really

Minimum detectable effect sounds technical, but the idea is simple. It's a planning threshold, not a result. Before you run the test, you choose the smallest effect that would be worth detecting, then size the experiment around that.

!An infographic explaining Minimum Detectable Effect in A/B testing using a fishing net analogy.

Think of MDE like a fishing net

The easiest way to think about minimum detectable effect is a fishing net.

If the holes are wide, you'll catch big fish and miss small ones. In testing terms, that means your experiment can detect large improvements, but smaller real gains may slip through. If the holes are tighter, the test becomes more sensitive, but now you need more time, more traffic, or both.

That's why MDE is a trade-off lever. You are deciding whether this test is built to catch whales or minnows.

For app teams, that trade-off shows up everywhere:

• Creative tests can often tolerate a larger MDE because you're looking for clear winners.
• Onboarding tests often need tighter sensitivity because small changes compound across paid traffic.
• Retention and revenue tests are harder because the metrics are noisier and slower.

A quick visual can help if you want the concept explained another way.

The planning choice most teams skip

The most important thing to understand is that minimum detectable effect is chosen before the test starts. It's tied to sample size, significance, and power. Eppo's documentation puts it plainly: minimum detectable effect is a power-analysis threshold determined by sample size, significance level, and desired power, and common defaults are 5% significance and 80% power when sizing experiments (Eppo on MDE and sample size planning).

That matters because MDE is not some hidden property of your idea, your paywall, or your ad. It's a property of the experiment design.

Minimum detectable effect is where statistics becomes operations. You're choosing how sensitive the test should be, then paying for that sensitivity with time and traffic.

Here are the four concepts that sit together in practice:

1. MDE is the smallest effect you care to detect.
2. Sample size determines how much data you collect.
3. Significance level controls how strict you are about false positives.
4. Power reflects your chance of detecting a real effect at or above the chosen MDE.

If one setting changes, the others move with it. That's why experienced growth teams don't ask, “What's the right MDE?” in isolation. They ask, “What effect size matters enough to justify the runtime this app can support?”

The Four Levers That Control Your MDE

Once you stop treating minimum detectable effect like a textbook definition, it becomes operational. You have levers. Pull one, and something else moves.

!A visual guide illustrating the four key factors that influence the Minimum Detectable Effect in statistical testing.

Sample size changes everything

The biggest lever is sample size. More users make estimates less noisy, which lowers the minimum detectable effect.

This relationship isn't linear. In a causal-inference power example using α = 0.05 and power = 0.8, the detectable effect falls from 0.5 at N = 100 to 0.15 at N = 1,000, 0.05 at N = 10,000, and 0.02 at N = 100,000. The same source also shows that 50/50 randomization is the most efficient design for sensitivity (power analysis examples and balanced assignment).

That's a key takeaway for an app team. Doubling traffic doesn't mean you halve your MDE in a neat, simple way. Getting more sensitive becomes progressively harder, which is why small-effect tests can become painfully slow.

If you're testing multiple ad concepts at once, structure matters too. This guide on multiple creative A/B testing in Meta Ads is useful because splitting traffic across too many variants can insidiously destroy sensitivity.

Alpha and power are business settings

Alpha and power are often treated like defaults inherited from a calculator. That's fine until the default creates a test that doesn't fit the business.

Alpha is about how cautious you want to be about false positives. Power is about how willing you are to miss a real winner. There's no free lunch here. If you want stronger detection, you usually pay in runtime or sample requirements.

A practical way to think about them:

• Lower tolerance for false positives: Better when shipping a product change is costly or hard to reverse.
• Higher power: Better when missing a real gain would be expensive.
• Faster tests: Usually require accepting a less sensitive setup or targeting only larger effects.
• Strict settings everywhere: Sounds disciplined, but can make routine experimentation too slow to be useful.

The right settings depend on decision cost. A cheap creative test and a major pricing change shouldn't always be held to the same operational standard.

Metric noise matters more than people expect

This is the lever people notice last. Some app metrics are relatively stable. Others are messy, skewed, and heavily affected by timing, user mix, or seasonality.

That's why the same app can run a clean test on onboarding completion and struggle to read anything from early revenue. Minimum detectable effect isn't fixed. It moves with variance, sample size, power, significance, and design choices such as clustering, as discussed in this overview of MDE, variance, and underpowered studies.

For mobile app teams, that has direct consequences:

The mistake is pretending all metrics deserve the same test design. They don't.

How to Choose the Right MDE for Your App

A UA manager launches a retention test on Monday, asks for a 1 percent lift target, and by Friday the test still has no clear read. The problem usually is not the experiment tool. The team picked a sensitivity target that does not fit the traffic, the metric, or the decision.

Choosing minimum detectable effect is planning the smallest result that would change what your team does. If a win at that size would not change spend, rollout, paywall design, or creative production, it is too small to anchor the test. If the target is so ambitious that the team would ignore anything smaller, it is probably too large.

Start with the decision, then work backward

The clean question is simple: what is the smallest lift worth acting on?

For app teams, that answer depends on economics and speed. A 3 percent gain in onboarding completion can matter if it affects every paid user you acquire. A 3 percent gain in D30 revenue may sound attractive but still be a poor primary target if it takes too long to read and delays five other tests.

I usually pressure-test MDE choices with four operational questions:

• Would this result change a real decision? If the answer is no, the MDE is too small.
• Can we reach a read before the test becomes stale? If seasonality, campaign changes, or roadmap deadlines will hit first, the MDE is too small for the available traffic.
• Is this metric clean enough to support that level of sensitivity? Onboarding events often can. Revenue and long-window retention often cannot.
• What are we not testing while this runs? Long tests carry a pipeline cost, not just a stats cost.

That last point gets missed a lot. For mobile apps, experiment speed has real value. A test that needs weeks to detect a tiny effect may be technically correct and still be the wrong choice for a fast-moving UA program.

Analytics Toolkit makes the trade-off clear. Tests planned around smaller lifts need much more traffic and time than tests planned around larger lifts, even with the same significance and power settings (trade-off between small and large planned lifts).

Choose differently by metric, not by habit

Teams often reuse one default MDE across every test. That creates bad plans.

Onboarding conversion can justify a tighter MDE because volume is usually higher and the metric is closer to the top of the funnel. Small gains here can improve downstream paid efficiency, especially for apps buying traffic at scale.

Paywall or trial start rate usually deserves a middle ground. The metric matters commercially, but eligible traffic is narrower and segment mix can change fast. It is often worth detecting moderate lifts, not the tiniest possible ones.

Feature adoption usually does not need extreme sensitivity. Product and growth teams rarely need proof of a tiny movement to decide whether a feature deserves more distribution, more education, or less roadmap attention.

Retention forces harder trade-offs. It matters a lot, but it is slower and noisier. For many apps, a very small retention MDE sounds disciplined and ends up producing tests that run too long to be useful.

LTV, ARPU, or ARPDAU should be handled with humility. These metrics matter to the business, but they are often poor choices for fast experiment reads. In practice, many teams use a cleaner leading indicator for the main call and check revenue metrics as a secondary validation layer.

Good MDE selection is not about maximum precision. It is about getting a decision-grade answer fast enough to use.

A practical way to set an MDE

A workable process for UA teams looks like this:

1. Name the business decision. Example: scale a creative concept, ship a paywall change, or keep the current onboarding.
2. Set the smallest lift that would change that decision. Use margin, spend, and roadmap cost. Do not use wishful thinking.
3. Check runtime before you commit. If the required sample pushes the test past a useful window, raise the MDE or switch to a cleaner proxy metric.
4. Match the primary metric to the test type. Creative and onboarding tests can often read on earlier events. Monetization and retention changes may need more patience.
5. Write down the trade-off. “We are choosing a larger MDE so we can test three ideas this month instead of one” is a valid strategy.

That is the practical version of MDE planning. It is not academic purity. It is choosing how much certainty you need before taking action, while keeping the testing pipeline healthy.

Example MDEs for common app metrics

The table below stays qualitative on purpose. Apps vary too much for fixed benchmarks to be useful here.

A simple rule works well in practice. Use smaller MDEs for high-volume, stable metrics tied closely to business value. Use larger MDEs for noisy, slow, or directional metrics where speed matters more than precision.

Common MDE Pitfalls and How to Avoid Them

Most MDE mistakes don't come from bad math. They come from teams using the right term in the wrong way.

!An infographic illustrating three common MDE pitfalls in A/B testing and their corresponding solutions for statistical accuracy.

Mistaking MDE for a post-test hurdle

This is the most common misunderstanding. A team plans a test around one target effect, then later looks at the observed lift and says, “It didn't beat the MDE, so it doesn't count.”

That's wrong.

Analytics Toolkit's guidance is clear: the observed effect does not have to exceed the MDE to be meaningful. MDE is a design-time sensitivity threshold for planning, not a post-test cutoff, and a significant result can still matter even if the observed effect is smaller than the original planned MDE (why observed lift doesn't need to beat planned MDE).

The operational correction is simple. Use MDE to plan the test. Use the actual statistical result to interpret the test outcome.

If the test is significant, treat that result on its own terms. Don't invent a second hurdle after the data arrives.

Design mistakes that quietly wreck sensitivity

Other pitfalls are less obvious because they happen before the dashboard ever loads.

Here are the ones that show up constantly in app growth work:

• Choosing an MDE that your traffic can't support: Teams plan for very small effects even though user volume makes the required runtime unrealistic.
• Ignoring MDE entirely: The test launches because the feature is ready, not because the experiment is properly sized.
• Using a noisy primary metric: Revenue, retention, or blended monetization gets picked when a cleaner upstream metric would have produced a usable decision faster.
• Changing the target mid-test: Once pressure builds, teams start redefining what they were hoping to detect.
• Breaking balance across groups: Uneven allocation can reduce sensitivity when a balanced split would have been cleaner.

A better approach looks like this:

1. Define the meaningful effect before launch. Tie it to the business decision, not your optimism.
2. Check whether the metric is stable enough. If not, move up-funnel or accept a larger MDE.
3. Protect the design during the run. Don't keep rewriting the experiment because early results are uncomfortable.
4. Read the outcome correctly. Planning thresholds and post-test interpretation are not the same thing.

A disciplined team doesn't avoid inconclusive tests entirely. It avoids avoidable inconclusive tests.

Making MDE Your Competitive Advantage

Most app teams think their edge comes from better ads, better product instincts, or faster execution. Those matter. But a quieter advantage sits underneath them: the ability to tell which changes are proving effective.

That's where minimum detectable effect becomes useful beyond statistics. It forces clarity before launch. What size lift matters? How long are you willing to wait? Which metric is stable enough to trust? When should you use a proxy instead of pretending LTV will give a fast answer?

Teams that answer those questions well make better decisions with the same traffic. They waste less time on tests that were doomed to be inconclusive. They stop killing good ideas too early. They also stop celebrating meaningless movement.

There's a broader lesson here for modern growth teams. AI can help generate more creative, more copy, more test ideas, and more variants at speed. That lowers the cost of producing experiments. It doesn't lower the cost of bad measurement. If anything, faster execution makes disciplined experiment design more important, not less.

Minimum detectable effect is part of that discipline. It's how you connect creative ambition to decision quality.

A practical habit helps. Before the next onboarding test, paywall experiment, or creative iteration goes live, write down three things: the metric, the smallest effect worth caring about, and the runtime you can realistically tolerate. If those three don't fit together, fix the design before you spend another day collecting ambiguous data.

That habit sounds small. Over time, it changes how a UA team operates.

If you're working on app growth and want sharper inputs for the kinds of conversion tests discussed here, Marketing For Apps By @designerants focuses on ad creative and app marketing assets for mobile teams. That can be useful when you're not just trying to generate more test variants, but trying to generate variants worth testing.