A/B Test Sample Size Calculator

An interactive application for determining required sample sizes in A/B tests based on desired effect size, statistical power, significance level, and variance.
Shiny
R
Author

Aleksei Prishchepo

Published

August 7, 2025

Project Overview

The A/B Test Sample Size Calculator is a small decision-support tool designed to help teams determine how much data is required to run a statistically meaningful A/B test. It addresses a core planning question in experimentation: how large does the experiment need to be to reliably detect a given effect?

By making sample size calculations interactive, the application helps balance statistical rigor with practical constraints such as traffic volume and experiment duration.

NoteRole

Product / Experimentation Analyst

NoteTools

R, Shiny

NoteDomain

Product analytics, experimentation, A/B testing

Key Features & Components

Sample size estimation for A/B tests

Calculates the required number of observations per group given target effect size, power, significance level, and variance assumptions.

Experiment planning support

Helps teams assess whether a proposed test is feasible with available traffic and within acceptable timeframes.

Design trade-off exploration

Makes explicit how stricter power requirements or smaller target effects increase sample size demands.

Read more about sample size estimation for A/B tests here

Live Demo

The application can be accessed online:

Implementation

  • Implemented in R using Shiny, translating standard frequentist sample size formulas into an interactive application.
  • Designed with a minimal interface to enable rapid iteration and clear understanding of statistical trade-offs.

See details of implementation in the project repository

Outcomes & Impact

TipExperimentation efficiency

Helps avoid underpowered or impractically large experiments.

TipInformed decision-making

Improves alignment between analytical requirements and business constraints.

TipExperimentation discipline

Demonstrates how focused analytical tools can improve experimentation discipline.

Skills Demonstrated

A/B testing • Sample size estimation • Experimental design • Power analysis • Applied statistics • Shiny app development

Apply This to Your Business

If your organization runs A/B tests or experiments, feel free to reach out via contact page to discuss how similar tools can be developed to enhance your experimentation workflows.

See Also

No matching items
Back to top