A/B Test Sample Size Calculator

An interactive application for determining required sample sizes in A/B tests based on desired effect size, statistical power, significance level, and variance.
Shiny
R
Author

Aleksei Prishchepo

Published

August 7, 2025

Project Overview

The A/B Test Sample Size Calculator is a small decision-support tool designed to help teams determine how much data is required to run a statistically meaningful A/B test. It addresses a core planning question in experimentation: how large does the experiment need to be to reliably detect a given effect?

By making sample size calculations interactive, the application helps balance statistical rigor with practical constraints such as traffic volume and experiment duration.

NoteRole

Product / Experimentation Analyst

NoteTools

R, Shiny

NoteDomain

Product analytics, experimentation, A/B testing

Key Features & Components

Sample size estimation for A/B tests

Calculates the required number of observations per group given target effect size, power, significance level, and variance assumptions.

Experiment planning support

Helps teams assess whether a proposed test is feasible with available traffic and within acceptable timeframes.

Design trade-off exploration

Makes explicit how stricter power requirements or smaller target effects increase sample size demands.

Read more about sample size estimation for A/B tests here

Live Demo

The application can be accessed online:

Implementation

  • Implemented in R using Shiny, translating standard frequentist sample size formulas into an interactive application.
  • Designed with a minimal interface to enable rapid iteration and clear understanding of statistical trade-offs.

See details of implementation in the project repository

Outcomes & Impact

TipExperimentation efficiency

Helps avoid underpowered or impractically large experiments.

TipInformed decision-making

Improves alignment between analytical requirements and business constraints.

TipExperimentation discipline

Demonstrates how focused analytical tools can improve experimentation discipline.

Skills Demonstrated

A/B testing • Sample size estimation • Experimental design • Power analysis • Applied statistics • Shiny app development

Apply This to Your Business

If your organization runs A/B tests or experiments, feel free to reach out via contact page to discuss how similar tools can be developed to enhance your experimentation workflows.

See Also

Back to top