Treatment Effect Detection with Controlled FDR under Dependence for Large-Scale Experiments

Autor:	Bao, Yihan, Han, Shichao, Wang, Yong
Rok vydání:	2021
Předmět:	Statistics - Methodology Statistics - Applications
Druh dokumentu:	Working Paper
Popis:	Online controlled experiments (also known as A/B Testing) have been viewed as a golden standard for large data-driven companies since the last few decades. The most common A/B testing framework adopted by many companies use "average treatment effect" (ATE) as statistics. However, it remains a difficult problem for companies to improve the power of detecting ATE while controlling "false discovery rate" (FDR) at a predetermined level. One of the most popular FDR-control algorithms is BH method, but BH method is only known to control FDR under restrictive positive dependence assumptions with a conservative bound. In this paper, we propose statistical methods that can systematically and accurately identify ATE, and demonstrate how they can work robustly with controlled low FDR but a higher power using both simulation and real-world experimentation data. Moreover, we discuss the scalability problem in detail and offer comparison of our paradigm to other more recent FDR control methods, e.g., knockoff, AdaPT procedure, etc. Comment: 9 pages, 5 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2110.07279 Zobrazit plný text záznamu View this record from Arxiv