Enterprise-Level Controlled Experiments at Scale: Challenges and Solutions
| Autor: | Sophia Liu, Aleksander Fabijan, Pawel A. Janowski, Somit Gupta, Wen Qin, Michael Furchtgott, Pavel Dmitriev |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Outreach
Enterprise level Trustworthiness Computer science Scale (chemistry) 0202 electrical engineering electronic engineering information engineering Mobile apps Software product development 020206 networking & telecommunications 020207 software engineering 02 engineering and technology Data science |
| Zdroj: | SEAA |
| DOI: | 10.1109/seaa.2019.00013 |
| Popis: | Online controlled experiments, known as A/B tests, have delivered tremendous value for businesses and customers. Traditionally, online controlled experiments running in search engines, websites, mobile apps, and other Business-to-Consumer (B2C) products use individual customers as randomization units. However, this may not be possible in Business-to-Business (B2B) products that are sold to, and used by, other enterprises. B2B products typically have constraints such as the need to provide identical experiences to all users within an enterprise. In other scenarios, such as features that need to be used by multiple users to be meaningful or the metrics of interest can be only obtained by showing the same experience under one enterprise, experimentation randomized by enterprise is needed. In this paper, based on experiences from Microsoft and Outreach, we introduce the concept of Enterprise Level Experiments (ELE) in software product development. We discuss the unique technical challenges in designing and analyzing ELEs and present solutions to overcome them. Furthermore, we present and discuss a real-world example of ELE run by Microsoft. Our aims in this paper are to provide practical guidance on running trustworthy experiments in B2B scenarios, and to stimulate further research in this new and important area of experimentation. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|