Vicky Yu
May 23, 2021

--

For example, we have an experiment to test copy changes on an upsell screen to increase trial starts. There is Control, and 3 variants we’ll call Test1, Test2, and Test3 with each group allocated at 25% of users that log in. One week after the experiment launch, we see Test1 trial starts is 25% lower versus control and we stop showing Test1 to users. Even if we haven’t reached statistical significance there’s no reason to let Test1 continue to be shown because it’ll bring down the company’s trial starts KPI. Since Test2 and Test3 are showing results better than Control we’ll evaluate the experiment results of those variants versus control when the sample size reaches statistical significance.

--

--

Vicky Yu
Vicky Yu

Written by Vicky Yu

Musings of a data scientist turned data analyst. Sharing my data experiences one story at a time.

Responses (1)