Refine
Has Fulltext
- yes (1)
Year of publication
- 2019 (1)
Document Type
- Master's Thesis (1)
Language
- English (1) (remove)
Is part of the Bibliography
- yes (1)
Keywords
- Sequential statistical Testing (1) (remove)
Institute
Sequential Statistical Testing Procedures as an Early Stopping for Binomial Bandit Experiments
(2019)
This study tries to provide an early stopping procedure on binomial bandits which is a type of multi-arm bandit experiment. In addition to that, it presents sequential statistical testing procedures which can be used as early stopping criteria for A/B experiment. The paper searches for the applicability of these procedures for binomial bandit case because multi-arm bandit experiment and A/B experiment are similar in the sense that rewards can be simulated as independent identical Bernoulli distribution.
It is often claimed that multi-armed bandit which use Thompson sampling requires dramatically less sample size than A/B testing while still controlling the type 1 and type 2 error rates on alpha and beta due to the concept of always switching to the better arm. However, it is also claimed that Bayesian procedures are not immune to peeking (early stopping) because of the structure of sequential testing.
The sequential statistical testing procedures reduce the required number of observations and allow the experiment to stop early when the collected data is good enough to make a conclusion. In this work, Wald’s SPRT and Max SPRT-I-AA (a modified version of Max SPRT) sequential statistical testing procedures are studied for the A/B testing, and their applicability to binomial bandit case is researched.
According to simulation results, Max SPRT-I-AA sequential statistical testing procedure perform well for the A/B test scenarios. The type 1 and type 2 error rates are remains on the acceptable level and the experiment time is reduced more than 50 %. However, for the binomial bandit case results are not so satisfactory and brings more questions about the applicability of Max SPRT-I-AA to multi-arm bandits because of the performance of upper boundary calculation.