We design new policies that ensure both worst-case optimality for expected regret and light-tailed risk for regret distribution in the stochastic multi-armed bandit problem. Recently, arXiv:2109.13595 showed that information-theoretically optimized bandit algorithms as well as standard UCB policies suffer from some serious heavy-tailed risk. Inspired by their results, we further show that heavy-tailed risk actually exists for all "instance-dependent consistent" policies. In particular, any policy that incurs an instance-dependent $O(\ln T)$ expected regret must incur a linear regret with probability $\Omega(\text{poly}(1/T))$. With the aim to ensure safety against such heavy-tailed risk, starting from the two-armed bandit setting, we provide a simple policy design that (i) has the worst-case optimality for the expected regret at order $\tilde O(\sqrt{T})$ and (ii) has the worst-case tail probability of incurring a linear regret decay at an optimal exponential rate $\exp(-\Omega(\sqrt{T}))$. Next, we improve the policy design and analysis to the general $K$-armed bandit setting. Specifically, the worst-case probability of incurring a regret larger than $x$ is upper bounded by $\exp(-\Omega(x/\sqrt{KT}))$. We also enhance the policy design to accommodate the "any-time" setting where $T$ is not known a priori. A brief account of numerical experiments is conducted to illustrate the theoretical findings. We conclude by extending our proposed policy design to the general stochastic linear bandit setting and obtain light-tailed regret bound. Our results reveal insights on the incompatibility between consistency and light-tailed risk, whereas indicate that worst-case optimality on expected regret and light-tailed risk on regret distribution are compatible.
翻译:我们设计了新政策,确保最坏情况的最佳性,既符合预期的遗憾,也符合在多武装匪徒问题中进行遗憾分配的轻尾风险。最近,arxiv:2109.135995显示,信息-理论优化强盗算法和标准的UCB政策存在一些严重的重尾风险。根据它们的结果,我们进一步显示,所有“依赖内向性一致”政策实际上都存在最坏情况的最佳性。特别是,任何产生依赖内向性的美元(美元)预期的遗憾分配的政策,必须产生直线性遗憾。最近,arx:2109.13595显示,信息-理论优化强盗算法以及标准的UCBCS-Ilorth 政策,从两武装强的轮向下,我们提供了一个简单的政策设计(i)最坏情况的最佳最佳情况,我们一般内向O(sqrt{t}风险最坏情况。(i)美元)和(ii)在最短的内向内向内向内向内流的内向内向内流最坏情况最坏情况最坏情况最坏的情况。(i)政策,我们最坏的内向内向后,我们最坏的内向内向内向后,我们最坏的内向后最差的内向内向内向内向内向内向内向内向内向内向内向内向内向内向内向内向内向内流的内向内流的内向内向内向内流最坏的内向内流的内流的内向内向内向内向内向内向内向内向, 直的内向内向内向内向内向。