Contextual multi-armed bandits have been studied for decades and adapted to various applications such as online advertising and personalized recommendation. To solve the exploitation-exploration tradeoff in bandits, there are three main techniques: epsilon-greedy, Thompson Sampling (TS), and Upper Confidence Bound (UCB). In recent literature, linear contextual bandits have adopted ridge regression to estimate the reward function and combine it with TS or UCB strategies for exploration. However, this line of works explicitly assumes the reward is based on a linear function of arm vectors, which may not be true in real-world datasets. To overcome this challenge, a series of neural-based bandit algorithms have been proposed, where a neural network is assigned to learn the underlying reward function and TS or UCB are adapted for exploration. In this paper, we propose "EE-Net", a neural-based bandit approach with a novel exploration strategy. In addition to utilizing a neural network (Exploitation network) to learn the reward function, EE-Net adopts another neural network (Exploration network) to adaptively learn potential gains compared to currently estimated reward. Then, a decision-maker is constructed to combine the outputs from the Exploitation and Exploration networks. We prove that EE-Net achieves $\mathcal{O}(\sqrt{T\log T})$ regret, which is tighter than existing state-of-the-art neural bandit algorithms ($\mathcal{O}(\sqrt{T}\log T)$ for both UCB-based and TS-based). Through extensive experiments on four real-world datasets, we show that EE-Net outperforms existing linear and neural bandit approaches.
翻译:多武装的土匪已经研究了几十年,并适应了在线广告和个性化建议等各种应用。为了解决土匪的剥削-探索交易,提出了三种主要技术:epsilon-greedy、Thompson Sampling(TS)和Clop Inful Bound(UBB)。在最近的文献中,线性背景土匪采用了山脊回归法来估计奖励功能,并将其与TS(UCB)或UCB勘探战略结合起来。然而,这一行的工作明确假定奖励是基于手臂矢量的线性功能,这在现实世界的数据集中可能不是如此。为了克服这一挑战,已经提出了一系列基于神经的土匪算法,其中指派了一个神经网络来学习基本的奖励功能,T或T或UCB来进行探索。在这个文件中,我们提出了“E-Net”基于星系的波纹(Explational net) 网络(E-ral-ral-ral-ral 网络) 来学习奖赏功能,而现在的Oral-al-al-al-ral-ral-al-ral-al-al-al-al-al-al-al-al-al-al-al-al-al-commal-commal-al-commal-al-al-commal-al-commal-l) 网络则是我们目前的模型, 和我们当前学习了一个测试数据, 的模型, 和不断的模型的模型的模型的模型,用来测量测算数据-al-sal-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-sal-al-al-al-al-al-al-l-l-l-l-l-l-l-l-l-l-al-l-l-l-l-l-l-al-al-al-al-al-al-al-al-al-al-al-al-al-l