In online advertising, auto-bidding has become an essential tool for advertisers to optimize their preferred ad performance metrics by simply expressing the high-level campaign objectives and constraints. Previous works consider the design of auto-bidding agents from the single-agent view without modeling the mutual influence between agents. In this paper, we instead consider this problem from the perspective of a distributed multi-agent system, and propose a general Multi-Agent reinforcement learning framework for Auto-Bidding, namely MAAB, to learn the auto-bidding strategies. First, we investigate the competition and cooperation relation among auto-bidding agents, and propose temperature-regularized credit assignment for establishing a mixed cooperative-competitive paradigm. By carefully making a competition and cooperation trade-off among the agents, we can reach an equilibrium state that guarantees not only individual advertiser's utility but also the system performance (social welfare). Second, due to the observed collusion behaviors of bidding low prices underlying the cooperation, we further propose bar agents to set a personalized bidding bar for each agent, and then to alleviate the degradation of revenue. Third, to deploy MAAB to the large-scale advertising system with millions of advertisers, we propose a mean-field approach. By grouping advertisers with the same objective as a mean auto-bidding agent, the interactions among advertisers are greatly simplified, making it practical to train MAAB efficiently. Extensive experiments on the offline industrial dataset and Alibaba advertising platform demonstrate that our approach outperforms several baseline methods in terms of social welfare and guarantees the ad platform's revenue.
翻译:在网上广告中,汽车招标已成为广告商通过仅仅表达高级别竞选目标和限制来优化其首选的性能衡量标准的一个重要工具。以前的工作考虑从单一代理人的角度设计自动投标代理商,而不以代理商之间的相互影响为模范。在本文中,我们相反地从分布式多代理人系统的角度来考虑这一问题,并提议一个通用的多代理人强化学习框架,即MAAB,以学习自动招标战略。首先,我们调查汽车投标代理商之间的竞争与合作关系,并提议为建立混合合作竞争模式而分配温度正规化信贷。通过在代理商之间谨慎地开展竞争与合作交易,我们可以达到一个平衡状态,不仅保证个体广告商的效用,而且保证系统绩效(社会福利)。 其次,由于在合作背后的低价格竞标中观察到的串通行为,我们进一步建议酒吧代理商为每个代理商设置个性化的投标条条,然后缓解收入的退化。第三,在大型广告系统上部署MAAB,与数以百万计计的广告商的基线交易,我们建议通过一个简化的投资者标准,在标准上,向高级的投资者展示一种简化的税收的汇率,我们的一个标准。