Figure skating scoring is challenging because it requires judging the technical moves of the players as well as their coordination with the background music. Most learning-based methods cannot solve it well for two reasons: 1) each move in figure skating changes quickly, hence simply applying traditional frame sampling will lose a lot of valuable information, especially in 3 to 5 minutes long videos; 2) prior methods rarely considered the critical audio-visual relationship in their models. Due to these reasons, we introduce a novel architecture, named Skating-Mixer. It extends the MLP framework into a multimodal fashion and effectively learns long-term representations through our designed memory recurrent unit (MRU). Aside from the model, we collected a high-quality audio-visual FS1000 dataset, which contains over 1000 videos on 8 types of programs with 7 different rating metrics, overtaking other datasets in both quantity and diversity. Experiments show the proposed method achieves SOTAs over all major metrics on the public Fis-V and our FS1000 dataset. In addition, we include an analysis applying our method to the recent competitions in Beijing 2022 Winter Olympic Games, proving our method has strong applicability.
翻译:图表滑冰评分具有挑战性,因为它要求判断玩家的技术动作以及他们与背景音乐的协调。大多数基于学习的方法无法很好地解决这个问题,原因有二:(1) 每一个动作都快速地以图滑冰方式变化,因此简单地应用传统框架抽样将失去大量有价值的信息,特别是在3至5分钟长的视频中;(2) 之前的方法很少考虑其模型中重要的视听关系。由于这些原因,我们引入了一个叫作滑冰-Mixer的新颖结构。它将MLP框架扩展成一个多式联运模式,并通过我们设计的记忆经常单元(MRU)有效地学习长期表现。除了模型之外,我们收集了一个高质量的视听FS-1000数据集,其中包含了8种节目的1000多段视频,有7个不同的评分尺度,超过了数量和多样性上的其他数据集。实验显示,拟议方法在公共Fis-V和我们的FS1000数据集的所有主要衡量尺度上都取得了SOTA。此外,我们还包括了对最近北京2022冬季奥林匹克运动会的竞赛应用方法的分析,证明我们的方法具有很强的适用性。