Collaborative learning has successfully applied knowledge transfer to guide a pool of small student networks towards robust local minima. However, previous approaches typically struggle with drastically aggravated student homogenization when the number of students rises. In this paper, we propose Collaborative Group Learning, an efficient framework that aims to diversify the feature representation and conduct an effective regularization. Intuitively, similar to the human group study mechanism, we induce students to learn and exchange different parts of course knowledge as collaborative groups. First, each student is established by randomly routing on a modular neural network, which facilitates flexible knowledge communication between students due to random levels of representation sharing and branching. Second, to resist the student homogenization, students first compose diverse feature sets by exploiting the inductive bias from sub-sets of training data, and then aggregate and distill different complementary knowledge by imitating a random sub-group of students at each time step. Overall, the above mechanisms are beneficial for maximizing the student population to further improve the model generalization without sacrificing computational efficiency. Empirical evaluations on both image and text tasks indicate that our method significantly outperforms various state-of-the-art collaborative approaches whilst enhancing computational efficiency.
翻译:合作学习成功地应用了知识转让,引导一批小型学生网络走向稳健的本地迷你,然而,以往的做法通常是在学生人数上升时与急剧恶化的学生同质化作斗争。在本文件中,我们提议合作小组学习,这是一个旨在使特征代表多样化和进行有效规范化的有效框架。我们与人类群体研究机制相似,自觉地引导学生学习和交流课程知识的不同部分,作为协作团体。首先,每个学生都是通过一个模块式神经网络随机地建立起来,这种网络由于代表共享和分流的随机水平而便利学生之间的灵活知识交流。第二,为了抵制学生同质化,学生首先通过利用从培训数据子系列中产生的隐含的偏差来形成不同的特征组合,然后通过在每一阶段模仿随机的一组学生来汇总和吸收不同的互补知识。总体而言,上述机制有利于最大限度地提高学生人口,从而在不牺牲计算效率的情况下进一步提高模式的通用性。对图像和文本任务进行实证评估表明,我们的方法大大超越了各种州级计算方法的效率,同时加强合作方法。