We present \emph{TabRet}, a pre-trainable Transformer-based model for tabular data. TabRet is designed to work on a downstream task that contains columns not seen in pre-training. Unlike other methods, TabRet has an extra learning step before fine-tuning called \emph{retokenizing}, which calibrates feature embeddings based on the masked autoencoding loss. In experiments, we pre-trained TabRet with a large collection of public health surveys and fine-tuned it on classification tasks in healthcare, and TabRet achieved the best AUC performance on four datasets. In addition, an ablation study shows retokenizing and random shuffle augmentation of columns during pre-training contributed to performance gains.
翻译:我们提出了TabRet,这是一种可以用于表格数据的可预训练Transformer模型。TabRet旨在处理包含未见过列的下游任务。与其他方法不同,TabRet具有额外的学习步骤,称为\emph{重标记},该步骤基于掩码自编码损失对特征嵌入进行校准。 在实验中,我们使用大量的公共卫生调查数据对TabRet进行了预训练,并在医疗保健分类任务上进行了微调。TabRet在四个数据集上均获得了最佳的AUC性能。此外,消融研究显示,预训练期间的重标记和随机重排列的增强有助于提高模型性能。