We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
翻译:我们建议进行一项新的测试,以测量文本模型的多任务准确性。 测试涵盖57项任务, 包括基础数学、 美国历史、 计算机科学、 法律等等。 要实现这一测试的高度精确性, 模型必须拥有广泛的世界知识和解决问题的能力。 我们发现, 虽然最近的模型几乎具有随机精确性, 但最大的GPT-3模型平均比随机概率高出近20个百分点。 然而, 在57项任务中, 最佳模型在达到专家级准确性之前, 仍然需要大幅改进。 模型的性能也偏斜, 并且经常不知道何时出错。 更糟糕的是, 它们在某些社会重要主题, 如道德和法律上仍然拥有近乎随机的准确性。 通过全面评估模型学术和专业理解的广度和深度, 我们的测试可以用来分析跨越许多任务的模型, 并找出重要的缺陷。