Automated essay scoring (AES) is a challenging task in cross-prompt settings due to the diversity of scoring criteria. While previous studies have focused on the output of large language models (LLMs) to improve scoring accuracy, we believe activations from intermediate layers may also provide valuable information. To explore this possibility, we evaluated the discriminative power of LLMs' activations in cross-prompt essay scoring task. Specifically, we used activations to fit probes and further analyzed the effects of different models and input content of LLMs on this discriminative power. By computing the directions of essays across various trait dimensions under different prompts, we analyzed the variation in evaluation perspectives of large language models concerning essay types and traits. Results show that the activations possess strong discriminative power in evaluating essay quality and that LLMs can adapt their evaluation perspectives to different traits and essay types, effectively handling the diversity of scoring criteria in cross-prompt settings.
翻译：自动作文评分（AES）在跨提示场景下因评分标准的多样性而成为一项具有挑战性的任务。以往研究多关注大语言模型（LLMs）的输出以提高评分准确性，但我们认为中间层的激活也可能提供有价值的信息。为探究这一可能性，我们评估了LLMs激活在跨提示作文评分任务中的判别能力。具体而言，我们利用激活拟合探针，并进一步分析了不同模型及LLMs输入内容对此判别能力的影响。通过计算不同提示下各特质维度上作文的方向向量，我们分析了大语言模型针对作文类型与特质的评价视角变化。结果表明，激活在评估作文质量方面具有强大的判别能力，且LLMs能够针对不同特质与作文类型调整其评价视角，有效应对跨提示场景下评分标准的多样性。
相关内容
微信扫码咨询专知VIP会员