We introduce JBE-QA, a Japanese Bar Exam Question-Answering dataset to evaluate large language models' legal knowledge. Derived from the multiple-choice (tanto-shiki) section of the Japanese bar exam (2015-2024), JBE-QA provides the first comprehensive benchmark for Japanese legal-domain evaluation of LLMs. It covers the Civil Code, the Penal Code, and the Constitution, extending beyond the Civil Code focus of prior Japanese resources. Each question is decomposed into independent true/false judgments with structured contextual fields. The dataset contains 3,464 items with balanced labels. We evaluate 26 LLMs, including proprietary, open-weight, Japanese-specialised, and reasoning models. Our results show that proprietary models with reasoning enabled perform best, and the Constitution questions are generally easier than the Civil Code or the Penal Code questions.
翻译：我们介绍了JBE-QA，一个用于评估大语言模型法律知识的日本司法考试问答数据集。该数据集源自日本司法考试（2015-2024年）的多项选择题（択一式）部分，为日本法律领域的大语言模型评估提供了首个综合性基准。它涵盖了《民法典》、《刑法典》和《宪法》，超越了先前日本资源主要关注《民法典》的范围。每个问题被分解为独立的真/假判断，并带有结构化的上下文字段。该数据集包含3,464个条目，标签分布均衡。我们评估了26个大语言模型，包括专有模型、开放权重模型、日语专用模型以及推理模型。我们的结果表明，启用推理功能的专有模型表现最佳，且《宪法》相关的问题通常比《民法典》或《刑法典》的问题更容易。
相关内容
微信扫码咨询专知VIP会员