자연어 설명의 판단 품질 측정: 예측 토너먼트 증거
이 논문은 Forecasting tournaments에서 제공되는 natural-language rationales의 품질을 측정하기 위해 Explanation Quality Markers (EQMs)라는 새로운 지표를 제안합니다. LLM을 활용하여 60가지의 theory-guided reasoning patterns을 점수화하고, 55,000개 이상의 forecast-rationale pairs를 대상으로 분석을 수행했습니다. 연구 결과, EQMs는 기존의 text-analysis 방법보다 정확도(accuracy)를 더 잘 예측하며, 특히 underperformers를 식별하는 데 탁월한 성능을 보였습니다.
arXiv:2606.30987v1 Announce Type: cross Abstract: Decision-makers routinely rely on expert judgments accompanied by written explanations, yet explanation quality is difficult to measure at scale. Forecasting tournaments offer a natural testing ground: probabilistic judgments are paired with natural-language rationales and scored against realized outcomes. We introduce Explanation Quality Markers (EQMs), a set of sixty theory-guided reasoning patterns scored by large language models (LLMs). In a pre-registered analysis of over 55,000 forecast-rationale pairs from a multiyear forecasting tournament, EQMs predict accuracy at both the forecast and forecaster levels, consistently outperforming pre-LLM text-analysis methods. More than 90% of statistically significant pattern-level EQM-accuracy correlations match our directional hypotheses. The signal is asymmetric: EQMs identify likely underperformers more reliably than they distinguish the very best forecasters. Benchmarked against traditional indicators of forecasting skill, EQMs are the strongest predictor at the forecast level and competitive at the forecaster level, though weaker than prior accuracy. Human ratings of rationale quality are less consistently correlated with accuracy and place disproportionate weight on rationale length. Results transfer to an independent forecasting study. EQMs provide a scalable, interpretable method for extracting judgment-relevant information from written explanations.