Decision model matched language model's accuracy at 1/55 of the cost
Researchers tested whether decision models built on large language models abstain when the correct legal authority is missing, using an extended version of CaseHOLD with 1,670 items where the correct holding was replaced out of 5,314 total items.
When no option is correct, generative language models still select an answer in 38-86% of items, while the most cautious decision model does so in 30%.
A decision model matched the best generative language model's performance at 88.1% versus 88.2% accuracy on standard CaseHOLD while operating at 1/55 of the cost.
Accepting a language model's answer only when a decision model agrees halves the number of wrong citations.
Sources behind this report
Report history
No earlier revision dates are retained here. This does not establish that the report has never changed; read any dated updates and corrections.
This brief was made by AI and checked against its cited public sources. How the newsroom works