LegalQuants member releases LegalForecastBench, a benchmark scored against real court rulings
The open-source benchmark gives AI models the briefs a federal judge received on a motion to dismiss, asks them to forecast the ruling, and scores each forecast against what the judge decided.
John J. Hughes, III, a member of LegalQuants, today released LegalForecastBench, an open-source benchmark developed with LegalQuants. It tests AI models on work litigators do every day: reading the briefs in a case and forecasting how the court will rule.
Each model receives the same briefs a federal judge received on a real motion to dismiss. It gives a probability that the judge dismissed each challenged claim against each defendant. The rulings are on the public record, so every forecast is scored against the actual outcome.
The first cycle, released as a beta, covers 91 federal cases, 387 scored claims and 18 models. GPT-6 Sol from OpenAI scored best overall, followed by Anthropic's Claude Fable 5.1 and SpaceXAI's Grok 4.6. Mr Hughes notes that the sample is not yet large enough to separate those three with statistical confidence. Claude Fable 5.1 was the best calibrated: all 81 of its forecasts made with 90% or more confidence were correct. OpenAI models also made up the observed cost-quality frontier, with its Luna models giving the strongest results at the lowest prices.
The benchmark code is open source under the Apache 2.0 licence and its data is released under CC BY 4.0, so anyone can reproduce the results. LegalForecastBench is versioned. Mr Hughes plans to test further models, including Google's Gemini Argon, as access allows.
“Predicting litigation outcomes requires analyzing the facts, law, and procedural posture. It's a task typically performed by senior partners.”
“John built this the way a litigator would. The right answer is whatever the judge decided, and every forecast is checked against it. Lawyers who build their own tools should hold them to that standard.”
About LegalQuants
LegalQuants trains lawyers to build their own tools. It runs the LegalQuants Residency, a four-week programme in which each lawyer builds a working AI tool with a mentor, and a community where members keep building. Its members contribute open-source work to Anthropic's Claude for Legal and to OpenAI for Law. LegalQuants was founded by Jamie Tso and Raymond Sun.
Press enquiries: admin@legalquants.com