Youssef

laya-evals

LLM-as-judge and calibration auditing for System 1 decision models: rubric scoring in CI plus reliability diagrams, ECE and coverage/accuracy thresholds by option count

Code on GitHub PyPI

Language
Python
Licence
Apache-2.0
Last push
1 Oct 2026
Stars
1

What it is

An answer confidence is only safe to automate on if it has been calibrated against gold labels. That meaning can change with the question shape: in our measured runs, a 3-option XNLI question reached a 90% target at a 0.8074 gate, while a 20-option MASSIVE question needed 0.9944. One global min_confidence is not a policy.

laya-evals couples a cheap, batched rubric judge with the evidence needed to decide when to trust it.

Results

Capability Why it matters
Batched rubric judge Ask choice, ordered score, and boolean noul questions in one laya pass.
Calibration audit Compute ECE, Brier score, reliability bins, and coverage/accuracy curves from (confidence, outcome) pairs.
Threshold advisor Select the lowest gate that meets a target accuracy, separately for each question shape.
Judge comparison Compare against gold or another judge with agreement, Cohen’s kappa, weighted kappa, accuracy, and cost-per-1k fields.
CI regression gate Fail builds when accuracy drops or calibration worsens; deliberately exclude unstable wall-clock measurements.

Try it

pip install laya-evals
# or: uv add laya-evals