What it is
An answer confidence is only safe to automate on if it has been calibrated against gold labels. That meaning can change with the question shape: in our measured runs, a 3-option XNLI question reached a 90% target at a 0.8074 gate, while a 20-option MASSIVE question needed 0.9944. One global min_confidence is not a policy.
laya-evals couples a cheap, batched rubric judge with the evidence needed to decide when to trust it.
Results
| Capability | Why it matters |
|---|---|
| Batched rubric judge | Ask choice, ordered score, and boolean noul questions in one laya pass. |
| Calibration audit | Compute ECE, Brier score, reliability bins, and coverage/accuracy curves from (confidence, outcome) pairs. |
| Threshold advisor | Select the lowest gate that meets a target accuracy, separately for each question shape. |
| Judge comparison | Compare against gold or another judge with agreement, Cohen’s kappa, weighted kappa, accuracy, and cost-per-1k fields. |
| CI regression gate | Fail builds when accuracy drops or calibration worsens; deliberately exclude unstable wall-clock measurements. |
Try it
pip install laya-evals
# or: uv add laya-evals

