Youssef

clef-evals

Calibration-first evaluation toolkit for Cloudflare's Clef decision models. Judge cheap, audit confidence.

Code on GitHub pypi.org

Language
Python
Licence
Apache-2.0
Last push
1 Oct 2026
Stars
1
clef-evals: Calibration-first evaluation toolkit for Cloudflare's Clef decision models. Judge cheap, audit confidence.

What it is

Most evaluation harnesses stop at accuracy. clef-evals also verifies whether a model’s stated probabilities deserve to be trusted. Run typed Clef decisions against your dataset, measure Expected Calibration Error (ECE), Brier score, latency and cost, then make regressions fail CI before they reach users.

Results

Evaluate Understand Enforce
Choice, binary and custom decision tasks through sync or async judges. Accuracy, ECE, binary and multiclass Brier, latency percentiles, token use and estimated input cost. A CLI gate and reusable GitHub Action that compare a fresh run with a committed baseline.

Try it

pip install clef-evals
export CLEF_ACCOUNT_ID=your_account_id
export CLEF_API_TOKEN=your_api_token