What it is
Quick start · Metrics · Your suite · Run on Kaggle
Changing image encoding can change what a vision API gets right. This Python CLI runs a labeled image suite through eight delivery variants, repeats each request, and turns answer correctness into regression metrics and CI gates.
How it works
- Variants. Each fixture image is re-encoded into two classes:
identical (
original,png-reencode,exif-strip) and lossy (jpeg-75,jpeg-40,resize-50,grayscale,webp). Before an identical variant is used, the tool verifies the pixels really did not change: dimensions, band set, per-pixel diff. The variant name alone is never trusted as proof. - Runs. Every fixture/variant pair is asked
--runstimes (default 3). A provider answers, an exact assertion judges each raw answer, and there is no LLM judge. - Metrics. Cell accuracy is correct answers over N. The variant
aggregate is the mean over eligible fixtures. Delta is the variant
aggregate minus the original aggregate over the same fixtures. A fixture
whose
originalmixes pass and fail is flaky: it gets excluded from strict metrics and the denominators say so. - Gates compare the strict aggregates against your thresholds.
The demo value of identical_delta_min is -1/6 because one seeded failure
in six fixtures moves the aggregate by exactly one fixture. That number is
the tool demonstrating itself.
Results
| Metric | Definition | Interpretation |
|---|---|---|
| Cell accuracy | Correct answers / repeats for one fixture/variant | Reliability for that transformed input |
| Variant accuracy | Mean cell accuracy over eligible fixtures | Accuracy for a delivery transformation |
| Delta vs. original | Variant accuracy − original accuracy over the same fixtures | Change from the baseline |
identical_delta_min |
Lowest aggregate delta among non-original identical variants | Worst correctness drop with unchanged pixels |
lossy_accuracy_min |
Lowest aggregate accuracy among lossy variants | Worst accuracy under delivery degradation |
flaky_count |
Fixtures whose original mixes pass and fail | Baselines excluded from strict aggregates |
Try it
pip install vision-input-check
vision-check demo

