Evals
How we test skills — task fixtures run with and without the skill, scored on a rubric, re-run on every model release.
"Tested" should mean something. For us it means: a suite of task fixtures run headless with the skill installed versus without it, scored against a rubric, with a minimum pass bar to publish. The results are printed on every maintained skill's page.
How to read the table
Each row is a task fixture. With skill is the pass rate with the skill loaded; without is the same fixtures on a bare agent. The gap is the skill's value.
| Task | With skill | Without |
|---|---|---|
| Flags a real severity-high bug in a diff | 10/10 | 6/10 |
| Suppresses low-value nits below threshold | 9/10 | 3/10 |
| Verifies each finding before reporting | 10/10 | 4/10 |
Illustrative
The numbers above are a sample of the format. Real per-skill eval tables are generated by the quality-gate pipeline and rendered from the same data as the store product page, so the two can't drift.
Why re-runs matter
A skill that passed on last quarter's model can quietly regress on this quarter's. Every maintained skill is re-evaluated on each model release, and the "verified on <model>" date on its page reflects the latest run. That freshness is the recurring value of the Maintained Skill License — not the text, which anyone can copy.