Spend agent skills where
models are weak

Which recurring errors should an agent-skills budget address first? SkillsBench measures gains across 84 tasks, seven model-harness configurations and 7,308 trajectories. Human-curated skills add 16.2 percentage points on average, with healthcare at +51.9pp and software engineering at +4.5pp. Gains differ substantially by domain, while self-generated skills have a negative average gain. Our judgment: investigate missing procedural knowledge in your own workflows before assuming that another model upgrade will resolve the failures. Coding skills may still be valuable; local tasks should determine their priority.

1. Skills become a measurable independent variable

SkillsBench (arXiv 2602.12670) defines skills as procedural knowledge for a class of tasks, organized around SKILL.md and portable through the filesystem between compatible harnesses. Code, templates and resources are optional assets. System prompts, few-shot examples, RAG retrievals and tool documentation are outside the paper’s definition. This separation makes skills versionable, reviewable and reusable. Adding validation logic is our engineering recommendation for enterprise use, rather than a prerequisite in the paper’s definition.

The paper compares three conditions: no skills, human-curated skills and model self-generated skills. Three commercial harnesses—Claude Code, Codex CLI and Gemini CLI—form seven model-harness configurations, with repeated trials graded by deterministic pytest. This is an evaluation of specific tasks and configurations, not a fully crossed experiment covering every model-harness pairing. Full text: arxiv.org/html/2602.12670v1.

The mean pass rate rises from 24.3% without skills to 40.6% with curated skills; the paper reports an average gain of +16.2pp. Its failure-analysis section also reports overall failure proportions of 78.4% and 61.1%. These use a different aggregation and should not be treated as direct complements of the mean pass rates. Retain the task set, configuration and denominator before comparing metrics.

2. Domain heterogeneity: locate missing procedural knowledge

By domain, gains vary more than tenfold: healthcare 34.2% to 86.1% (+51.9pp), manufacturing 1.0% to 42.9% (+41.9pp), cybersecurity +23.2pp; mathematics 41.3% to 47.3% (+6.0pp), and software engineering 34.4% to 38.9% (+4.5pp).

The paper discusses thinner pretraining coverage and specialized procedures as one explanation for larger gains in areas such as clinical data standardization and manufacturing scheduling. Software engineering and mathematics are more common in public text. That interpretation still requires validation for a company’s private toolchain and internal processes; domain labels do not establish what a deployed model actually knows.

Our judgment: allocate budget using the local distribution of errors. A task already performed reliably may offer less improvement headroom than a repeatedly failing internal workflow. Domain differences in benchmark pass rates are not financial returns at equal investment, and they do not demonstrate a tenfold enterprise ROI. Skills are candidates for filling observed knowledge gaps, not a universal replacement for model, tool or process improvements.

3. Self-generated skills need validation, not assumed efficacy

The paper’s self-generated skills decline by 1.3 percentage points on average. Claude Code with Opus 4.5 scores 21.6% versus 22.0% without skills. Codex CLI with GPT-5.2 scores 25.0% versus 30.6%: a decline of 5.6 percentage points. The table’s −8.1% is normalized gain, not an 8.1-point absolute decline. Claude Code with Haiku 4.5 is flat at 11.0%, while the Opus 4.6 configuration improves by 1.4 points.

These results do not support treating self-generated skills as a reliable source of gains in the tested setup. They also do not prove that agents can never improve their skill libraries. The paper evaluates particular generation procedures, tasks and model configurations. Other improvement methods require new experiments; one benchmark cannot rule out every future or alternative approach.

Our judgment: this is both an engineering and a governance issue. A skill library should retain its author, approver, evaluation version and retirement conditions. Treat generated skills as candidates, evaluate them on held-out tasks not used to write them, and review them before production. Human authorship alone does not guarantee quality either.

4. The unit of evaluation is model plus harness

Results also vary across complete model-harness configurations. Claude Code combinations improve from +13.9pp (Opus 4.6) to +23.3pp (Opus 4.5); Gemini CLI combinations improve +13.6pp to +17.4pp, with Gemini 3 Flash reaching 48.7% with skills. The paper records cases of Codex CLI neglecting supplied skills and implementing another solution. Availability of a skill is not evidence that the execution path used it as intended.

This comparison does not hold the same model fixed while changing only the harness. Cross-configuration differences therefore cannot be attributed solely to harness choice, nor compared causally with most model upgrades. Procurement and acceptance should still evaluate the complete model, harness and skill-library combination. On local workloads, control variables and measure task success, cost and rework together.

5. Skill-count tradeoffs and the experiment’s capability boundary

In the grouped analysis, two to three skills show the largest average gain (+18.6pp), one skill +17.8pp, and four or more +5.9pp. Moderate-length documentation outperforms comprehensive documentation. Additional skills consume context and maintenance effort, but these groups do not establish a universal causal rule from adding skills one by one to the same fixed task. Use local ablations to choose the appropriate quantity and length.

Sixteen of 84 tasks (19%) achieve zero success in the tested configurations and conditions. This marks unsolved tasks within that experiment, not impossibility for every system. On cost, the paper reports Gemini 3 Flash consuming 2.3 times the input tokens per task of Gemini 3 Pro (1.08M versus 0.47M) while costing about 44% less at the prices used in the paper ($0.55 versus $0.98). Those are workload and pricing snapshots, not current quotes. Substituting a smaller model plus skills requires acceptance on quality as well as cost.

6. Our judgment and an allocation checklist

Our judgment: begin capability investment by checking whether useful procedural knowledge has been written as an evaluable asset. A skill is one form of that asset. Its effect depends on the task, knowledge quality, harness and validation process. Models, tools and workflow design also matter; none should be treated as a universal substitute for the others.

Action: select a recurring industry task that your team fails, encode a senior practitioner’s procedure as a skill, and run the A/B comparison on held-out tasks. Record outcomes and cost in an evidence card. Let budget follow reproducible reductions in error rather than leaderboard position or the sheer number of skills in a repository.

References: SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (arXiv 2602.12670 v1; correction checked against Tables 3 and 10 and the skill definition) arxiv.org/abs/2602.12670 | full paper. Measurements from the paper and our implementation recommendations are distinct; results depend on tasks, configurations and aggregation.