Spend agent skills where
models are weak

The worst place to add skills to an agent is coding — the model already knows how. The best place is your own industry workflow, where the model will never know on its own. SkillsBench turned skills into a measurable independent variable across 84 tasks, 7 model-harness configurations and 7,308 trajectories: human-curated skills add 16.2 percentage points on average, but the return varies more than tenfold by domain — healthcare +51.9pp versus software engineering +4.5pp. Self-generated skills return nothing, or less. Our judgment: the accounting unit for agent capability is domain procedural knowledge, not model upgrades.

1. Skills become a measurable independent variable

SkillsBench (arXiv 2602.12670) defines a skill as a structured package of instructions, code templates, resources and verification logic, and explicitly excludes system prompts, few-shot examples, RAG retrievals and tool documentation. The definition matters more than the headline: it separates skills from prompt engineering and turns them into versionable, reviewable, reusable assets.

Methodologically, every task runs three times — no skills, human-curated skills, and model self-generated skills — across three commercial harnesses (Claude Code, Codex CLI, Gemini CLI) and seven frontier models, five trials per task, 7,308 trajectories, graded by deterministic pytest. Full text: arxiv.org/html/2602.12670v1.

Result: the mean pass rate moves from 24.3% with no skills to 40.6% with curated skills, and the overall failure rate falls from 78.4% to 61.1%. Skills work — but the next two findings are the ones buyers should read.

2. Domain heterogeneity: skills are cheap where models are strong

Split by domain, returns differ more than tenfold: healthcare 34.2% to 86.1% (+51.9pp), manufacturing 1.0% to 42.9% (+41.9pp), cybersecurity +23.2pp — against mathematics 41.3% to 47.3% (+6.0pp) and software engineering 34.4% to 38.9% (+4.5pp).

The mechanism is direct: the biggest gains land where pretraining coverage is thin and specialised procedural knowledge is required — clinical data standardisation, manufacturing scheduling optimisation. In domains saturated in public text, such as software engineering and mathematics, models have already internalised the operating procedure, so skills only add marginal polish.

Our judgment: the implication for budget allocation is counterintuitive. Spending a skills budget on prompt-tuning a general coding agent is remedial tutoring for a model that already knows the job; the same money on your own industry workflow can return an order of magnitude more. Skills are not a general-purpose model enhancement — they are a specialised part that fills a pretraining gap.

3. Models cannot write their own operating procedures

The second finding is more sobering: model self-generated skills return near zero or negative — an 80.9% overall failure rate, worse than the 78.4% with no skills at all. Per configuration: Claude Code with Opus 4.5 scores 21.6% self-generated versus 22.0% with none; Codex CLI with GPT-5.2 scores 25.0% versus 30.6%, 8.1 points lower; Claude Code with Haiku 4.5 is flat at 11.0%.

The paper’s conclusion fits in one line: models cannot reliably author the procedural knowledge they benefit from consuming. That kills a popular assumption — that agents can self-improve their own skill library. A model can write code, but it cannot write a better operating procedure than the one it lacks, because what it lacks is exactly the domain experience absent from public text.

Our judgment: this is a governance problem, not a technical one. A skill library needs human review — who authors, who approves, who retires, all with names attached. Letting self-generated skills into production institutionalises the defect that models do not know what they do not know.

4. The unit of evaluation is model plus harness

The same skills also perform very differently across harnesses. Claude Code has the highest skills utilisation, improving +13.9pp (Opus 4.6) to +23.3pp (Opus 4.5); Gemini CLI has the highest raw score (Gemini 3 Flash at 48.7%), improving +13.6pp to +17.4pp. Codex CLI shows the most instructive failure mode: it frequently neglects provided skills — the agent acknowledges the skill content but often implements its own solution instead. The skill was read, not executed.

Switching harness at a fixed model can move absolute scores by roughly twenty points, more than most model upgrades deliver. So “we bought the best model” is not a deliverable capability claim. What can be procured and accepted is the combination of model, harness and skill library.

5. More skills do worse, and 19% of tasks defeat everyone

On quantity, two to three skills deliver the largest gain (+18.6pp), one skill +17.8pp, and four or more collapse to +5.9pp; on length, moderate skills beat comprehensive documentation. Skill count and length are liabilities — each extra skill costs context budget, adds conflict probability and needs maintenance.

The boundary is equally clear: 16 of 84 tasks (19%) score 0% across all seven configurations and all three conditions. Skills cannot rescue tasks beyond the capability frontier. One cost-side observation rounds this out: Gemini 3 Flash consumes 2.3 times the input tokens per task of Gemini 3 Pro (1.08M versus 0.47M), substituting iterative exploration for reasoning depth, yet costs about 44% less per task at official pricing ($0.55 versus $0.98). A small model with skills can clear the no-skills baseline of a larger one — direct evidence of trading skills for model tier.

6. Our judgment and an allocation checklist

Our judgment: the first question in agent capability investment is not which model to buy, but whether our own procedural knowledge has been written down as an asset. Skills are one measurable form of that asset — and their return depends on domain scarcity, on human curation, and on whether the harness actually executes them.

Action: the smallest move available this week — pick one industry task your team repeatedly fails, write the senior practitioner’s operating procedure into a single skill, and run the A/B. The question to leave with: in your agent capability budget, what share goes to model upgrades versus writing down your own process? If it is overwhelmingly the former, you are buying what everyone else can buy.

References: SkillsBench: A Benchmark for Evaluating Agent Skills (arXiv 2602.12670; 84 tasks, 11 domains, 7 model-harness configurations, 5 trials per task, 7,308 trajectories, deterministic pytest grading) arxiv.org/abs/2602.12670 | full text arxiv.org/html/2602.12670v1. All figures cited above come from the paper text and its tables.