Dynamic Agent Skills: Resolving Facts at Call Time

Most agent skills today are static. Someone reads the truth once, writes it into a markdown file, and ships it. The procedure and the facts get frozen together in the same paragraph: which table is canonical, what the join key is, which filter drops the test accounts. Then the table gets deprecated, and the skill doesn’t know.

Nobody thinks this scales. But watch what the industry reaches for, because it’s always the same drawer. A registry. An owner field. A version scheme, a quarterly review, a linter that checks frontmatter. We shipped a couple of those at modus, where I work on the context layer behind a multi-tenant AI data platform. They bought us tidier metadata on top of knowledge that was still wrong.

What we eventually admitted is that a static skill is a cache with no invalidation protocol. It caches a retrieval result — the facts someone happened to look up while writing it — behind a key that’s a natural-language task description. No dependency tracking, no TTL, no way to notice a source moved.

Every complaint about skill libraries falls out of that one property. They go stale because uninvalidated caches go stale. They multiply because copying the cache is the cheapest way to serve a slightly different task. They contradict each other because two copies drift on their own schedules. They bloat because an author who can’t predict which facts the model will need writes down all of them, just in case.

So skill inflation is the rational local move when the architecture makes every skill carry its own snapshot of the warehouse. The scolding I did about it for a while was misdirected.

The alternative we run now: the authored file keeps intent, procedure, output contract, guardrails, and a scope over the company’s knowledge. Every fact gets resolved against a live context layer at the moment the agent asks. The markdown the model reads still exists — it’s just a build artifact with a lifetime of one call.

Where This Sits Next to Things You Already Know

Two comparisons are worth making explicitly, because this design is adjacent to both and the differences are the whole argument.

RAG with metadata filters solves a nearby problem and stops one step short. There, the filter is a per-query argument assembled at the call site, and the unit of retrieval is a chunk ranked by similarity. Both are load-bearing differences. A scope in our model is authored configuration that compiles once per call into a single filter clause shared by every retrieval path, and gets re-applied to results on the way out — which matters because a filter expressed as a query argument gets re-derived slightly differently by the eleventh code path that needs it, and because a metadata filter naming a deleted asset typically matches everything rather than nothing. The deeper split is what gets ranked: chunk similarity has no opinion about the fact that a column belongs to a table which belongs to a schema, so it cannot trade a table’s full detail for three tables’ identity lines when the budget gets tight. That trade is most of what our composer does.

MCP resources are the closer cousin, and the distinction is about who selects. MCP correctly moves the fetch to call time: the client asks for a resource by URI and gets current content instead of a stale transcription. But selection stays with the client model, which has to already know which resource to ask for, from a list that is itself an authored enumeration — so the staleness moves from the content into the index of what exists, and the model pays for its own exploration in round trips. A resource’s content is also the same for every reader. Composing server-side means selection happens against the caller’s principals under a token budget the server controls, which is why the same skill renders a different document for a finance analyst than for a contractor. If you already speak MCP: this is what we serve through an MCP tool call, not a replacement for it.

Part 1: What a Human Still Writes

The split follows rate of change. Procedure changes when a team changes its mind, so maybe quarterly. Facts change when the systems that own them change: a migration lands at 2am, someone archives a channel, a dashboard’s query gets rewritten by an analyst who has since left. If it moves on the second clock, a person shouldn’t be maintaining it by hand.

So “use dim_customers_v3, it’s the canonical one” stops being prose and becomes configuration. No table name in there. When dim_customers_v3 gets deprecated, the glob resolves to a different set on the next call, and nobody edited the skill, because the skill was never about that table. The static_ids selection is the escape hatch for the handful of items you really do want pinned by identity — a note that says refunds double-count before 2024, or a saved query someone validated.

This is also where the pressure to write one skill per task drains away. When we read our own sprawling library, most siblings had identical procedures and differed only in which facts they quoted. Resolve facts dynamically and they collapse into one skill with a wider scope, because the composition layer picks the relevant subset per question instead of the author guessing months in advance.

Two details are critical out of proportion to their size. A rule pointing at a deleted asset must match nothing rather than everything, and globs need expanding into concrete paths before the fetch, so a rule matching no real asset fails closed. And the scope filter gets re-applied on the way out, because graph traversals and parent-repair fetches will happily drag in neighbors that were never in scope.

Part 2: Composition Is a Graph

If facts arrive at call time, something has to choose them. A real question cuts across levels of different hierarchies sitting in different indexes: a revenue question wants warehouse schema, a validated SQL example, the wiki page defining “revenue,” and ideally the thread where someone noticed the refund double-count.

What held up is a graph of strategies, each declaring a predicate and contributing candidates only when it fires. Here’s the executed graph from one run, which we emit on every call:

Entity extraction feeds keyword and embedding search. The hierarchy gate walks down from selections to tables to columns in batches, routing on how many candidates came back. Orphan repair fetches a parent when only children matched. Relatives expand around hits. The shape is different on the next query, and that’s the point — it’s more machinery than a fixed pipeline, and what it buys is one engine serving wildly different sources without becoming an if-else forest.

The discipline that matters most inside it is knowing when not to call a model. Count thresholds route each stage: a handful of candidates gets fetched wholesale, hundreds get an LLM relevance filter, thousands get vector search plus a cross-encoder rerank first. Serializing ten thousand column names into a prompt is expensive and useless. Everything also runs under per-strategy timeouts and output-node caps, which exist because one tenant’s pathological schema found the branch we hadn’t bounded and stalled the whole composition.

Part 3: The Budget Is Part of the Skill

A static skill has no budget model. It’s however long its author felt like being, and you pay that on every call whether the question needed it or not. The cost is per invocation but the decision was made once, at authoring time, by someone not thinking about tokens.

Our default markdown budget is 15,000 tokens with a hard ceiling of 50,000. The ceiling has a guardrail event attached, composer_final_summary_truncated, which exists because we needed an alert for the day a single item’s payload blew through it rather than a mystery about why an answer got worse.

The idea that unlocked the budget for us was ranking relevance at the section level rather than per item. Group resolved candidates into numbered sections following the natural hierarchy — integration, then selection, then object, then detail — rank the sections, then take whatever prefix of that ranking fits. Any section over 5,000 tokens on its own gets exploded into its children first, because otherwise one blob competes head-to-head with a one-line note and the ranker has no way to be useful.

Two rankers run over those sections. An LLM ranker chunks the section list at 15,000 tokens per chunk and takes two independent views, fusing them with reciprocal rank fusion at k=60 so a section’s fate doesn’t depend on which chunk it landed in. The other path is a pure cross-encoder that skips LLM ranking entirely for latency-sensitive callers. Same sections, same budget cut, different cost. A markdown file offers no such dial.

Rank 3 is the whole argument in one line. A 68-token human note outranks four tables, and no author would have thought to quote it in a skill about revenue.

Detail then degrades in two independent directions. Across sections, the budget decides what’s included at all. Within a section, a prompt level decides how much of each item renders — we have eight, from identity-only up to full payload. Ranking runs on the cheap short bodies and only survivors render at complete. Oversized values — a 40k-row CSV or some enormous document body — get offloaded to files the agent fetches on demand with a placeholder left behind, so one fat item can’t eat the budget.

The budget is also where quality actually lives, and we can put numbers on that. On a 21-case evaluation suite, we swapped the reranker feeding section ranking and measured recall against labeled ground truth at four token budgets:

	recall@2,500	recall@5,000	recall@10,000	recall@15,000	end-to-end recall
candidate A vs control	+0.0657	+0.1336	+0.0326	+0.0225	+0.0225
candidate B vs control	+0.0273	+0.0070	−0.1239	−0.1334	−0.1004

Candidate A improved recall by 2 points end-to-end and by 13 points at a 5,000-token budget — the same change, six times larger when the budget is tight, because a better ranker’s entire job is getting the right sections above the cut. Candidate B is the more instructive row: it improves recall at the smallest budget and loses 13 points at the largest. One number would have called that a win or a disaster depending on which budget you happened to measure at, and both readings would have shipped.

Part 4: Rendering Is a Contract

The moment markdown is generated instead of written, its formatting is code, and code that emits prompts deserves the same care as any other interface. We treated it as string concatenation for a while and paid for it.

Keep two responsibilities apart. The composer owns document structure — headings, grouping, ordering, the budget cut. Per-item body rendering belongs to a template bound to an exact content schema, keyed by content type and a hash of that schema. That hash is the piece worth stealing if you’re starting over: when a schema changes, the hash changes, and any cached render is automatically invalidated rather than silently serving a prompt built against a shape that no longer exists. Generated prompts have the same dependency problem as static skills if you don’t track what they were built from — and this approach ensures you always do.