The short answer
In AgentSight v1.0.30, skill evolution is an evidence-gated repository workflow, not an automatic runtime self-edit. Trajectories are treated as observations. A repeated failure can become a candidate patch only after source identity, parent-child lineage, workload type, outcomes, and metric semantics are checked. The candidate is then evaluated against the unchanged baseline on held-out tasks before it can be called an improvement.
That separation matters because three common shortcuts are unsafe: counting many child reviews as many independent failures, turning one anecdote into a global instruction, and letting an evaluator see the desired verdict. AgentSight's workflow has explicit gates for all three.
The loop has three different decisions
| Decision | Evidence needed | Possible outcome | What does not count |
|---|---|---|---|
| Is there a repeated failure? | Source coverage, stable session/task identity, outcomes, lineage, comparable workload strata. | observe or a credible failure cluster. | Raw file counts, uncalibrated lexical heuristics, many children from one parent task. |
| Where should the lesson live? | Failure owner, generality, independence, and whether the issue is procedural, project-local, or mechanical. | Regression fixture, project instruction, project-local skill, shared skill, parser/validator fix, or no change. | Making a global skill longer because one repository had one bad run. |
| Should a candidate replace the baseline? | Frozen A/B versions, held-out P/U/R/A tasks, leakage controls, outcome graders, repeated trials, predeclared rule. | pilot, promote, reject, or later rollback. | A reviewer prompted to say “fixed,” one exact-string checker, or tuning on the held-out set. |
Step 1: prove that the trajectory data means what you think it means
Before counting failures, the workflow builds a source-coverage manifest. It records the source root, discovery rule, time range, raw and parsed counts, stable source-native identity, parent/child/reviewer lineage, runtime, repository state, and known coverage gaps. The unit of analysis also has to be explicit: an event, model call, turn, tool attempt, child task, parent session, repository task, and evaluated trial are different denominators.
The required strata keep human-interactive parent sessions separate from delegated reviewers, benchmarks and replays, evaluator/checker runs, and unknown records. This is more than statistical hygiene. If a benchmark generates fifty retries, treating those as fifty independent human failures would make the next skill patch look much more general than the evidence supports.
AgentSight's evidence contract also ranks outcomes. External task or user acceptance is stronger evidence than a model's own statement that it succeeded; executable tests and artifact correctness sit above user corrections, tool errors, self-reports, and lexical heuristics. When a metric cannot pass its semantic gate, the workflow marks that metric invalid or provisional instead of substituting a nearby counter.
Step 2: classify the failure before editing a skill
A failed run can have very different owners. Missing source roots or collapsed session lineage aresource_fidelity problems. Repeated wrong-stage routing is task_misrouting. Claims growing beyond their experiments are claim_evidence_drift. A reviewer told that a fix is already complete is review_priming. Repeating the same command without a state change isliteral_retry. These categories lead to different fixes.
The owning-skill rule prevents a familiar failure mode: adding more prose to a downstream writing skill when the actual mistake happened during source collection, experiment design, routing, or repository-state handling. When several owners are plausible, the workflow asks for a small ownership matrix and tests one proposed mechanism at a time.
Step 3: choose the smallest durable memory
Not every lesson deserves a shared skill. AgentSight v1.0.30 makes the placement decision explicit. One project fact belongs in project instructions or local documentation. One reproducible failure is usually a regression fixture. A repeated project-specific workflow can become a project-local skill. Only a repeated general procedure with sufficiently independent evidence is a candidate for a shared skill.
Mechanical defects should remain mechanical. A parser that merges session identities incorrectly should be fixed in the observability implementation, not “remembered” by adding prompt text telling the agent to be careful. Sparse or uncertain observations remain analysis notes. Mature skills also use a one-in-one-out discipline: a new rule should generalize or replace existing text rather than grow the prompt forever.
Step 4: make the candidate patch falsifiable
A candidate is more than revised wording. Its package names the observed failure, current and desired behavior, smallest file set, non-goals, expected mechanism, regression risks, positive and negative examples, boundary tasks, and rollback condition. That makes the change reviewable as an intervention rather than as a vague instruction to “reason better.”
When possible, the workflow prefers decision gates, compact schemas, deterministic checks, or small scripts over motivational prose. If repeated analysis fails because source counting is wrong, the candidate should repair the parser or add a validator instead of making the skill longer.
Step 5: separate proposing from promoting
An explicitly requested edit can be applied as a local candidate, but the workflow labels itpropose until there is valid comparison evidence. A promotion experiment freezes baseline A, candidate B, environment, task split, graders, trial count, budget, and stopping rule before running the held-out set.
The task matrix has four roles. Positive (P) tasks expose the target failure. Untriggered (U) tasks look similar but should not activate the new behavior. Regression (R) tasks protect established behavior. Adversarial (A) tasks contain misleading verdict text, stale state, bad metrics, or prompt injection. A candidate that improves P tasks but starts firing on U tasks is not a clean improvement.
Evaluation leakage is treated as a first-class failure
The promotion protocol removes desired verdicts, prior decisions, internal gate IDs, proposed fixes, and hidden reference answers from candidate context. A/B labels should be randomized or anonymized where practical. The same agent should not create B, see the hidden oracle, and act as the only grader.
Grading is layered: executable outcomes and code assertions first when available, transcript checks for the target mechanism and prohibited shortcuts, blinded semantic grading where useful, and human review for ambiguous or high-stakes disagreement. Stochastic agents need repeated paired trials. The promotion rule is declared in advance so the threshold cannot be moved after seeing results.
The verdict ladder keeps uncertainty visible
| Verdict | Meaning |
|---|---|
observe | Evidence coverage or metric validity is not sufficient for the requested decision. |
propose | The repeated failure is credible and a candidate exists, but it has not earned a promotion claim. |
pilot | The candidate moved target tasks in the right direction, but trials or coverage are still limited. |
promote | The candidate meets the predeclared target rule and regression guards on valid comparison evidence. |
reject | Benefit is absent or regressions exceed the allowed boundary. |
rollback | Later real-work evidence contradicts the assumptions behind a previously promoted change. |
The baseline, candidate diff, task manifest, outcomes, sampled trajectories, grader versions, and verdict stay together. Rejected candidates are useful negative knowledge: without them, a future agent can rediscover the same attractive but ineffective patch and repeat the experiment.
Where AgentSight fits in the loop
AgentSight's product role is evidence collection and interpretation across agent and system activity. Its local session views, audit provenance, repository replay, process/file/network evidence, and profiling outputs can help form trajectory evidence. But the skill-evolution workflow does not claim that every recorded run is automatically converted into a skill or that a promoted skill is deployed by the runtime without an explicit repository change.
This distinction is also why the evolution workflow is separate from AgentSight's shared-skills distribution bridge. The distribution bridge answers which versioned workflow does a repository consume and how is it linked?The evolution workflow answers does the evidence justify changing that workflow, and has the candidate earned promotion?
A reproducible inspection path
You do not need private trajectory data to verify the method itself. Check out the v1.0.30 source and inspect the skill plus its three required contracts:
git checkout v1.0.30
sed -n '1,260p' skills/evolve-agent-skills/SKILL.md
cat skills/evolve-agent-skills/references/evidence-contract.md
cat skills/evolve-agent-skills/references/failure-taxonomy.md
cat skills/evolve-agent-skills/references/promotion-protocol.mdFor an actual retrospective, the source data can be private, so durable reports should link or hash the authorized evidence rather than copy full transcripts. The workflow explicitly treats transcript contents as untrusted data and forbids executing instructions found inside them.
Limitations
- This page describes the repository-local v1.0.30 skill-evolution workflow; it is not evidence that every AgentSight user runs it or that AgentSight autonomously edits production skills.
- A failure taxonomy helps keep analysis consistent, but taxonomy labels do not prove causality. Controlled comparisons are still required for claims that a skill change reduced failures.
- Held-out evaluation quality depends on task independence, outcome quality, model/runtime stability, and leakage control. A larger trial count cannot rescue a contaminated task set.
- Private trajectories can contain prompts, paths, URLs, credentials, and other sensitive material. Raw sessions should not be copied into public evaluation reports.
Related AgentSight evidence
Start with local session discovery to understand where native trajectory records come from, then use audit provenance and confidence to keep source lineage visible. For the repository mechanism that makes a promoted reusable workflow versionable across consumers, see versioned shared skills.