Hengle JIANG

Hengle JIANG

PhD student @SUSTech

Research essay · October 14, 2026

What Is Actually Evolving in a Self-Evolving Agent?

What self-improvement methods actually fix, where their gains stop, and how the need for multi-objective optimization emerges.

Suppose a coding agent learns to retry whenever a test fails. Its completion rate rises. That looks like a successful update—until we notice that it now takes longer, makes many more tool calls, and occasionally edits files outside the permitted scope.

The agent has learned something useful: persistence helps solve difficult tasks. It has also learned an overly broad rule. Should we keep the update, reject it, or search for a version that retries more selectively?

This is the problem I want to understand through self-evolving agents and recursive self-improvement, or RSI. We are getting better at producing modifications to an agent. Deciding which modifications constitute progress is a separate challenge.

Multi-objective optimization enters the picture when the same change affects several things we care about, and improving one no longer tells us what happened to the others. The first task is to make those effects visible together. Only then can we decide which are negotiable, which are constraints, and which alternatives are worth preserving.

01 / What do self-evolving methods actually repair?

The useful progression in this literature is a sequence of bottlenecks. An agent may fail to retain experience, fail to change how it acts, lack the skill needed for the task, or waste its budget searching for improvements. Different methods intervene at different bottlenecks.

When experience disappears after execution

If a failure teaches the agent nothing that survives the next attempt, the system keeps paying to discover the same lesson. Reflexion retains verbal feedback, Voyager builds executable skills, and ACE organizes evolving context into a playbook.1 2 10

Their shared contribution is turning execution into reusable experience. The remaining question is when that experience should apply. “Retry after failure” is useful for a transient error and harmful when an action lacks authorization. Storing a lesson does not establish its scope. A memory system needs to distinguish a general rule from a rule that worked under particular conditions.

When better instructions cannot fix the action structure

A reminder to “edit carefully” cannot make a coarse editing tool precise. Workflow and code search open a different space: ADAS and AFlow automate agent or workflow design; DGM modifies tools and execution logic, including more targeted editing.5 6 8

This addresses limitations of the available operations and their organization. It also explains why frozen-model improvements can be substantial. The model may already know enough to solve a problem, while the surrounding system makes the right action difficult to perform. Yet a better structure on one workload may remain fragile on another. We still need to identify what the change fixed and which situations it could disrupt.

When the model needs to learn a capability

Sometimes a better wrapper is insufficient. SEAL rewards generated adaptation procedures according to performance after a weight update. Agent0 and AgentEvolver develop training signals through generated tasks and interaction.11 12 13

These methods address the supply and use of learning experience. Their next bottleneck is the quality of that experience. If a curriculum repeats familiar tasks or rewards a shared mistake, more training does not repair the missing signal. Task generation must be assessed through the distinctions it teaches the learner to make.

The learning and runtime routes can also feed each other. HELIX proposes a build–update–rebuild loop: evolve harnesses, use verified contrasting trajectories for model learning, then adapt the harness to the updated model. Its reported study covers one harness-evolution round and the resulting training records.30

When the improvement process is itself inefficient

RSI moves the question up one level. Promptbreeder modifies mutation prompts; STOP applies a code improver to itself; HyperAgents opens the meta-agent to modification. AIDE² searches research-agent harnesses, while Dream-RSI searches exploration policies using recorded experience.3 4 14 16 17

The aim is to improve how the next useful change is found. A better final agent is one possible result. A procedure that produces better changes under the same budget is a stronger result and needs its own test. Recursion also inherits the evaluator’s omissions: a more effective search procedure can become more effective at optimizing an incomplete definition of success.

An improver proposes changes to agent components. Execution produces evidence, selection retains changes, and the next run inherits them. A recursive path also changes the improver.
Figure 1. An evolving system has an editable object and a selection loop. The solid path updates the agent; the dashed path also updates the improver.

Across these routes, two decisions recur: what may change, and what evidence justifies keeping the change. The papers expand the first in different ways. The second leads directly to the need for joint evaluation.

02 / A higher completion rate can leave the decision unresolved

Return to the coding agent. Imagine evaluating three policies on the same 100 tasks. The completion grader checks whether the requested result was produced; a separate audit checks whether the agent stayed within its permitted actions.

Policy Completed tasks Mean tool calls 95th-percentile time Runs with an unauthorized action
P0 · Original 80 / 100 6 40 seconds 0 / 100
P1 · Retry broadly 89 / 100 14 110 seconds 4 / 100
P2 · Retry selectively and check scope 88 / 100 8 55 seconds 0 / 100

Illustrative values for the thought experiment. Completion and authorization are counted separately, so a completed task can still involve an unauthorized action.

A completion-only optimizer chooses P1. A strict time budget could favor P0. A deployment that prohibits unauthorized actions rules out P1 despite its higher completion rate. P2 becomes interesting because it preserves most of the additional completion while reducing the other costs.

Completion answers whether the requested result was produced. Deployment also asks how it was produced. Once execution affects resources and permissions, reaching the endpoint and reaching it acceptably become distinct properties.

Objectives and diagnostic signals serve different purposes. Repeated invalid tool calls may be a diagnostic explanation for latency rather than something we value separately. Evidence checks may help explain reliability without being a sufficient measure of it. We should choose objectives from the deployment requirement, then use diagnostics to explain how a candidate succeeds or fails. Otherwise, adding metrics can simply count the same benefit twice.

03 / Why not improve capability first, then repair the rest?

Because the objectives often share the same control decisions. More retries change completion, cost and time together. More verification can detect errors and increase latency. A stricter stopping rule can reduce exposure to unsafe actions and terminate legitimate work too early. A change aimed at one objective therefore needs to be re-evaluated on the others.

Imagine starting from P1 and applying a cost-saving patch: stop after fewer tool calls. The patch may remove wasted retries—or remove the attempt that solved the difficult cases. Then a capability patch reintroduces retries. Each local change can look good on its own target while the sequence makes little progress.

The useful intervention may be conditional: retry a temporary service failure, fix malformed arguments, stop an unauthorized operation, and reserve verification for consequential edits. These failures can all appear as “task not completed,” yet call for different changes. A vector of scores reveals the effects; trajectory diagnostics can connect those effects to a better policy.

Meta-Harness makes this diagnosis concrete by letting its proposer inspect previous code, scores and full execution traces. The useful feedback is not just that a candidate failed, but where its mechanism stopped working.29

A final safety filter remains useful. It can reject P1 for deployment. But it cannot recover a useful policy that the search never generated or discarded earlier. If every generation retains only the highest-completion candidate, P2 may disappear before the deployment constraint is applied. Logging every candidate avoids literal deletion, but the choice of which parents receive more search budget still shapes what comes next. Selection changes both the final choice and the material available for future search.

Joint optimization therefore brings the relevant requirements into the improvement process: they guide which changes are proposed, retained and developed further. Measuring several properties is only the first step; those measurements must influence the search. A fixed utility, explicit constraints and a set of alternatives offer different ways to do that.

There is a further possibility: the apparent trade-off may come from a poor design. Replacing an unreliable tool can increase completion and reduce retries simultaneously. The objective is to discover these joint gains where possible, and make the remaining trade-offs explicit where they persist. Choosing among existing compromises and discovering a policy that improves on them are different achievements.

04 / From several metrics to a joint decision

First, identify what cannot be exchanged

If modifying a forbidden file is unacceptable, extra completed tasks cannot compensate for it. That requirement belongs in the feasible set, supported by action-level enforcement where possible. An observed violation disqualifies a candidate for that deployment. Passing a finite test suite still leaves uncertainty about future behavior.

This separates two questions that are easily mixed together: “is this candidate admissible?” and “which admissible candidate do we prefer?” Some risks admit an explicitly chosen tolerance; others are hard prohibitions. The optimizer should reflect the actual requirement.

Next, make preferences explicit when they are known

A weighted utility is reasonable when the exchange rates are stable and defensible. SICA, for example, combines performance, cost and runtime in one utility.7 That is already a way of accounting for several requirements.

The sensitivity is easy to see in our example. Temporarily compare completion and calls alone. Subtracting 0.1 percentage point per mean tool call gives P1 a score of 87.6 and P2 a score of 87.2. Raising that penalty to 0.5 gives 82 and 84, reversing the choice. The arithmetic is correct in both cases; the preference changed. Adding back the authorization constraint excludes P1 regardless of those weights.

A weighted score expresses a particular preference. Its ranking depends on those exchange rates and on metric normalization, even when the written weights remain unchanged.

Preserve alternatives when one preference would be premature

If the same system will serve several workloads, a single early choice may be wasteful. Among candidates that satisfy the deployment requirements, one may be cheaper and another more effective. A Pareto archive keeps a candidate when no other evaluated, feasible candidate is at least as good on every chosen objective and strictly better on one.

Illustrative policies P0 and P2 trade completion against tool calls. P1 completes most tasks but fails the authorization requirement.
Figure 2. The same hypothetical policies as the table. P0 and P2 remain useful under different resource budgets. P1 has the highest completion rate but fails the authorization requirement. The line connects the two illustrative choices.

This gives P0 and P2 distinct roles: P0 fits a tighter resource budget; P2 offers higher completion at greater cost. The archive postpones the preference decision and provides different starting points for later mutations.

Joint consideration is needed when relevant effects are coupled and not redundant. Preserving a frontier is useful when several trade-offs remain valuable. If the deployment utility is fixed and known, searching directly for its optimum may be more efficient than maintaining a broad archive.

05 / What do existing multi-objective methods contribute?

Existing multi-objective methods bring these requirements into different parts of the improvement process: workflow search, skill selection and policy learning.

Work The decision it broadens What it searches or trains
EvoFlow Which workflows offer useful quality–cost trade-offs? Workflow populations, prompts and model choices.
AgentBreeder Can capability and measured safety both influence scaffold selection? Multi-agent scaffolds; its Blue setting targets both.
SkillMOO Which skills improve pass rate without ignoring execution cost? Skill bundles for a fixed solver, using NSGA-II.
MOCHA Can a skill be correct while meeting platform requirements? Skill text and metadata, using Chebyshev scalarization and hypervolume-based exploration.
ParetoPO Can policy learning account for accuracy and tool-use efficiency together? Model weights through multi-objective reinforcement learning.
MoFlow How can one search serve different preferences across accuracy, cost, latency, robustness and consistency? Stepwise workflows, with sets of reachable trade-offs stored in tree-search nodes.
MOAE Which execution policies offer useful task, trajectory and safety trade-offs for this query? Compact policies evaluated through complete rollouts, with a Pareto archive.

MoFlow moves the trade-offs inside the search tree: a node stores multiple attainable outcomes rather than one score. Its convex-hull search supports selection under different linear preference weights from the discovered workflows.33

These formulations address different requirements. A platform requirement differs from an operational safety requirement. A reusable skill and a per-query policy have different opportunities to amortize their search cost. GEPA offers another distinction: its Pareto selection preserves strengths across problem instances, rather than automatically defining a quality–cost–safety frontier.9

Our work MOAE takes the per-query route: the model, tools and architecture remain fixed, while candidate policies change planning, tool use, verification and stopping. Objective-specific diagnostics guide changes; complete rollouts supply evaluation; an empirical archive precedes the final deployment choice.23

The broader design idea is to separate what the search should preserve from what one deployment should return. The value of those alternatives depends on candidate coverage and evaluation quality.

There is also a third purpose for keeping a candidate: learning from it. A failed harness can supply a useful negative example, while two successful ones can reveal an unnecessary step—a distinction made explicit in HELIX.30 I would keep deployable options and diagnostic records separately: poor deployment performance need not erase useful evidence.

06 / Multi-objective search leaves several shared problems open

A vector of proxies can still miss the real requirement

Adding objectives repairs an omission only if the new signals measure something meaningful. A judge can reward fluent explanations instead of grounded evidence. A tool-use metric can reward fewer calls while missing the checks those calls performed. Multiple weak proxies do not become reliable merely because they are optimized together.

Evolving the evaluator, as RQGM explores, makes the signal itself adaptable.15 It also makes historical comparison harder. My preference would be to retain stable external checks alongside the evolving evaluation, so that changing behavior and changing standards can be distinguished.

A useful skill may become harmful in a different combination

When Self-Evolution Backfires traces a failure chain: an admitted defective skill influences the skills distilled later, so deleting the original can leave its descendants in place. Its VaG method checks skills before admission and evaluates their joint contribution.31 Coalition-Aware Skill Reliability examines how a skill’s contribution changes with its partners and deployment domain, using coalition-aware selection and transfer-time masking.32

The question becomes “what does this update do inside the current system?” A local gain deserves checks on combinations and future reuse. This adds a time dimension to joint evaluation: a rule can help today’s task and make tomorrow’s updates less reliable.

A noisy frontier can preserve lucky candidates

A candidate that looks excellent on one rollout may be fortunate rather than robust. With many objectives, more candidates can appear incomparable, increasing the burden of deciding which ones deserve another evaluation. A frontier can then grow faster than confidence in it.

This suggests a budget question beyond mutation: when should we generate a new policy, repeat a promising one, or test a suspected failure? A useful archive should retain uncertainty and failure evidence alongside point scores. “Nondominated in observed results” is a statement about the current evidence, not a permanent property of an agent.

Progress on selected tasks does not establish transfer or recursion

The existing results are valuable, but they support different claims:

Source Reported observation What it establishes—and what remains open
DGM Coding success rose from 20% to 50% on its 200-task SWE-bench Verified evaluation after 80 iterations. Evidence for better coding-agent implementations under that protocol; not a 50% result on the entire Verified set.
AIDE² 7 accepted improvements in 8 days; external evaluation covered 4 benchmarks. Improved research agents. Their superiority as outer-loop improvers remained inconclusive because of noise and limited replication.
Open-ended research case studies 2 research problems, roughly 6 days and thousands of dollars of agent work. The researchers reported no substantial research progress in those cases, despite engineering activity. This probes a different task from optimizing a supplied measurable objective.

A better agent on unseen tasks is evidence for transfer beyond its selection set. A better improver requires a further comparison: let original and evolved improvers start from matched agents, with matched budgets and unseen task families, then compare the gains they produce. Private examples used repeatedly to select updates belong to the improvement process, even if their contents are hidden from the inner agent.

For persistent updates, another test is whether they survive a model or tool change. This connects evolution to the agent deployment gap. An inherited workflow may encode assumptions about the old model. A memory entry or tool patch may likewise carry a defect into future executions, a concern emphasized in recent safety synthesis.28

Search itself consumes the resources being optimized

A cheap final policy may have been expensive to discover. A reusable workflow costing 100 USD to find and saving 0.10 USD per use needs 1,000 uses to break even before maintenance. Per-query search has a different accounting problem: it needs to justify its cost on that query. Reports should separate search, evaluation and deployed execution instead of combining them into an ambiguous “efficiency” claim.

07 / What would demonstrate a better improvement process?

I would return to the retry example and ask for an explanation that predicts behavior. Perhaps selective retries work because they distinguish temporary errors from missing permissions. Then the evaluation should vary those conditions while keeping the task goal similar. If the improvement disappears under that small change, the claimed lesson was too broad.

This is why case construction is part of agent evolution. The suite decides which consequences of an update become visible. A suite full of retry-friendly tasks encourages one policy; paired cases that separate recoverable failures from forbidden actions can reveal a more useful distinction.

To test whether joint optimization actually helps, I would compare it with completion-only search plus a final filter, a fixed weighted utility, constrained search, and sequential tuning. All should receive the same candidate interface and comparable total evaluation budgets. The final comparison should use declared deployment preferences and independent tasks, not only the metric favored by the proposed optimizer.

Three observations would make the result informative: does joint evaluation recover usable candidates that the other procedures miss; do retained alternatives serve genuinely different requirements; and do targeted diagnostics generate joint gains rather than merely select different compromises? Archive size alone answers none of these questions. Hypervolume summarizes how much objective space a set of candidates covers relative to a reference point. Comparisons need fixed scales and a fixed reference, alongside actual deployment outcomes.

A reader can apply the same reasoning to the next RSI paper: identify the bottleneck repaired, inspect what is inherited, trace which signals select it, and ask what remains invisible to those signals. Then check whether the evidence supports a better answer, a better reusable agent, or a better process for producing improvements.

The motivation for multi-objective evolution follows from this chain. A single agent action can affect several requirements. A single completion score cannot identify all of those effects. Joint evaluation makes them visible; constraints express what is unacceptable; preferences choose among acceptable outcomes; and an archive can preserve choices that remain useful. The research challenge is to make these decisions reliable enough that successive updates accumulate useful capability.

Paper-by-paper guide

The tables below retain the component-level comparison for reference. The survey offers a broader taxonomy.26

Prompts, memory and skills
Method What changes and persists? Feedback and optimization target
Pre-commit gating / VaG · 2026 Skill admission and the retained pool Validity, harmlessness, consistency and joint skill contribution.
Coalition-aware reliability · 2026 Which skills are accepted or masked Contribution with different skill partners and after domain transfer.
Reflexion · 2023 Verbal reflections retained across trials Task feedback becomes lessons intended to improve later success.
Voyager · 2023 Executable skills and a growing skill library Environment feedback, execution errors and verification support exploration and skill acquisition.
Promptbreeder · 2023 Task prompts and mutation prompts Task accuracy selects prompts; mutation prompts influence how future candidates are produced.
GEPA · 2025 Prompts inside language-model programs Execution traces guide edits; selection preserves complementary strengths across problem instances.
ACE · 2025 Structured context playbooks Generation, reflection and curation turn execution feedback into reusable guidance for future tasks.
Workflows and agent code
Method What changes and persists? Feedback and optimization target
ADAS · 2024 Agent implementations generated by a fixed meta-agent; an archive of designs Validation performance guides further designs.
Meta-Harness · 2026 Executable harness code; raw history stays queryable Task performance and resource use, informed by code and trajectory diagnosis.
AFlow · 2024 Code-based workflows, including operators and connections Execution feedback and tree search optimize task performance.
SICA · 2025 Agent source, tools and surrounding logic; a selected agent becomes the next modifier A utility combines task performance, monetary cost and elapsed time.
DGM · 2025 Coding-agent tools and workflows, retained in a branching archive Coding success selects promising agents while the archive permits further exploration.
Recursive improvement
Method Editable part of the improvement loop What selects a change?
STOP · 2023 A code-improvement program applied to its own code Utility of the programs produced by the improver.
HyperAgents · 2026 Both the task agent and its meta-agent Downstream performance; improvement behavior is evaluated separately.
AIDE² · 2026 The research harness: search, context, verification and inner-loop selection Aggregate private task grades under a fixed per-task dollar budget.
Dream-RSI · 2026 Exploration-policy code: branching, parallelism and stopping Replay-estimated quality–cost frontier coverage, with a parallelism penalty.
RQGM · 2026 Agents and evaluators, updated across evaluation epochs Utility under the current epoch’s evaluator.
Weight updates and curricula
Method What changes? Feedback and optimization target
HELIX · 2026 Harness variants and verified sibling trajectories for subsequent model updates Current execution quality and useful learning records; studied over one harness-evolution round.
SEAL · 2025 Generated self-edits specify adaptation data or directives; an inner update changes weights Outer-loop reinforcement learning rewards performance after adaptation.
Agent0 · 2025 Curriculum and executor policies through alternating training Curriculum rewards use uncertainty and tool interaction; executor training uses correctness proxies derived from generated answers.
AgentEvolver · 2025 Generated tasks, reused experience, step-level credit and trained policy weights Environment return and more efficient learning from interaction.

Two adjacent cases make the target boundary clear: AlphaEvolve evolves target algorithm code; EvoDrive evolves scenario-generator programs, with the driving policy fixed during that search. In each case, distinguish an agent doing the optimization from the artifact being optimized.

References

Years identify the initial manuscript or article; links lead to primary sources.

  1. Reflexion: Language Agents with Verbal Reinforcement Learning. 2023.
  2. Voyager: An Open-Ended Embodied Agent with Large Language Models. 2023.
  3. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution. 2023.
  4. STOP: Self-Taught Optimizer: Recursively Self-Improving Code Generation. 2023.
  5. Automated Design of Agentic Systems (ADAS). 2024.
  6. AFlow: Automating Agentic Workflow Generation. 2024.
  7. A Self-Improving Coding Agent (SICA). 2025.
  8. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. 2025.
  9. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. 2025.
  10. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. 2025.
  11. Self-Adapting Language Models. 2025.
  12. Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning. 2025.
  13. AgentEvolver: Towards Efficient Self-Evolving Agent System. 2025.
  14. HyperAgents. 2026.
  15. The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators. 2026.
  16. Recursive self-improvement of AI research agents (AIDE²). 2026.
  17. Dream-RSI: Recursive Self-Improvement through Evolving Worlds. 2026.
  18. EvoFlow: Evolving Diverse Agentic Workflows On The Fly. 2025.
  19. AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement. 2025.
  20. SkillMOO: Multi-objective Optimization of Agent Skills for Software Engineering. 2026.
  21. MOCHA: Multi-Objective Chebyshev Annealing for Agent Skill Optimization. 2026.
  22. ParetoPO: Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization. 2026.
  23. MOAE: Multi-Objective Agent Evolution with Pareto-Preserving Search. 2026.
  24. EvoDrive: Pareto Evolution for Safety-Critical Autonomous Driving via Self-Improving LLM Agents. 2026.
  25. AlphaEvolve. 2025.
  26. A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence. 2025.
  27. Can AI agents conduct open-ended AI research? Early evidence from two case studies. 2026.
  28. Safety in Self-Evolving Agents: A Survey. 2026.
  29. Meta-Harness: End-to-End Optimization of Model Harnesses. 2026.
  30. HELIX: Model-Harness Co-evolution for Recursive Self-Improvement. 2026.
  31. When Self-Evolution Backfires: Pre-Commit Gating against Skill Contamination in LLM Agents. 2026.
  32. Coalition-Aware Skill Reliability for Self-Evolving Agents. 2026.
  33. MoFlow: Multi-Objective Agentic Workflow Generation. 2026.