EvoHarness-RL: Learning Runtime Harness Coordination for Self-Evolving Agents

Overview

Long-horizon agents depend on an external harness — memories, state trackers, planners, verifiers. Today each harness is hand-built for one environment and driven by fixed rules, so the agent and its harness are never optimized together.

Can heterogeneous, task-specific harnesses be unified into one interface — and can the agent learn when to use it?

Key insights

1One BPE workflow for every dataset

Belief / Progress / Experience defines roles, not implementations. A thin environment adapter grounds them in ALFWorld, WebShop, and WebArena; the policy always sees the same interface.

frontier models +10.0 pts, zero training

2Harness use is an on-the-fly action

Four meta-actions share the action space and step budget with environment actions. The policy decides at each step whether to read or write external state.

A = A_env ∪ {track, commit, recall, note}

3Two findings about RL with a scaffold

(a) The BPE scaffold makes GRPO find successful rollouts faster. (b) Early diversity / exploration prevents premature pruning and lets RL converge to a better coordination rule.

2.2× / 1.6× / 2.9× faster · w/o diversity −2.1

4Internalization → harness annealing

Repeated patterns the scaffold provides get absorbed into the policy through RL, so harness calls drop while success keeps rising. What remains external is what is genuinely hard to internalize.

calls −67% / −83% / −15%, success ↑

One BPE workflow, every environment

Instead of a new harness per benchmark, every environment plugs into the same three-role workspace $\mathcal{H}_t = (B_t, P_t, E_t)$. Only the adapter changes.

Environment

ALFWorld · WebShop · WebArena

Env. adapter

grounds roles from agent-visible observations

Shared interface

Belief · Progress · Experience

Policy

same 4 actions, any environment

Role

ALFWorld

WebShop

WebArena

Beliefwhat the agent believes · in-episode

Object states, locations, receptacle relations, inventory

Request constraints & compared candidates across pages

Current page, navigation trail, active controls, past values

Progresshow the task is going · in-episode

Bounded record of manipulation subgoals

Which product requirements are resolved / unverified

Completed vs. pending requirements across pages

Experiencewhat can be reused · cross-episode

General principles + task-specific procedures & failure patterns, maintained by a solver–evolver loop (Add / Update / Archive / Skip) — identical structure in all three.

No variant sees privileged environment state; all content is derived from normal agent interaction.

+10.0

Plugging BPE in as an always-on scaffold (EvoHarness-Base) already lifts Claude Opus 4.6, GLM-5, and GPT-5 by 10.0 pts on average with no parameter updates — while hand-designed memory baselines on Qwen3-8B are mixed or even negative across environments.

Action protocol: using the harness on the fly

Always-on BPE fixes what lives outside the model. EvoHarness-RL instead turns every harness access into a policy decision, made step by step during the rollout.

STEP t

One action, joint space

$a_t \sim \pi_\theta(\cdot \mid o_t, \mathcal{V}_t, c_t)$ over $\mathcal{A}_{\mathrm{env}} \cup \mathcal{A}_{\mathrm{bpe}}$.

IF ENV

Advance the world

The environment steps; adapters silently update Belief / Progress stores in the background.

IF BPE

Read / write, then refresh $\mathcal{V}_t$

The call costs one step of the same budget — so it must be worth it.

trackREAD · B

Selective read of Belief, e.g. [object], [price], [candidates], earlier page values.

commitWRITE · P

Write the current subgoal to Progress; returns the updated done / remaining plan.

recallREAD · E

Query Experience for relevant principles, procedures, known mistakes, or search hints.

notePROPOSE · E

Flag a discovery or a contradiction to recalled Experience; buffered and consolidated only at epoch end.

Illustrative ALFWorld trace — “put two tissue boxes on the dining table” (adapted from the paper’s appendix example).

rollout

…ENVtake tissuebox 2 from coffeetable 1

BPEtrack [tissuebox]

tissuebox 1 on diningtable 1 · tissuebox 2 held_by agent · tissuebox 3 on coffeetable 1 · searched: diningtable 1, coffeetable 1

BPEcommit [place held box on diningtable 1, then verify two distinct boxes]

✓ place first box ✓ pick up second box ☐ place + verify count

BPErecall [pick two]

Do not treat the first placement as completion of a two-object goal; verify each new relation.

ENVgo to diningtable 1

ENVput tissuebox 2 in/on diningtable 1

BPEnote [tissue boxes often on coffee tables; verify after each placement]

buffered → evolver decides Add / Update / Archive / Skip after the rollout batch

The observation alone no longer says where box 1 went or which receptacles were searched — Belief keeps it, Progress keeps the “two boxes” constraint explicit, Experience supplies the reusable check.

Training pipeline: GRPO rollouts interleave environment actions with BPE meta-actions; composite reward of base, exploration, and specialization terms.

Fig. 2Training pipeline of EvoHarness-RL. The policy generates rollouts that interleave environment actions with BPE meta-actions, accessing and updating belief, progress, and experience states. GRPO updates the policy with a composite reward that combines the base objective, an annealed exploration bonus, and an efficiency incentive, encouraging a transition from broad harness exploration to selective, task-adaptive coordination.

Learning how, then when

Stage 1 · How

Supervised initialization

Teacher trajectories that interleave env actions with BPE calls teach the syntax and semantics of each action. Without it, RL cannot explore the new action space from sparse reward (ALFWorld 95.0 → 50.0).

Stage 2 · When

Cost-aware GRPO

RL compares coordination strategies and keeps only the calls worth their step cost, instead of copying the teacher’s pattern.

$R(\tau;u) = R_{\mathrm{GRPO}}(\tau) + \underbrace{\lambda_{\mathrm{div}}(u)\,R_{\mathrm{div}}(\tau)}_{\text{explore (annealed)}} + \underbrace{\lambda_{\mathrm{eff}}\,R_{\mathrm{eff}}(\tau)}_{\text{specialize}}$

$R_{\mathrm{div}}$: fraction of BPE action types used in $\tau$, with a cosine weight that reaches 0 at horizon $U$. $R_{\mathrm{eff}} = \mathbf{1}[\mathrm{succ}]\,(T_{\max}-|\tau|)/T_{\max}$: shorter successful trajectories win, so a redundant harness call costs the same as a redundant env action.

Two findings about RL with a harness

Finding A

The scaffold makes GRPO rollout-efficient

Externalized state, progress, and experience let GRPO discover successful trajectories earlier, so each rollout batch carries more learning signal.

Same supervised-data budget: reaches 85 / 90 / 93% validation success in 2.2× / 1.6× / 2.9× fewer GRPO steps than vanilla SFT + GRPO.
It starts lower (57.8 vs. 79.7%) — it must also learn four new actions — then overtakes. The gain is not from a stronger SFT checkpoint.
Versus plain GRPO: hits 80% training success by step 19 and averages ~23 pts higher over the last 10 steps.

Why: conditional rollout-efficiency bound

If BPE returns decision-relevant information used with bounded error, the rollouts needed to find a success with prob. $\ge 1-\rho$ satisfy

$$N_{\mathrm{BPE}}(x) \le \Bigl\lceil \tfrac{1}{1+\alpha(x)/p_0(x)}\, N_{\mathrm{base}}(x)\Bigr\rceil,$$

where $p_0$ is the baseline success probability and $\alpha>0$ the net benefit of BPE information — largest when $p_0$ is small.

GRPO steps to validation threshold: EvoHarness-RL vs SFT+GRPO.

Fig. 3GRPO steps to reach ALFWorld validation-success thresholds (evaluated every 5 steps), same supervised-data budget.

Finding B

Diversity & exploration let RL converge to a better coordination rule

Efficiency pressure alone prunes new harness actions before their value is discovered. A transient diversity bonus keeps strategies alive early, then anneals away so efficiency can specialize them.

Harness-action dynamics under full reward, without diversity, and without efficiency.

Fig. 4Per-action harness use under different rewards. Without diversity, use anneals too early; without efficiency, it stays high late in training.

w/o diversity: harness use collapses early — premature pruning, not specialization. ALFWorld 95.0 → 92.9.
w/o efficiency: calls stay high late in training; nothing separates useful from redundant calls. 95.0 → 86.4.

Complementary in time: diversity broadens early exploration; efficiency converts it into selective use.
note peaks early to bootstrap an empty Experience bank, then fades — its distilled knowledge keeps being recalled.

Why the bonus doesn’t bias the final policy: convergence with diversity annealing

With $\mathcal{B}^*$ the optimal coordination rules and $\Delta_J$ the gap to the best non-optimal rule,

$$J^* - J(\mu_u) \le \varepsilon_{\mathrm{opt}}(u) + \lambda_u, \qquad \Pr(b\in\mathcal{B}^*) \ge 1 - \tfrac{\varepsilon_{\mathrm{opt}}(u)+\lambda_u}{\Delta_J}.$$

Since $\lambda_u = 0$ for $u \ge U$, retained harness use is ultimately decided only by task reward, efficiency, and optimization error.

Analysis

How training reshapes the division of labor between the model and its harness — five findings, one at a time.

Analysis 1

Internalization & harness annealing

The scaffold’s repeated patterns get absorbed into the policy by RL. On ALFWorld, mean harness calls fall from ~5 to ~1 while success keeps rising — the harness is kept only for what is still worth externalizing.

75%fewer harness calls, early → final

60%higher training success

+6.4pts over always-on BPE

Phase 1

Early explore

Broad reliance on BPE to stabilize long-horizon execution; every action gets tried.

Phase 2

Annealing

Recurring routines move inside the model; routine calls are pruned while success rises.

Phase 3

Stable

Calls plateau above zero — what remains is genuinely useful outside the model.

Not just fewer calls: perturbing the learned choice and timing hurts even at matched frequency.
Always-on BPE (no learned annealing) trails EvoHarness-RL by 6.4 pts.

Mean harness calls over ALFWorld training: early explore, annealing, and stable phases.

Fig. 5Mean harness calls during ALFWorld GRPO training, split into early-explore, annealing, and stable phases.

Per-action breakdown across all three environments

Per-action harness use declines while success rate rises in ALFWorld, WebShop, and WebArena.

Each action anneals at its own rate while success (dashed) rises; the internal/external boundary differs by environment.

Analysis 2

SFT teaches how; RL teaches when

Harness use rises with SFT, then falls with RL — while success rises at both stages, on every benchmark.

3.0×fewer calls than SFT (ALFWorld)

+26.4pts success over SFT

Base

Interface exists, unused

Prompting alone: the pretrained policy barely calls BPE.

calls1.96

success52.1%

SFT · learn how

Heavy scaffold reliance

Imitation teaches the actions — and copies the teacher’s heavy call pattern.

calls9.03

success68.6%

RL · learn when

Selective coordination

Redundant calls pruned; task-relevant support retained.

calls2.98

success95.0%

Harness calls per episode and success rate for Base, SFT, and EvoHarness-RL across three benchmarks.

Fig. 6Harness calls per test episode (bars) and test success (line). SFT → RL: calls 9.03→2.98 (ALFWorld), 6.87→1.15 (WebShop), 2.00→1.70 (WebArena); success 68.6→95.0, 43.8→80.6, 16.0→26.0.

Analysis 3

Coordination specializes to the environment

Harness use drops everywhere, but the same recipe keeps different actions depending on task structure.

92%recall share · ALFWorld

89%track share · WebShop

66–75%fewer calls overall

ALFWorld — recurring household routines → reusable experience matters: recall is 92% of final calls.
WebShop — a fresh constraint combination every episode → keep the current request: track is 89%.
WebArena — long workflows with shifting subgoals → no single state suffices: mixed, led by commit.
Total calls fall 66–75% while training success rises 60–83%.

Early vs final meta-action composition across ALFWorld, WebShop, WebArena.

Fig. 7Early vs. final meta-action composition. $H$ = harness actions per episode, SR = training success rate.

Analysis 4

The coordination rule transfers across domains

Initialize ALFWorld training from a WebShop-trained checkpoint and adapt for 100 steps.

EvoHarness-RL transfer adapts fast and approaches the in-domain policy on both training and validation.
GRPO transfer shows strong negative transfer: training partly recovers, validation stays far lower.
Dropping target-side SFT also hurts — interface initialization still helps adaptation.
What transfers is when to externalize, not a fixed action mix — the two domains end with very different mixes (Analysis 3).

Cross-domain adaptation from WebShop to ALFWorld: training and validation success.

Fig. 8WebShop → ALFWorld adaptation over 100 target-domain steps. Left: training success; right: validation success.

Analysis 5

The harness evolves with the policy

Experience is continually maintained, not append-only: it moves from acquisition to refinement.

94.8unseen success, co-evolved bank

+8.2pts over a fixed bank

Early cycles are dominated by Add; as the bank fills, Update takes over to fix stale or conflicting entries.
Co-evolving beats a fixed bank, most on unseen tasks (94.8 vs. 86.6).
Self-consolidation by Qwen3-8B itself works without a frontier consolidator.

ALFWorld avg. success (%) of separately trained Experience variants.

Experience-bank composition and evolver operations over consolidation cycles.

Fig. 9Experience-bank composition (bars) and evolver Add / Update operations (lines) over consolidation cycles.

Results

ALFWorld, WebShop, and WebArena. EvoHarness-RL (Qwen3-8B) vs. the strongest open-source baseline, SkillRL:

ALFWorld Avg

95.0

vs. 85.0 · +10.0

WebShop SR

80.6

vs. 73.2 · +7.4

WebShop Score

90.8

vs. 85.5 · +5.3

WebArena Avg

26.0

vs. 18.0 · +8.0

Δ vs. ReAct

+43.9

vs. +36.3 · +7.6

Main results

Success rates (%). ∗ frozen, trainable; blue = column best per backbone; shaded rows = ours; Δ averages the four aggregate-metric gains over paired ReAct; † ALFWorld/WebShop numbers transcribed from SkillOS.

Unseen ALFWorld tasks

EvoHarness-RL keeps 94.8% — +22.2% relative over SFT + GRPO, close to seen-split performance.

Ablations (ALFWorld Avg)

Dashed line = full EvoHarness-RL (95.0); ▼ = drop when the component is removed. Axis starts at 40%.

BibTeX

@article{ning2026evoharness,
  title   = {EvoHarness-RL: Learning Runtime Harness Coordination for Self-Evolving Agents},
  author  = {Ning, Xuying and Fu, Dongqi and Wei, Tianxin and Bei, Yuanchen and Yang, Xiyuan and
             Xu, Wujiang and Song, Yueqi and Li, Bingxuan and Li, Zihao and Zeng, Hanqing and
             Shen, Xiang and Wang, Yajuan and Wu, Yifan and Wang, Qifan and Liu, Jiayi and
             Li, Hong and Xia, Yinglong and Fan, Xiangjun and Tong, Hanghang and He, Jingrui},
  journal = {arXiv preprint arXiv:2608.05446},
  eprint  = {2608.05446},
  archivePrefix = {arXiv},
  year    = {2026}
}