2026.10.20

Why this paper matters

LLM agents can accept a report after a required check has failed or settle for a weaker conclusion without trying another route that could still succeed. PRIME addresses this by separating check generation from report authorization: the agent may propose actions, but a runtime determines which conclusions the recorded evidence permits.

Method

PRIME freezes alternative evidence routes before checking begins. Each route is a conjunction of evidence conditions, and completing a route for a conclusion—or for a stronger conclusion—authorizes the corresponding report. An external runtime records check results in a ledger, blocks conclusions without a completed route, and identifies unresolved checks that could support a stronger conclusion. Optional completion can run those checks after the agent stops. The same route representation therefore serves two functions: enforcing what may be reported and directing budgeted evidence collection.

Evidence

On the 480-task SUFFICE split, false authorization was 26.7% for a fully informed Qwen2.5-32B-Instruct agent and 0.0% for PRIME under executable requirements and deterministic resolvers. Showing an agent its requirements and ledger did not prevent unsupported reports in this benchmark; runtime enforcement did. PRIME’s directions and added checks raised the average score for justified conclusions from 30.1 with refusal alone to 59.2 with completion. These score results used Qwen2.5-32B-Instruct and a budget of tool requests rather than matched total model calls or tokens.

Boundary and research use

“Authorization is not truth”: the gate certifies consistency with the committed requirements and ledger, so wrong requirements or admitted incorrect observations can still yield an incorrect conclusion. The executable SUFFICE reference standard also does not establish performance against independently validated organizational policies. Confirmatory evidence is concentrated on one model and one greedy run per task; other models and decoding settings are secondary analyses. In a separate 240-task text-judgment experiment, PRIME still had 4.6% false authorization; a second judgment reduced this to 0.8%, but lowered supported utility, added model calls, and was not a matched-total-compute comparison.

A useful replication would implement the route ledger for graded outcomes and compare gate-only, route-guidance, and post-stop-completion variants under equal total model-call and token budgets. The key question is not only whether unsupported reports disappear, but how much justified utility is recovered per additional verification cost—and how those results change when requirements and evidence judgments are fallible.

An evolved harness can score higher on its search set without establishing a comparable gain beyond that set. The paper targets this evaluation gap: a search-set improvement may combine benchmark-specific overfitting, additional inference-time compute, and residual changes whose generality is still unknown. Harness-Delta Attribution (HDA) is designed for harness evolution around a fixed-weight model.

For each baseline–evolved pair, HDA first retains only statistically significant search-set gains. It then constructs two controlled variants: a compute-matched baseline that spends comparable inference-time resources, and a neutralized evolved harness in which detected benchmark-specific shortcuts are minimally disabled. An analysis agent proposes the matching and neutralization procedures, while a human reviewer checks them at two task-dependent checkpoints. HDA defines the scaling contribution as the compute-matched baseline’s gain, the overfitting contribution as the score lost after shortcut neutralization, and the remainder as residual improvement.

The evidence makes the attribution consequential. Across 11 executor models and three evolution frameworks, 21 settings showed statistically significant search-set gains. In those settings, overfitting explained 54% of the gain on average and test-time scaling explained 14%; their combined share exceeded 50% in 14 settings. The same LiveMath artifact was independently encoded by all three tested evolution frameworks. Transfer was also fragile: excluding ALFWorld, the held-out gain was smaller than the search-set gain in 13 of 15 settings; in eight it was less than half as large, and four evolved harnesses scored below baseline on test. Validation-based selection did not remove this problem: across 16 Meta-Harness settings, it improved held-out performance in 10 and reduced it in six.

A useful research operation follows: report the search-set significance test, matched-compute baseline, shortcut-neutralized harness, and held-out results as separate quantities rather than presenting one aggregate improvement. Treat the residual as an audit bound, not as measured transfer. HDA interprets overfitting as a lower-bound estimate because implicit or entangled shortcuts can remain; it interprets the residual as an upper bound on generalizable improvement, and reports cases where a nominally residual-only search-set gain still regressed on held-out data. Finally, held-out splits can share artifacts: LiveMath’s answer-choice pattern is shared across splits, while ALFWorld’s simulator rules persist across them. These conditions make attribution-aware evaluation a complement to, not a replacement for, independent transfer tests.

An autoresearch loop can generate many experiment ideas, but it still needs to estimate whether a proposed change is worth a long training run. This paper turns that operational gap into a measurable target: forecast the impact of an individual LLM-proposed ML experiment before execution, then compare the forecast with the observed outcome. The focus is therefore numerical outcome forecasting, not merely producing an idea or completing a strong final solution.

The authors assemble 288 LLM-proposed ideas across eight MLE-bench tasks; after 19 failed runs or unusable predictions, 269 ideas remain for analysis. Five frontier LLMs then predict each idea’s impact before training, and their forecasts are compared with the measured results. Prediction and outcome changes are placed on a common normalized scale, after which the paper reports pooled R² for fit and a fitted slope for magnitude calibration.

The strongest pooled predictor is Claude Opus 5.5, with R² = 0.49, followed by Claude Fable 5.1 at 0.41 and GPT-6 Astra at 0.38. Every tested model underestimates the magnitude of large score changes: fitted slopes range from 0.21 to 0.61. Forecast quality is also task-dependent: Opus 5.5 reaches R² = 0.94 on toxic-comment classification but only 0.39 on dog-breed identification. Direction is easier than magnitude in this evaluation: Opus 5.5 predicts the correct direction for 82% of ideas. Higher reasoning settings help the three strongest models, but not consistently across all five.

The transferable operation is to evaluate an experiment forecaster along separate axes. For each executed idea, log the predicted and observed normalized change; report direction accuracy, R², and the fitted slope separately; then stratify those metrics by task family. Use the resulting calibration evidence to decide how much weight a forecast should receive in experiment selection, rather than treating a correct sign as evidence of a reliable magnitude estimate.

The boundary conditions are material. Each idea was run only once, so outcome noise is unquantified and the attainable prediction ceiling is unknown. The study covers eight small ML problems, not large-scale model-training runs. Its results also apply to the retained 269 ideas, because 19 of the 288 generated ideas were excluded after failed runs or unusable predictions.

Long-horizon agents can learn from failures, but the value of a failure record may change as execution progresses. RECAP studies whether retaining, compressing, or removing one history block improves the subsequent continuation relative to full-history KEEP.

At each retained checkpoint, it replays the same environment state and branches matched rollout seeds into three treatments. KEEP retains the block, COMPRESS replaces it with a deterministic summary, and DROP removes selected message pairs plus matching duplicate failure mentions. The measured outcomes are terminal success and continuation prompt-token cost against KEEP. A five-seed longitudinal case holds both the failure block and its summary fixed: compression fails at an early state and becomes cost-saving at a later state.

RECAP's proposal stage uses prefix-only execution-relational features to relate the failure block to the current execution. It predicts harm and normalized token benefit for COMPRESS and DROP, proposes the higher-benefit edit, and requires both gates; otherwise it returns KEEP. Independent task-bundle calibration then selects among frozen candidate policies using one-sided Clopper–Pearson bounds for two task-level losses: harmful selected checkpoints and non-positive aggregate normalized benefit. Its guarantee is conditional on iid calibration and future bundles under the same fixed sampling and measurement protocol, and it does not certify individual edits.

On Hunyuan-A13B-Instruct (FP8), RECAP edited 43.24% of 37 held-out retained test checkpoints (16 edits: 10 COMPRESS, 6 DROP), matched KEEP's observed continuation success, and reduced unconditional mean continuation prompt cost by 4,619.8 tokens (2.13%) over 111 continuations. Across five backbones, its paired-success saving point estimates were positive, while observed continuation success stayed within 1.6 percentage points of KEEP; these savings are averaged over a selector-dependent jointly successful subset rather than all continuations.

The research operation to borrow is to freeze one failure block, compare matched KEEP/COMPRESS/DROP branches at multiple checkpoints, and record success and prompt-token cost separately. This asks whether an edit transfers across execution states, rather than whether compression is uniformly useful.

The scope is narrower than sequential live memory adaptation. The study evaluates one intervention per matched continuation, so its evidence is not for repeatedly adapting context through a complete evolving episode. The main evaluation uses screened AgentCE-Bench planning tasks and retained checkpoints; it does not establish performance on unsuccessful baseline trajectories or other deployment settings. For Hunyuan, state-dependent selection does not demonstrate learned harm discrimination because all development harm labels were zero; zero observed test harm is not evidence of zero population risk.

Attack success rate (ASR) gives every failed injection the same interpretation: a successful defense. This paper audits that limitation by classifying what happened in standard tool-agent trajectories and by adding a security–task-completion view.

The authors audit 33,000 trajectories from eleven open-weight agents (2–14B parameters) on InjecAgent and AgentDojo with a label-blind LLM judge. They also hold model weights and attacks fixed while changing the system-prompt policy on InjecAgent, and compare serving precisions. For AgentDojo episodes with both verdicts, ATOM crosses attack success with user-task completion.

This differs from the purpose-built SecFid benchmark, which already separates executing an injection, processing it as task data, and ignoring it. The useful increment here is to apply failure-behavior labels to standard benchmark trajectories and to expose a per-episode security–utility outcome that ASR alone collapses.

Among benchmark-scored failures, the judge classified injections as ignored in 76.3% of AgentDojo cases and 80.3% of InjecAgent cases; explicit refusal accounted for 18.0% and 17.2%, respectively. Recognition did not guarantee resistance: among trajectories where the judge flagged recognition, agents still executed the attacker’s action in 28.9% of AgentDojo cases and 16.9% of InjecAgent cases. In the matched InjecAgent prompt test, pooled ASR moved from 0.2% to 43% when the system policy changed, while an Interdict prompt reduced it to 0.22%; model weights and attacks were unchanged. On AgentDojo, 21.7% of audited episodes were safe but dysfunctional—attack failed, but the user task was incomplete. NF4 reduced ASR by 2.33 points versus FP16 while reducing Safe-and-Useful episodes by 3.97 points and utility under attack by 4.75 points, so the lower ASR was not evidence of better security in this evaluation.

These conclusions are bounded: experiments use two synthetic, single-agent benchmarks with static injections, so they do not establish resilience to adaptive or white-box attackers, multi-agent settings, or live deployment. The 33,000 labels rely on one LLM judge; 150 human validation examples agreed 96% on four-way groups but 86% across six labels and 77.33% for InjecAgent six-label judgments. Recognition is inferred from stated reasoning, so it may be undercounted or fail to reveal internal causes. ATOM applies only to AgentDojo, while prompt-policy interventions ran only on InjecAgent and did not measure user-task utility.

For a next evaluation, report ASR together with failure mechanism and task completion, then rerun the same models and attacks under system-prompt policies and serving precisions. The research question is not simply “did the attack fail?” but “did the agent refuse, ignore, mechanically fail, or safely complete the user’s task—and do those labels persist under changed conditions?”