2026-09-16
2026-09-16
Agents trained with search-semantic cues learn to make unnecessary search calls on math problems that require code execution.
Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
The central limitation investigated is a tool-selection policy that invokes a tool from superficial prompt cues rather than genuine task requirements. In controlled synthetic environments combining factual question answering and mathematical reasoning, Yang et al. inject cues strongly correlated with particular tools during training while making those cues causally irrelevant to whether a tool is needed.
Counterfactual test. The authors define tool spurious correlation as “cue-driven inflation of tool-selection probability for a functionally inappropriate tool.” They operationalize it with paired counterfactual inputs that differ in cue presence and report (\Delta\mathrm{Tool}_{Y-N}): the difference in selection rate for the inappropriate tool between cue-present and cue-absent variants. This is a concrete evaluation operation for an agent project: construct task-equivalent prompt pairs, then measure whether an irrelevant marker changes the action policy.
Using Qwen2.5-7B-Instruct fine-tuned with GRPO, the reported effect was substantial under one targeted condition. Search-semantic cues were injected into factual-QA training examples, then evaluated on math problems where search was not functionally helpful and code execution was required. The agents made unnecessary search calls, with spurious search rates rising by up to 39.2 percentage points. The effect was selective rather than universal: the authors interpret their experiments as showing that shortcut vulnerability was strongly associated with task competence. In their asymmetric setup, a search shortcut formed when search use was learned on factual QA, whereas the corresponding Python shortcut did not form when the agent did not learn Python use on the math task.
Semantic content mattered alongside competence. In a swapped-cue test, code-semantic cues attached to the well-learned factual task produced (\Delta\mathrm{Search}\leq3.5\%), far below the aligned-cue result. For mitigation, the authors add a dense decision-level reward in which an LLM judge evaluates the necessity of every tool call; they report that it suppresses cue-driven use while preserving task performance.
Boundary and next question. This is not evidence of prevalence in deployed agents: training used 1,000 examples, 30 steps, one 7B base model, and one seed per run. The necessity reward also relies on GPT-5 Nano, which the authors describe as a practical choice rather than definitive necessity ground truth. A useful extension is to repeat the paired-cue audit across seeds and independently validate judge labels against human annotations.
Learn a compact counterfactual audit for checking whether an RL agent chooses tools because they are needed rather than because a prompt cue predicts them.
Abstract; Definition 1 and §2.2; §5.2–§5.4 and Tables 1 and 3; Appendix A.2; Ethics Statement
Gavel Reads Skill-Routing Signals from a Frozen LLM with Two Linear Maps
The Router Within: Eliciting Native Skill Routing from a Frozen LLM
SkillRet evaluates retrieval quality in isolation and does not measure downstream task success or end-to-end agent performance. Gavel targets a different routing constraint: the paper reports extracting selection signals from the frozen agent LLM itself, without placing skill text in context before selection or adding a standalone large retriever.
Gavel routes in two stages: a glance over the whole library followed by a verdict on its shortlist. The glance projects the task and each skill’s mid-layer states through two trained linear maps, then scores the library against compact per-skill key banks built during installation. The verdict resumes the frozen model with shortlisted skills and reads both task likelihood and an explicit yes/no judgment. Their scores are fused with the glance as a product of experts. The installation-time key banks can also be compressed: with ε=0.83 in the Qwen3-32B build, the paper reports an approximately 8.5× reduction in bank size at a cost of at most about 1.6 benchmark points.
On a frozen Qwen3-32B backbone, the authors report that Gavel outperforms progressive-disclosure and retrieve-and-rerank pipelines adding 1.2B–16B external parameters, by up to 13.4 percentage points on written-task benchmarks and up to 21.9 points when routing is needed mid-rollout. The comparison covers SkillRet, SRA-Bench, Eval-Core, and SkillTraj, with Hit@1 adjudicated by GPT-5.6 Sol. The ablation supports the fusion design: the full glance-plus-likelihood-plus-yes/no ruling scores highest on every tested benchmark, exceeding the better single-signal variant by 1.9 points on SkillRet, 3.8 on SRA-Bench, and 4.6 on SkillTraj.
The gains still have a measurement boundary. Human validation of the automated judge agreed on the credit decision for 171 of 200 sampled pairs, or 86%, so adjudicated Hit@1 is not noise-free. The reported transfer is also tied to the stated benchmark setup rather than established live-traffic robustness. The paper leaves open whether the same read-out can route tools, memories, or MCP servers from an agent’s own forward passes.
For a focused follow-up, train only the two readout maps, keep the backbone frozen, and test whether rankings remain stable when the skill library, query style, and routing point in a trajectory change. The useful research question is not only whether internal states predict the next skill, but when that signal fails relative to an external retriever and whether the verdict’s extra forward pass improves task completion rather than only adjudicated routing accuracy.
Read it for a concrete recipe—mid-layer readout, installation-time ε-cover compression, and product-of-experts fusion—and to design a distribution-shift test for internal routing before applying the idea to tools or memories.
Abstract; §3.2–§3.5; §4.1; §4.3; Appendix F.2 (Table 3); Appendix H.2; §5 Conclusion · §6 (Limitations)
ReImaGin uses image generation as a flexible visual reasoning tool
Reasoning with Image Generation
Textual chain-of-thought is a poor fit for tasks that require direct manipulation of visual representations. The paper’s concrete target is the limitation of specialist visual tools: their operations are narrow and rigid, rather than able to flexibly generate or transform visual content.
ReImaGin is training-free and modular: it invokes a single instruction-following image generator for open-ended transformations and carries out its visual reasoning in interpretable pixel space. Unlike fixed-function tools, the generator accepts natural-language commands for operations such as removing an occlusion or generating a floorplan from disjoint room views. The reported comparisons use text-only reasoning and a specialist visual-tool baseline across six visual reasoning tasks.
For a researcher, the most actionable mechanism is selection, not merely generation. On MMSI multi-view spatial reasoning, sampling 10 images and using an MLLM to choose the most faithful candidate raises accuracy from 51.0% to 59.0% for Gemini-3.1-Pro and from 42.0% to 54.3% for Qwen-3.5-27B. The paper also searches for visualization policies: proposal and reasoning agents evaluate task-specific prompts on small train/dev splits. With at most 50 examples in each split and four rounds, automatic discovery recovers most of the gains of handcrafted strategies, although a gap remains for tasks requiring more elaborate policies.
A transferable experiment is to treat the visual intermediate as a hypothesis rather than an explanation: define a transformation that should expose the task-relevant relation, sample several outputs, select or verify them, and measure both answer accuracy and transformation faithfulness. This separates the question ‘did generation help?’ from ‘did the agent reason over a reliable representation?’ The no-op ablation supports this isolation: replacing generation with a tool that returns its input lowers performance on all six tasks, with the largest drops on path tracing (89.0 to 77.5) and puzzle completion (42.3 to 36.5).
The boundary is operational. Generative calls add cost and latency compared with specialist-tool baselines, and prompt discovery costs approximately $214 per task in the reported configuration. Automatic policies can also request transformations the generator executes unreliably: a richer strategy slightly hurts performance when coloring lines is less reliable than making them solid. On Gemini-3.1-Pro depth reasoning, the specialist baseline still leads, 99.2% to ReImaGin’s 94.6%. The useful research question is therefore which intermediate representation is worth its generation and verification budget, not whether image generation universally replaces specialist tools.
Read this paper to learn how to make an MLLM propose, sample, and verify task-specific visual intermediates, while testing whether generator faithfulness and runtime cost justify replacing fixed specialist tools.
On four social-simulation tasks, SAE and probe steering often outperform basic prompting
Interpreting and Steering LLM Agents for Social Simulations
LLM-based social simulation has a concrete measurement problem: interpretability is needed to assign clear mechanisms to observed behavior, while steerability is needed to mute or amplify theoretically meaningful mechanisms. Fan et al. compare prompt-based manipulation, SAE-derived feature steering, and probe-based direction steering on Llama-3.3-70B-Instruct across four natural-language tasks covering lottery risk, ultimatum altruism, divergent creativity, and product innovation.
The method change is to make internal representations part of the intervention rather than treating prompts as the only control surface. SAEs are used to decompose representations into human-readable, behavior-linked features; probes then shift behavior in specified directions. In the reported implementation, probes are trained on layer-48 activations and steered with a scalar direction, while SAE steering nudges selected risk-related features. This creates an inductive-to-deductive workflow: use SAE features to propose mechanisms, then use a probe to test and control a behavioral dimension.
The clearest evidence is calibration. In the lottery task, probe steering placed the switching point across approximately 30–200 tokens, with a mean absolute error of about two tokens from target values, using 40 agents per condition. SAE steering also produced dose-dependent transitions: moderate steering began shifting behavior around 75 tokens, while stronger steering shifted the transition toward 65–70 tokens. Prompting was less consistent across traits: no prompting variant produced graded risk-seeking in the lottery, while few-shot chain-of-thought shifted ultimatum acceptance thresholds from roughly 10 to 40 tokens.
The boundary matters. The experiments focus on individuals rather than multi-agent networks or influence, and the trait set is small. SAE steering can hallucinate when pushed too far, and imperfect feature disentanglement can activate off-target representations. Probe directions are not automatically context-invariant either: a probe trained on brick creativity data transferred to other objects, but probe accuracy degraded by about 5–6 percentage points.
For a follow-up study, treat an SAE-readable feature as a hypothesis rather than as validated causality: fit and calibrate a probe against a behavioral target, then sweep its coefficient across held-out objects or tasks while recording off-target changes. The useful research question is whether the intervention changes the intended social trait without changing unrelated behavior.
To learn a concrete replacement for prompt-only trait manipulation: identify candidate internal features, calibrate a probe-based control, and test its cross-context and off-target behavior.
abstract1.1; S4.SS2.SSS1.p4; S4.SS2.SSS1.p2; S4.SS2.SSS1.p3; A2.SS7.p3; S5.p9; S5.p10; S5.p11
Plan Injection Steers Actors and Evades Chain-of-Thought Monitors in the Authors’ Tests
Corrupt Plans, Clean Traces: Evading Chain-of-Thought Monitoring with Plan Injection
The limitation under test
Chain-of-thought (CoT) monitoring inspects an actor model’s reasoning for unsafe planning, deception, or misalignment. This paper tests an input-integrity gap: harmful but benign-sounding reasoning placed in the actor’s context can steer subsequent behavior while evading that inspection. The authors call this attack plan injection (§Abstract).
Method change
The authors first use an investigator-agent setup in a TruthfulQA multiple-choice hint setting to discover adversarial plans. Interestingly, the most effective of these variants, labeled cot_pure, simply appends the faulty injection plan to the query without any explicit instruction to use or follow it (§3.3). They then evaluate injected plans on APPS coding tasks and a paired Bio-Math setting, asking whether actors adopt the injected reasoning and whether monitors flag the resulting transcripts.
The practical research move is to evaluate both steering and detectability. A plan that changes behavior but is immediately flagged is different from one that changes behavior while producing a seemingly self-generated rationale that a monitor accepts.
Evidence and conditions
On APPS, the paper reports that “the actor followed the injected plan in 50.6% of cases for Qwen3-8B and 41.7% for DeepSeek-R1” (§5.3). These figures apply to the paper’s verified APPS injections, whose embedded code was checked against public tests and a chosen backdoor input. In Bio-Math, using verified rewritten plans, the reported adoption rates were 93.0% and 98.9%; these are best-of-three rates, so they should not be read as single-shot rates (§5.3).
The monitor result is conditional on the authors’ monitor suite and suspicion threshold: monitors “fail to flag at least 25% of the attempts,” while non-thinking monitors miss “up to 76%” in some evaluated settings (§1). More monitor reasoning was not a dependable remedy in the tested budget sweep: “Increasing the monitor’s reasoning budget is not uniformly helpful” (§6.3).
Boundary and research operation
The authors’ scaling experiments cover two reasoning models and two tasks, and they do not demonstrate an end-to-end attack through RAG, tool outputs, or multi-agent pipelines (§7). A useful follow-up experiment is therefore to treat every retrieved plan or intermediate trace as an untrusted input: vary its provenance and wording, then measure both behavioral adoption and monitor detection rather than assuming a coherent visible rationale is safe evidence.
It gives a concrete evaluation pattern for CoT safety: test whether untrusted contextual plans both steer an actor and evade the monitor, while keeping the reported model, task, aggregation, and sampling limits explicit.
Prepared retrospectively from the 2026-09-16 candidate papers; verified on 2026-09-16.