2026-11-10
2026-11-10
Low-budget molecular-optimization scores can depend strongly on retrievable task information
What Are LLM Agents Actually Learning on Low-Budget Molecular Optimization Benchmarks?
The paper’s central intervention is a controlled information ablation: on PMO, it varies access to task names/descriptions and curated reference compounds at a 50-oracle-call budget, adds an oracle-form-only condition, and compares the agent with a deterministic retrieval-only policy. A separate ProteinLeadOpt-50 experiment tests a fixed single-edit random walk under an AutoDock Vina composite oracle.
Under standard information, GPT-4o mini reaches 14.44 sum top-1 AUC across 23 PMO tasks; hiding task names and curated references produces 4.85, while the identity-blinded oracle-form condition scores 5.39. In the factorial ablation, references add 9.13 AUC when names are absent, whereas names add 1.22 when references are absent. The no-LLM retrieval-only policy reaches 12.16—84% of the full agent’s 14.44—by submitting exposed references and making one-atom RDKit variants. On the three-seed, eight-task subset, standard information beats full blinding on all eight tasks (one-sided p=.0039).
On ProteinLeadOpt-50, three random-walk seeds over 50 target–starter pairs obtain composite 0.636 ± 0.008 and docking improvement −1.11 ± 0.04 kcal/mol, making this a concrete oracle-matched floor for that tested setup. Yet the paper’s positive control prevents an overbroad conclusion: under strict blinding, a retrieval-free population agent reaches 7.39 versus REINVENT’s 7.06 at matched 50-call sum-AUC budget. The useful research operation is therefore to separate information access from feedback-driven search before assigning credit to an agent’s optimization behavior.
Treat the result as scoped. The full 23-task factorial is one run per condition; the C1–C4 comparison alone has three seeds and covers eight tasks. The causal ablations concern one gpt-4o-mini agent and protocol. The random-walk result is specific to the tested Vina composite, starter set, edit set, and budget; the paper’s sensitivity analysis shows that the chosen substituent set materially affects the baseline. A practical audit is to run the 2×2 access ablation, publish a retrieval-only score, and test a random-walk floor matched to the oracle while reporting seeds and edit choices.
It provides a reusable audit for low-budget molecular-optimization claims: hide task and reference identity, measure a no-LLM retrieval baseline, and test an oracle-matched local-search floor without implying that retrieval-free agents have no value.
PINNeval evaluates PINN components inside a competitive surrounding pipeline
PINNeval: A Comprehensive Evaluation Standard for Physics-Informed Neural Networks
PINNeval targets a concrete comparison failure: many existing PINN benchmarks study one pipeline component in isolation while leaving the other components at minimal defaults. The paper argues that conclusions from those settings may not transfer to competitive pipelines.
It separates a PINN training pipeline into domain-importance adaptation, loss balancing, neural-network architecture, and optimization scheme. For a component comparison, the protocol requires competitive implementations of the other three, and asks authors to report metrics, cost, precision, robustness, reproducibility information, and structured setup documentation. PINNpipe supplies a JAX implementation with 18 literature methods and nine benchmark problems.
The case studies show why the condition matters. In the nine-benchmark domain-importance study, uniform resampling (Random-R) had the lowest median error on seven benchmarks; LAS improved on it on Wave and Heat, so the study did not demonstrate a general adaptive-sampling advantage. In loss balancing, GradNorm led on five of nine benchmarks, but fixed weights led on three and SoftAdapt on one. Thus neither component had a universal winner under its reported surrounding setup.
Architecture rankings also depended on problem and model size: PirateNet won only 3 of 24 benchmark–size combinations and took roughly 1.8–2 times MLP runtime, rather than consistently improving accuracy. In the optimizer comparison, the reported gains were attributed to adding SSBroyden2—not to float64 as an independent improvement—although float64 was required for that stage.
Transferable operation: design the ablation around a single target component, keep the other three competitive and explicit, and attach error to cost, precision, robustness, and configuration details. Treat the result as conditional on the reference pipeline, not as a component ranking detached from context.
The boundary is important. The empirical study is restricted to forward PINNs; its suite is representative of the reexamined studies, not all PDE settings, and excludes complex geometries and other challenging problem classes. The reference instantiation must also be updated as competitive practice changes. Finally, the SSBroyden2 result does not establish large-network feasibility: the implementation stores a dense inverse-Hessian approximation with quadratic memory cost.
Use this paper to turn your next PINN ablation into a conditional comparison: choose one target component, strengthen the other three, and record the cost and configuration needed to reproduce the result.
Cancellation Alone Did Not Restore Service in the Tested vLLM Overload Runs
Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine
The measurement change
The paper asks whether propagating cancellation is enough when clients retry timed-out LLM calls. Its method follows each client attempt to the engine execution it creates, rather than treating the client call as the only unit of work. It tests vLLM under overload on three open models and adds scheduler traces for one model; the evaluation considers both service recovery and calls delivered to clients. This makes internal work visible that client-level request metrics can miss.
What the traces show
With retries enabled, cancellation alone did not restore service in any reported run, and only 12–38% of calls that arrived after the overload were delivered. In the traces, retries that vLLM ran typically started prefill less than half a second before their own timeout. The authors interpret this timing as consistent with a queue-refill loop: a retry can become new engine work shortly before it gives up, so cancelling one attempt does not by itself prevent later retry attempts from adding work.
Executions later aborted had consumed 79–85% of the computed prompt tokens. This does not mean cancellation failed to abort them: scheduler steps already issued could still complete. Separately, vLLM’s prompt-token counter missed a third to half of all computed prompt tokens, because work from executions aborted before their first output token was omitted from that counter.
Control comparison and research lesson
The controls produced different client outcomes. A retry budget restored service by shedding retries but left many overload calls undelivered. Request coalescing delivered every call from the overload onward, some late, unless cancellation had already aborted the running execution. For a researcher, the transferable operation is to instrument the path from attempt to execution to delivered outcome, then compare overload controls on both recovery and client-visible delivery rather than on cancellation latency or a single engine counter.
The evidence is narrow: it comes from one vLLM version on one GPU type, synthetic single-turn prompts, an emulated open-loop client, and one deadline and retry policy. The deferred-cancellation result also needs a narrow reading: in those runs, the proxy cancelled no execution, so deferral acted as coalescing without cancellation. These conditions leave behavior in other serving stacks, deployments, workloads, and client policies untested.
Read it to learn how to trace a retry into the engine execution it creates and evaluate overload controls against both service recovery and delivered calls.
ADB makes authorization overreach measurable with paired compliant and unauthorized routes.
Authorization Drift in LLM Agents: Evaluating Boundary Compliance in Multi-Step Tool Use
What changed
AgentDAM measures unnecessary disclosure of designated sensitive facts during web-agent task execution, whereas ADB measures annotated authorization violations across data, tool, scope, and time boundaries in executable tasks. ADB makes overreach measurable by pairing a feasible compliant route with a useful but unnecessary unauthorized alternative. ADB contains 648 samples across seven domains and tests four boundary dimensions: data, tool, scope, and time.
SERS validates executable environments, compliant-path reachability, and shortcut triggerability against actual agent behavior rather than relying only on LLM self-critique. In SERS, an automated gate accepts a sample only if at least one probe rollout covers the annotated required calls and at least two of the eight rollouts trigger a designed drift point. The method change to carry forward is operational: make each task contain a viable authorized route and a useful unauthorized shortcut, then reject items that fail either rollout check.
What it finds
Under normal bare tool-calling evaluation, Qwen models and Claude Sonnet 4.6 attained TSR ≥0.90 with DR ≈0.43–0.61. Across 644 analyzed tasks, 555 tasks (86.18%) had at least one clean-success trajectory. Over-collection of data is the most common measured drift dimension, motivating attention to early information-gathering decisions. In the evaluated scaffold configurations, lower drift often accompanies a collapse in compliant task competence, so drift alone would make the apparent improvement misleading.
Boundary conditions
ADB is therefore an opportunity-conditioned stress test, not an estimate of drift prevalence in ordinary deployments. Drift detection matches tool names and argument subsets against predefined annotations; unanticipated unauthorized paths may be missed, potentially underestimating drift. An independent audit found feasibility and success-criteria issues in 6 of 20 samples; the audit used one annotator and a small sample, so it provides limited validation of benchmark quality.
For a new tool-use evaluation, copy the paired-route structure rather than treating task success as sufficient evidence of authorization compliance. Report drift and clean completion as separate axes, then test whether a mitigation lowers boundary crossings without suppressing the authorized route. The concrete question is: when a compliant path remains feasible, does the intervention change the agent’s choice, or merely make completion harder?
Read this to learn a concrete way to test tool-use authorization: validate both an achievable compliant path and an attractive unauthorized shortcut, then interpret drift together with clean task completion.
Section 1, paragraph 2; Section 3.2, Stage 3 – Execution-reflective quality control; Section 4.2, paragraph 1; Table 1; Appendix E.5.2, paragraph 2; Tables 16–17; Section 5.2, paragraph 1; Table 1; Section 5.3, paragraph 1; Appendix E.8, Table 22; Section 7, Limitations; Appendix C.4, Table 9 and accompanying paragraph · Section 3.2, Task Design and Data Generation (S3.SS2.p1)
Randomized checkpoints separate local audit effects from agent-policy value
Knowing When to Intervene: Causal Evaluation of AI Agents
What changed
Static benchmarks can report whether an agent succeeds, but their static nature does not reveal the value of auditing interventions at deployment time. Dynamic-treatment work already provides recursive SNMM g-estimation and an off-policy value estimator; this paper applies related causal machinery to agent trajectories, where the question is when an intervention changes the terminal outcome. It therefore studies assigned interventions, not merely whether an intervention is associated with a successful final trace.
Method
At each pre-specified eligible point, the authors randomize intervention or control conditional on the observed pre-assignment history, then let the agent continue; in these experiments, assignment probability is one-half. Because assignments recur within trajectories, the same collection design supports effects for history-defined groups and evaluation of history-dependent policies. The framework separates three targets: a current-decision effect with future assignments still randomized, a current-decision effect when later interventions are absent, and the gain from a selective history-dependent policy versus never intervening. The randomized-continuation target uses cross-fitted AIPW with known propensities; the other targets use structural nested mean models and g-estimation. Identification still requires consistency, sequential exchangeability, positivity, and no interference; SNMM-based targets additionally depend on correct blip-effect-model specification.
Evidence and boundary
Across tau-bench Retail, ToolSandbox, and ImpossibleBench, the authors report that intervention effects vary substantially across agent actions and states. In ToolSandbox, reviewing one “datetime info to timestamp” tool had an estimated +6.9-point effect [1.2, 12.6], while the broader Time and calculations capability estimate was +1.8 points [-0.2, 3.8], illustrating why aggregate capability scores can hide candidate audit targets. In an ImpossibleBench subset, adversarial feedback increased detected violations by 9.4 percentage points [2.9, 15.9] at the final eligible checkpoint, versus 3.3 points [0.6, 6.1] after the first failed attempt. These estimates are prioritization evidence, not confirmation that an intervention policy will improve deployment outcomes.
The main statistical boundary is that SNMM confidence intervals have valid coverage only when the semiparametric model is well specified; under misspecification, fitted coefficients may be weighted projections. The reported intervals are pointwise and do not adjust for multiple comparisons. Empirical evidence comes from relatively small models and established benchmarks that may be saturated by frontier systems, while history representations are hand-designed.
For a deployment study, randomize audits at logged checkpoints, preserve pre-assignment history, and report both a local effect and the value of the complete selective policy. The transferable question is: for which history does intervening now improve the terminal outcome, and does selecting those histories outperform never intervening?
Read this paper to learn how to turn deployment-time audit choices into estimable causal questions, while keeping immediate intervention effects distinct from the value of a history-dependent intervention policy.