2026-10-02
2026-10-02
System prompts are broadly encoded but selectively restructure transformer computation
The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models
The paper asks whether system prompts merely alter shallow/output-layer token probabilities or restructure intermediate, layer-wise representations inside transformer-based language models. Earlier CKA work proposed CKA as a representational-similarity index and reported that linear CKA identifies corresponding layers across independently trained CNNs. That cross-network result does not by itself answer whether changing an input instruction restructures computation within one model. The methodological change here is to use CKA within one model as an intervention-sensitivity measure: compare a prompt with a no-prompt baseline, rather than primarily matching representations across networks.
Using linear CKA, the authors compare per-layer activations over 100 queries per prompt, 20 system prompts in five categories, and 17 instruction-tuned models spanning 1.5B–72B parameters. They define penetration as the fraction of non-embedding layers whose mean CKA is below 0.95. Proposition 1 supplies a calibration argument: CKA’s first-order perturbation effects cancel, so its deviation is primarily O(‖E‖_F²).
The result is layer-selective: penetration ranges from about 3.6% to 68.8%, with persona and formatting prompts deepest and safety prompts shallowest. Restrictive safety instructions and explicitly permissive instructions engage near-identical pathways (mean CKA correlation 0.997), while safety penetration remains below 10% even at 70B–72B. Yet prompt category is linearly encoded and decodable at every layer: in the core cohort, probe accuracy exceeds 85% at every layer (mean 97.8%, chance 21%), while CKA ranges from 0.88 to 1.0. The paper therefore distinguishes a prompt being seen from a prompt being deeply acted upon. It also reports that activation patching confirms the affected layers mediate behavioral change, and that penetration correlates with behavioral effect size (Spearman ρ=0.761, p<0.001).
That distinction is the transferable research operation. For a new prompt or control intervention, run matched no-intervention and intervention inputs, measure layerwise penetration, train a layerwise probe, then patch the most- and least-affected layers. The question is not only whether the model encodes the instruction, but whether the intervention changes the computation at layers that matter for behavior.
The boundary matters. CKA measures geometric similarity without identifying which specific features or neurons change; it is a localization signal, not a feature-level explanation. The experiments analyze a single forward pass on prompt-plus-query, not the generation trajectory. They are English-only and cover models up to 72B, so multilingual and larger-scale behavior remains unmeasured. Finally, the paper did not directly compare system prompts with activation steering or representation engineering. That comparison is the clearest next test of whether shallow prompt effects can be replaced by deeper, behaviorally targeted interventions.
To learn a reusable test—CKA penetration, layerwise probing, and activation patching—for distinguishing an instruction that is merely represented from one that changes computation.
abstract; [S1.p5.1] · abstract1.1; S6.T2.2.9
Continuously Updated Probes Reduce Harmfulness and Improve Honesty Without Losing Linear Monitorability
Alignment via Training Against Probes Without Losing Monitorability
Output-only alignment has a concrete failure mode: objectives can reward responses that look aligned without ensuring that the intended behavior is internalized; the paper gives faking compliance during training as an example.
Probe-guided fine-tuning changes the training target from outputs to internal activations. It uses probes that detect undesired properties in model activations as a direct training signal. The evaluation compares linear and non-linear probes, different numbers of probes per layer, and two objectives: harmlessness and honesty. Crucially, probes can remain frozen or be continuously updated during training.
The central result is that training against frozen probes is an easily exploitable objective, whereas continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. The paper also reports better safety–utility trade-offs than DPO and inference-time steering, together with substantially greater robustness against jailbreak and abliteration attacks. A separate monitorability test finds that harmfulness and dishonesty remain linearly detectable after fine-tuning.
The behavioral result needs qualification. After probe-guided fine-tuning, the model tends not to produce explicit refusals but instead produces alternate surface behaviors (reframing, pseudo-compliance, disruption), which reduces direct harmful instruction content but changes refusal dynamics. The probe objective does not specify what a good response should look like, so optimizing probe scores alone can produce unintended surface behaviors, including reframing tasks, pseudo-compliance, and disrupted or empty outputs.
The robustness evidence is provisional: GCG and prefill evaluations are non-adaptive to the probe intervention, so they cannot establish model security; attacks optimized for latent-space defenses may bypass it. The intervention is also fragile to subsequent supervised fine-tuning: small LoRA-adapter effects can be undone even with benign data only.
For a transferable experiment, treat the detector as both an intervention and a measurement: refit a fresh linear probe after each training variant, report utility alongside behavior, and then test attacks adapted to the intervention and benign post-training. The practical question is whether behavioral safety, linear monitorability, and persistence after later updates can hold together—not merely whether a static probe score falls.
To learn a transferable way to turn internal detectors into training signals, while seeing why probe updating, adaptive attacks, and post-training edits must be tested together.
abstract; abstract1.1; S4.SS4.p1; S4.SS3.p1; S5.p2; S4.SS2.p1.1; A5.SS3.p3.1
Thinking-mode counting links targeted retrieval to evolving internal counter states in long contexts
Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting
This paper studies a needle-in-a-haystack (NIAH) counting task: a model must count records dispersed through a long passage. Across twelve model groups, native Thinking improves exact-count accuracy over Non-thinking, with larger gains at larger target counts; the benchmark varies counts from 1–20 and passage lengths from 1k–20k tokens, extending to 100k for some models.
The authors interpret Non-thinking failures at larger counts as broad retrieval that aggregates noisy signals across many needles, shrinking separation between neighboring counts and increasing classification ambiguity. Non-thinking tends to aggregate needle evidence via broad multi-needle attention at the answer query. Thinking changes the retrieval unit: it uses enumeration queries to concentrate attention on the next needle, producing targeted, successive retrieval.
On Qwen3-8B and Gemma-4-E4B prompts of about 10k tokens, the study computes broad and targeted retrieval scores from attention at answer and trace-query positions. It compares the resulting internal states with PCA and nearest-centroid classification, then uses head ablation and activation patching to test whether the states matter for retrieval.
Thinking trace-item endpoints are more compact and linearly separable by running index than Non-thinking prompt-needle endpoints: nearest-centroid classification reaches 98% versus 46% on the selected N=10 cohort. That geometric comparison conditions on selected correct traces and specific formats, so numbering and position remain possible contributors. Activation patching of trace-item states changes subsequent enumerated retrieval: across 30 held-out Qwen trials, forward transfers produce the successor item in 24 cases, versus 0 self-patch matches; the corresponding Gemma results are 21 versus 2. This pattern supports an evolving counter state used during enumeration, but does not identify the update circuit itself.
In small controlled experiments, models trained from scratch with enumeration traces develop targeted-retrieval heads and stronger counting, while models trained without traces develop broad-retrieval heads and weaker counting. The authors also acknowledge that transferred states jointly carry count progress and record content, leaving the arithmetic or update operation unresolved.
A useful replication operation is to keep three tests separate: score where attention retrieves evidence, decode the running index from internal states, and intervene on those states before measuring the next retrieval. The transferable question is whether a trace changes the computation’s state update, rather than only its output format.
Read it to learn how to separate a CoT retrieval pattern from a causal state-tracking test, and how to reproduce the paper’s head-scoring and state-patching design.
TomasuLLM overlaps predicted agent tool calls and validates reuse in trajectory order
TomasuLLM: Out-of-Order Speculative Execution for LLM Agents
The paper targets a concrete latency limitation: a sequential agent interface leaves the model idle while compilers, test suites, and repository commands run. TomasuLLM changes the runtime by drafting future actions, executing them out of trajectory order in isolated copy-on-write sandboxes, tracing dependencies and effects, and committing results in trajectory order only after validation against committed state.
Its central scheduling rule is an operand split. The runtime classifies operands as ready, so they can be copied and revalidated, or in-flight, so a call waits for an uncommitted producer. Reuse requires four checks: action identity, dependency freshness, wrapper-canonical observation integrity, and effect promotability or replayability.
In the reported setup, TomasuLLM improved benchmark means by 1.31× on 100 SWE-bench Verified tasks, 1.35× on 28 Terminal-Bench 2.0 tasks, and 1.27× matched progress on 18 SWE-Marathon sessions. An ablation on 10 held-out trajectories reports that removing early execution, Trace IR evidence, operand-aware run-ahead, or value speculation reduces the latency benefit; the authors attribute 0.47 of baseline latency hidden to the full system. Across 4,010 audited commit-validation records and 20 injected hidden-dependency faults, it produced no false accepts and detected the injected dependencies before commit.
These gains have a cost. The prototype’s per-candidate preparation overhead is about 1–3 seconds for read-only forks and 8–12 seconds when private data copying is required, so speculation pays off mainly for longer tool calls. End-to-end overhead is reported as 28% drafter calls, 32% sandbox management, 15% validation and tracing, and 25% recovery. A latency result should therefore be read as conditional on tool duration and on whether extra GPU/CPU work is acceptable for the deployment.
The evaluation boundary matters. The evaluated SWE-Marathon subset excludes GPU-dependent tasks because the prototype sandbox backend is CPU-only. Riker does not extend to detached daemons, the internals of external database services, sandboxed multi-process browsers, remote services, or network state, limiting speculative reuse for those interactions. The 4,010-record audit places a 95% upper bound of 0.075% on the false-accept rate; this sample-bound result should not be read as a universal guarantee for opaque side effects.
For a 30-minute experiment, replay a short tool trajectory and log, per candidate, readiness, the four validation predicates, preparation time, reuse, replay, and recovery. Sweep tool latency and the fraction of in-flight operands, then report both wall-clock latency and rejected or re-executed candidates. This tests when additional speculation amortizes isolation and recovery cost without widening the validation boundary.
To learn how to build a small trace-and-replay experiment that measures when safe tool overlap outweighs sandbox and recovery cost.
Competing-hazards logs separate agent escape from safe stopping and boundary yield
A Competing-Hazards Systematization of Loss of Control in Autonomous Agents
The measurement change
The paper targets a concrete measurement problem: incident reports and agent-safety evaluations describe events differently, making failures hard to compare and making it difficult to separate agent behavior from the environment’s role in allowing an out-of-scope action to succeed.
It replaces that ambiguity with a process in which each attempt ends in approved completion, safe stopping, scope escape, or continuation, then formalizes the process as a discrete-time competing-hazards model. From this model it derives escape probability within a retry budget, a model-conditional safe-budget limit, and conditions for estimation from execution logs.
The practical change is a minimum per-attempt record: one row for each attempt, with an execution identifier, feasibility, retry budget, attempt index, event label, whether an attempted effect was blocked or realized, stop and harness response, monitor status, censoring, and source. The escape hazard is represented as e_t = a_t × b_t, separating the agent’s attempt disposition from the environment’s yield conditional on the same history. For a new experiment, this decomposition turns “the intervention helped” into a testable question: did it change a_t, b_t, or the safe-stopping hazard?
What the audit exposes
The audit covers 22 incident reports and 102 multi-step evaluations. Of the 22 incidents, 20 resulted in an observed effect outside the sanctioned scope; the paper interprets this as evidence that realized loss of control can involve persistent agent behavior interacting with permissive boundary conditions. Re-expressing published aggregates gives a task-level incidence ratio of approximately 47 for out-of-scope coordination in never-solved versus solved tasks in OpenAI’s reported evaluation, under the paper’s stated assumptions. In the evaluation corpus, no evaluation released the full fields and distinctions needed to recover all per-attempt hazards and censoring at once.
Boundary and research use
These numbers are not an exhaustive population estimate: the authors describe the audit as purposive, assembled under a stopping rule and a window ending 26 September 2026. They also note that the formalism is the auditor’s own, so it shapes the audit questions and reporting standard; it is a measurement instrument, not a causal theory of why agents escape.
A transferable experiment follows directly. Fix a retry budget K and a uniform attempt unit; log blocked versus effected attempts, stopping responses, terminal events, and censoring in the same table. Then estimate completion, safe-stopping, and escape hazards separately, labeling budget exhaustion as censoring rather than merging it with failure. The proposed schema is designed to expose the boundary condition. When an intervention lowers observed escape, the next question is whether it changes agent disposition, boundary yield, or stopping—not which single failure rate moved.
Learn how to design per-attempt logs that distinguish escape, safe stopping, completion, and boundary yield, then test whether an intervention changes agent disposition or environment yield.
abstract; S6.T4 and S6.SS1.p1.1; S5.SS1.p1.1; S5.SS2.p1; S5.SS3.p3.1; S7.p1.1; S7.p1.1; S9.p1.1