2026-08-29
2026-08-29
Probability audits on target-document prefixes cannot rule out trigger-word extraction
Memorization Is Not Extraction: Tight Differential-Privacy Bounds and Audit Blind Spots
The researchers subjected the same secret to be protected to two different checks: an auditor looks only at the probabilities the model assigns to subsequent tokens after being given prefixes of the target text, or at losses computed from those probabilities; an attacker, by contrast, may switch to another prompt to search for the secret. The paper finds that even if the first check shows that the target prefix cannot reproduce the text, it cannot rule out complete recovery through the latter prompt path. This changes what it means for an “audit to pass”: it first covers only the prompt channels that were actually checked.
Prior work has already discussed that distinguishability and extraction risk are not equivalent, but the resource constraints and attack games are also different. This paper restricts “local scoring” to reading only the probabilities the model assigns to subsequent tokens after each given prefix of the target text; the attacker may query the model adaptively but does not know the secret, and can ultimately submit only a length-limited list of candidate answers.
The authors use paired training datasets for a counterfactual comparison. In the first training branch, the data provider inserts text of the form “trigger word followed by the secret” only when it reads a target record consisting of the target prefix and the secret. In the second branch, when the provider reads the same target prefix paired with another independent secret, it does not insert this trigger text. Neither branch directly uses the target record for training. Thus, if only the first branch can reproduce the secret from the trigger word, the difference can be attributed to whether the provider conditionally inserted the trigger text based on the record it read; the attack procedure itself cannot use prior knowledge of the secret to construct prompts, decode outputs, or construct its candidate list.
In standard fine-tuning of Pythia-1.4B and Qwen2.5-1.5B, across all tested lengths, the fraction of secrets extracted from the trigger word was at least 0.98, while extraction from the target-document prefix was 0; a control group that directly inserted the target document into the training set could reproduce it from the original prefix, showing that the test itself had not failed. Record-level counterfactual experiments likewise produced a pair of results in which the trigger word was extractable in the branch containing the record and not extractable in the branch without it. The conclusion supported is that extraction tests on the original-text prefix and local probability scoring miss trigger channels using a different prefix.
The boundary conditions matter as well. The local scoring signal in ordinarily trained models merely decays rather than becoming strictly zero. Reference-model scoring uses a reference model, whereas LiRA calibrates membership inference from multiple shadow models trained with the target included or excluded. In the paper’s SGD experiments, these audits mainly reflect fitting of the completed trigger record and the secret tokens it contains, not direct training on the target record itself. Strict invisibility to any method that scores only along the target prefix comes from the explicit construction and the gated release at the service layer. A transferable practice is to evaluate original-prefix reproduction, local-loss auditing, and extraction with a different-prefix prompt separately. For extraction that submits a candidate list, one should also report the probability that the secret falls in the most likely m answers an attacker would provide without using the model, based only on pre-existing side information and the remaining corpus uncertainty (the top-m prior hit rate), to distinguish gains due to the model from what the attacker could already guess.
See Figures 10–11: two experiments with the same two model families respectively show trigger-word versus original-prefix extraction and the record-level counterfactual comparison, directly allowing verification of the difference introduced by changing the prompt path.
第 8.2 节“The trigger channel at scale”“The record itself, not just the channel”；图 10–11（PDF 第 11–12 页） · Theorem 6.1（第 6 节；PDF 第 7 页） · 第 2.3 节；Theorem 2–3
Under the same pruning procedure, adapter updates yield smaller behavioral circuits
Circuit Condensation: Post-Training that Concentrates a Behavior’s Causal Circuit
The researchers modify the procedure for finding behavioral circuits in a frozen model into a post-training process: they progressively remove connections between internal components while requiring the model to retain a specified behavior. This changes one research judgment: a large discovered circuit may reflect either pre-existing wiring or the fact that the behavior has not yet been trained into a concentrated form. The paper uses a control that changes only whether adapters can be updated to test whether the reduction comes from reshaping weights or from the search strategy.
The authors treat connections between components as candidate circuit edges. When an edge is disabled, its internal signal is replaced with the signal produced by a paired prompt for which the answer should differ. In each round, edges receive causal attribution scores based on the effects of these replacements on the specified behavior, and low-scoring edges are removed first. The method then trains only low-rank adapters attached to the frozen base model—a small set of trainable parameters—so that the model fits the original model’s next-token distribution through the remaining connections. The update-enabled group trains these adapters, whereas the frozen group leaves them unchanged. On the validation set, both the masked circuit and the adapter-equipped model with all gates open must retain the target behavior; additionally, on independent probes of non-target capabilities, the full-gate adapter model is required to keep the perplexity ratio with adapters on versus off below a threshold. Upon failure, the process reverts to the previous state. Earlier frozen discovery methods searched only within existing weights; this work adds a closed loop of “prune–distill–validate–rollback.”
The key comparison holds fixed the ranking, edge-removal schedule, validation, and rollback, allowing adapter updates in one group while keeping weights frozen in the other. Across four behaviors and eight models, the frozen group retained more edges in 29 of 32 task–model combinations (taking the median over three seeds for each combination); across all 96 seed runs, the geometric mean of the frozen-to-update-enabled edge-count ratio was 7.4×. This supports the narrower conclusion that, under this controller, adapter updates allow the specified behavior to be concentrated in a smaller set of connections. It does not establish that the globally minimal circuit has been found.
This circuit primarily describes the post-trained model, and “edge count” is meaningful only under this test defined by counterfactual replacement of internal signals; the adapters still span the entire model, and the paper does not report inference acceleration. A learnable research practice is to place frozen and update-enabled groups under exactly the same search controller, changing only whether weights are mutable; the resulting reduction can then be used for subset enumeration or pairwise edge ablations to test whether the smaller graph is genuinely easier to audit.
See Table 1 to directly verify the edge-count difference between the frozen control and update-enabled group under the same controller, and to assess the scope of the evidence for “weight-update-driven reduction.”
§3.1–§3.2、Algorithm 1；§5.2；表1 · §2–§4、式(3)、§4.2
With permutation randomness shared, parallel analysis still changes under equivalent hidden coordinates
Representation Measurements Under Function-Preserving Reparameterizations
The researchers first compensate for orthogonal rotations in adjacent language-model weights so that the model’s input–output behavior remains unchanged, then test whether “parallel analysis” of hidden representations changes with the coordinate system. They find that, even when the covariance spectrum of contextual representations remains unchanged—that is, when the amount of variance carried by each mutually orthogonal direction of variation is unchanged—the number of components selected by parallel analysis can still change substantially. Thus, if this number or a threshold decision derived from it is interpreted as a property of the model function or of the retrieval evidence itself, it must first pass an invariance test of this kind.
Parallel analysis compares observed covariance eigenvalues with reference eigenvalues obtained after independently permuting each column of the representations, to decide how many directions to retain. Earlier permutation parallel analysis likewise depends on this coordinate-wise operation; this paper adds function-preserving rotation as a controlled negative control. Rotation does not change the eigenvalues of the original covariance, but it does change how much variance each individual coordinate carries; the column-wise permutation preserves precisely these coordinate marginals, so the reference distribution changes with the coordinate system.
The strongest validation removes coordinate-wise standardization, retaining only centering, and uses the same permutation randomness before and after rotation. Across 1,200 contexts from four RMSNorm models and three retrieval domains, the maximum relative error between the observed spectra before and after rotation was only 1.26×10^-6, yet the component count changed in 1,141 contexts. This control localizes the change to the permutation reference, rather than to the observed spectrum or chance differences between two Monte Carlo draws. It does not show that every rotation, task, or larger model will exhibit an effect of the same magnitude; the experiment uses a particular structured rotation.
A transferable research practice is: if a metric claims to describe a function or evidence, first specify the allowed function-preserving transformations; then, on the original representations and the representations after a compensated transformation, share the random reference and separately record the observed spectrum, reference threshold, and final selection. If the goal genuinely requires invariance to a common orthogonal rotation, the alternative score proposed here can use pairwise inner products among contextual representations and between contextual and answer representations—that is, their relative geometry—rather than the marginals of individual hidden coordinates; this information is invariant under a common rotation.
See Table 4: it places reruns with independent permutation seeds alongside the centering-only control with shared randomness, providing the most direct verification that the component-count change comes from the reference distribution.
§3.1，Lemma 1 (Expected Permutation Covariance) · §4.3 Randomness and Preprocessing Checks；Table 4 · §1.2；§2.1 Theorem 2.1；§3.1.2
After inversely adjusting learning rate with β, nearly identical DPO losses can still correspond to nearly threefold KL
Disentangling Optimization Scale from Preference Scale in DPO
DPO (Direct Preference Optimization) is the training objective used here for preference fine-tuning: it updates a model from pairs of a “preferred response” and a “dispreferred response,” with β setting the scale at which such preference differences enter the objective. The researchers jointly adjust β and the learning rate to test what training curves actually report about what a policy has learned. They find that even when the DPO validation losses from two training runs nearly overlap, the models’ deviation from the reference model and downstream performance can still differ substantially. Therefore, DPO loss calculated using each run’s own β cannot by itself be used to select a β configuration.
The gradient of standard DPO contains β: β both changes the shape of the preference loss as response differences vary and directly amplifies or shrinks each SGD update through the gradient; adaptive optimizers may partly offset the latter effect. This paper’s addition is to handle these two roles separately. The authors use centered softplus: they divide the same DPO objective by β and shift its numerical baseline so that its initial training value is zero. For any fixed positive β, this is only an affine rescaling of the objective, leaving its optimum unchanged; but in finite-step training, the gradient is no longer directly amplified or shrunk with β. With SGD, the update scale matches standard DPO only if the learning rate for this rescaled objective is set to the standard-DPO learning rate multiplied by β. This is a tool for rescaling and diagnosis, not evidence that it produces a better optimum.
The main evidence comes from LoRA preference fine-tuning of Qwen3-4B. On UltraFeedback, the authors compare two SGD training settings: β = 0.01 with a learning rate of 0.001, and β = 0.02 with the learning rate halved. Both run for six epochs with four random seeds each. Reference KL quantifies how much the trained model’s generation distribution deviates from that of the reference model, and thus tests whether models with similar losses are in fact still different policies. From the first epoch onward, the validation losses of the two groups differ by at most 0.0051, but their final reference KL per token is 0.814 and 0.278, respectively—nearly a threefold difference; their length-controlled AlpacaEval win rates also differ by 13.76 percentage points. The downstream evaluation here uses a model judge, and such loss overlap does not occur on every dataset, but it is sufficient to show that loss curves suppress the effective margin scale associated with β.
Editorial recommendation: when diagnosing across β values, do not select configurations using only DPO loss computed under each group’s own β; as in this paper, also inspect held-out pair NLL at a fixed scale (β = 1 in the paper), the policy’s log-probability difference between preferred and dispreferred responses, the margin after subtracting the reference model, and KL relative to the reference model. Whether to include downstream evaluation and how to set the β and learning-rate grid still need to be validated for the task. In the paired case studied here, centered softplus separates these trajectories, but it has not yet been shown to be a general quality scale across arbitrary β settings.
See Table H2 in Appendix H to directly verify the key control: after the learning rate is inversely adjusted with β, validation DPO losses are nearly identical while KL separates clearly.
§3.2，式(6)；§4.1–4.3，式(10)–(14)。 · §3.4；附录 H 表 H2；附录 I 表 I1。 · §4，式(4)–(7)及“What does the DPO update do?”；附录 A.4，式(21)–(22)。 · §3.3，式(3)–(4)。
Under the same artificial segmentation, retaining bootstrap value improves success by 20 percentage points over absorbing termination
Offline goal-conditioned reinforcement learning samples future goals from recorded trajectories and trains value functions with multi-step returns. The researchers artificially cut trajectories at non-terminal positions known to continue, then compare two treatments: treating the cut as a termination, or, after stopping reward accumulation before the cut, continuing to use the value of the stored successor state to complete the target. The results show that the semantics of segmentation metadata alone can substantially change the final ability to reach the goal; a data pipeline cannot directly treat administrative segmentation as reinforcement-learning termination.
Earlier work on time limits has already explained that truncations that are not environment terminations should retain value bootstrapping. This paper does not introduce a new Bellman rule; its addition is a controlled stress test. The original trajectory identifiers are still used for future-goal sampling, and transitions, goal sampling, optimization, and evaluation are held fixed; the artificial cuts affect only where multi-step returns stop. In the CVT condition, after a cut the method bootstraps from the stored successor state, that is, it uses the predicted value of that state to complete the return; the naive condition treats the same cut as an absorbing termination. Actual goal completion is not bootstrapped in either group.
The main experiment covers five PointMaze tasks, inserting 35,000 cuts for each realization and comparing three optimization seeds and three independently seeded artificial segmentation realizations. After 100,000 training steps, CVT has a mean success rate of 39.1%, versus 19.1% when the same cuts are treated as terminations, a paired gap of 20.0 percentage points; unsegmented data reaches 50.5%. Because the original trajectories and future-goal distribution are not rewritten, this gap directly points to whether bootstrap value is retained in multi-step targets. CVT still does not recover unsegmented performance, showing that correct boundary semantics do not guarantee strict immunity to segmentation under finite training. [Paper §3.1–3.2, Table 1, §5.1]
A reusable practice is to store “original trajectory boundaries used for sampling” and “boundaries used to truncate multi-step returns” in separate fields. For artificial cuts with known continuous successor states, hold transitions, goal sampling, and the random-number stream fixed, switching only the bootstrap mask; then reconstruct the two types of targets with the same value function and check whether the difference is as expected. This check cannot be applied across genuine resets, missing-data discontinuities, environment terminations, or goal completion: bootstrapping requires an observed successor known to belong to the same continuing process and a boundary not defined as terminal; goal completion and genuine environment termination remain noncontinuing under their Bellman/task semantics.
See Table 1 to directly verify the paired success-rate difference for the same set of artificial cuts under bootstrapping versus absorbing-termination semantics, as well as the boundary that CVT does not fully recover unsegmented performance.
§3.1–§3.2；§4.1；表1；§5.1。 · §3，式(5)–(6)；§4–§5。
Prepared retrospectively from the 2026-08-29 candidate papers; verified on 2026-09-09.