2026-09-14
2026-09-14
Later HuBERT and wav2vec 2.0 layers retain word-identity signal after local phoneme residualization
Do speech foundation models really learn words?
Why the probe is ambiguous
A successful word probe does not by itself show a specialized word representation: discrimination could come from encoding phonemic form rather than form-independent word identity or syntactic/semantic properties. [arxiv:2609.10434v1, Abstract]
Experiment
Huo and Dunbar ran frame-level linear word-identity probes on LibriSpeech dev-clean representations from the final convolutional layer and all 12 transformer layers of HuBERT-base and wav2vec 2.0-base. [arxiv:2609.10434v1, S4.p2; Figure 1 (S3.F1)] Before probing, they standardized embeddings and fit ridge regressions from one-hot phoneme, diphone, or triphone factors to the embeddings, then subtracted the predictions; frames where the factor fully contained the word were excluded. [arxiv:2609.10434v1, S4.p1; Table 1 (S3.T1)] Five-fold linear probes then tested whether word identity remained recoverable, while phoneme probes checked whether the nuisance information had actually been reduced. [arxiv:2609.10434v1, S4.p1; Table 1 (S3.T1)]
What remained
Residualization substantially reduced the ability of a linear probe to classify individual phonemes. [arxiv:2609.10434v1, S4.p1; Table 1 (S3.T1)] Yet residualizing out phonemes did not change the overall word-prediction pattern: the convolutional layer contained little word-identity signal, while transformer layers peaked at somewhat more than 90% accuracy in HuBERT layers 9–10 and wav2vec 2.0 layers 7–8. [arxiv:2609.10434v1, S4.p2; Figure 1 (S3.F1)] Removing triphone information reduced word classification substantially overall, but some middle and later layers remained well above the paper’s simple baselines. [arxiv:2609.10434v1, S4.p3] The authors interpret this residual performance as evidence that word classification is not merely a reflection of short local phoneme sequences.
Downstream check
The probe result also transferred to an unsupervised word-discovery pipeline: on HuBERT layer 9, phoneme residualization improved NED, token F1, and R-value for boundary detection followed by k-means clustering. [arxiv:2609.10434v1, S5.p1-S5.p4; Table 2 (S5.T2)] This gain was specific to the tested residualization choice: diphone and triphone residualization degraded segmentation performance. [arxiv:2609.10434v1, S5.p5]
Boundary and transferable operation
The preprocessing is not incidental. Omitting standardization caused residualization to fail to remove phoneme-identifying cues, according to the ablation. [arxiv:2609.10434v1, S4.p1; Table 1 (S3.T1)] The main practical limitation is that the method needs aligned phonemic labels, which are not easily available for most tasks. [arxiv:2609.10434v1, S8.p1] The result should therefore be read as residual word signal under these tested local controls, not as a demonstration of context-independent lexical tokens. For a new representation audit, the reusable operation is concrete: name a plausible confounder, regress it out with an explicitly tested preprocessing pipeline, verify nuisance-probe reduction, and then re-evaluate both the target probe and a downstream task.
Read this for a reusable residual-probing design: regress out a named local-form confounder, verify that the confounder probe weakens, and then test whether the target representation and downstream behavior still hold.
Plain Text Outperformed POS and RST Token Enrichment on GCDC Coherence Classification
Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures
The limitation and the intervention
Large language models can produce text that is grammatically correct yet semantically incoherent, with contradictions or disrupted logical flow. This paper tests whether explicitly adding syntactic and rhetorical annotations helps predict such incoherence, rather than assuming that more linguistic structure will automatically improve a Transformer classifier.
The researchers compared three XLM-RoBERTa Longformer pipelines: plain text, POS enrichment, and RST enrichment. POS enrichment appends a part-of-speech token after each word. RST enrichment inserts special tokens for elementary discourse-unit boundaries, rhetorical relations, and nuclearity. The tokenizer and model embeddings were extended for the new symbols, with new embeddings initialized to the vocabulary mean. Models were trained for binary coherence classification with weighted binary cross-entropy and assessed using balanced accuracy and Brier Score Loss.
What the comparison showed
On GCDC, the Plain pipeline achieved the highest balanced accuracy, 72.4 ± 0.1%, averaged over five runs, and outperformed both enriched pipelines. The result does not mean POS information was uniformly useless: on Brier Score Loss, the POS pipeline performed best overall while Plain performed worst, indicating better-calibrated probabilities despite lower balanced accuracy.
The same ranking held in a constrained transfer test. Models trained on English GCDC were applied without further training to the Portuguese FakeTrueBR news subset; Plain reached 73.1 ± 0.8% balanced accuracy and exceeded the RST- and POS-enriched models. This is evidence about this backbone, token-insertion design, datasets, and zero-shot setting—not evidence that syntactic or discourse information cannot help coherence assessment in other representations.
The authors argue that flat token sequences may be structurally mismatched to hierarchical RST information because Transformer mappings can flatten that structure. Treat this as an author hypothesis, not the measured mechanism: the reported comparison establishes worse performance after enrichment, but does not directly isolate why it occurred. A further boundary matters for the disinformation application: coherent text can still be biased, misleading, or factually incorrect. In addition, the DMRST parser used for RST annotations supports only six languages.
For an annotation-based project, make plain text the required baseline and measure both decision quality and calibration. The transferable research question is: does the model use the proposed structure when it is encoded as tokens, or does the representation require an architectural mechanism that preserves its hierarchy?
Read this for a directly testable baseline: adding POS or RST labels as flat input tokens reduced balanced accuracy here, while POS improved probability calibration.
Abstract; Sections 3.1–3.3; Tables 5, 7, and 8; Discussion; Ethics Statements and Limitations
Belief-State Engine separates Bayesian state estimation from LLM action selection in partially observable planning
Belief-State Engine: Augmenting LLMs for Principled Planning Under Partial Observability
The design move
The paper identifies a specific failure mode in common LLM agents: they act as history-conditioned policies without an explicit belief over hidden state. Under ambiguous feedback, the agent may commit too early, update onto the wrong hypothesis, or let its policy drift as history grows (arxiv:2609.10036v1, Abstract). BSE moves state estimation outside the LLM: given a POMDP model with transition and observation models T and Z, it maintains posterior b_t with a Bayes filter, serializes only that posterior into the prompt, and withholds raw action-observation history from the policy (arxiv:2609.10036v1, Abstract). The LLM therefore receives a belief-to-action interface rather than a transcript to interpret.
What is actually guaranteed
The key theorem is conditional. If T and Z are known at run time, the Bayes update is used, the LLM policy is belief-measurable, and raw history is hidden, BSE-plus-LLM is a sound Markov policy on the induced belief MDP and inherits classical Bellman-optimality guarantees (arxiv:2609.10036v1, Section IV-G, Theorem 9). This is a composition result under the stated interface constraints; it is not a guarantee that those conditions hold when the model is unknown or the LLM sees raw history.
Evidence and boundary
In the canonical Tiger experiment (T=20, gamma=0.95, 40 paired seeds, gpt-4o at temperature 0.3), BSE’s reported mean discounted return was 3.06 (95% CI [-4.41, 8.24]), versus -12.00 ([-25.75, 1.75]) for both Reactive and NL-Tracker (arxiv:2609.10036v1, Section VII-B, Table I). The paper reports that Reactive and NL-Tracker also had identical aggregate outcomes—32/40 successful episodes, mean return -12.00, and zero listen actions—so the free-text belief did not improve this comparison (arxiv:2609.10036v1, Section VII-B, S7.SS2.p2 / Table I). On the attack-graph instance, BSE’s median JSD across belief-equivalent history pairs was 0, versus 0.043 for NL-Tracker, while absolute returns overlapped among methods (arxiv:2609.10036v1, Section VII-C, Table III, S7.SS3.p2). That result needs a strong qualification: the implemented transition kernel was the identity for every action, with exploit and patch dynamics left unimplemented (arxiv:2609.10036v1, Appendix C, A3.SS2.p3; Section VII-C, S7.SS3.p5). Thus the transferable research operation is to separate inference from action selection and test history invariance directly, but to validate it first in environments where state transitions are implemented.
The paper’s empirical scope is also narrow: budget constraints reduced the preregistered protocol to three of six baselines; main comparisons used N=40, ablations N=25, and reported runs used one LLM-sampling seed (arxiv:2609.10036v1, Section VI, S6.p2; Section VII-A, S7.SS1.p1). Finally, BSE requires a POMDP model at run time; when the model is unknown or misspecified, the stated guarantees do not hold (arxiv:2609.10036v1, Section VIII-A, S8.SS1.p1-p4).
To assess whether an explicit external Bayesian belief module plus an LLM provides conditional POMDP guarantees and measurable gains under partial observability, while checking the evaluation scope and attack-graph implementation boundary.
A2ABreak Builds a Unified FSM from A2A Protocol Prose for Adversarial Security Analysis
A2ABreak: Systematic Security Analysis of the A2A Protocol
A2ABreak’s comparison exposes a concrete limitation of direct, zero-shot analysis: a zero-shot LLM operating over the same A2A specification produced zero confirmed findings under the same adversarial verification procedure. The method change is to make the specification an intermediate, inspectable model rather than the only prompt context.
A2ABreak uses three stages: Stage A formalizes the natural-language specification into structured statements, Stage B constructs a unified finite-state machine, and Stage C discovers and validates protocol-level vulnerabilities from that machine. The pipeline constrains LLM assistance with a domain-specific schema, places human checkpoints at stage boundaries, and verifies candidate attacks against four conditions: grounding in the FSM, executability under full compliance, absence of normative prevention, and genuine absence of the security primitive from the specification. For a researcher, the transferable idea is to make each proposed attack traceable to an explicit protocol path before asking whether it is exploitable.
The resulting model contains 929 formalized statements, 37 states, and 76 transitions. The paper reports 11 new vulnerabilities, each exploitable by a specification-compliant adversary without requiring an implementation flaw. Examples include cross-client context injection when context identifiers lack ownership binding or access control, credential harvesting when identity is not propagated across delegation hops, and sensitive-task exposure through self-asserted AgentSkill claims without attestation.
The decisive evidence includes both discoveries and error accounting. After automated verification, independent experts validated 11 of 16 candidates, identified one duplicate, and overturned four false positives. A2ABreak achieved 73.3% precision and 84.6% F1 against independent expert review; the zero-shot baseline on the same specification produced zero confirmed findings. Thus, the reported result supports formal grounding as a useful condition for this evaluation, not a claim that unconstrained LLM review is generally ineffective.
The boundary is important: every vulnerability claim assumes that all normative “must,” “should,” and “may” requirements are satisfied, implementations are correct, and transport security is established. The analysis is therefore about protocol-level normative gaps under a fully authenticated, compliant adversary. No open-source A2A reference implementation was available for corroborating these findings with fuzzing, static analysis, or symbolic execution. A practical follow-up is to release the statement corpus, FSM, traces, and reviewer decisions, then test whether independent researchers can reproduce each attack path under the same compliance assumptions.
Read it to learn a concrete workflow for converting protocol prose into an auditable FSM and testing whether proposed attacks remain executable when every normative requirement is honored.
[abstract1.1]; [S4.p1]; [S5.p4]; [S4.SS3.p1]; [S1.p4]; [S1.p7]
ILP-Based Splitting and Negative Sampling Reduce Measured Shortcuts in PPI Datasets
Why the audit matters
Random splitting can make a PPI model look strong for the wrong reason. In the paper’s maximally biased configuration, a random 80/10/10 split of positives, uniform negatives from the complement, and a Random Forest over concatenated mean-pooled ESM-2 or ProtT5 embeddings produce near-perfect performance. This setup is therefore a concrete warning that predictive performance can reflect dataset construction rather than biological signal.
When train-test protein overlap is removed, usable shortcuts still come from self-interactions, taxonomic identity, and functional relatedness, and their prevalence depends on the data source. The method change is to make both split assignment and negative selection explicit integer-linear-programming problems. At the split stage, KaHIP clustering is followed by an ILP that minimizes discarded PPIs while targeting user-specified split fractions; CD-HIT-2D removes residual validation/test pairs above 40% identity. At the negative-sampling stage, the ILP selects from a stratified candidate pool of roughly four times the positive count and matches self-interaction counts, per-protein degree, taxon-pair counts, and mean GO-BP Jaccard index.
Jointly applying the ILP-based split and negative sampler reduces measured attribute mutual information and brings the embedding-based Random Forest baseline to chance-level performance across nearly all datasets, except those with irreducible self-interaction biases. For PDB-Dimers and PINDER, the self-interaction shortcut dominates, making these structural datasets a different interaction population from assay-oriented data. Sequence similarity was a weaker shortcut than prior work suggested, but suppressing inter-split similarity remains necessary, particularly when structural data contain repeated protein variants.
A transferable research operation is to audit a new benchmark before training its main model: measure the same attribute separations, construct a similarity-reduced split, and compare convenience negative samples with an ILP-selected set. Report the target model on both constructions alongside residual attribute dependence. That experiment tests whether a claimed gain survives when known label-construction routes are explicitly matched.
Boundaries
Matching GO overlap is not neutral: proteins in the same pathway, cellular component, or complex genuinely interact more often, so this constraint can remove real interaction-related signal along with artifact. The pipeline supports only ESM-2 and ProtT5 embeddings, and the negative-sampling ILP uses a stratified candidate pool of about four times the positive count because solving over the full complement is computationally prohibitive; the full-pool optimum may differ. Even the most rigorous mitigation leaves residual bias in PDB-Dimers and PINDER, and complementary strategies such as positive downsampling or loss reweighting were not evaluated empirically.
Read it to learn how to turn a shortcut audit into constrained split and negative-sampling experiments, while seeing where debiasing can remove genuine interaction-related signal.
abstract1.1; S2.SS2.p1; S2.SS2.p3; S2.SS2.p4; S2.SS4.p1; S3.p1; S3.p3; S3.p4
Prepared retrospectively from the 2026-09-14 candidate papers; verified on 2026-09-16.