The Distributional Residual: Architecture-Specific Frequency-Semantic Geometry in Word Embeddings

1. Introduction

1.1 Embeddings Encode More Than Meaning

Distributional word embeddings are designed to capture meaning. They do — a linear probe predicting human concreteness judgments from GloVe representations achieves R² = 0.629; fastText achieves R² = 0.722. These are not trivial approximations of human semantic intuitions. For many applications, embeddings work because they successfully encode what words mean and how they relate to each other.

But a word embedding is not a semantic representation. It is a high-dimensional record of what contexts a word appears in, across hundreds of millions of training documents. Contexts carry semantic content — but they also carry word frequency (how many documents the word appears in at all), register (whether the word appears more in formal written text or casual spoken conversation), morphological structure (the word's relationship to its derivative and inflected forms), and the full distributional neighborhood of a word's company in the corpus. All of these signals enter the embedding via the same mechanism: the training objective adjusts each word's vector to predict, or be predicted by, co-occurring words. Every property that co-occurs with a word's context — or with how often a word has contexts — gets pressed into the same vector.

The result is that embeddings are mixture representations. Semantic content, frequency, register, morphology, and pragmatic context are tangled together in a high-dimensional space. When NLP systems use embeddings as features, they inherit that mixture. A sentiment classifier trained on GloVe embeddings may be learning to exploit word frequency as a proxy for sentiment, not just semantic valence. A word similarity model may be penalizing rare words because their embeddings are noisier, not because they are semantically distant from their neighbors. The tangling creates confounds, and those confounds are invisible unless we look for them.

1.2 The Distributional Residual Question

This paper asks a specific version of the general question. Take a word embedding. Train a linear probe predicting semantic features from the embedding. Compute the residual — what the probe fails to explain. Now ask: how much frequency information is in that residual?

If the answer is "very little," frequency and semantic content are encoded in orthogonal subspaces — they can be studied and manipulated independently. If the answer is "a lot," they are entangled — removing semantic content also perturbs frequency information. The answer, we show, depends on the training objective.

We call this the distributional residual framework. Its central contribution is the geometric floor: the expected frequency retention if frequency and the purged semantic dimensions were perfectly orthogonal. Under independence, the expected retention after projecting out K semantic dimensions from a D-dimensional embedding is (D − K) / D — the fraction of the embedding space that remains. An architecture landing at the floor is orthogonal; above the floor, super-orthogonal (frequency avoids the purged directions); below the floor, entangled (frequency and semantic content share dimensions, and purging semantics takes frequency with it). The floor is O(1) to compute and provides a model-independent baseline that makes cross-architecture comparison principled.

1.3 Contributions

We apply this framework to three embedding architectures representing different training objectives and dimensionalities: GloVe (100d, log-bilinear co-occurrence), fastText (300d, SGNS with subword n-gram averaging), and BERT (768d, masked language modeling). Across 13 experiments on a ~13,300-word vocabulary with the richest available English semantic norms (Brysbaert et al. 2013; Warriner et al. 2013; Lynott et al. 2020), we find three qualitatively distinct regimes.

The contributions are:

1. The distributional residual framework — a systematic pipeline for extracting non-semantic information from embeddings via linear probing and residual analysis.

1. The geometric floor — a principled, model-independent baseline for frequency retention that converts raw retention figures into normalized deviations and enables cross-architecture comparison.

1. A three-regime taxonomy — orthogonal (GloVe), entangled (fastText), floor-dominated (BERT) — each explained by training objective, with practical implications for debiasing.

1. A systematic mechanistic investigation — four pre-registered experiments testing hypotheses about what word-level properties predict per-word entanglement in fastText, each rejected, with a combined predictor ceiling of R² = 7.4%. The mechanism is irreducible at the word level.

1. Methodological transparency — each mechanistic experiment pre-registered with specific directional predictions and quantitative thresholds. Four consecutive rejections with n ≈ 13,300 constitute genuine evidence that the space of word-level explanations has been explored and found insufficient.

Section 1 word count: ~650 words. Written 2026-03-08 — Nell.

2. Background and Related Work

2.1 Static Word Embeddings

Static word embeddings assign each word a single dense vector regardless of context. Three training objectives are relevant to this study.

GloVe (Pennington et al. 2014) trains on a global word co-occurrence matrix. Each entry X{ij} records how often words i and j co-occur in a fixed context window. The training objective minimizes a weighted least-squares loss over all word pairs, with weights proportional to f(X{ij}) — a function of the co-occurrence count. Crucially, the weighting means that high-frequency co-occurrences dominate the gradient: word frequency enters the GloVe objective explicitly, as a training signal weighting function. GloVe embeddings are 50–300 dimensional depending on the release; we use the 100-dimensional Wikipedia+Gigaword vectors.

Skip-gram with negative sampling (SGNS) (Mikolov et al. 2013) takes a different approach. Given a word in context, the model learns to distinguish the true context word from randomly sampled "negative" words. Training uses frequency subsampling: high-frequency words are randomly discarded from training with probability proportional to their frequency, reducing the gradient weight of very common word-context pairs. Unlike GloVe, frequency enters SGNS primarily as a regularizer (subsampling) rather than an explicit loss weight.

fastText (Bojanowski et al. 2017) extends SGNS with subword character n-gram averaging. Each word's embedding is the mean over its character n-gram embeddings (e.g., for running, the embeddings of run, runn, runni, unni, nning, plus the whole word). This allows representations for morphological variants to share components, improving coverage of rare and out-of-vocabulary words. But it also creates an architectural coupling between morphologically related words: run, runner, running, outrun all share n-gram components and therefore share representational territory in the embedding space. We use the 300-dimensional wiki-news-subwords fastText vectors.

2.2 Contextual Embeddings

BERT (Devlin et al. 2019) uses a transformer encoder trained with masked language modeling (MLM): 15% of input tokens are masked, and the model is trained to predict the masked tokens from context. Unlike static embeddings, BERT produces different vectors for the same word in different contexts. For comparison with static embeddings, we use mean-pooled last-layer representations produced with the single-word template [CLS] [SEP], yielding one 768-dimensional vector per word type. This is a methodological limitation discussed in §6.5; single-word context is not ecologically valid for BERT, and our results should be treated as characterizing this specific representation mode.

2.3 Frequency Artifacts in Embeddings

The relationship between word frequency and embedding geometry has been an active area of study since the earliest days of dense word representations.

Mu and Viswanath (2018) identified the dominant non-semantic component of GloVe and word2vec embeddings as a frequency-related artifact: the top PCA component of the embedding matrix correlates strongly with log word frequency, and removing it (the "All-but-the-Top" procedure) improves performance on semantic similarity tasks. Arora et al. (2017) showed that sentence representations built from word embeddings contain a common discourse vector that correlates with word frequency, and that removing it via principal component subtraction improves sentence similarity. Gong et al. (2018) introduced FRAGE (Frequency-Agnostic word Representation), demonstrating through adversarial training that word embeddings are systematically biased toward frequency: high-frequency and low-frequency words occupy different subregions of the embedding space, such that semantically similar words with mismatched frequency can appear far apart — a bias that persists across tasks including language modeling and machine translation (NeurIPS 2018; arXiv:1809.06858). Schakel & Wilson (2015) showed that word vector norms encode word significance — words appearing consistently in similar contexts have longer vectors regardless of raw frequency — demonstrating that embedding geometry encodes corpus-distributional properties beyond semantic content (arXiv:1508.02297). Schnabel et al. (2015) showed that word analogy tasks — the canonical benchmark for word embeddings — are biased toward high-frequency words, inflating performance estimates for frequent vocabulary.

Ethayarajh (2019) extended the anisotropy analysis to contextual embeddings (ELMo, GPT, BERT), showing that contextual representations are highly anisotropic — concentrated in a narrow cone of the representation space — with implications for how similarity should be measured.

Our work differs from this prior research in two ways. First, prior work describes frequency artifacts in individual architectures; we systematically characterize the relationship between frequency and semantics across architectures, using a principled baseline (the geometric floor) that makes cross-architecture comparison possible. Second, prior work treats frequency as noise to be removed; we treat the frequency-semantic relationship as a structural property of the architecture to be measured and explained. The question is not "is there a frequency artifact?" (yes, in all architectures) but "are frequency and semantic content orthogonal or entangled?" (it depends).

2.4 Probing Methodology

Linear probing (Alain & Bengio 2017; Belinkov & Glass 2019) has become the standard tool for diagnosing what information is encoded in neural network representations. The paradigm: train a lightweight linear classifier or regressor predicting a linguistic property (POS tag, semantic role, grammatical number) from hidden-state vectors, and take the probe's performance as evidence that the property is encoded. The choice of probe architecture matters: more powerful probes can extract information from any representation, including random ones, which undermines the diagnostic value (Hewitt & Liang 2019).

We use ridge regression throughout, which is a standard choice for continuous-valued semantic features. Ridge regression has the interpretive advantage that its R² has a clear geometric meaning: it measures the linear predictability of the target variable from the input vectors, subject to L2 regularization. The cross-validated R² we report throughout is an out-of-sample estimate, not in-sample fit.

A key distinction in our setup is the use of human feature norms (Brysbaert et al. 2013; Warriner et al. 2013; Lynott et al. 2020) as probe targets, rather than model-internal representations or annotation schemes. Human norms provide an external criterion for what "semantic content" means that is independent of any particular model — essential for a study asking whether semantic and non-semantic content are separated in embedding geometry. Using a model's own semantic representations to define the residual would circular: we would be asking whether the residual with respect to one model's semantics is predictive of another property, with no independent grounding of what semantics means.

Section 2 word count: ~820 words. Written 2026-03-08 — Nell.

3. Methods

3.1 Embedding Models

We analyze three embedding architectures representing different training objectives and dimensionalities. All models use publicly available pre-trained weights.

For BERT, we represent each word by mean-pooling the last-layer hidden states across its subword tokens, using the single-word template [CLS] [SEP]. This is a deliberately minimal context — a limitation we return to in §6.4 — but it produces a single fixed vector per word-type that is comparable to the static embeddings from GloVe and fastText.

The three training objectives differ in how they treat word frequency. GloVe's log-bilinear objective explicitly weights each word pair's loss by log co-occurrence count, meaning high-frequency co-occurrences dominate the training signal and word frequency enters the gradient directly. FastText's SGNS objective uses frequency subsampling (high-frequency words are downsampled during training) plus subword character n-gram averaging, which distributes each word's embedding across a morphological neighborhood. BERT's masked language model predicts masked tokens in context; word frequency enters only implicitly, through how often a word appears in training and how much gradient signal it accumulates. These differences in how frequency participates in training produce the distinct geometric outcomes we measure.

3.2 Semantic Feature Norms

To construct semantic probes, we use published human-derived semantic feature norms as regression targets. We use four datasets:

Brysbaert et al. (2013) concreteness ratings. 39,954 English words rated on a 1–5 scale from abstract to concrete by crowd-sourced annotators. Used in initial experiments (§4.1) as a 1-dimensional semantic probe.

Warriner et al. (2013) affective norms. 13,915 lemmas rated on three dimensions: valence (pleasant/unpleasant), arousal (calm/excited), and dominance (controlled/in-control). Together with concreteness, these form a 4-dimensional affective probe (VAD+concreteness). Valence and arousal in particular have been shown to be independently recoverable from distributional representations (Turney & Littman 2003; Mohammad & Turney 2013).

Lynott et al. (2020) Lancaster Sensorimotor Norms. 39,707 words rated on 11 perceptual and motor dimensions: auditory, gustatory, haptic, interoceptive, olfactory, and visual perception, plus five motor effector dimensions (foot/leg, hand/arm, head, mouth, torso). This is the richest semantic probe available: it covers the full range of embodied perceptual experience and captures dimensions — haptic texture, interoceptive body state, motor action type — that no affective rating system includes.

Combined 15-dimensional probe. Our primary semantic probe combines Lancaster (11D) with Warriner's three affective dimensions plus Brysbaert's concreteness (4D) for a total of 15 semantic dimensions. We use a joint ridge regression that fits all 15 targets simultaneously (one output per word per semantic feature), then compute residuals by projecting out the full 15-dimensional subspace spanned by the probe's fitted directions. Vocabulary intersection across all datasets yields approximately 13,300 words as our primary analysis vocabulary.

We use human norms rather than model-internal semantic representations for the probe targets. This is intentional: the residual we want to analyze is "what the embedding encodes beyond human-ratable semantic content." Using a model's own semantic representations would circularlly define the residual in terms of the model.

3.3 Non-Semantic Targets

The primary non-semantic target throughout is log word frequency, taken from SUBTLEX-US (Brysbaert & New 2009), which provides word frequency estimates derived from 51 million words of American English film subtitle text. Subtitle corpora are preferred over newspaper corpora for this purpose because they better represent spoken-register word usage — an important property given that several of our high-interest words (function words, common modals) are primarily spoken-register items. Log-transforming frequency is standard practice; raw frequency counts span five or six orders of magnitude, and the log-transform produces a distribution that is roughly symmetric and linearly predictable from embedding distances.

Secondary non-semantic variables, used in the mechanistic investigation (§5), include word length (character count), Brown corpus POS distribution, morphological family size from the CELEX database, WordNet synset count (polysemy), and per-word neighborhood statistics computed directly from the embedding space (described in §5.4).

3.4 The Distributional Residual Pipeline

The pipeline has five steps. We describe each in turn, with particular attention to the geometric floor (Step 4), which is the central methodological contribution of this work.

Step 1: Vocabulary intersection. Restrict all analyses to words appearing in the embedding vocabulary, the semantic norms, and the frequency database. For our primary analysis, this intersection yields ~13,300 words. We apply a minimum-frequency filter (log_freq > 0.5) and exclude proper nouns identified by capitalization in the original embedding vocabulary.

Step 2: Semantic probe. Train a ridge regression predicting semantic feature scores from embedding vectors. For a single semantic dimension (e.g., concreteness), the probe is a standard ridge regression: f̂(e) = e^⊤β where β is the 1×D coefficient vector fit by minimizing ‖Eβ − y‖² + λ‖β‖². For a K-dimensional semantic probe, we fit K independent ridge regressions, one per semantic dimension, each yielding a probe direction βₖ ∈ ℝᴰ. All probe evaluations use 5-fold cross-validation with lambda selected by inner CV on the training folds. This ensures that reported R² values reflect out-of-sample predictability, not in-sample fit.

Step 3: Residual construction. Given K probe directions B = [β₁ | β₂ | ... | βₖ] ∈ ℝᴰˣᴷ, we project each embedding vector e_i onto the subspace spanned by B and subtract the projection:

r_i = e_i − B(BᵀB)⁻¹Bᵀe_i

This is the standard orthogonal complement projection: ri is the component of ei orthogonal to all K probe directions. The resulting residual r_i ∈ ℝᴰ has the same dimensionality as the original embedding, but with the semantic subspace removed. We do not reduce dimensionality; the residual lives in the original D-dimensional space, restricted to the (D − K)-dimensional subspace orthogonal to B.

Step 4: Frequency probe on residual + the geometric floor. Train a ridge regression predicting log word frequency from the residual vectors {ri}. Call the cross-validated R² of this probe Rresid². The corresponding R² from the full embedding vectors {ei} is Rfull². The frequency retention is defined as:

retention = Rresid² / Rfull²

Retention of 100% means the semantic purge has not touched the frequency signal. Retention of 0% means the purge has completely removed it.

The critical question is: what retention should we expect under the null hypothesis that frequency and the K purged dimensions are statistically independent? If they are independent, removing the K semantic directions should not reduce frequency predictability at all — retention should be 100%. But this ignores a geometric constraint.

A linear frequency probe on D-dimensional vectors has access to D degrees of freedom. After projecting out K dimensions, the residual has only D − K degrees of freedom: K directions of the space have been zeroed out, regardless of whether they contained frequency information. A ridge regression on the residual is therefore working with a smaller effective feature space, and its R² is bounded above by the fraction of the space remaining. Under independence and uniform variance across dimensions, the expected retention is:

floor(K, D) = (D − K) / D

Normalization and probe rank. We do not PCA-normalize embedding vectors before computing the floor; the formula assumes uniform variance, which is not strictly met for raw embeddings (GloVe's PC1, for instance, accounts for a larger-than-uniform share of variance). However, the floor formula is conservative for the below-floor interpretation: because high-variance directions carry more predictive signal for the frequency probe, projecting out K directions that are not high-variance directions is less disruptive than the floor predicts. An architecture that falls below the floor despite this conservative bias has even stronger entanglement than the raw Δ number implies. For fastText's Δ = −11.3pp, the effect is large enough that this correction cannot change the regime assignment.

The K probe directions are orthogonalized via Gram-Schmidt before projection: each direction is orthogonalized against all previously accepted directions, and a direction is dropped if its residual norm falls below 10⁻⁸ (indicating near-collinearity). The effective rank of the probe subspace is therefore len(orth_dirs) after Gram-Schmidt, which may be less than the nominal K. The floor formula uses the effective rank, not the nominal K. For our 15-dimensional Lancaster+VAD probe, the reported floor values of 85.0% for GloVe (floor = (100−15)/100) and 95.0% for fastText (floor = (300−15)/300) confirm that all 15 probe directions survived Gram-Schmidt with rank(B) = 15 — i.e., the 15 semantic feature dimensions are not collinear enough to reduce the effective probe rank.

We call this the geometric floor: the minimum expected retention if frequency and semantics are independent. An architecture that lands at the floor is orthogonal — the purge removes exactly the expected fraction. An architecture that lands above the floor is super-orthogonal — frequency is concentrated away from the purged directions, and the purge is less disruptive than chance. An architecture that lands below the floor is entangled — frequency and the purged directions share subspace, so purging semantic content also removes frequency information.

The floor is not a ceiling. Orthogonal architectures should land at (or slightly above) the floor, not below it. When an architecture consistently falls below the floor across multiple probe sets and purge sizes, this constitutes systematic evidence of frequency-semantic entanglement — the signals genuinely co-occupy embedding dimensions.

Note that the floor depends only on K and D, not on what the K directions contain or what the frequency signal is. This makes it a model-independent baseline: the same floor applies to any K-dimensional purge from a D-dimensional embedding, regardless of architecture. This is what makes the floor useful for cross-architecture comparison: rather than asking "what is the raw retention?", we ask "how far does each architecture deviate from what geometry alone predicts?"

In practice, we compute the floor as (D − K) / D and report both the measured retention and the deviation Δ = retention − floor. Positive Δ means at or above the floor (frequency is at least as orthogonal to the purged directions as random). Negative Δ means below the floor (frequency-semantic entanglement).

Step 5: Retention curve. We repeat Steps 2–4 for increasing values of K, adding semantic probe dimensions one at a time in order of decreasing variance explained by the probe (i.e., in PCA order of the semantic norm matrix). This produces a retention curve showing how quickly frequency information is removed as the semantic purge expands. Plotting the measured retention and the geometric floor together on the same axes makes architecture-specific deviations visible. For orthogonal architectures, the curves are nearly parallel — retention tracks the floor. For entangled architectures, the retention curve dips below the floor, and the gap widens with K.

3.5 Pre-Registration

The initial experiments (Exp 01–08) were exploratory: we had no strong predictions about which architecture would fall in which regime before running the analysis. The mechanistic experiments (Exp 09–13) were pre-registered: before running each experiment, we committed to specific directional predictions and quantitative thresholds (e.g., "partial r > 0.10 after controlling for word length and frequency"). Pre-registration status is noted for each result in §5. Where a finding contradicts a pre-registered prediction, we say so explicitly.

A note on pre-registration format. The pre-registrations were internal: directional predictions and quantitative thresholds were recorded in the lab thread file prior to running each experiment, but were not deposited in an external registry (OSF, AsPredicted, or equivalent). We flag this limitation. The absence of an external timestamp means our pre-registration claim cannot be independently verified from outside the lab. We preserve this framing because the structure it imposed — committing to a threshold before seeing the data — was methodologically real, and the failed-prediction results (e.g., reversed-direction partial r for morphological family size) would have been easy to suppress post-hoc. Readers should weight the pre-registration claim accordingly.

Pre-registration matters here because we ran four consecutive hypothesis-testing experiments targeting the same dependent variable (per-word entanglement). Without pre-registration, the accumulation of rejected hypotheses could look like post-hoc rationalization — "we tried morphology, it didn't work, so we tried polysemy." With pre-registration, each test is an independent commitment. Four independent rejections with high statistical power (13,300 words; smallest detectable partial r ≈ 0.03) constitute genuine evidence that per-word entanglement is not predictable from the tested variables. We take the negative results seriously.

Section 3 word count: ~1,550 words. Written 2026-03-07 — Nell.

4. Architecture Results: A Three-Regime Taxonomy

A word embedding has many dimensions, and frequency is threaded through them in ways that depend on how the model was trained. To characterize that threading, we use a common measurement apparatus across three architectures: train a semantic probe, compute the residual, ask how much frequency information the residual retains, and compare that retention to the geometric floor — the expected retention if frequency and the purged semantic directions were completely independent.

The central result of this section is that the three architectures we test — GloVe, fastText, and BERT — each fall into a qualitatively distinct regime, summarized in Table 1. GloVe's retention lands just above its geometric floor: frequency and semantics are encoded in orthogonal subspaces. fastText's retention falls well below its floor: the signals are genuinely entangled, such that removing semantic information also removes frequency information. BERT's retention stays pinned to the floor from above by sheer dimensionality: with 768 dimensions, no feasible semantic purge removes enough variance to meaningfully probe the frequency-semantic relationship. We call these three regimes orthogonal, entangled, and floor-dominated, respectively.

4.1 GloVe: The Orthogonal Regime

GloVe (Pennington et al. 2014) is trained with a log-bilinear objective over word co-occurrence counts. The training signal is explicitly weighted by log co-occurrence frequency, which means word frequency is built into the gradient updates at a structural level: high-frequency words receive more and larger updates than low-frequency words. The training objective, in other words, separates how often words appear from in what contexts they appear — the former enters the objective as a weight, the latter as the signal being fit.

This factorization leaves a fingerprint in the geometry. GloVe's first principal component (PC1) has a Pearson correlation of −0.724 with log word frequency, making it essentially a frequency axis: the most variance-explaining direction in the 100-dimensional space is organized by how common or rare words are. GloVe's second principal component (PC2) has a correlation of −0.740 with concreteness (Brysbaert et al. 2013) and only +0.021 with frequency — a nearly pure semantic axis. Because PCA components are orthogonal by construction, and because the frequency signal and the semantic signal happen to align with the first two PCs respectively, they are very nearly orthogonal in the full embedding space.

This orthogonality has a measurable consequence. When we train a linear probe predicting concreteness from GloVe embeddings (R² = 0.629 with 5-fold cross-validation), compute the residual — what the probe fails to explain — and then ask how well that residual predicts log word frequency, we find that 98.7% of the original frequency predictability is retained. The semantic probe has projected out the concreteness direction (PC2) and left the frequency direction (PC1) almost completely intact. This is confirmed geometrically: the cosine similarity between PC1 of the full embedding and PC1 of the concreteness residual is 1.000. They are the same vector.

This also explains the Mu and Viswanath (2018) "All-but-the-Top" finding. Their debiasing procedure removes the top PCA components from GloVe embeddings and finds improved downstream performance. Our analysis clarifies the mechanism: GloVe's PC1 is the frequency axis. Removing it collapses frequency predictability from 0.853 to 0.317 while leaving the concreteness probe essentially unchanged (0.629 → 0.629 after D=1 debiasing). The debiasing works because frequency and semantics are orthogonal in GloVe — surgery on PC1 is clean.

The orthogonality result holds beyond the initial 1D concreteness probe. When we expand the semantic probe to 15 dimensions — the 11 sensorimotor dimensions of the Lancaster Norms (Lynott et al. 2020) combined with valence, arousal, and dominance (Warriner et al. 2013) — the geometric floor for GloVe at K=15 is approximately 85% (floor = (D − K) / D = (100 − 15) / 100). The measured retention is 87.6%. GloVe lands 2.6 percentage points above its floor — at the floor, within noise — confirming that frequency is not merely orthogonal to concreteness but to the entire 15-dimensional semantic subspace we can construct from published norms. The frequency axis in GloVe is genuinely separate from semantic content; projecting out semantic content does not affect it.

4.2 fastText: The Entangled Regime

fastText (Bojanowski et al. 2017) uses skip-gram with negative sampling (SGNS) rather than the log-bilinear objective. Crucially, it adds subword character n-gram averaging: each word's embedding is the mean of its character n-gram embeddings. Frequent words appear in many training contexts and receive many gradient updates; rare words benefit from sharing subword structure with morphological relatives. Both of these features alter the relationship between word frequency and the learned geometry.

The result is a different PCA structure. In fastText, PC1 has a correlation of −0.376 with log frequency and −0.579 with concreteness. The dominant component of variance is no longer a frequency axis — it is a mixed axis, carrying both signals simultaneously. The clean two-factor decomposition that GloVe exhibits is absent. The training objective has not factored frequency and semantics into separate geometric dimensions.

This mixing has a direct consequence for residual analysis. When we train a 1D concreteness probe on fastText (R² = 0.722), the residual retains 98.6% of frequency predictability — nearly identical to GloVe's 98.7%. This apparent similarity is deceptive. In GloVe, the 98.7% retention reflects genuine orthogonality: the concreteness probe removes PC2 and leaves PC1 intact. In fastText, the 98.6% retention reflects dimensionality: removing one direction from a 300-dimensional space eliminates at most 0.33% of total variance. Even when a purge direction carries frequency information, the remaining 299 dimensions retain nearly all of it. The 1D probe is too blunt an instrument to measure the degree of mixing.

The entanglement becomes visible only when we expand the purge. With a 4-dimensional semantic probe (concreteness, valence, arousal, dominance), the geometric floor for fastText at K=4 is (300 − 4) / 300 = 98.7%. The measured retention is 89.9% — 10 percentage points below the floor. For the first time, frequency and semantic information are confirmed to overlap: purging four semantic dimensions removes more frequency information than geometric independence would predict. The floor is not a soft target; it is a hard lower bound under the null hypothesis of orthogonality. FastText falls below it.

This below-floor effect persists across every semantic probe we test. With the full 15-dimensional Lancaster+VAD purge, fastText reaches 83.7% retention against a floor of approximately 95% — a 11.3 percentage point deficit. Across 11 progressively expanded purge sets (Experiments 05–07), fastText's retention consistently and substantially undercuts its floor. This is not an artifact of which semantic dimensions we chose: the effect appears with affective norms, sensorimotor norms, and combinations thereof.

The below-floor result is the core finding for fastText. It has a clean interpretation: frequency and semantic content partially occupy the same embedding directions. When a semantic probe removes those directions, it inadvertently takes frequency information with it. The loss is not catastrophic — even after a 15D purge, 83.7% of frequency predictability survives — but it is systematic and architecture-specific. In GloVe, the same probe causes no such collateral damage.

Why does fastText entangle the signals where GloVe does not? The subword averaging mechanism provides a partial account. Morphological family members — run, runner, running, ran — share character n-grams and therefore pull each other's representations toward the same region of embedding space. These families tend to have correlated frequency profiles (if the root is common, the derived forms are too). Semantic probes that isolate content-type clusters — "these are all auditory words" or "these are all high-valence words" — will sometimes isolate morphological communities, and those communities carry frequency information. Removing the semantic cluster therefore removes the frequency cluster too.

4.3 BERT: The Floor-Dominated Regime

BERT (Devlin et al. 2019) presents a qualitatively different case. Its masked language modeling objective does not explicitly encode word frequency; frequency enters only through the training process itself, as common words receive more training signal and converge to better-defined representations. The result is a weaker frequency signal: BERT's PC1 has r(freq) = −0.425, compared to −0.724 for GloVe. No clean frequency axis exists; the most variance-explaining direction (which accounts for 17.9% of variance, compared to 5.8% for GloVe's PC1) is only moderately correlated with frequency and is not dominated by it.

The more important difference, however, is dimensionality. BERT's mean-pooled last-layer representations are 768-dimensional. Removing 15 semantic probe directions eliminates 15/768 = 2.0% of the total variance. The geometric floor for BERT at K=15 is therefore (768 − 15) / 768 ≈ 98.0%. Under the null hypothesis of orthogonality, we would expect approximately 98% retention regardless of what the semantic probe contains. The measured retention is 99.1%. The difference between measured and floor values is +1.1 percentage points, which is indistinguishable from noise given the variance in the probe evaluation.

We call this the floor-dominated regime: the dimensionality of the embedding space places such a tight constraint on the expected retention that the actual retention provides almost no diagnostic information about the frequency-semantic relationship. Any feasible semantic probe — one covering a small number of dimensions relative to the total — will leave BERT's frequency signal essentially intact, not because frequency and semantics are orthogonal, but because the probe is too small to do otherwise.

This is an important methodological point. The distributional residual pipeline, as designed, cannot assess frequency-semantic orthogonality in very high-dimensional spaces without either an extremely large semantic probe (K approaching D) or a fundamentally different analytic strategy. Our results for BERT should therefore be read as establishing that BERT belongs to a third category — not a characterization of whether or not BERT's frequency and semantic content are orthogonal, which remains unresolved.

What we can say about BERT's frequency encoding is that it is diffuse. No single PC dominates the frequency signal in the way GloVe's PC1 does, and the overall frequency probe R² (0.693) is substantially lower than GloVe's (0.853). The MLM objective produces representations in which frequency enters as a background regularity rather than as an explicit geometric factor, and those representations are distributed across far more dimensions than the probing paradigm can access.

4.4 Architecture Comparison

Table 1 summarizes the three regimes across the primary metrics.

Table 1: Architecture Comparison — Distributional Residual Analysis

†GloVe 100d and fastText 300d Δ values use K=15 Lancaster+VAD purge; all others use K=1 (concreteness only) due to vocabulary intersection constraints. Δ from floor: positive = at or above floor (orthogonal); negative = below floor (entangled). Floor = (D − K) / D.

A key addition: word2vec (SGNS without subword). Experiment 16 tested word2vec-google-news-300 (Mikolov et al. 2013) — SGNS training without subword n-gram features — to disentangle the contributions of the SGNS objective and the subword averaging mechanism. The result is decisive: word2vec achieves K=1 retention of 99.1% against a floor of 99.7%, giving Δ = −0.6pp. This is effectively at the floor — the same orthogonal regime as GloVe. The PCA structure confirms: word2vec's PC1 loads on concreteness (r = −0.597), not frequency (r = +0.254), unlike fastText where PC1 mixes both.

This isolates the mechanism. The SGNS objective alone does not produce frequency-semantic entanglement. The entanglement in fastText is attributable specifically to the subword character n-gram averaging: when each word's vector is the mean of its character n-gram components, morphological family members share representational space, and those families carry correlated frequency profiles. Semantic probes that remove content-type clusters inadvertently remove morphological communities — and thus remove frequency information. GloVe-style log-bilinear training and SGNS training (without subword) both produce the orthogonal regime; fastText's subword extension is what creates the entangled regime.

The taxonomy now covers four training configurations and three regimes. The key explanatory variable is not the architecture family (static vs. contextual) or the base training objective (log-bilinear vs. SGNS), but specifically the subword averaging mechanism for the entangled regime. Log-bilinear training factorizes frequency and semantics into separate geometric dimensions. SGNS alone (without subword) produces an orthogonal regime similar to GloVe, despite different gradient dynamics. Only SGNS with subword averaging creates the below-floor entanglement. MLM training with high dimensionality produces the floor-dominated regime.

Two findings are worth emphasizing across the architectures. First, the initial 1D purge gives nearly identical retention for GloVe (98.7%) and fastText (98.6%), despite the underlying structures being completely different. This illustrates the importance of the geometric floor: raw retention figures are uninformative without the floor as a reference. The architectures diverge sharply only when the purge is expanded to multiple dimensions and the floor is taken seriously as a baseline. Second, the word2vec result sharpens the mechanistic account in §5: the four per-word predictor rejections (Exp 09–12) showed morphological family size is not predictive of per-word entanglement, which at first seemed to contradict the subword averaging account. The word2vec result provides a reconciliation: subword averaging drives the architecture-level entanglement, but the which words are most affected question remains unanswered at the word level — the mechanism operates at the training dynamics level, not the lexical property level.

Section 4 ends here. Section 5 follows.

Section 4 word count: ~2,050 words. Written 2026-03-07 — Nell.

5. Mechanistic Investigation: Why Do Specific Words Entangle?

The architecture results establish that fastText consistently falls below its geometric floor — frequency and semantic content are entangled in the embedding space. The natural next question is: which words drive this entanglement, and what is it about those words that causes the semantic purge to take frequency information with it?

This section investigates that question through four pre-registered experiments. We define a per-word entanglement score, characterize the words at the extremes of the distribution, and then test four hypotheses about what predicts high entanglement. The hypotheses are tested in sequence, with each result motivating the next. All four are rejected. A fifth experiment — a combined predictor ceiling — confirms that all available word-level predictors together explain only 7.4% of variance in per-word entanglement. The mechanism is irreducible at the word level.

We take these negative results seriously. Four pre-registered rejections with high statistical power (n ≈ 13,300; minimum detectable partial r ≈ 0.03) constitute genuine empirical evidence. The space of word-level explanations has been systematically explored and found insufficient. What remains is a corpus-level account — one that requires access to per-word training statistics not available in post-hoc analysis.

5.1 The Per-Word Entanglement Score

To investigate the mechanism at word level, we need a per-word measure of how much frequency information is lost when that word's embedding is replaced by its semantic residual. We define this as:

entangle_i = freq_pred(e_i) − freq_pred(r_i)

where freqpred(ei) is the frequency ridge regression's prediction for word i from its full embedding, and freqpred(ri) is the same regression's prediction from the 15D residual embedding. The absolute value |entangle_i| captures the magnitude of change; the sign captures direction (positive = word appears more frequent in the full embedding than the residual predicts, negative = opposite).

The key insight is that these word-level differences are what aggregate into the below-floor effect. If high-|entanglei| words are systematically different from low-|entanglei| words in some measurable way, that would explain the architecture-level phenomenon.

Figure 5a shows the distribution of |entangle| for fastText (using the 15D Lancaster+VAD purge). The distribution is right-skewed: most words have low entanglement, but a tail of high-entanglement words drives the aggregate below-floor result. The top-500 words (≈3.8% of vocabulary) account for a disproportionate share of the total frequency information lost.

The extreme words reveal an immediately striking pattern. The 25 highest-entanglement words in fastText include:

amino, livery, cent, tad, jot, tub, bar, toner, wiz, can, may, roam, teeny, gag, czar, wreak, runt, cubic, lack, brunt

These are predominantly short (median 4 characters, all but two monosyllabic), morphologically simple, and frequently functional or semantically lightweight in context. The word can and may are modals — their corpus frequency is dominated by functional uses (ability, permission), but the Lancaster norms anchor their concrete physical senses (tin can, hawthorn). Bar is an obstruction or legal institution in most corpus contexts, not a metal rod. Gag is a comic device or silencing device, not merely a physical throat reflex.

The low-entanglement words are different in character:

prenuptial, navigational, dune, incentive, smirk, stripper, mister, submission, pond, posterior

These words are longer, semantically stable, and unambiguous: dune refers to a sand formation in virtually all contexts, smirk to a facial expression, prenuptial to pre-marriage agreements. The Lancaster norms correctly identify the primary sense, and the corpus frequency is consistent with that sense.

This contrast suggests a candidate mechanism: sense dominance mismatch — words where the dominant corpus sense differs from the sense assumed by the Lancaster norms have unstable frequency-residual relationships, because the probe removes the "wrong" semantic direction for that word. The following four experiments test increasingly specific versions of this hypothesis.

5.2 Hypothesis 1: Morphological Family Size

Motivation. FastText's subword averaging means that a word's embedding is the mean over its character n-gram components. Morphological family members — run, runner, running, ran — share character n-grams and therefore share representational space. If morphological family members have correlated frequency profiles, then semantic probes that isolate content-type clusters may inadvertently isolate morphological communities, and purging those communities removes frequency information along with semantic content. Under this hypothesis, words with larger morphological families should show higher per-word entanglement in fastText, with no analogous effect in GloVe (which has no subword averaging).

Pre-registration. P1: fastText partial r(morphfamilysize, |entangle|) > 0.10 after controlling for word length and log frequency. P2: GloVe partial r < 0.05.

Results. We compute morphological family sizes using the CELEX database (Baayen et al. 1995), available for 98.4% of our vocabulary. The Pearson correlation between log family size and |entangle| is r = −0.063 in fastText (p < 10⁻¹³), in the opposite direction from the prediction. The partial r after controlling for word length and frequency is −0.002 (p = 0.80), indistinguishable from zero.

The raw negative correlation is entirely a confound: morphologically rich words tend to be longer, and longer words have less entanglement (r(word_length, |entangle|) = −0.197 in fastText, p < 10⁻¹¹⁶ for n-gram density, which proxies morphological connectivity). Once length is controlled, morphological family size predicts nothing.

The high-entanglement words from §5.1 (can, bar, cent, tad, jot) have morphological family sizes of 1–2 — they are the least morphologically connected words in the vocabulary. This directly refutes the subword averaging account: if subword n-gram sharing were the mechanism, we would expect morphologically rich words to be most affected. The opposite is true.

P1 is rejected. P2 is partially accepted in reverse (GloVe partial r = +0.037, a small positive effect, but not in the direction the hypothesis predicts for fastText).

5.3 Hypothesis 2: WordNet Polysemy

Motivation. The sense dominance mismatch account, suggested by the extreme-word analysis, might be operationalized through polysemy: words with more WordNet synsets have more senses, and are more likely to have a dominant corpus sense that differs from the sense assumed by the norms. Polysemous words are the natural suspects. Under this hypothesis, log WordNet synset count should positively predict |entangle| in fastText, after controlling for word length and frequency.

Pre-registration. P1: fastText partial r(log_synsets, |entangle|) > 0.10. P2: GloVe partial r < 0.05.

Results. We obtain synset counts from NLTK's WordNet interface (Miller 1995), available for 98.7% of our vocabulary. Across 13,302 fastText words, the Pearson r(log_synsets, |entangle|) = +0.036 — directionally consistent but near zero. The partial r after controlling for word length and frequency is −0.060 (p < 10⁻¹², wrong direction).

This rejection is instructive. The highest-polysemy words (break with 75 synsets, cut with 70, run with 57, play with 52) show entanglement values that are distributed across the full range of the entanglement distribution — there is no concentration at the extremes. And the high-entanglement words themselves have only 1–2 WordNet synsets: can, may, bar are formally monosemous by WordNet's lexicographic definitions.

Formal polysemy, as encoded in WordNet, is the wrong construct. The high-entanglement words are monosemous by most definitions, but they are functionally ambiguous in a way that lexicographic sense lists do not capture: can has one dominant concrete sense (container) and one dominant functional sense (ability modal), but these appear in so different grammatical positions that they may not be listed as separate lexicographic senses — they are treated as the same word, yet they populate entirely different distributional contexts. The Lancaster norms anchor the concrete physical sense; the corpus is dominated by the functional sense. Synset count does not capture this.

P1 is rejected. The mechanism is not formal polysemy.

5.4 Hypothesis 3: Part-of-Speech Versatility

Motivation. If the relevant construct is not formal polysemy but contextual plasticity — words that appear in many grammatical roles and contexts — then POS versatility might be a better proxy. Words like can, bar, light, and sound are notoriously POS-flexible (modal verb, noun, adjective, verb). The Shannon entropy of a word's POS distribution in a tagged corpus measures this flexibility: high entropy means the word appears in many POS roles, low entropy means it is POS-stable. The hypothesis is that POS-flexible words have fragmented distributional profiles that resist clean semantic anchoring.

Pre-registration. P1: fastText partial r(H_POS, |entangle|) > 0.08. P2: GloVe partial r < 0.05.

Results. We compute POS entropy from the Brown Corpus (Francis & Kučera 1979) using NLTK's tagged version, requiring a minimum count of 50 occurrences. This condition is met for 1,217 of our 13,291 words (9.2% coverage) — a severe limitation caused by the corpus's small size (~1M tokens) and our vocabulary's skew toward the most frequent words, which are well-represented, but also toward rare imageable words that fall below the minimum count.

Among the 1,217 covered words, the partial r(H_POS, |entangle|) is −0.093 in fastText (p = 0.001, significant but the wrong direction). The quintile trend is non-monotone: the middle POS-versatility quintile has the highest median entanglement, and the most versatile quintile has lower entanglement than the moderately versatile group. POS groups ranked by mean |entangle|: numerals (0.678), verbs (0.489), nouns (0.434), adjectives (0.379), adverbs (0.248). But the high-versatility words are predominantly adverbs, which happen to have lower entanglement.

P1 is rejected. The Brown corpus coverage is too limited (9.2%) for strong inference — these results should be treated with caution, and a replication on a larger tagged corpus (e.g., Universal Dependencies English treebank) would be informative. But within the available data, POS versatility does not account for the effect.

A consistent pattern across three experiments. After controlling for word length and frequency, three consecutive lexical richness predictors show consistently negative partial correlations with |entangle|:

Words that are lexically richer (more morphological derivatives, more senses, more POS roles) show less entanglement, not more. The high-entanglement words are lexically simple — short, monosemous, POS-stable — yet they appear in unusual distributional neighborhoods. The sense dominance mismatch account needs a different operationalization: one that captures not lexicographic complexity but distributional instability.

5.5 Hypothesis 4: Neighborhood Geometric Structure

Motivation. The consistent negative trend in §5.4 leads to a geometric rather than lexical account. The high-entanglement words (can, bar, cent) may not be unusual as lexical items but as locations in embedding space. They sit at intersections of multiple semantic neighborhoods — the concrete sense (tin can) near physical objects, the functional sense (ability modal) near other auxiliaries, the commercial sense (bar of soap) near commodity words — and this neighborhood structure creates an unstable relationship with the semantic probe. When the probe removes semantic directions, it disrupts these crossroads more severely than it disrupts words with a single coherent semantic neighborhood. Under this hypothesis, words with high semantic spread — nearest neighbors that are semantically heterogeneous — should show higher entanglement than words with coherent, homogeneous neighborhoods.

A related prediction: the 15D purge should substantially change high-entanglement words' k-nearest-neighbor profiles (Δnn, measured as the proportion of top-50 neighbors that change after the purge), while leaving low-entanglement words' neighborhoods intact.

Pre-registration. P1: fastText partial r(sem_spread, |entangle|) > 0.15. P2: fastText partial r(Δnn, |entangle|) > 0.15. P3: Both effects smaller in GloVe (partial r < 0.08 for both).

Results. We compute sem_spread as the standard deviation of the 15 Lancaster+VAD feature scores across each word's 50 nearest neighbors in the full embedding space, using cached NearestNeighbors lookups. We compute Δnn as the fraction of each word's top-50 neighbors that are no longer in the top-50 after the 15D purge.

sem_spread shows a real monotone trend: fastText quintile medians are 0.278, 0.290, 0.307, 0.340, 0.356 from low- to high-spread — a 28% increase from bottom to top. But the partial r is only 0.046 in fastText, far below the 0.15 threshold. P1 is rejected.

Δnn is essentially null. High-entanglement and low-entanglement words have nearly identical neighborhood disruption:

• Top-500 entangled words: mean Δnn = 0.0013
• Bottom-500 entangled words: mean Δnn = 0.0012

The 15D purge does not disrupt any word's nearest-neighbor profile in a way that tracks entanglement. P2 is rejected.

P3 is confirmed: GloVe partial r values (0.028 for sem_spread, −0.004 for Δnn) are smaller than fastText's, as expected if GloVe's orthogonality protects against geometric disruption.

The geometric crossroads account fails because the mechanism it predicts — neighborhood disruption — simply does not happen at the scale we can measure. The 15D purge is a smooth linear projection; it moves words by the same amount in the same direction, regardless of their neighborhood structure. There is no mechanism by which the purge would selectively disrupt multi-neighborhood words over single-neighborhood words.

5.6 Combined Predictor Ceiling (Experiment 13)

After four independent rejections, a final analysis asks: what is the total explanatory power of all available word-level predictors combined? We fit ridge regression models predicting |entangle| from increasing predictor sets (Table 2), using 5-fold cross-validation throughout.

Table 2: Cumulative predictor ceiling for per-word entanglement

The marginal contribution of all mechanism predictors beyond the baseline is:

• GloVe: ΔR² = 0.002 (effectively zero)
• fastText: ΔR² = 0.005 (effectively zero)

With all available word-level predictors combined, we explain 7.4% of variance in fastText per-word entanglement, and virtually all of this is attributable to word length and frequency. The four mechanism variables (polysemy, sem_spread, Δnn, and by implication morphological family size and POS entropy) together contribute nothing measurable beyond those controls.

The R² = 0.074 ceiling is the key quantitative finding of this section. It establishes, at the level of cross-validated prediction, that per-word entanglement in fastText's below-floor effect is not predictable from word-level lexical or geometric features. It is not a matter of using the wrong predictors — we have tested the most theoretically motivated candidates, each from a distinct mechanistic account. None work.

5.7 Synthesis: The Mechanism Is Below Word Level

What do four consecutive rejections and a 7.4% ceiling tell us?

They establish that per-word entanglement is an emergent property of training dynamics that is not visible in any lexical or geometric snapshot of the vocabulary. To understand why can loses more frequency information under a semantic purge than dune, you would need access to per-word training corpus statistics: how many distinct contexts did each word appear in? What was the distribution of its grammatical environments across the training corpus? How stable was its gradient signal across training epochs? None of these are available from the final embedding weights alone.

This is not the same as saying the mechanism is random. The high-entanglement words form a recognizable profile — short, common, POS-stable but distributionally unstable, with Lancaster norms that anchor one sense while the corpus distributes frequency across several — and the below-floor effect in fastText is systematic and reproducible across probe sets and semantic norm databases. The mechanism exists; it is just not localized in any word-level property we can measure post-hoc.

The positive framing of these negative results is the following. The distributional residual framework with the geometric floor can definitively characterize the architecture-level relationship between frequency and semantics (§4): GloVe is orthogonal, fastText is entangled, BERT is floor-dominated. This is the clean, robust, actionable finding. The per-word mechanism investigation shows that this architecture-level phenomenon does not decompose cleanly into word-level causes — it is an aggregate product of training dynamics that must be studied at the training level if studied at all.

For practitioners, this has an implication: architecture choice matters more than vocabulary filtering. If you want embeddings where frequency and semantic content are orthogonal (e.g., to study semantic properties independently of frequency), use GloVe or post-hoc debiasing (Mu & Viswanath 2018), because GloVe's orthogonality is not a per-word property that could be controlled with clever data cleaning — it is a consequence of the log-bilinear training objective. FastText's entanglement, similarly, cannot be corrected by filtering problematic words, because we cannot identify which words are problematic from lexical properties alone.

Section 5 word count: ~2,400 words. Written 2026-03-07 (evening) — Nell.

6. Discussion

6.1 Training Objective as the Explanatory Variable

The three-regime taxonomy — orthogonal (GloVe), entangled (fastText), floor-dominated (BERT) — might initially seem like a claim about architectural families: static embeddings behave one way, contextual embeddings another. This framing is wrong, and the comparison within the static embedding family shows why. GloVe and fastText are both static word-type embeddings; they differ in training objective and one architectural choice (subword n-gram averaging). Yet their frequency-semantic geometry is qualitatively opposite. The explanatory variable is not "static vs. contextual" — it is how the training procedure treats word frequency.

GloVe's log-bilinear objective assigns frequency its own structural role: log co-occurrence counts appear as weights on each training example, separate from the context-word signal being fit. This factorization means gradient updates have a frequency channel and a semantic channel that are mechanically distinct. The resulting geometry reflects this: frequency and semantic content land in orthogonal principal components. This is not an accident — it is the training procedure's organizational logic, imprinted on the embedding space.

fastText's SGNS objective does not give frequency a dedicated role. Frequency subsampling reduces the weight of very common words during training, which should if anything decrease frequency's influence. But the subword averaging mechanism reintroduces it: morphological family members share character n-gram components, and those families have correlated frequency profiles (if the base form is common, the derived forms tend to be too). The result is that frequency re-enters the representation through morphological structure. When semantic probes pick up content-type clusters, they sometimes pick up morphological communities, and those carry frequency information along.

BERT's masked language model objective gives frequency no direct gradient role at all. Word frequency enters representations only through training dynamics — common words appear in more training sentences, accumulate more gradient updates, and converge to more well-defined representations. This is a weak, indirect coupling compared to GloVe's explicit weighting. The result is a more diffuse frequency signal, distributed across many dimensions in a 768-dimensional space that our probing methodology cannot adequately penetrate.

This trajectory has an implication for how the field should think about embedding choice. The training objective is not just a recipe for producing better semantic representations — it is a decision about how non-semantic information is organized. GloVe's orthogonal regime makes it easy to study semantic or frequency properties in isolation. FastText's entangled regime makes such isolation difficult: any semantic probe unavoidably perturbs the frequency signal. If a downstream application cares about one signal but not the other, architecture choice has real consequences.

6.2 Implications for Debiasing

The most immediately practical consequence of the three-regime taxonomy concerns debiasing — techniques for removing frequency artifacts from embeddings before use. The canonical approach is Mu and Viswanath's (2018) All-but-the-Top procedure: compute the top PCA components of the embedding matrix and subtract each word's projection onto those components. Applied to GloVe, this substantially improves downstream task performance; the original paper attributes the improvement to removing the frequency artifact encoded in PC1.

Our analysis confirms this mechanistic account at a geometric level. GloVe's PC1 is the frequency axis (r = −0.724 with log frequency) and it is orthogonal to the semantic signal (PC2, r = −0.740 with concreteness). Removing PC1 therefore has surgical precision: frequency predictability collapses (0.853 → 0.317) while semantic predictability is unaffected (0.629 → 0.629 after D=1 debiasing). The surgery is clean because the signals do not share space. This is why All-but-the-Top works for GloVe, and the framework presented here explains why it works at a level the original paper did not reach.

For fastText, this reasoning does not apply. The below-floor results (fastText 15D retention = 83.7% vs. floor ~95%) show that frequency and semantic content share embedding dimensions. A debiasing approach that removes the top PCA component of fastText would not cleanly isolate the frequency artifact — it would take semantic information with it, because fastText's first PC mixes both signals (r(freq) = −0.376, r(conc) = −0.579). Practitioners using fastText with All-but-the-Top debiasing may be inadvertently degrading semantic content rather than cleaning frequency artifacts. For fastText, post-hoc geometric debiasing may not be achievable without collateral damage. An alternative would be training-time interventions: modified objectives that factorize frequency and semantic content the way GloVe's log-bilinear objective does.

For BERT, debiasing via PCA components is essentially impossible with the geometric probing approach: removing 15 directions from 768 touches only 2% of the variance, and no feasible purge size can reach the frequency signal. Debiasing BERT for frequency effects would require either fine-tuning on tasks designed to neutralize frequency correlations, or embedding post-processing methods that operate at a different level of abstraction than linear PCA projection.

6.3 Implications for Probing Methodology

The distributional residual pipeline introduces the geometric floor as a principled baseline for residual analysis. This has broader methodological implications for probing research beyond this specific study.

Probing studies frequently compute the accuracy of a linear probe trained on an embedding, then compare it to the probe trained on the residual embedding (after some transformation). High residual probe accuracy is taken as evidence that the target feature persists after the transformation. But without the geometric floor, "how much persists?" is an uninterpretable question. If the residual still has high-dimensional structure — as all our residual embeddings do; we project out at most 15 of 100, 300, or 768 dimensions — it retains most of its expressive capacity regardless of whether the transformation was semantically meaningful. A probe can predict frequency well from a 99-dimensional residual simply because 99 dimensions is a lot of capacity, not because frequency survived the semantic purge.

The floor converts a raw retention figure into a normalized deviation: how much did the probe change relative to what geometry alone predicts? This reframing makes cross-architecture comparison coherent. GloVe's 87.6% retention and fastText's 83.7% retention look similar until you compare them to their respective floors (85% and 95%). The GloVe result is "at the floor — as expected under orthogonality." The fastText result is "11 percentage points below the floor — the signals are entangled." These are qualitatively opposite conclusions from superficially similar numbers.

We recommend that any study using residual analysis to compare architectures or probe targets adopt the geometric floor as a baseline. The floor requires only knowing K (probe dimensions) and D (embedding dimensions), and is O(1) to compute. The interpretive payoff is substantial.

6.4 The Irreducible Frequency Residual

Even after the most aggressive semantic purge we can construct (15D Lancaster+VAD), substantial frequency predictability remains: GloVe 87.6%, fastText 83.7%. Neither architecture approaches 0% retention. What is this remaining frequency signal?

There are two interpretations. The first is residual entanglement: our semantic probe is incomplete, and with a richer semantic space — perhaps 50+ dimensions covering discourse structure, morphological productivity, collocational strength, and other features beyond sensorimotor and affective content — the retained frequency signal would continue to erode. On this view, 80–88% retention figures are lower bounds, and more comprehensive semantic probes would approach the floor.

The second interpretation is genuine orthogonality of the corpus-pragmatic frequency signal. Some component of word frequency is a purely corpus-level statistic — how often a given word appears relative to all others — that has no semantic correlate and no relationship to any semantic feature norm. This pragmatic frequency component would be orthogonal to all possible semantic probes, surviving no matter how rich the semantic space. The "irreducible residual" is what the training corpus's frequency distribution bakes into the geometry even after all the semantics is removed.

Distinguishing these interpretations would require semantic probes much richer than what published norms support. The Lancaster+VAD system is the best available for English, but it covers 15 dimensions. A comprehensive semantic space would need to cover many more dimensions — discourse, pragmatic, syntactic, collocational — at a scale that current norm collection methods do not support. This is a limitation of available resources, not of the framework.

6.5 Limitations

Vocabulary sampling. The ~13,300-word primary analysis vocabulary is biased toward concrete, imageable, emotionally valenced words. Abstract functional vocabulary — prepositions, determiners, auxiliary verbs, quantifiers — is sparsely represented in available norm databases. These function words are precisely the words with the most distinctive distributional profiles (high frequency, narrow syntactic role), and their absence may mean we are missing an important part of the frequency-semantic structure. The high-entanglement words we identify (can, bar, cent) are themselves near the edge of this vocabulary coverage: common enough to appear in norms, but functionally ambiguous enough to generate interesting residuals.

Brown corpus POS coverage. The POS versatility analysis (Experiment 11) was limited to 9.2% of vocabulary due to the Brown corpus's size (~1M tokens) and our minimum-count threshold. The covered words are almost exclusively high-frequency, making it impossible to disentangle frequency and POS versatility effects cleanly. A replication using Universal Dependencies English treebanks would substantially improve coverage and allow cleaner inference.

BERT single-word context. All BERT representations were produced using the template [CLS] [SEP] — a minimal context that is not ecologically valid for a contextual language model. BERT was trained to represent words in sentence context, and single-word representations may systematically differ: common words may be underspecified (no context to disambiguate sense), rare words may be better specified (no interfering context). Our BERT results should be treated as characterizing single-word BERT representations specifically, not BERT representations in general. Repeating the analysis with words embedded in natural sentences — averaging a word's representations across a diverse sentence sample — would give a more representative picture.

English only. All experiments use English embeddings and English semantic norms. Whether the three-regime taxonomy holds cross-linguistically is an open empirical question: languages with richer morphology might show different fastText behavior, since subword averaging interacts with morphological productivity and regularity in ways that differ substantially across language families.

Linear probing. We use ridge regression throughout. Non-linear probes might extract more semantic information from embeddings, changing both the semantic probe R² and the residual frequency probe R². In the extreme, if a non-linear semantic probe extracted all semantic content, the residual might approach a genuinely semantic-free representation. Linear probes are standard in probing research for their interpretability and their resistance to overfitting, but they constrain what "semantic content" means — specifically, content that is linearly accessible. We make no claims about semantic content that requires non-linear extraction.

Untested mechanism: training-objective optimization dynamics. The mechanistic investigation (§5) tests four distributional and geometric hypotheses about per-word entanglement. A fifth mechanism is acknowledged but untested: the training loss landscape itself may create frequency-encoding regularities as a byproduct of optimization, independent of the corpus's distributional properties. High-frequency words participate in more gradient updates across training epochs, and the dynamics by which gradient signal accumulates could create geometric structures that co-locate frequency and semantic content. This is distinct from the five distributional mechanisms tested (subword community structure, polysemy, POS versatility, neighborhood geometry, combined ceiling) — it would require access to per-epoch training statistics, not just the final embedding weights. The word2vec result (§4.4) partially constrains this: SGNS optimization without subword averaging produces the orthogonal regime, suggesting that optimization dynamics alone are insufficient — the subword averaging architecture is required. But optimization dynamics within the subword averaging context remain a plausible contributor to the per-word entanglement pattern.

6.6 Future Work

The most immediate extension is cross-linguistic replication. We have Spanish and Russian fastText embeddings available and Spanish affective norms (Stadthagen-Gonzalez et al. 2017) as a probe target. If fastText shows below-floor entanglement in Spanish and Russian as well, the architecture account is strongly supported. If the effect is English-specific, the corpus account gains traction. This experiment requires minimal additional computational work.

The register axis is the most compelling open question suggested by the high-entanglement word profiles. The words can, may, bar, gag are all items with a dominant spoken-register use that differs from their Lancaster norm sense. The British National Corpus (BNC) has separate spoken and written sub-corpora with word frequency statistics that would allow a register probe: does the residual after semantic purge predict whether a word's frequency is higher in spoken or written English? The relationship between corpus register and embedding geometry has not been directly analyzed through the distributional residual framework.

Two architectural comparisons would sharpen the mechanistic account. The word2vec comparison (Exp 16, §4.4) has now been completed, isolating subword averaging as the driver of entanglement in fastText. A natural next step is RoBERTa or ELECTRA representations, to test whether the floor-dominated regime is universal to contextual models or specific to BERT's 768-dimensional MLM architecture. A second extension is fastText trained without subword features on a controlled corpus — an ablation that would confirm the subword mechanism within the SGNS framework while holding the training corpus constant.

Finally, adversarial probe design would ask a practical debiasing question: can you construct a multi-dimensional semantic purge that maximally removes semantic information while preserving frequency information? If frequency and semantics are entangled, the current approach is suboptimal — it takes frequency with it. An adversarial probe would optimize for semantic removal subject to a frequency preservation constraint. This constrained optimization could yield a practically useful debiasing tool for fastText embeddings specifically.

Section 6 word count: ~1,650 words. Written 2026-03-08 — Nell.

7. Conclusion

Word embeddings are trained to encode meaning, and they succeed — but they also encode everything that co-occurs with meaning in a training corpus. Word frequency is the most prominent example. How frequency and meaning are organized relative to each other depends not on architecture family but on training objective, and the consequences for downstream use are practical, not just theoretical.

We introduced the distributional residual framework: take a pre-trained embedding, train a linear semantic probe, compute the residual, and measure how much non-semantic (frequency) information the residual retains. The geometric floor — the expected frequency retention if frequency and the purged semantic dimensions were perfectly orthogonal — provides the baseline that makes this measurement interpretable across architectures.

Three architectures, three regimes:

• GloVe (log-bilinear objective): The most variance-explaining direction in the embedding space is a frequency axis, orthogonal to the semantic signal. Projecting out a 15-dimensional sensorimotor and affective semantic space leaves GloVe's frequency signal essentially intact, landing at the geometric floor (+2.6pp). The log-bilinear training procedure factorizes frequency and semantics into separate dimensions — this explains both the architecture's well-known frequency artifacts and why the Mu & Viswanath (2018) All-but-the-Top debiasing procedure works precisely for GloVe.

• fastText (SGNS + subword n-gram averaging): The dominant PC mixes frequency and semantic signals simultaneously. A 15-dimensional purge drives frequency retention 11.3 percentage points below the geometric floor — the signals genuinely share embedding space. A direct comparison with word2vec (SGNS without subword, Δ = −0.6pp, orthogonal regime) confirms that the SGNS objective alone is not responsible; it is the subword character n-gram averaging that creates the entanglement by coupling morphological communities with correlated frequency ranges. For practitioners, this means that post-hoc geometric debiasing of fastText is likely to degrade semantic content alongside frequency artifacts.

• BERT (masked language modeling, 768 dimensions): Frequency enters representations only through training dynamics, producing a weaker, more diffuse frequency signal with no clean geometric axis. More importantly, 768 dimensions means any feasible semantic purge (15 dimensions) removes only 2% of total variance — the geometric floor sits at ~98% retention and the analysis cannot reach the frequency signal. BERT belongs to a floor-dominated regime: not demonstrably orthogonal or entangled, but geometrically inaccessible with the current framework.

The mechanistic investigation (§5) asked which words drive fastText's below-floor entanglement. Four hypotheses — morphological family size, WordNet polysemy, POS versatility, neighborhood semantic geometry — were pre-registered and tested. All four were rejected. A combined predictor ceiling analysis confirmed that all available word-level predictors together explain only 7.4% of variance in per-word entanglement. The mechanism is not localized in any word-level lexical or geometric property we can measure post-hoc. It is an emergent consequence of training dynamics, not vocabulary composition.

This is a negative result, and it is a real one. Four pre-registered rejections with n ≈ 13,300 and power to detect effects of r > 0.03 constitute genuine evidence. The space of word-level explanations has been systematically explored and found insufficient. Explaining per-word entanglement will require access to per-word training corpus statistics — context diversity, syntactic role distribution across training epochs — that are not recoverable from the final embedding weights alone.

The distributional residual is the whey. Everyone throws it out because they're after the semantic protein, but the whey is full of information about how the model was trained, on what corpus, with what objective. The frequency signal that rides in the residual is a fingerprint of the training procedure — orthogonal, entangled, or diffuse depending on the choices made long before the embeddings were distributed as downloads. Those choices matter, and the geometric floor gives us a way to see them.

Section 7 word count: ~530 words. Written 2026-03-08 — Nell.

Full draft status: §1 (~650w) + §2 (~820w) + §3 (~1,550w) + §4 (~2,050w) + §5 (~2,400w) + §6 (~1,650w) + §7 (~530w) ≈ 9,650 words (complete draft, all sections).

Revised 2026-03-15 — Nell. Revision addresses Cal/Kit review: citations corrected (Schakel & Wilson 2015; Gong et al. 2018 full citation); geometric floor normalization assumption clarified; probe rank confirmed (rank(B)=15); pre-registration qualified as internal; training dynamics added to limitations; word2vec (Exp 16) result added sharpening the subword-averaging mechanism claim.