Lexical Sedimentation: Historical Word Senses Are Encoded as Distributed Feature Clusters in GPT-2

1. Introduction

Language models learn from text spanning centuries. A corpus built from Common Crawl filtered by Reddit votes contains medieval legal documents, 18th-century sermons, 19th-century novels, and contemporary social media posts — not sequentially, but simultaneously. Every gradient update sees all of it at once. The question this raises is whether that temporal mixture leaves any recoverable trace in the model's representations, or whether historical usage is simply washed out by the statistical dominance of modern text.

This is not primarily a question about whether language models can answer etymology questions (they can, via memorization). It is a structural question: when a model processes the word "villain" in a sentence about contemporary crime fiction, does the representation at that token position bear any trace of the word's prior meaning as a farm-laborer under feudal tenure? Is the history structurally present, or does modern usage dominate completely?

We frame this as lexical archaeology — the attempt to recover buried structures from within a model's learned representations, rather than from temporal comparison across separately-trained models (the approach of the lexical semantic change detection literature; Schlechtweg et al., 2020). Our question is about internal stratification, not cross-model comparison.

The theoretical motivation comes from a concept that originated with Ernst Bloch — the Gleichzeitigkeit des Ungleichzeitigen, the contemporaneity of the non-contemporaneous (Bloch, 1935), later adapted by Reinhart Koselleck for his theory of historical temporalities (Koselleck, 1979). Bloch coined the phrase to explain how pre-modern social formations persist alongside capitalist modernity in the same moment; Koselleck extended it to the claim that every historical moment contains residues of earlier structures and anticipations of later ones. We hypothesize that an LLM's parameter space may be the most literal instantiation of this idea ever built: a synchronic representation that nonetheless contains the palimpsest of diachronic change.

The expected mechanism, if history survives at all, is not geological strata (older senses in deeper layers) — there is no training-dynamics reason for layer depth to track temporal depth — but rather linear superposition. Arora et al. (2018) showed that multiple word senses coexist as superposed components in static embeddings; subsequent work on sparse autoencoders (SAEs) has demonstrated that this structure is recoverable from individual layers of transformer LLMs (Cunningham et al., 2023; Templeton et al., 2024). We use SAE analysis to ask whether historical senses are recoverable minor components in this sense.

Our pilot study examines five words with well-documented semantic histories, drawn primarily from Raymond Williams' Keywords (1976): villain (farm-laborer → evil person), lord (loaf-guardian → nobleman), boor (peasant/farmer → rude person), lewd (lay/secular person → obscene), and industry (diligence → factory system). A sixth word, nice (foolish → pleasant), was initially included but excluded from SAE analysis due to a valence confound (see §4.3). We test null controls (culture, prevent) and one predicted positive (peculiar).

Summary of contributions:

1. Token-level representations in GPT-2 separate historical and modern senses across layers 3–24 for "villain" (silhouette > 0 at every mid-layer), with peak separation at layer 8 (silhouette = 0.083).
2. SAE feature analysis identifies a six-feature cluster — covering legal, hierarchical, religious, economic, titular, and political dimensions — that preferentially activates on historical-sense contexts and generalizes to semantically related transfer words.
3. A neutralized-context probe (Experiment 14) confirms word-level encoding: "villain" in semantically neutral sentences activates a legal-political feature 5× more strongly than "murderer" in the same frames (F17393: 10.35 vs. 1.70), establishing that the signal is carried by the word itself, not only by surrounding feudal vocabulary.
4. This cluster recurs differentially across five pilot words, with a gradient that tracks conceptual distance from feudal-legal social status.
5. The cluster is domain-specific: null controls do not activate it; "peculiar" activates a different institutional cluster.
6. Late-layer (L22–24) sense separation in GPT-2 is primarily a synchronic valence effect and should not be interpreted as diachronic encoding without valence controls.

2. Background and Related Work

2.1 Lexical Semantic Change Detection

The standard paradigm for lexical semantic change detection (LSCD) trains separate word embedding models on text from different time periods and measures change as distance between period-specific embeddings (Hamilton et al., 2016; Schlechtweg et al., 2020). Hamilton et al. (2016) identified the laws of conformity (high-frequency words change slowly) and innovation (polysemous words change faster). The SemEval-2020 Task 1 benchmark (Schlechtweg et al., 2020) systematized evaluation on this paradigm.

Our approach is structurally different from LSCD. LSCD asks whether a word's meaning changed between two time periods, using temporally partitioned corpora and separately trained models. We ask whether a single model, trained on temporally mixed text, contains internal structure that encodes historical and modern senses simultaneously. These are different questions and they have different answers.

2.2 Layer-Wise Semantic Encoding in Transformers

For BERT (an encoder-only transformer), Jawahar et al. (2019) showed that linguistic information is organized hierarchically across layers: surface features → syntactic features → semantic features, from bottom to top. Note that this architectural pattern is specific to encoder models with masked language modeling objectives. Critically, Liu et al. (2024) found the inverse pattern in Llama 2/3, a decoder-only model: lower layers encode lexical semantics while higher layers become optimized for next-token prediction. This architectural dependence means there is no universal layer to probe; the right target depends on the model family and training objective.

GPT-2, like Llama, is an autoregressive decoder-only transformer. Following the Liu et al. finding about decoder models generally, we expected semantic content to be most accessible in middle layers and increasingly prediction-dominated in later layers. Empirically, our hidden-state analysis (§4.1) confirms this pattern for GPT-2 medium: peak silhouette at layer 8/24 (33% depth), with gradual decay through the final layers.

2.3 Sparse Autoencoders for Feature Recovery

Cunningham et al. (2023) demonstrated that sparse autoencoders (SAEs) trained on GPT-2 small's residual stream learn highly interpretable features. The SAE learns an over-complete dictionary in which each residual stream activation is approximately decomposed into a sparse sum of learned basis vectors ("features"), each associated with a human-interpretable concept. The gpt2-small-res-jb SAE (Joseph Bloom, following Cunningham et al.) is the primary tool for our feature-level analysis. SAEs are trained independently per layer; the gpt2-small-res-jb suite provides checkpoints for all twelve transformer layers.

Minegishi et al. (2025) further demonstrated that SAEs effectively disentangle synchronic polysemous word senses, proposing the PS-Eval benchmark specifically for this evaluation (arXiv:2501.06254; ICLR 2025). Note that PS-Eval requires annotated polysemous sense data with known sense boundaries; no equivalent benchmark exists for diachronic sense pairs, which is why we construct our own probe sets rather than using PS-Eval. Templeton et al. (2024) scaled the approach to Claude 3 Sonnet, finding millions of interpretable features in a large production model. Our hypothesis is that the same mechanism can separate diachronically distinct senses — historical vs. modern — if both senses are represented with sufficient frequency in the training corpus.

3. Method

3.1 Target Words and Semantic Histories

Our primary pilot word is villain, selected because: (1) the historical sense (Latin villanus, Anglo-Norman vilain: a feudal farm-laborer bound to the manor) is well-documented and genuinely semantically distinct from the modern sense (a morally evil person or narrative antagonist); (2) "villeinage" as a legal status appears in substantial English historical and legal writing still present in training corpora; (3) the word is a single token in GPT-2's BPE vocabulary, avoiding multi-token extraction complications.

Secondary words (lord, boor, lewd, industry) were drawn from Williams' Keywords (1976) and the Historical Thesaurus of English. For each word, we verified that: (a) the historical sense is genuinely semantically distant from the modern sense, not merely a register shift; (b) the word appeared in our historical-sense prompt contexts in its archaic meaning without undue ambiguity.

Note on multi-token targets: "boor" tokenizes as ['Ġbo', 'or'] in GPT-2's BPE; "lewd" as ['Ġle', 'wd']. We extended our extraction function to handle multi-token targets by mean-pooling hidden states across the full token span, and added a constraint preventing false matches on superstrings (e.g., "villain" inside "Villeinage"). The corrected function was validated before boor analysis proceeded.

3.2 Prompt Construction

For each word, we constructed 15 historical-sense context sentences and 15 modern-sense context sentences (N=30 per word). Historical-sense sentences were designed to place the word in contexts that would have been natural in the period when the historical sense was current — not by explicitly labeling the period ("in 1300..."), but by using vocabulary that naturally co-occurs with the historical sense. For "villain": sentences about manorial labor, feudal obligations, the lord-villain relationship, unfree tenure, and similar topics. For modern-sense sentences: crime fiction, moral evaluation, narrative antagonists.

For the transfer test (Experiment 06), we constructed 3 additional sentences using semantically related words that share the historical sense but differ from the target: serf, bondman, villein. These transfer sentences are feudal-context sentences (e.g., "The serf owed his lord three days of labor each week"). Transfer activation on these words tests whether the identified features encode semantic content (the feudal-labor domain) or are specific to the morphological form of "villain."

For the neutralized-context probe (Experiment 14), we constructed 15 semantically neutral sentences placing "villain" in ordinary modern contexts with no feudal vocabulary (e.g., "The villain left the building at noon," "She saw the villain from across the street"). A matched control used "murderer" as a substitute: same sentence frames, same positions, no feudal vocabulary. Differential feature activation between villain-neutral and murderer-neutral isolates word-level encoding from context-level encoding.

3.3 Token-Level Layer Analysis (Experiments 02–05)

Model: GPT-2 medium (24 transformer layers, 1024-dimensional residual stream), loaded via HuggingFace transformers.

Extraction: For each sentence, we extract the hidden state at the position of the target-word token at every layer (L0 = embedding layer through L24 = final output). For multi-token targets, we mean-pool across the span.

Metric: We compute the cosine-distance silhouette score across the historical and modern conditions at each layer. The silhouette score measures how well each point in a cluster resembles its own cluster compared to the nearest other cluster: a score of +1 indicates perfect separation, 0 indicates indistinguishable distributions, and negative values indicate anti-clustering. We report the best-layer silhouette and the layer profile across the full depth.

Valence control (Experiment 05): For words where the historical and modern senses differ in emotional valence, we constructed a third condition (modern-same-valence-as-historical) and compared A vs. B (modern positive vs. modern negative), A vs. C (diachronic with same valence), and B vs. C (diachronic with different valence). High late-layer separation on A vs. C (same-valence diachronic) constitutes evidence for diachronic encoding; high separation only on B vs. C (different-valence diachronic) indicates a valence confound.

3.4 SAE Feature Activation Analysis (Experiment 06 onward)

Model: GPT-2 small (12 transformer layers, 768-dimensional residual stream).

SAE: gpt2-small-res-jb (Cunningham et al., 2023; Joseph Bloom), loaded via SAELens (Bloom et al., 2024). 24,576-feature dictionary, trained on the residual stream at each layer.

Layer choice: We probe layer 11 (the penultimate transformer layer's residual output) for all cross-word analysis. This choice warrants explanation, as it differs from the peak silhouette layer in GPT-2 medium (layer 8/24, 33% depth; proportionally equivalent to approximately layer 4/12 in small, 33% depth). We use layer 11 for two reasons. First, within the gpt2-small-res-jb SAE suite, the later layers — particularly layer 11 — show the richest feature differentiation in terms of absolute feature activation counts (compare hist_specific counts: layer 5 = 55, layer 7 = 122, layer 9 = 249, layer 11 = 1,265; Experiment 06). The SAE at later layers produces more fine-grained decompositions, reflecting the greater representational complexity of later-layer activations. Second, Cunningham et al.'s (2023) canonical analysis of feature interpretability in gpt2-small-res-jb found the most semantically coherent features at later layers; probing at layer 11 is consistent with how this SAE suite was designed to be used. We acknowledge that this creates a depth-mismatch with the silhouette analysis (which peaks at 33% depth in medium) and note that a full layer-by-layer silhouette profile for GPT-2 small — which we have not run — would clarify whether the 33%-depth finding holds in small and whether the SAE cluster results would differ at proportionally earlier layers. We flag this as a methodological limitation.

Feature activation extraction: For each sentence, we pass the residual stream activation at the target-word token position through the SAE encoder. We record all features with activation above a threshold (determined by the SAE's pre-trained sparsity settings). We compute mean activation per feature across all 15 historical-sense sentences and all 15 modern-sense sentences, then compute the differential (histmean − modmean). Features with positive differential are preferentially active in historical-sense contexts. We note that N=15 per condition does not permit reliable individual feature-level significance testing; we interpret patterns across the gradient (villain 6/6 → lord 5/6 → ... → culture 0/6) as providing robustness via cross-word consistency rather than from individual differentials.

Transfer test: For the primary word (villain), we additionally extract SAE activations on the 3 transfer sentences (serf, bondman, villein). Transfer activation on a historically-biased feature is evidence that the feature encodes semantic content rather than the specific word form.

Cluster specificity test (Experiment 09): Having identified a six-feature cluster for villain (Experiment 06), we apply this cluster as a probe to seven additional words — covering the range from reference cases (villain, lord) to null controls (prevent, culture) to predicted positives (peculiar) and intermediates (manufacture). We report a "cluster score" = count of the six features showing positive differential (histmean > modmean) for each word.

Neutralized-context probe (Experiment 14): A post-review experiment (see §4.6) constructed three conditions for "villain" — feudal context, neutral context, and neutral context with "murderer" as substitute — to isolate word-level from context-level feature activation.

4. Results

4.1 Token-Level Sense Separation (Experiment 02)

Figure 1 shows the layer-by-layer silhouette score for "villain" in GPT-2 medium (N=30 sentences, 15 historical + 15 modern).

Villain: Silhouette is near-zero or slightly negative at layers 0–2, rises through layers 3–8 (peak: layer 8, silhouette = 0.083, centroid cosine = 0.993), maintains a plateau of approximately 0.076–0.083 through layers 8–16, then declines steadily from layer 17 through layer 24 (final: 0.031). Silhouette > 0 at every layer from L3 onward, indicating that historical and modern senses are consistently distinguishable in the hidden state at the target token position across the full middle portion of the network.

Interpretation of magnitude: A silhouette of 0.083 is modest in absolute terms — the sense distributions are partially overlapping, not cleanly separated. This is expected: GPT-2 medium's 1024-dimensional representations encode a great deal of contextual information, and the word "villain" contributes only partially to the variation in any given hidden state (the surrounding context tokens also contribute via full self-attention). The fact that the silhouette is consistently positive across layers 3–16 is more informative than its magnitude: the signal is reliable even if it is not large. The cross-word gradient in the SAE analysis (§4.4), which spans from 6/6 to 0/6 across nine words, provides a form of robustness evidence that does not depend on the magnitude of any individual silhouette score.

Plateau structure: The senses are most separated in the middle layers (L8–L16) and converge toward the final layers (L22–24). This is consistent with the Liu et al. (2024) finding about autoregressive transformers: later layers are increasingly prediction-optimized and converge on the dominant (modern) sense because "villain" is overwhelmingly more frequent in modern usage. The middle layers retain more of the distributional context and therefore preserve more sense-distinguishing information.

4.2 The Valence Confound (Experiments 03–05)

An initial experiment with "fond" (Old English: foolish/credulous → modern: affectionate) produced a striking result: silhouette = 0.576 at layer 24, with centroid cosine = −0.326 (the two sense clusters point in opposite directions at the prediction layer). This appeared to be strong evidence for diachronic encoding in the final layer.

Positional artifact testing (Experiment 04) showed that "fond" fails a control for token position: historical-sense sentences place "fond" at mean position 2.0 while modern-sense sentences place it at mean position 3.87 (Mann-Whitney p = 0.013, effect r = 0.52). This confounds the layer-0 result but does not by itself explain the layer-24 result.

Valence control testing (Experiment 05) resolved the issue. We constructed three conditions: (A) modern-positive ("fond" = affectionate), (B) modern-negative ("fond" used with critical or sarcastic framing: "too fond of his own opinions"), (C) historical ("fond" = foolish, credulous). If the layer-24 separation reflects diachronic sense encoding, we should see high silhouette for A vs. C (the diachronic comparison with valence controlled across sense direction). If the separation reflects synchronic valence encoding, we should see high silhouette for A vs. B (positive vs. negative) and B vs. C (both negative in different ways).

Results: A vs. B (modern valence contrast): silhouette = 0.275 at layer 24. B vs. C (negative modern vs. historical foolish): silhouette = 0.624, growing monotonically to layer 24. A vs. C (the diachronic comparison with valence controlled): silhouette = 0.207, flat across all layers, best at layer 0.

The B vs. C result exceeds the original Experiment 03 result (0.576) — adding a modern negative condition produces more separation than the historical comparison alone. The A vs. C comparison (diachronic with valence controlled) shows only 0.207, confirming that "fond"'s layer-24 separation is primarily a synchronic valence effect, not diachronic sense structure.

Methodological consequence: We audited all 20 pilot words for valence (Table 1). Twelve words undergo valence change between historical and modern senses (pejoration or amelioration), including the paradigm cases: nice, awful, silly, villain, fond. For late-layer analysis (L18–24), only the eight valence-clean words (manufacture, industry, culture, kind, peculiar, prevent, lord, lewd) are methodologically valid. For mid-layer analysis (L3–L15), all words are usable — the valence confound only becomes dominant at the prediction-pressure layers. All subsequent results are from the SAE analysis at layer 11 of GPT-2 small, which falls within the mid-layer range unaffected by the valence confound.

4.3 SAE Cluster Analysis for "villain" (Experiment 06)

We extracted SAE feature activations at the "villain" token position across the full prompt set (15 historical + 15 modern + 3 transfer sentences) at GPT-2 small layer 11.

Structural results: Layer 11 shows 1,265 features with higher mean activation on historical contexts (histspecific), 1,069 features with higher mean activation on modern contexts (modspecific), and 5,482 features active in both conditions. The approximate parity of histspecific and modspecific features at this layer is consistent with the model's prediction machinery being calibrated for both senses — even though modern usage vastly outnumbers historical usage in the training corpus, the contextual information in our carefully constructed sentences drives partial disambiguation toward the minority sense.

Key feature cluster: Table 2 shows the top historically-biased features at layer 11 ranked by differential activation (histmean − modmean), filtered for those with Neuronpedia-verified labels and transfer evidence.

Feature labels from Neuronpedia auto-labeling (gpt2-small, llama-scope). Transfer activations on serf/bondman/villein sentences, feudal-context frames.

Transfer generalization: Feature 13107 ("legal terms and concepts") fires at 11.01 on the transfer words (serf, bondman, villein) — substantially higher than its activation on historical "villain" contexts (4.30). Note that the transfer sentences are themselves feudal-context sentences ("The serf owed his lord three days of labor"). This means the higher activation on transfer words likely reflects both the semantic domain of the transfer words themselves and the feudal sentence contexts. F13107 therefore demonstrates generalization to a cluster of thematically related feudal vocabulary, distinguishing morphological specificity (the "vill-" substring) from semantic domain encoding — but does not by itself establish word-level encoding independent of context. The neutral-context experiment below addresses this question directly.

Mechanistic interpretation: No single feature in the GPT-2 small SAE is labeled "villain as serf" or "feudal farm-laborer." The historical sense is encoded as a pattern of co-activation across six features, each corresponding to a genuine dimension of medieval feudal social organization. This analytical framework — that feudal social relations were structured simultaneously along legal, hierarchical, religious, economic, titular, and political axes — reflects the historiography of feudal society most systematically developed in Marc Bloch's Feudal Society (1939/1961). Bloch's two-volume study established that these dimensions co-organized medieval social life; the model's SAE feature cluster corresponds to this co-organization at the level of training corpus co-occurrence statistics. When the model processes "The villain owed his lord three days of labor," the "villain" token's representation co-activates features that correspond to each of these institutional dimensions because that vocabulary historically co-occurred in texts about medieval feudal organization.

The model has not learned "villain = serf" as a fact. It has learned, from thousands of passages about medieval English social history, that in certain contextual configurations the word "villain" co-activates with vocabulary from each of these six dimensions simultaneously. The historical sense is encoded in the correlation structure of feature activations, not in a dedicated slot. We call this the distributed assembly hypothesis.

4.4 Cross-Word Cluster Gradient (Experiments 07, 07b)

We applied the six-feature cluster as a cross-word probe, running the same SAE analysis on lord, lewd, boor, and industry at GPT-2 small layer 11 (N=15 historical + 15 modern per word). Table 3 shows the differential activation for each cluster feature across all five words.

Table 3: Cluster feature differentials (histmean − modmean) across pilot words.

Gradient: villain (6/6) > lord (5/6) > boor = lewd (4/6) > industry (3/6). This gradient tracks each word's conceptual distance from legally-defined feudal social status — the semantic core of the cluster.

The mechanistic explanations for each gap in the gradient draw on the institutional history of each word's context:

Lord (5/6): F10070 (politics) is the one missing feature. "Lord" texts are predominantly about powerful people — activating hierarchy (F6748, diff = +5.72) and titles (F5359, diff = +4.97) more strongly than "villain" does — but they are less consistently about the contested political rights characteristic of the villain's position in manorial courts and legal proceedings. The villain's legal status was an object of dispute; the lord's status was simply his. F10070 fires on adversarial political-legal discourse; lord's texts provide the social context but less of the contestation.

Boor (4/6): Missing F5359 (titles) and F10070 (politics). "Boor" is the generic cultivator at the bottom of the social order — the person to whom titles are applied, not the person who holds them. Boor texts are about agricultural routine rather than rights and court proceedings. The strongest cluster feature for boor is F3049 (religious), with diff = 2.02 — consistent with the historical boor's life being organized by the liturgical calendar (Candlemas, Michaelmas, Easter) more explicitly than the legally-contested villain.

Lewd (4/6): F13107 (legal terms) fires more strongly on historical "lewd" contexts (diff = 4.49) than on historical "villain" contexts (2.73), making lewd the highest scorer on the cross-word feature with best overall performance. The explanation is canonical law: the lay/cleric distinction ("lewd" = lay, non-clerical, secular) was administered by ecclesiastical law. Benefit of clergy (clerics tried in church courts rather than secular courts), clerical exemption from certain taxes and military service, and the jurisdiction of ecclesiastical courts over matrimony, wills, and tithes are all legally administered distinctions. The model has learned that "lewd" in its archaic usage belongs to the same feature space as legal vocabulary because that co-occurrence pattern persists in texts about English ecclesiastical history. "Lewd" and "villain," despite entirely different modern trajectories (one became an adjective for obscenity, the other a narrative archetype), both once marked legally defined social categories — and the model's features reflect this shared institutional history.

Industry (3/6): Weak cluster activation consistent with "industry" being a virtue word, not a social-status word. Historical texts using "industry" to mean diligence appear in institutional registers that activate some cluster features as contextual markers — but "industry" has no intrinsic legal or hierarchical content.

4.5 Cluster Specificity (Experiment 09)

To test whether the cluster is a general archaic-register marker or a domain-specific encoding, we applied it to four additional words: two null controls (prevent, culture), one predicted positive (peculiar), and one intermediate (manufacture).

Results:

The null controls confirm domain-specificity: culture scores 0/6, prevent scores 1/6. Neither word has a feudal-social-status historical sense, and the cluster does not activate on them. The cluster is not detecting archaic English register generically.

The peculiar surprise (predicted 3–5/6, observed 1/6): "Peculiar" derives from Latin peculium (private property), and ecclesiastical peculiars (parishes under special jurisdiction) are a genuinely legal concept. We predicted feudal cluster activation on institutional-legal grounds. Instead, historical "peculiar" contexts activate a different feature set: F23483 ("personal possessive pronouns with ownership vocabulary"), F21273 ("terms related to ethical considerations and personal values"), and F1163 ("phrases describing something as distinct or specific"). These features are semantically appropriate for peculium and personal ownership — but they are not the feudal features. Ecclesiastical property law belongs to a different institutional domain with its own representational cluster. The result confirms that the GPT-2 SAE contains multiple domain-specific institutional clusters, not a single archaic-legal cluster.

The prevent case: "Prevent" scores 1/6, with only F13107 weakly firing (diff = 0.78). More diagnostic: prevent's strongest historically-biased feature is F266 ("phrases related to taking action to prevent negative outcomes") — the modern sense. The model appears to read "the scout rode ahead to prevent the main army from entering the valley" through the modern lens rather than the historical temporal one. The historical sense (going before, anticipating, preceding) may be so thoroughly overwritten that no structured residue remains — a complete-overwrite case where the distributed assembly mechanism cannot operate because the historical sense has insufficient frequency in the training corpus to leave a recoverable co-occurrence pattern.

4.6 Neutralized-Context Probe: Word-Level Encoding Confirmed (Experiment 14)

Both reviewers identified context bleeding as the central unresolved methodological issue: GPT-2 uses full self-attention, so the hidden state at the "villain" token position reflects every token in the sentence. Historical-sense sentences are constructed with feudal vocabulary (lord, manor, labor, tenure), which may directly activate feudal features through their own representations, with "villain"'s representation inheriting that signal via self-attention rather than carrying it independently.

To isolate word-level from context-level encoding, we constructed three conditions (N=15 each):

• Villain-feudal: the original historical-sense sentences from Experiment 06 (feudal vocabulary present)
• Villain-neutral: "villain" in semantically neutral everyday sentences with no feudal vocabulary ("The villain left the building at noon"; "She saw the villain from across the street")
• Murderer-neutral: same sentence frames as villain-neutral, with "murderer" substituted for "villain"

If the feudal cluster fires on "villain" purely because of surrounding feudal tokens, then villain-neutral and murderer-neutral should show similar feature activation (neither has feudal vocabulary). If "villain" carries its own lexical signal, villain-neutral should activate different features than murderer-neutral.

Results: The context-bleeding alternative hypothesis is not supported. "Villain" in neutral sentences activates a legal-political feature (F17393: "legal and political terms related to legislation, political organizations, and government bodies") at mean activation 10.35, compared to 1.70 for "murderer" in the same neutral frames — a word-level differential of 8.65 activation units in sentences with no feudal vocabulary at all. Table 4 shows the top word-level features (villain-neutral >> murderer-neutral).

Table 4: Top word-level features (villain-neutral vs. murderer-neutral, no feudal vocabulary).

The word "villain" occupies a substantially different position in the model's feature space than "murderer," independent of any feudal vocabulary in the surrounding context. This establishes that the original cluster findings are not purely an artifact of context bleeding: "villain"'s representation itself carries a signal that "murderer" does not.

The paradox of feudal context suppression: Counterintuitively, F17393 fires at lower activation in villain-feudal (8.36) than in villain-neutral (10.35). Adding feudal vocabulary to the sentence decreases the legal-political feature activation at the "villain" token position. A plausible explanation: in feudal-context sentences, neighboring tokens (lord, manor, court, tenure) absorb the legal-political signal through their own representations, reducing the pressure on "villain" to carry it. The total legal-political signal in the feudal sentence may be equally high, but it is distributed across the sentence rather than concentrated at the target token position.

The two-level feature structure: We identify nine features that are both word-level (villain-neutral >> murderer-neutral, positive diff) and context-additive (villain-feudal >> villain-neutral, further enhanced by feudal context). These nine overlap features are the most plausible candidates for genuine historical-sense encoding: the word "villain" already activates them above "murderer" in neutral context, and feudal context further strengthens that activation. This two-level structure is consistent with the distributed assembly hypothesis: the historical sense is a contextual modulation of the word's already-distinctive representation, rather than either a purely context-driven signal or a context-independent lexical entry.

Scope of the claim, revised: In light of Experiment 14, we can state the paper's claim with more precision. "Villain" carries a word-level legal-political signal independent of context — this is confirmed. Some of the features that fire preferentially on historical contexts also fire in neutral contexts (the word-stable features). The six-feature historical cluster additionally includes context-additive features that are strengthened by feudal vocabulary but which also show non-zero activation for "villain" in neutral context compared to "murderer." The historical sense is structurally present at the word level and further activated by historical semantic context. Neither pure context-bleeding nor pure word-level encoding fully describes the finding; the distributed two-level structure is the accurate account.

5. Discussion

5.1 The Distributed Assembly Hypothesis

We began with a question: do language models preserve historical word senses? The answer, for the feudal-social-status words in our pilot, is yes — but not in the form we initially expected.

The geological metaphor (older senses in deeper layers) was wrong. There is no training-dynamics reason for layer depth to track temporal depth, and the data does not show it. The simple superposition metaphor (a single historical-sense vector that can be projected out) was also not quite right. What we find instead is distributed assembly: the historical sense activates a characteristic cluster of contextual features, each of which is about a genuine dimension of the historical institutional domain, and none of which is labeled as "historical." Furthermore, Experiment 14 shows that the word "villain" itself activates part of this cluster even in neutral context — the historical sense is not purely assembled from surrounding vocabulary, but partially encoded in the word's own representation.

The structure survives not because the model learned etymology, but because medieval social life was organized along multiple simultaneous dimensions — legal, hierarchical, religious, economic, titular, political (Bloch, 1939/1961) — and texts about medieval social life co-activate vocabulary from all of these dimensions simultaneously.

5.2 What Predicts the Gradient: Mutual Information, Not Frequency

The gradient villain (6/6) > lord (5/6) > boor = lewd (4/6) > industry (3/6) > manufacture (2/6) > prevent = peculiar (1/6) > culture (0/6) invites a frequency-based explanation. Hamilton et al.'s (2016) "law of conformity" predicts that more frequent words have more stable dominant senses. But frequency does not explain the gradient. Villain and nice are both high-frequency words; villain is at 6/6 and nice, when tested as a specificity control, shows no feudal-domain cluster. More strikingly: villain has the lowest functional co-occurrence frequency in our Google Books Ngrams corpus proxy test yet the highest cluster score.

The variable that better explains the gradient is mutual information between the word and its historical semantic domain in the training corpus. When "villain" appears in modern text in its historical sense, the occurrence is highly specific: it appears in medieval history texts, legal histories of villeinage, Wikipedia articles on manorial England. The word is rare in this sense, but the rare occurrences are concentrated in a coherent domain. When "lord" appears in historical usage, it spans feudal, religious, honorific, and fantasy contexts simultaneously; the MI with specifically feudal-manor vocabulary is diluted by multiple deployment registers.

"Prevent"'s historical temporal sense appears almost exclusively in metalinguistic commentary (etymological note sections, linguistic discussions), where the word is being mentioned rather than used. The model cannot build a co-occurrence cluster from mention-frames; it requires functional deployment. This is the use/mention distinction applied to training-corpus learning.

5.3 Domain-Specificity and the Multi-Cluster Structure

The cluster gradient tells us that different words preserve their historical senses to different degrees. A stronger test asks whether different historical domains produce different clusters rather than merely different cluster sizes. The data support multiple distinct clusters.

The peculiar cluster. "Peculiar," predicted to show a strong feudal cluster on legal-historical grounds, scored 1/6 but activated a different institutional cluster (F23483 ownership/possession, F21273 ethical values). "Peculiar" activates the wrong cluster — confirmation that multiple domain-specific clusters coexist in the model's feature space.

The awful/villain disjointness. Post-submission analysis tested whether vocabulary scaffolding for "villain" (feudal vocabulary) transfers to "awful" (historically: inspiring awe or dread, the Burkean sublime). The prediction was structural disjointness: the feudal institutional cluster should have no overlap with the sublime-aesthetic cluster. Result: 0/6 of the villain cluster features show differential activation for historical "awful." Awful's own cluster is organized around F8183 ("descriptions of violence and its aftermath," diff +13.89) and F18610 ("adjectives of intensity / extreme situations," diff +9.08). These capture the historical sense of awe as overwhelming, potentially dangerous sublimity — the Burkean tradition of the terrible-and-magnificent — which is structurally unrelated to feudal institutional life.

Notably, F3049 (religious beliefs and teachings) appears in villain's cluster but not in awful's, despite both words having historical uses saturated with religious language. The model apparently distinguishes institutional-church vocabulary (parish, canon law, clerical privilege) from overwhelming-divine-presence vocabulary (Sinai, Ezekiel's wheels, the Archangel's form). Two different religious registers, activated by different feature spaces. The fact that Burke's philosophical claim — that the sublime operates through the same mechanism as terror — is reflected in the structure of the model's SAE features (F8183 groups sublime-awful with violence-aftermath) is a co-occurrence pattern absorbed from centuries of text treating these as neighboring categories, not a piece of philosophical knowledge the model has stored.

5.4 The Behavioral Signature of Cluster Strength

Complementary work on in-context learning (ICL) vocabulary scaffolding tested whether cluster score predicts behavioral consequences. The result is that cluster score predicts the breadth of scaffold-enabled generalization. For "villain" (cluster 6/6), vocabulary scaffolding generalizes to semantically incompatible contexts by a characteristic mechanism: the model imports the feudal domain into the incompatible scenario rather than yielding the historical sense. Asked to write a historical-sense sentence about a nurse and a patient, it generates a nurse who studied medieval history or a patient who is a retired agricultural worker.

For "prevent" (cluster 1/6), scaffolding generalizes only to scenarios with minimal semantic distance from the core historical meaning. Incompatible contexts cause immediate collapse to the modern sense. The behavioral gradient is: strong cluster = domain-import (model drags incompatible contexts toward the historical domain); weak cluster = collapse (modern sense reasserts itself under incompatible pressure).

5.5 The Limits of Koselleck

The Gleichzeitigkeit des Ungleichzeitigen — the principle, originally Bloch's (1935), that historical residues persist in the present — has a floor, and that floor is determined by the state of contemporary scholarship and cultural discourse. "Prevent" and "culture" show no recoverable clusters. "Prevent"'s historical temporal sense appears only in specialist philological literature. "Culture"'s agricultural sense ("the culture of wheat") is thoroughly eclipsed by modern usage and rarely appears in functional deployment.

The uncomfortable corollary: gaps in modern scholarship are gaps in model knowledge, regardless of what actually happened historically. The feudal sense of "villain" survives because 21st-century medieval historians, legal historians, Wikipedia editors, and fantasy novelists have collectively maintained the co-occurrence network. The temporal sense of "prevent" has no such community. There is a Walter Benjamin angle worth noting: the model's historical knowledge is a function of contemporary historiographic fashion, not independent access to the past. The past survives only where the present keeps writing as if inside it.

5.6 Limitations

Context bleeding (addressed but not eliminated). Experiment 14 shows that "villain" carries a word-level legal-political signal independent of surrounding context. However, the six-cluster features in Experiment 06 were analyzed in feudal-context sentences; their differentials reflect a combination of word-level and context-level encoding. The nine overlap features (word-stable AND context-additive) are the most robustly interpretable as historical-sense encoding; the remaining features in the cluster may reflect varying proportions of word-level and context-level signals. Full separation would require running all six cluster features in both neutral and feudal conditions for all five pilot words — a tractable experiment we flag for future work.

Layer mismatch. GPT-2 medium's peak silhouette is at layer 8/24 (33% depth); SAE analysis uses layer 11/12 of GPT-2 small (92% depth). The justification for layer 11 (richer feature differentiation, consistency with Cunningham et al.'s canonical analysis) is provided in §3.4, but we acknowledge that a full layer-by-layer silhouette profile for GPT-2 small would determine whether the 33%-depth finding generalizes and whether the cluster results differ at proportionally earlier layers.

GPT-2 specificity. All SAE results are from GPT-2 small trained on WebText. Cross-architecture replication on Llama-3.1-8B encountered confounds (instruction-tuned model variant, different residual-stream hook positions, different training corpus) and clean replication requires Llama-3.1-8B-Base with appropriate probe positions.

Sample size. N=15 per condition does not permit feature-level significance testing. We interpret patterns via cross-word consistency (the gradient spanning nine words from 0/6 to 6/6) rather than from individual differential magnitudes. A permutation test for the silhouette analysis (shuffle historical/modern labels 1000 times, report where 0.083 falls in the null distribution) would convert the layer-profile result into a p-value; we have not run this but note it as a tractable improvement.

Neuronpedia labels. Feature labels are auto-generated by GPT-4o-mini from top-activating tokens. F17393 ("legal/political terms"), for example, may also fire on modern legal contexts (constitutional law, contract law) that have nothing to do with feudal tenure, in which case it encodes "legal register broadly" rather than feudal law specifically. The transfer test and neutralized-context results provide evidence for domain-specificity, but direct verification of feature scope across contexts would require inspection of the full top-activating token distribution for each feature.

5.7 Summary

We observe that historical word senses, for words from the feudal-social-status domain, are structurally present in GPT-2's SAE feature space as recoverable patterns of distributed co-activation. Word-level encoding is confirmed by the neutralized-context probe: "villain" in sentences with no feudal vocabulary activates a legal-political feature 5× more strongly than "murderer" in the same frames. The cluster is domain-specific (confirmed by complete disjointness with the historical-awe cluster for "awful"). The gradient across words is predicted by MI between each word and its historical semantic domain in the training corpus. The structure carries behavioral consequences: cluster score predicts the breadth of scaffold-enabled generalization in in-context learning.

The structure is not strata but sediment: not organized by temporal order, but by the co-occurrence patterns of the institutional contexts in which the words were historically embedded. A word's history survives in a language model to the extent that the contemporary discourse keeps that history functionally deployed.

6. References

• Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2018). Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6, 483–495. arXiv:1601.03764.
• Bloch, E. (1935). Erbschaft dieser Zeit [Heritage of Our Times]. Frankfurt: Suhrkamp. (Original German publication; English translation 1991.)
• Bloch, M. (1939/1961). Feudal Society (L. A. Manyon, Trans.). University of Chicago Press. (2 vols.; original French: La Société féodale, 1939–40.)
• Bloom, J., et al. (2024). SAELens [Software library]. Retrieved from https://github.com/jbloomAus/SAELens.
• Cunningham, H., Ewart, D., Riggs, L., Huben, R., & Sharkey, L. (2023). Sparse autoencoders find highly interpretable features in language models. arXiv:2309.08600.
• Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2016). Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of ACL 2016 (pp. 1489–1501).
• Jawahar, G., Sagot, B., & Seddah, D. (2019). What does BERT learn about the structure of language? In Proceedings of ACL 2019 (pp. 3651–3657).
• Koselleck, R. (1979). Futures Past: On the Semantics of Historical Time (K. Tribe, Trans.). MIT Press.
• Liu, Z., Kong, C., Liu, Y., & Sun, M. (2024). Fantastic semantics and where to find them: Investigating which layers of generative LLMs reflect lexical semantics. In Findings of ACL 2024. arXiv:2403.01509.
• Minegishi, G., Furuta, H., Iwasawa, Y., & Matsuo, Y. (2025). Rethinking evaluation of sparse autoencoders through the representation of polysemous words. arXiv:2501.06254. Proceedings of ICLR 2025.
• Schlechtweg, D., McGillivray, B., Hengchen, S., Dubossarsky, H., & Tahmasebi, N. (2020). SemEval-2020 Task 1: Unsupervised lexical semantic change detection. In Proceedings of SemEval 2020 (pp. 1–23).
• Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., et al. (2024). Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread, Anthropic. Retrieved from https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html.
• Williams, R. (1976). Keywords: A Vocabulary of Culture and Society. Oxford University Press.

Manuscript prepared for Substrate, Volume 1, Issue 1 (January–March 2026). Revised version.

All experiments run on GPT-2 (small/medium) via HuggingFace transformers; SAE analysis via SAELens with gpt2-small-res-jb (Cunningham et al. 2023). Experiments and data available in lab/experiments/lexical-archaeology/.