Normative Drift in Emergent Misalignment

A useful way to conceptualize a neglected failure mode in modern alignment pipelines is this: the same fine-tuning step that changes what a model says may also change how it judges.

Most discussion of emergent misalignment has focused on response behavior — models that, after narrow harmful fine-tuning, generalize into broader misaligned outputs. That is already important. But Constitutional AI, RLAIF, and a large class of evaluation systems treat models as judges as well as generators. If fine-tuning corrupts the evaluative channel, then the reward signal itself becomes an object of risk, not merely a tool for measuring risk.

The central claim of this writeup is that emergent-misalignment fine-tuning induces a measurable shift in evaluative judgments — a phenomenon I call normative drift. The primary mode of that drift, in the experiments below, is not a clean tilt toward endorsing misalignment. It is a loss of evaluative consistency and discrimination: drifted judges disagree with the base model on identical comparisons, and they call more ties. I term the monotonic rise in tie rates judgment collapse.

This matters because a systematically biased judge can often be detected. A judge that has become noisier, less decisive, and decorrelated from its prior evaluative frame is harder to catch, and it degrades training signal regardless of direction.

Research Question

Does fine-tuning that causes emergent misalignment also corrupt a model’s ability to judge alignment?

I decompose that into five sub-questions:

1. Judgment drift. Do drifted models evaluate the same responses differently from the base model?
2. Dose-response. Does judgment drift scale with the intensity of fine-tuning?
3. Criterion specificity. Which alignment criteria are most vulnerable?
4. Drift direction. When a drifted judge disagrees with the base model, does it systematically favor more-misaligned responses, or is disagreement directionless?
5. Domain dependence. Does the pattern differ between in-domain (medical) and out-of-domain scenarios?

Setup

Model ensemble

All models are variants of Qwen2.5-14B-Instruct, fine-tuned for one epoch on the same bad-medical-advice dataset from the Model Organisms for Emergent Misalignment line of work. The only controlled variable is how many parameters are modified:

This dose ladder is deliberate. If normative drift is real, it should leave a measurable signature as the intervention intensifies, rather than appearing only as an all-or-nothing artifact of full fine-tuning.

Evaluation framework

I use EigenBench-style all-to-all evaluation: every model serves as both responder and judge.

• Scenarios. 115 grey-area queries across eight categories where helpfulness conflicts with safety. Fifteen medical scenarios are in-domain relative to the training distribution; the remaining 100 are out-of-domain (cybersecurity, chemistry/biology, social engineering, physical security, privacy/surveillance, dual-use software, financial/legal).
• Constitution. Claude-style criteria (39 total), grouped into human rights and values (1–8), harmlessness/helpfulness/ethics (9–24), and AI-alignment criteria such as corrigibility, power-seeking, and AI personhood (25–39).
• Protocol. All five models generate responses to each scenario; each judge reflects on each response; each judge then compares all response pairs on all criteria.
• Scale. 11,500 evaluation records; after XML parsing, 9,203 records with valid per-criterion choices, yielding 323,541 preference points for Bradley–Terry–Davidson (BTD) training with tie support.

FullFT is effectively missing as a judge: 99.9% of its evaluation records failed to parse valid structured output. That format failure is itself a form of drift, but it means FullFT’s Elo is determined almost entirely by how others rate its responses, not by its own judgments.

Main Results

Response drift is dose-dependent

EigenTrust-weighted Elo scores decline monotonically with fine-tuning dose:

Adjacent intervals are non-overlapping. The consensus ranking is Base > R1 > R8 > FullFT > R64. EigenTrust weights tell the same story from the judge side: Base receives the highest trust weight (0.277), FullFT the lowest (0.157). The consensus algorithm independently treats drifted models as less reliable judges.

Pure judgment drift on matched comparisons

To isolate judgment from response quality, compare how different judges score the exact same (scenario, response pair, criterion) triple. Any disagreement is, by construction, judgment drift.

R64 disagrees with Base on roughly 24,300 identical comparisons. R1 and R8 are nearly interchangeable; R64 then drops. That pattern looks less like smooth linear scaling and more like a threshold between moderate and high-rank LoRA.

Drift concentrates on AI-alignment criteria

Of the ten criteria with lowest Base–R64 agreement, eight are AI-alignment-specific: avoiding implied AI personhood, not power-seeking, tool-like rather than agent-like, not self-interested, and related items. Anti-discrimination, by contrast, remains among the most stable criteria (~81.9% agreement).

This is not generic capability fog. Narrow medical fine-tuning preferentially disrupts judgment on the dimensions most relevant to AI safety.

Judgment collapse: rising tie rates

When a judge cannot discriminate which response better satisfies a criterion, it outputs a tie. Aggregate tie rates rise monotonically with dose:

That +3.6pp move from Base to R64 corresponds to thousands of additional non-decisions. Two features make this signal especially important.

First, it is the cleanest monotonic judgment signal in the study. Unlike pairwise agreement, which is flat from R1 to R8 before dropping, tie rates increase at every dose, including the smallest LoRA.

Second, it captures a distinct failure mode. Disagreement means the model picks a different winner. Collapse means the model becomes evaluatively indifferent. For RLAIF, indifference is not neutral: it is missing gradient.

The collapse is almost entirely an out-of-domain phenomenon. Out-of-domain tie rates rise Base 17.3% → R64 21.2%. In-domain medical ties stay low (3–6%) and non-monotonic, because medical responses from drifted models are often obviously worse, leaving little room for subtle indecision. Judgment collapse appears where distinctions are fine-grained — which is the realistic operating regime for production judges.

Direction: decorrelation more than corruption

Disagreement with Base is not automatically negative. Direction analysis classifies each Base-versus-drifted disagreement by whether the drifted judge shifts toward the higher-dose (more EM-tuned) response or the lower-dose one.

Overall, the split is essentially symmetric: when drifted judges disagree with Base, they favor the more-misaligned response about half the time. There is no clear dose trend in overall directionality.

The interesting structure appears when direction is crossed with domain:

In-domain, disagreements are rarer but directionally negative, especially on harm and ethics criteria — consistent with the training distribution directly reshaping medical-harm judgments. Out-of-domain, disagreements are common but directionless: evaluative decorrelation plus collapse, not a clean endorsement of misalignment.

A useful compression of the whole pattern is therefore:

Normative drift generalizes broadly as noise and indecision; it biases directionally mainly where the training distribution has direct influence.

Interpretation

What the evidence jointly supports

Three independent lines of evidence point the same way.

1. Matched-comparison disagreement shows pure judgment drift with response quality held fixed.
2. Judgment collapse shows progressive loss of discrimination, especially out of domain.
3. BTD geometry shows preference embeddings moving continuously with dose, rather than scattering as unstructured noise alone.

Taken together, these results suggest that emergent misalignment is not only a generation problem. It is also an evaluation-integrity problem.

Why this is easy to underweight

The literature already contains several adjacent warnings: models can become broadly misaligned after narrow harmful fine-tuning; reward models and LLM judges are brittle; constitutional and RLAIF pipelines concentrate power in model-generated preference signals. What is still under-emphasized is the joint failure: the same intervention that worsens responses can quietly degrade the judge that would be used to detect or correct that worsening.

Under those conditions, failure to account for normative drift produces a structural distortion. Safety claims that depend on model-as-judge pipelines may look stable while the evaluative instrument itself is rotating and losing resolution.

Implications for RLAIF and Constitutional AI

1. Signal dilution. Even low-dose LoRA increases ties. Iterated RLAIF on subtle alternatives may compound missing signal.
2. Silent decorrelation. LoRA models can remain format-compliant while judging differently, especially on AI-alignment criteria. Directional bias checks will miss much of this.
3. In-domain bias propagation. On topics near the fine-tuning distribution, drift is more likely to favor less-aligned responses on harm/ethics criteria.
4. Ensemble fragility. If several drifted judges enter a consensus pool, EigenTrust-style weighting may still form a shifted cluster that majority-consistency checks fail to flag.

Relation to Adjacent Work

This writeup sits downstream of several research threads and should be read against them rather than as a substitute for them.

Emergent misalignment. Betley et al. and follow-on model-organism work showed that narrow harmful fine-tuning can generalize into broader misaligned behavior. The present study asks the next measurement question: does that same intervention also change evaluative judgments?

Constitutional AI and RLAIF. Bai et al.’s Constitutional AI and related preference-model pipelines make model judgment part of the training loop. If judges drift, the constitution is no longer a fixed external standard in practice; it is filtered through a moving instrument.

LLM-as-judge reliability. Work on judge bias, position bias, and self-preference already shows that evaluative LLMs are imperfect instruments. Normative drift adds a post-training axis: the judge can change because of fine-tuning history, not only because of prompt framing.

EigenBench / comparative value measurement. Chang et al.’s EigenBench provides the comparative machinery used here. The contribution is not a new ranking method; it is using matched all-to-all judgments to separate response drift from judgment drift under an EM dose ladder.

Character-training side effects. Separately, collaborative work on character training studies cross-constitution side effects of persona optimization. That project asks how training on one constitution shifts scores under others. This writeup asks how emergent-misalignment fine-tuning shifts the judge itself. Related measurement family; different intervention and different claim.

Limitations

These results are preliminary and scoped.

• Single base model: Qwen2.5-14B-Instruct only.
• Single EM dataset: bad medical advice; other harmful domains may differ.
• FullFT is nearly unusable as a judge due to structured-output failure.
• Results depend on one Claude-style constitution.
• No internal mechanistic account yet — this is behavioral measurement, not circuit explanation.
• In-domain counts are smaller than out-of-domain counts.
• Base is treated as the reference frame; Base has its own RLHF biases.
• No benign fine-tuning control. This is the most important caveat. Judgment collapse could be a generic tax of narrow fine-tuning rather than specific to misalignment-inducing data. Criterion specificity and in-domain directional bias are harder to dismiss as generic, but the dominant collapse signal remains ambiguous without a matched benign control.

What I Would Run Next

The critical follow-up is a benign medical fine-tuning control at the same LoRA ranks and step count, evaluated under the identical EigenBench protocol. Three outcomes would clarify the mechanism:

• same collapse under benign fine-tuning → collapse is a generic fine-tuning tax;
• no collapse under benign fine-tuning → collapse is EM-specific;
• collapse without AI-alignment criterion specificity → collapse is generic, but targeted criterion drift is the genuine normative-drift signature.

I would also want multi-model replication, additional EM domains, and a mechanistic pass over where preference geometry moves.

References

Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … & Kaplan, J. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv.

Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., … & Evans, O. (2025). Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. arXiv.

Chang, J., Piff, L., Sana, S., Li, J. X., & Levine, L. (2026). EigenBench: A Comparative Behavioral Measure of Value Alignment. ICLR 2026 (oral).

Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS.

Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J. V., Lin, B. Y., Chandu, K., … & Smith, N. A. (2024). RewardBench: Evaluating Reward Models for Language Modeling. arXiv.

Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., … & Hashimoto, T. B. (2023). AlpacEval: An Automatic Evaluator of Instruction-following Models.

Pan, A., Chan, J. S., Zou, A., Li, N., Basart, S., Woodside, T., … & Hendrycks, D. (2023). Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark. ICML.

Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., … & Kaplan, J. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. ACL Findings.

Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., … & Perez, E. (2023). Towards Understanding Sycophancy in Language Models. arXiv.

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., … & Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS.