Andrii Bidochko

• Updated: July 19, 2026
• 7 min read

Towards Isolated Interventions via Almost Orthogonal Features in Language Models

Direct Answer

The paper introduces a method for shaping the internal representations of large language models (LLMs) so that individual linear features are nearly orthogonal, dramatically reducing unwanted side‑effects when those features are edited. This matters because it brings us closer to truly modular, causally interpretable models that can be safely tweaked for specific tasks such as mathematical reasoning or bias mitigation.

Background: Why This Problem Is Hard

Mechanistic interpretability assumes that high‑level concepts—like “addition” or “sentiment”—are encoded as linear directions in a model’s activation space. In theory, if each concept lives on its own axis, we could intervene on one axis without disturbing the others. In practice, however, the learned features are entangled: rotating one direction often drags along several others, leading to feature interference. This interference makes localized edits unpredictable, limiting the usefulness of interpretability tools for real‑world system engineering.

Existing approaches to intervention typically fall into two camps:

• Post‑hoc probing. Researchers identify a direction after training and then apply a linear edit. The edit can improve a downstream metric but often degrades unrelated capabilities.
• Regularization during pre‑training. Techniques such as sparsity or low‑rank factorization encourage simpler representations, yet they do not explicitly enforce independence between concepts.

Both camps struggle with the Independent Causal Mechanisms (ICM) principle, which states that a system’s causal factors should operate independently. When the principle is violated, any causal intervention leaks into other mechanisms, breaking the promise of modularity. As LLMs become the backbone of AI agents, retrieval‑augmented pipelines, and enterprise copilots, the inability to safely edit internal knowledge becomes a critical bottleneck.

What the Researchers Propose

The authors propose to reshape the model’s internal feature dictionary so that its vectors are almost orthogonal. In plain language, they add a regularization term that penalizes the dot product between any pair of feature vectors, pushing them toward mutual perpendicularity while still allowing the model to learn useful representations.

Key components of the framework are:

• Feature Dictionary. A set of linear directions extracted from a Sparse Auto‑Encoder (SAE) that approximates the activation space of a target layer.
• Orthogonality Regularizer. An explicit loss term that minimizes the sum of squared cosine similarities across all pairs of dictionary vectors.
• Self‑Coherence Metric. A theoretical measure that captures how much a dictionary deviates from perfect orthogonality; lower values imply tighter isolation.

By training the SAE with this regularizer, the resulting dictionary satisfies the “almost orthogonal” condition, which the authors argue is sufficient to bound the propagation of interference when a single feature is edited.

How It Works in Practice

The workflow can be broken down into three stages:

1. Extraction. A pre‑trained LLM is frozen, and activations from a chosen layer are fed into an SAE. The SAE learns a compact set of basis vectors (the feature dictionary) that reconstruct the activations with minimal error.
2. Regularization. During SAE training, the orthogonality regularizer is applied. The loss function becomes:
   Loss = ReconstructionError + λ·OrthogonalityPenalty, where λ balances fidelity against independence.
3. Intervention. To edit a concept, a practitioner identifies the corresponding feature vector, adds a scalar offset (positive or negative), and passes the modified activation back through the decoder. Because the dictionary vectors are nearly orthogonal, the edit’s influence stays confined to the target concept.

What distinguishes this approach from earlier methods is the explicit, mathematically grounded guarantee that the “leakage” of an edit is bounded by the self‑coherence of the dictionary. In other words, the more orthogonal the dictionary, the closer the realized intervention mirrors an ideal, isolated edit.

Evaluation & Results

The authors evaluated the method on two fronts:

Mathematical Reasoning Benchmarks

They selected a suite of arithmetic and algebra problems where each reasoning step can be mapped to a distinct latent concept (e.g., “addition”, “multiplication”, “variable substitution”). Using the orthogonalized dictionary, they performed targeted interventions that amplified the model’s ability to execute a specific operation.

• Isolation Score. Measured how much the accuracy on non‑targeted operations changed after an intervention. The orthogonalized model showed a 30% reduction in unintended impact compared to a baseline SAE.
• Overall Accuracy. Despite the added regularization, the model retained > 98% of its original performance on the untouched test set, confirming that orthogonalization does not sacrifice core capability.

Generalization to Unseen Prompts

Interventions were also tested on prompts that combined multiple concepts (e.g., “solve 3 × (4 + 5)”). The orthogonalized model maintained the intended boost on the targeted operation while leaving the surrounding reasoning steps untouched, whereas the baseline often over‑corrected, leading to new errors.

These findings demonstrate two crucial points:

1. Enforcing near‑orthogonality yields a quantifiable reduction in feature interference.
2. The regularizer can be applied without degrading the model’s baseline competence, making it viable for production‑grade systems.

Why This Matters for AI Systems and Agents

For practitioners building AI agents, the ability to edit a model’s internal knowledge safely opens several strategic avenues:

• Targeted Skill Injection. Add or strengthen a specific capability (e.g., a new domain‑specific ontology) without retraining the entire model.
• Bias Mitigation. Suppress undesirable associations by nudging the offending feature vector, while preserving overall performance.
• Dynamic Adaptation. Deploy agents that can be “patched” on‑the‑fly in response to emerging regulations or user feedback.

These capabilities align directly with the needs of platforms that orchestrate multiple AI services. For example, the Workflow automation studio can now incorporate a “feature‑edit” block that adjusts a language model’s reasoning style without halting the entire pipeline. Similarly, AI marketing agents could be fine‑tuned to adopt brand‑specific tone by editing a single “tone” feature, ensuring consistency across campaigns.

What Comes Next

While the results are promising, several open challenges remain:

• Scalability. Extending orthogonalization to the billions of features present in the deepest layers of state‑of‑the‑art LLMs may require hierarchical or block‑wise regularization.
• Discovery of Conceptual Alignments. Mapping dictionary vectors to human‑interpretable concepts still relies on manual probing; automated alignment would accelerate adoption.
• Robustness to Distribution Shift. It is unclear how orthogonalized dictionaries behave when the model encounters out‑of‑distribution inputs.

Future research could explore hybrid approaches that combine orthogonal regularization with contrastive learning, or integrate the technique directly into the pre‑training objective. From an engineering perspective, integrating the method into existing AI stacks is straightforward: the UBOS platform overview already supports custom SAE modules, and developers can plug in the orthogonal regularizer as a drop‑in component.

Potential applications span beyond mathematics. Imagine a customer‑support bot whose “policy compliance” feature can be toggled in real time, or a code‑generation assistant that isolates “security‑aware” reasoning from general syntax handling. The ability to intervene cleanly could become a cornerstone of trustworthy AI deployment.

For those interested in reproducing the experiments or extending the codebase, the authors have open‑sourced their implementation at GitHub. The repository includes scripts for extracting dictionaries, applying the orthogonality penalty, and evaluating isolation metrics.

In summary, the paper provides a concrete, theoretically justified pathway to more modular language models. By making internal concepts nearly orthogonal, it reduces the risk of unintended side‑effects during intervention—a prerequisite for safe, controllable AI agents that can evolve alongside business needs.

Read the full study on arXiv for a deeper dive into the mathematics and experimental details.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.