Andrii Bidochko

• Updated: July 16, 2026
• 7 min read

TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models

Direct Answer

TypeProbe introduces a probing framework that extracts latent type representations from the hidden states of large pre‑trained code models, revealing that these models internally encode cross‑lingual type semantics even when trained on untyped source code. This matters because it opens a pathway to more reliable code understanding, safer code generation, and language‑agnostic tooling built on top of existing foundation models.

Background: Why This Problem Is Hard

Modern code models such as CodeBERT, Codex, and StarCoder achieve impressive benchmarks on code completion and synthesis, yet they operate as black boxes. Developers and researchers lack visibility into whether the models truly grasp the formal type systems that underpin languages like Java, Python, or TypeScript. Without this insight, several practical problems arise:

• Type safety violations: Generated snippets may compile but still violate expected contracts, leading to runtime errors.
• Cross‑language reuse: Enterprises that maintain polyglot codebases need assurances that a model trained on one language can respect the type discipline of another.
• Debugging and auditability: When a model suggests a refactor, engineers need to know whether the suggestion respects the original type intent.

Existing interpretability work on code models focuses on token‑level attention or syntax‑tree alignment, but these methods do not directly address formal type semantics. Moreover, most probing studies target natural language models, where type information is less rigorously defined. The lack of a systematic way to surface type knowledge from code models therefore remains a critical bottleneck for trustworthy AI‑assisted software development.

What the Researchers Propose

The authors present TypeProbe, a lightweight, language‑agnostic probing methodology that treats the hidden states of a pre‑trained code model as a source of latent type vectors. The core idea is to train a simple linear classifier—called a “probe”—on top of the model’s residual stream to predict the type of a variable or function return. The framework consists of three conceptual components:

1. Dataset Builder: A parallel corpus of Java and Python snippets where each function is annotated with its argument and return types. The corpus includes both typed and deliberately untyped examples to test emergent behavior.
2. Probe Trainer: A linear model that maps hidden representations to a fixed‑size type embedding space. Training is performed separately for each language, but the same probe architecture is reused across languages.
3. Cross‑lingual Evaluator: A set of experiments that train a probe on one language and test its ability to infer types in the other, thereby measuring the degree of shared type semantics.

By keeping the probe linear and shallow, the authors ensure that any successful classification can be attributed to information already present in the base model, rather than to the probe’s capacity.

How It Works in Practice

The practical workflow of TypeProbe can be broken down into four stages:

1. Tokenization and Embedding Extraction

Source code is tokenized using the same tokenizer as the underlying code model (e.g., Byte‑Pair Encoding). For each token, the model’s residual stream—a concatenation of hidden states from all transformer layers—is recorded.

2. Label Alignment

Each token that corresponds to a variable, parameter, or function name is aligned with its ground‑truth type label from the curated dataset. Types are normalized to a language‑agnostic taxonomy (e.g., int, float, object, list).

3. Linear Probe Training

A ridge‑regularized linear classifier is fitted to map the high‑dimensional residual vectors to the type taxonomy. Because the classifier is linear, it can be interpreted as a set of direction vectors that highlight where type information resides in the model’s space.

4. Cross‑lingual Transfer & Robustness Checks

After training on Java, the same probe is applied to Python residuals (and vice‑versa) without retraining. Additional experiments perturb the source code lexically (e.g., renaming identifiers) or syntactically (e.g., reordering statements) to test whether the type signal survives such changes.

This pipeline is deliberately simple, allowing practitioners to plug TypeProbe into any existing transformer‑based code model with minimal engineering effort.

Evaluation & Results

The authors evaluate TypeProbe on three fronts:

Dataset Coverage

A parallel corpus of 12,000 Java–Python function pairs was constructed, covering primitive, collection, and user‑defined types. Approximately 30 % of the examples were deliberately stripped of explicit type annotations to test emergent learning.

In‑language Probing Accuracy

When trained and tested on the same language, the linear probe achieved > 85 % top‑1 accuracy for primitive types and ~ 70 % for complex user‑defined classes. This demonstrates that the base model already stores a strong type signal.

Cross‑lingual Transfer

Probes trained on Java could predict Python return types with 78 % accuracy, and the reverse direction yielded 75 % accuracy. The drop relative to in‑language performance is modest, indicating that the model learns a language‑agnostic representation of type semantics.

Robustness to Perturbations

Lexical renaming of identifiers reduced accuracy by only 4 %, while syntactic reordering (e.g., moving a return statement) caused a 6 % drop. These results suggest that the type signal is tied to deeper semantic patterns rather than surface forms.

Overall, the experiments confirm three key claims: (1) pre‑trained code models encode type information even without explicit supervision, (2) this encoding generalizes across programming languages, and (3) the representation is resilient to common code transformations.

Why This Matters for AI Systems and Agents

Understanding that code models carry latent type knowledge reshapes how developers can build AI‑driven software tools:

• Safer code generation: Agents can query the hidden type vectors to verify that a suggested snippet respects expected contracts before emitting it to a developer.
• Language‑agnostic assistants: A single model can serve polyglot teams, with TypeProbe acting as a bridge that translates type expectations across languages.
• Dynamic type inference in IDEs: Integrated development environments can surface inferred types for dynamically typed languages, improving autocomplete and static analysis.
• Orchestrated workflows: When composing multiple AI agents—e.g., a code‑summarizer, a test‑generator, and a refactoring bot—TypeProbe provides a common semantic grounding that reduces mismatches.

Practitioners looking to embed these capabilities into production pipelines can leverage existing UBOS solutions. For example, the UBOS platform overview offers a modular runtime where a TypeProbe micro‑service can be registered alongside code generation models. The AI marketing agents showcase how specialized probes can be combined with domain‑specific agents to enforce business rules. Finally, the Workflow automation studio lets teams design end‑to‑end pipelines that automatically validate generated code against inferred type constraints before deployment.

What Comes Next

While TypeProbe opens exciting avenues, several limitations remain:

• Granularity of type taxonomy: The current experiments focus on coarse‑grained types. Extending the probe to capture generic type parameters (e.g., List<String>) will require richer label schemas.
• Scalability to large codebases: Probing every token in massive repositories could be computationally expensive; efficient caching strategies are needed.
• Integration with downstream tools: Turning probe outputs into actionable constraints for compilers or CI pipelines is an engineering challenge.

Future research directions include:

1. Training multimodal probes that combine hidden states with abstract syntax tree embeddings to improve precision on complex types.
2. Exploring self‑supervised objectives that explicitly encourage models to align their internal representations with type systems during pre‑training.
3. Extending cross‑lingual probing to less‑common languages (e.g., Rust, Go) to assess the universality of the discovered type space.

Enterprises interested in prototyping these ideas can start with the UBOS for startups program, which provides sandbox environments and API access to plug in custom probes. Larger organizations may consider the Enterprise AI platform by UBOS, which offers scalable deployment, monitoring, and security features required for mission‑critical code intelligence workloads.

References

Gorgone, G., & Carcassi, F. (2026). TypeProbe: Recovering Type Representations from Hidden States of Pre‑trained Code Models. arXiv preprint arXiv:2607.08339v1.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.