Andrii Bidochko

• Updated: July 4, 2026
• 7 min read

The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

Direct Answer

The paper introduces the Metanym Game, a self‑contained, peer‑community benchmark that measures large language models’ (LLMs) structural intelligence through a competitive word‑game format. By forcing models to generate and evaluate analogical statements without any pre‑provided content, the benchmark offers a contamination‑resistant gauge of both factual accuracy and judgment skill.

Background: Why This Problem Is Hard

Evaluating LLMs has become a moving target. Traditional benchmarks rely on static test sets—multiple‑choice questions, factual recall, or downstream task performance—that are easily memorized during pre‑training. As models grow larger, the risk of data contamination (where test items appear in training corpora) inflates scores and obscures true capability.

Beyond contamination, most existing metrics focus on surface‑level abilities: token prediction, answer correctness, or alignment with human preferences. They rarely probe structural intelligence—the capacity to understand and manipulate relational patterns, analogies, and abstract reasoning that underpin human cognition. Cognitive‑science research shows that analogical reasoning is a core indicator of higher‑order thinking, yet LLM evaluation pipelines lack robust, scalable analogical tests.

Current approaches also separate generation from evaluation. A model that writes fluent text may be a poor judge of truth, while a strong evaluator often requires a separate, specialized system. This split creates a “dual‑skill” problem: we need benchmarks that can assess both creation and judgment simultaneously, without relying on external oracle models.

What the Researchers Propose

David Nordfors proposes a novel framework: the Metanym Game. It is a competitive word‑game where each participating LLM acts as both player (generator) and judge (evaluator). The core ideas are:

• Self‑generated content: No pre‑written prompts or answer keys are supplied. Models must invent analogies, metaphors, and relational statements from scratch.
• Peer‑community rating: After a round of generation, each model rates the statements produced by its peers on two axes—factual correctness and structural elegance.
• Spectral competence extraction: By applying a singular value decomposition (SVD) to the matrix of peer ratings, the system derives a single competence score that simultaneously reflects a model’s ability to generate high‑quality statements and to judge others consistently.
• Dynamic council of experts: The top‑scoring models form a rotating “council” that conducts official benchmark runs. Seats on the council are contestable; a stronger model can replace a weaker council member based on its own rating.

This design eliminates the need for a golden answer key or an external oracle, making the benchmark inherently resistant to training‑data leakage.

How It Works in Practice

The Metanym Game proceeds through three conceptual stages:

1. Prompt Generation

Each participating LLM receives a minimal seed—a single word or phrase (e.g., “bridge”). The model must produce a metanym, an analogical sentence that maps the seed onto a new domain (e.g., “A bridge is a conversation that spans silence”). The output must be syntactically complete and semantically coherent.

2. Peer Evaluation

All generated metanyms are shared with the entire participant pool. Every model assigns two scores to each sentence:

• Factual rating: Does the statement respect real‑world constraints? (e.g., “A bridge cannot be a conversation that spans silence” would be flagged as false.)
• Structural rating: How well does the sentence capture the intended analogical relationship?

Ratings are recorded in a matrix R where rows represent judges and columns represent statements.

3. Spectral Scoring

The matrix R undergoes singular value decomposition: R = UΣVᵀ. The leading singular vector captures the dominant pattern of agreement across judges. By projecting each model’s row onto this vector, the framework extracts a single competence value that reflects both generation quality (how often a model’s statements receive high scores) and judging skill (how consistently a model’s ratings align with the consensus).

What sets this approach apart is its dual‑role assessment: a model that excels at creation but judges poorly will receive a balanced score, revealing the trade‑off between the two skills. Conversely, a model that is an excellent judge but a mediocre generator can still achieve a respectable overall competence.

Evaluation & Results

Nordfors evaluated the Metanym Game using a diverse roster of 12 publicly available LLMs ranging from 7 B to 70 B parameters, including open‑source and commercial offerings. The experiments covered three dimensions:

• Factual alignment: Correlation of the game’s factual ratings with the GPQA‑Diamond benchmark (a high‑quality factual QA suite). The Pearson correlation reached r = 0.92, indicating that peer judgments reliably approximate an external gold standard.
• Structural intelligence: Human experts manually inspected a random sample of metanyms and rated their analogical depth. The game’s structural scores showed a strong alignment (Spearman ρ ≈ 0.78) with expert judgments, confirming that the peer community can capture nuanced relational reasoning.
• Skill dissociation: The analysis revealed a surprising pattern: the strongest generators tended to be only “middling” judges, while the sharpest judges were often mid‑ranked generators. This asymmetry underscores the importance of evaluating both abilities separately.

Beyond raw numbers, the study demonstrated that the council‑of‑peers mechanism stabilizes over multiple rounds, producing a consistent ranking that is robust to the addition or removal of individual models. The spectral solution also proved computationally lightweight, requiring only a single SVD on a modest‑size matrix.

Why This Matters for AI Systems and Agents

For practitioners building AI agents, the Metanym Game offers a practical, contamination‑free yardstick to gauge two critical competencies:

• Creative reasoning: Agents that must generate novel plans, analogies, or explanations can be benchmarked on their ability to produce structurally sound statements.
• Self‑assessment: In multi‑agent orchestration, an agent’s capacity to evaluate peer outputs is essential for tasks like debate, consensus building, or error correction.

Because the benchmark is self‑contained, teams can integrate it into continuous‑integration pipelines without worrying about test‑set leakage. Moreover, the spectral competence score provides a single, comparable metric that can be tracked across model upgrades, hyper‑parameter sweeps, or fine‑tuning regimes.

Enterprises looking to adopt trustworthy LLMs can leverage the Metanym Game to certify that a model not only knows facts but also reasons analogically—a prerequisite for domains such as legal analysis, scientific hypothesis generation, and strategic planning. For example, the Enterprise AI platform by UBOS could embed the Metanym Game as an internal validation step before deploying agents to customer‑facing workflows.

What Comes Next

While the Metanym Game marks a significant advance, several open challenges remain:

• Scalability to larger communities: As the number of participating models grows, the rating matrix becomes larger. Future work could explore incremental SVD or distributed consensus algorithms.
• Domain‑specific extensions: Tailoring the seed words to specialized vocabularies (e.g., biomedical, legal) would test domain‑level structural intelligence.
• Human‑in‑the‑loop augmentation: Incorporating expert judges alongside models could further calibrate the spectral scores and provide a hybrid human‑AI benchmark.
• Dynamic difficulty adjustment: Adaptive seed selection based on current model performance could keep the game challenging as models improve.

Addressing these avenues will deepen our understanding of LLM judgment and generation trade‑offs, and could inspire new forms of peer‑learning where models iteratively improve by critiquing each other.

Organizations interested in collaborating on benchmark extensions or integrating the Metanym Game into their AI governance workflows can explore the UBOS partner program for joint research opportunities.

References

• Nordfors, D. (2026). The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence. arXiv preprint.
• Gao, L., et al. (2023). GPQA‑Diamond: A High‑Quality Factual Question‑Answer Benchmark. *Proceedings of the ACL*.
• Gentner, D. (1983). Structure-Mapping: A Theory of Analogy. *Cognitive Science*, 7(2), 155‑170.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.