Andrii Bidochko

• Updated: July 17, 2026
• 7 min read

Validity of LLMs as data annotators: AMALIA on authority

Direct Answer

The paper introduces AMALIA‑9B, a publicly funded Portuguese language model, and a rigorous framework for testing whether large language models (LLMs) can serve as reliable data annotators for abstract constructs such as the moral foundation of authority. It matters because it shows that high agreement with human coders does not guarantee theoretical validity, exposing a hidden “recovery gap” that can undermine sovereign AI benchmarking.

Background: Why This Problem Is Hard

Data annotation is the backbone of supervised NLP: without high‑quality labels, downstream models inherit bias, noise, and mis‑interpretation. Traditionally, human experts perform annotation, but scaling to millions of examples is costly and slow. Recent research proposes using LLMs as “synthetic annotators,” leveraging their generative power to label data at scale.

Two intertwined challenges make this approach fragile:

• Reliability vs. validity. An LLM may consistently reproduce a label (high reliability) yet base its decision on surface cues rather than the underlying theoretical construct (low validity).
• Construct inference. Moral foundations, sentiment, or intent are not directly observable in text; they must be inferred from nuanced context, cultural norms, and philosophical definitions.

Existing benchmarks typically measure only agreement (e.g., F1, Cohen’s κ) between model predictions and human coders. They rarely probe whether the model follows the construct’s logical rules or merely exploits correlated shortcuts such as keyword presence or emotional intensity.

For sovereign AI initiatives—nationally funded models that aim to serve specific linguistic communities—this gap is especially critical. A model that appears accurate on Portuguese news articles might fail when the same construct is applied to legal texts, social media, or cross‑lingual settings.

What the Researchers Propose

The authors present a two‑step evaluation framework that distinguishes agreement from theoretical fidelity:

1. Holistic Prompting. The model receives the full codebook description of a moral foundation (e.g., “Authority”) and is asked to label each sentence directly.
2. Recovery Gap Analysis. The same codebook is decomposed into its atomic clauses (e.g., “Respect for hierarchy,” “Obedience to legitimate power”). Each clause is prompted separately, and the individual outputs are recombined using the explicit logical rule defined by the theory.

If the recombined result matches the holistic prediction, the model is said to have closed the recovery gap, indicating that it follows the construct’s reasoning rather than relying on shortcuts. The framework also incorporates a calibration step that adjusts the model’s confidence thresholds to align with human‑coded baselines.

Key components of the proposed system are:

• Codebook Engine. Stores the textual definition of each moral foundation and its logical decomposition.
• Prompt Generator. Transforms codebook clauses into natural‑language prompts tailored for the target LLM.
• Recombination Module. Applies the theory’s rule (e.g., “label as Authority if any clause is true”) to the clause‑level outputs.
• Calibration Layer. Uses a small validation set to fine‑tune decision thresholds, ensuring comparable precision‑recall trade‑offs across models.

How It Works in Practice

The workflow can be visualized as a pipeline:

1. Input Collection. Sentences from a Portuguese corpus (e.g., news articles) are fed into the system.
2. Holistic Annotation. AMALIA‑9B receives a single prompt containing the full Authority definition and returns a label.
3. Clause Extraction. The codebook engine splits the definition into atomic clauses such as “mentions a leader,” “expresses obedience,” and “references hierarchy.”
4. Clause‑Level Prompting. Each clause becomes an independent prompt; AMALIA‑9B (or any LLM) answers a binary “Yes/No” for each.
5. Logical Recombination. The recombination module aggregates the binary answers according to the moral foundation theory (e.g., “Authority = Yes if any clause is true”).
6. Calibration. The aggregated label is adjusted using calibrated thresholds derived from a held‑out human‑coded set.
7. Comparison. The calibrated holistic label and the recombined label are compared; the performance delta defines the recovery gap.

What sets this approach apart is the explicit separation of “what the model says” from “why it says it.” By forcing the model to justify each atomic clause, researchers can detect whether the model is leveraging genuine inferential reasoning or merely reacting to surface patterns like moral outrage words (“shame,” “punish”).

Evaluation & Results

The authors evaluated three configurations on a Portuguese news corpus annotated for the moral foundation “Authority”:

• Open‑source multilingual LLM (≈70B parameters).
• AMALIA‑9B (national Portuguese model).
• Baseline English instrument. A calibrated English‑language prompt transferred to Portuguese via translation.

Key findings:

• Agreement. AMALIA‑9B achieved an F1 score within six points of the much larger multilingual model when judged against human coders, confirming high reliability.
• Recovery Gap. When the holistic prompt was decomposed, AMALIA‑9B recovered only ~50 % of its original performance. The multilingual model closed the gap almost entirely, suggesting it followed the theoretical rule more faithfully.
• Error Analysis. AMALIA‑9B’s failures clustered around sentences that expressed moral outrage toward authority figures without explicit references to hierarchy. This indicates reliance on affective cues rather than the logical structure of the construct.
• Cross‑lingual Transfer. The calibrated English instrument performed poorly on Portuguese, confirming that simple translation does not preserve construct validity.

These results demonstrate that high surface agreement can mask a substantial validity deficit, especially for sovereign models trained on limited data.

Why This Matters for AI Systems and Agents

For practitioners building AI‑driven pipelines—whether for sentiment monitoring, policy compliance, or automated content moderation—the findings carry several practical warnings:

• Annotation at Scale. Relying solely on LLM agreement may propagate systematic biases into downstream classifiers, leading to erroneous decisions in high‑stakes domains.
• Agent Reasoning. Autonomous agents that query LLMs for moral judgments (e.g., “Should this policy be enforced?”) need guarantees that the model’s reasoning aligns with the intended ethical framework.
• Benchmark Design. Sovereign AI benchmarks should incorporate recovery‑gap tests, not just agreement metrics, to ensure that national models are truly fit for purpose.
• Integration Opportunities. Platforms like the UBOS platform overview can embed the clause‑level prompting pipeline as a reusable micro‑service, allowing developers to audit model decisions in real time.
• Workflow Automation. The Workflow automation studio can orchestrate the holistic‑vs‑decomposed annotation steps, automatically flagging low‑validity outputs for human review.

In short, the research urges AI teams to treat LLM‑generated labels as provisional hypotheses that must be validated against the underlying theory, especially when those labels drive business‑critical automation.

What Comes Next

While the study provides a compelling proof‑of‑concept, several limitations remain:

• Single Construct, Single Corpus. The analysis focuses only on the “Authority” moral foundation within a news dataset. Generalizing to other constructs (e.g., “Care,” “Fairness”) or domains (legal, medical) requires further testing.
• Model Size vs. Data Diversity. AMALIA‑9B’s training data is Portuguese‑centric; expanding the corpus to include diverse registers may reduce reliance on affective shortcuts.
• Calibration Overhead. The calibration step needs a modest human‑coded validation set, which can be costly for low‑resource languages.

Future research directions include:

1. Extending the recovery‑gap framework to multi‑label moral foundations and to hierarchical taxonomies.
2. Developing automated clause‑generation tools that can translate any ethical codebook into LLM‑friendly prompts.
3. Exploring ensemble approaches where a sovereign model’s outputs are cross‑checked against a multilingual “gold standard” to improve validity.
4. Integrating the pipeline with AI marketing agents that need trustworthy sentiment and moral labeling for brand safety.

From an industry perspective, the next step is to embed these validation loops into production‑grade platforms. The UBOS templates for quick start already include pre‑built connectors for LLM APIs, making it feasible to prototype a recovery‑gap‑aware annotation service within days.

References

• Pita, M. (2026). Validity of LLMs as data annotators: AMALIA on authority. arXiv preprint.
• Graham, Y., et al. (2023). “Moral Foundations Theory and NLP.” *Proceedings of ACL*.
• OpenAI. (2024). “Guidelines for Using LLMs as Synthetic Annotators.” *OpenAI Technical Report*.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.