Andrii Bidochko

• Updated: June 14, 2026
• 7 min read

Measuring Progress Toward AGI: A Cognitive Framework – UBOS Blog

Direct Answer

The paper introduces a Cognitive Taxonomy that breaks general intelligence into ten well‑defined cognitive faculties, and pairs it with a rigorous evaluation protocol that produces a “cognitive profile” for any AI system. This matters because it gives the field a concrete, empirically grounded way to track AGI measurement and compare progress across models, moving the conversation from speculation to data‑driven assessment.

Background: Why This Problem Is Hard

Artificial General Intelligence (AGI) has become a rallying point for investors, policymakers, and researchers, yet the community still lacks a shared yardstick for progress. Existing benchmarks—such as language model leaderboards, game‑playing scores, or narrow task suites—capture isolated capabilities but fail to reflect the breadth of cognition that humans exhibit. Consequently, claims of “human‑level” performance are often ambiguous, making it difficult to:

• Distinguish genuine leaps in general reasoning from overfitting to benchmark quirks.
• Identify systematic gaps (e.g., planning vs. perception) that could become safety hazards.
• Provide regulators and investors with transparent, comparable metrics for responsible governance.

Traditional evaluation pipelines also suffer from “benchmark saturation”: once a model tops a leaderboard, the community quickly moves on, leaving the underlying capability under‑examined. Moreover, most metrics are single‑dimensional (accuracy, BLEU, win rate) and ignore the multidimensional nature of intelligence highlighted by decades of cognitive science.

What the Researchers Propose

The authors propose a two‑part framework:

1. Cognitive Taxonomy: A hierarchical classification of ten core cognitive faculties—perception, attention, memory, language, reasoning, planning, learning, social cognition, motor control, and meta‑cognition. Each faculty is defined using operational criteria drawn from psychology and neuroscience, ensuring that the taxonomy aligns with established human cognition research.
2. Evaluation Protocol: A systematic process that assembles a suite of held‑out tasks, each targeting a specific faculty. By running an AI system through the entire suite, researchers generate a vector‑like “cognitive profile” that quantifies strengths and weaknesses across the ten dimensions.

Key roles in the framework include:

• Task Curators who design or select benchmark tasks that isolate a single faculty while controlling for confounding factors.
• Profile Aggregators that normalize raw scores, apply statistical weighting, and output a standardized profile.
• Comparative Analysts who interpret profiles to map progress trajectories, identify emergent capabilities, and flag safety‑relevant blind spots.

How It Works in Practice

The workflow can be visualized as a pipeline:

1. Task Library Construction: Researchers compile a repository of 100+ tasks, each annotated with the faculty it targets (e.g., “visual object recognition” for perception, “multi‑step planning in a simulated kitchen” for planning).
2. Model Inference: An AI system is evaluated on every task under identical conditions (same prompts, same compute budget). Results are recorded as raw performance metrics (accuracy, success rate, latency, etc.).
3. Normalization & Scoring: Raw scores are transformed into a 0‑100 scale using faculty‑specific baselines (human average, random baseline, and state‑of‑the‑art model). This step removes task‑specific difficulty variance.
4. Cognitive Profile Generation: The normalized scores are assembled into a ten‑dimensional vector. Visualization tools (radar charts, heat maps) make the profile instantly readable.
5. Longitudinal Tracking: Profiles are stored in a versioned database, enabling researchers to plot trajectories over time, compare architectures, or assess the impact of training interventions.

What sets this approach apart is its insistence on held‑out tasks—tasks that the model has never seen during training—thereby reducing the risk of memorization masquerading as genuine capability. Additionally, the taxonomy’s grounding in cognitive science ensures that each dimension has a clear, interpretable meaning, unlike opaque composite scores.

Evaluation & Results

To validate the framework, the authors conducted three experiments:

1. Baseline Profiling of Existing Models

They applied the protocol to three widely used systems: a 175‑billion‑parameter language model, a multimodal vision‑language model, and a reinforcement‑learning agent trained on Atari games. The resulting profiles revealed expected strengths (language model excelled in language and reasoning) but also uncovered surprising weaknesses, such as poor meta‑cognition and limited social cognition across all three systems.

2. Cross‑Model Comparative Study

Using the same task suite, the team compared a newer 500‑billion‑parameter model against its predecessor. While raw benchmark scores improved modestly (2‑5% on average), the cognitive profile showed a substantial jump in planning (≈15 points) and learning (≈12 points), suggesting that scaling can yield non‑linear gains in specific faculties.

3. Ablation of Training Regimens

Researchers trained two variants of a transformer: one with standard next‑token prediction, another with an auxiliary “cognitive‑faculty” loss that explicitly penalized errors on a subset of tasks. The latter achieved a more balanced profile, closing gaps in attention and memory without sacrificing language performance.

Collectively, these results demonstrate that the Cognitive Taxonomy and evaluation protocol can surface nuanced capability shifts that traditional leaderboards miss. The framework also proved sensitive enough to detect the impact of modest training changes, making it a practical diagnostic tool for AI developers.

Why This Matters for AI Systems and Agents

For practitioners building next‑generation agents, the framework offers a concrete roadmap to assess whether a model truly possesses the breadth of abilities required for real‑world deployment:

• Targeted Improvement: By pinpointing low‑scoring faculties, engineers can allocate compute and data resources to the most critical gaps (e.g., adding social‑cognition datasets to improve user‑interaction bots).
• Safety & Alignment: A balanced cognitive profile reduces the risk of “skill‑specific blind spots” that could lead to unsafe behavior in high‑stakes environments.
• Product Differentiation: Companies can market agents with verified strengths in planning or meta‑cognition, backed by transparent, reproducible metrics.
• Orchestration & Workflow Automation: Platforms like the Workflow automation studio can ingest cognitive profiles to automatically route tasks to the most capable subsystem, optimizing overall system performance.
• Integration with Existing AI Stacks: The profile can be stored alongside model metadata in the Chroma DB integration, enabling rapid similarity searches for models that excel in a given faculty.

What Comes Next

While the Cognitive Taxonomy marks a significant step forward, several open challenges remain:

• Task Coverage Expansion: The current suite, though extensive, still under‑represents embodied motor control and long‑term meta‑cognition. Future work should incorporate robotics simulators and lifelong learning benchmarks.
• Standardization Across Labs: Widespread adoption will require a shared repository, version control, and community‑driven validation—similar to the ImageNet model for vision.
• Dynamic Profiling: Real‑time profiling of deployed agents could enable adaptive orchestration, where a system re‑allocates workloads based on its current cognitive state.
• Human‑AI Comparative Studies: Aligning the 0‑100 scale with human performance across faculties would make the profiles more interpretable for non‑technical stakeholders.

Addressing these gaps will likely involve collaborations between AI labs, cognitive scientists, and industry platforms. For example, the Enterprise AI platform by UBOS could host a federated profiling service, allowing companies to benchmark proprietary models without exposing raw data.

In the meantime, researchers and developers can start experimenting with the framework by downloading the task library, running the open‑source evaluation scripts, and publishing their own cognitive profiles. Such community‑driven data will accelerate the emergence of a shared AGI measurement language, turning speculative milestones into quantifiable progress.

For a deeper dive into the methodology and to access the full set of tasks, see the original arXiv paper. The authors also provide a public leaderboard where you can compare your model’s profile against the baseline systems presented in the study.

Conclusion

The Cognitive Taxonomy and its accompanying evaluation protocol give the AI community a practical, scientifically grounded tool for artificial general intelligence evaluation. By translating decades of cognitive research into a machine‑readable profile, the framework bridges the gap between abstract AGI discourse and concrete engineering decisions. As more organizations adopt this approach, we can expect a clearer, data‑driven narrative of how—and how fast—AI systems are moving toward truly general intelligence.

Call to Action

Explore how UBOS can help you integrate cognitive profiling into your AI pipelines:

• Visit the UBOS platform overview to learn about modular model hosting.
• Check out the UBOS templates for quick start on building evaluation dashboards.
• Join the UBOS partner program to collaborate on next‑generation AGI measurement standards.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.