Andrii Bidochko

• Updated: August 24, 2026
• 6 min read

On the Definition of Intelligence

Direct Answer

The paper On the Definition of Intelligence (arXiv) proposes a species‑agnostic, measurable notion of intelligence called ε‑concept intelligence, which evaluates an agent’s ability to reproduce entities that exemplify a given concept within a tolerable error margin. By framing intelligence as “entity fidelity,” the authors create a unified yardstick that can compare reinforcement learners, generative models, classifiers, and analogical reasoners on the same scale.

Background: Why This Problem Is Hard

Artificial General Intelligence (AGI) research has long been hampered by the lack of a common, operational definition of intelligence. Existing benchmarks—such as Atari scores, GLUE, or the AI‑Gym suite—focus on narrow tasks and implicitly assume that high performance on one benchmark translates to general capability. This assumption breaks down for several reasons:

• Task specificity: Metrics are tied to a single environment or data distribution, making cross‑domain comparison impossible.
• Conceptual ambiguity: “Intelligence” is used interchangeably for pattern recognition, planning, and creativity, without a clear boundary.
• Evaluation bias: Human‑crafted test suites often embed cultural or domain knowledge that favors certain model families.
• Safety blind spots: Without a universal yardstick, it is difficult to detect emergent failure modes that only appear when an agent is transferred to a new context.

Consequently, researchers and product teams lack a principled way to gauge whether a new architecture truly advances general intelligence or merely exploits a narrow benchmark.

What the Researchers Propose

Ng introduces a framework built around the intuition of entity fidelity: an intelligent system should be able to generate new instances that belong to the same conceptual class as a set of reference entities. The core idea is captured by two components:

1. Conceptual exemplar set: A collection of entities (images, text snippets, action trajectories, etc.) that concretely instantiate a target concept.
2. Admissible distinguisher family: A set of statistical or perceptual tests that a user deems acceptable for judging whether a generated entity belongs to the same concept.

Intelligence is then defined as the ability of a system to produce new entities that no distinguisher in the admissible family can separate from the original exemplars beyond a pre‑specified tolerance ε. This definition is deliberately agnostic to the underlying learning paradigm, allowing reinforcement learners, diffusion models, or symbolic reasoners to be evaluated under the same rubric.

How It Works in Practice

The operational workflow can be broken down into four stages:

1. Concept Specification

Domain experts curate a representative dataset of entities that embody the target concept. For a “cat” concept, this might include photographs, sketches, and short video clips.

2. Distinguisher Construction

A suite of admissible distinguishers is assembled. These could be:

• Pre‑trained classifiers (e.g., a ResNet that predicts “cat” vs. “non‑cat”).
• Human‑in‑the‑loop similarity judgments.
• Statistical tests on feature distributions (e.g., texture histograms).

3. Generation Phase

The candidate AI system—whether a diffusion model, a policy network, or a symbolic engine—produces a batch of novel entities intended to belong to the same concept.

4. Fidelity Evaluation

Each distinguisher scores the generated batch against the exemplar set. If the aggregate error across all admissible distinguishers stays below ε, the system is deemed ε‑intelligent for that concept.

What sets this approach apart is its explicit separation of what is being measured (conceptual fidelity) from how it is measured (the distinguisher family). This modularity enables researchers to plug in new distinguishers without redefining the intelligence metric, and it permits cross‑paradigm comparisons that were previously impossible.

Evaluation & Results

Ng validates the framework through three experimental families:

1. Visual concepts: Using ImageNet sub‑categories (e.g., “golden retriever”), the authors compare a GAN, a diffusion model, and a reinforcement‑learning‑based image composer. All three achieve ε‑intelligence at different ε thresholds, illustrating a trade‑off between sample diversity and fidelity.
2. Sequential decision concepts: For a “maze‑solving” concept, a classic Q‑learning agent and a transformer‑based planner are evaluated against a set of optimal trajectories. The planner reaches lower ε values, indicating higher conceptual alignment.
3. Language concepts: The authors test GPT‑4‑style language models on the concept “formal business email.” Human raters serve as distinguishers, and the models achieve ε‑intelligence comparable to human writers after a few fine‑tuning steps.

Key takeaways from the results:

• ε‑concept intelligence provides a single scalar that captures both quality and adherence to the target concept.
• Different model families can be placed on a common curve of ε versus computational budget, enabling cost‑benefit analyses.
• When ε is tightened, generative diversity drops, mirroring the classic precision‑recall trade‑off but now expressed in a concept‑centric language.

Why This Matters for AI Systems and Agents

For practitioners building AI‑driven products, the framework offers a practical evaluation loop that aligns directly with business outcomes:

• Unified benchmarking: Teams can compare a new diffusion model against an existing rule‑based system without building separate test suites.
• Safety screening: By selecting distinguishers that encode policy constraints (e.g., “no disallowed content”), ε‑intelligence can serve as an early warning signal for unsafe generations.
• Continuous monitoring: Deployments can periodically re‑run the fidelity test on live outputs, ensuring that model drift does not breach the ε tolerance.
• Orchestration of heterogeneous agents: In multi‑agent pipelines, each component can be assigned a concept‑specific ε budget, simplifying resource allocation.

These capabilities map directly onto existing UBOS offerings. For example, the OpenAI ChatGPT integration can be wrapped with a custom distinguisher that checks for brand‑compliant language, turning the integration into an ε‑intelligent conversational agent. Similarly, the Workflow automation studio can orchestrate multiple concept‑specific generators, each validated against its own fidelity threshold.

What Comes Next

While the ε‑concept intelligence framework marks a significant step forward, several open challenges remain:

• Scalability of distinguishers: Building exhaustive distinguisher families for high‑dimensional concepts (e.g., “creative storytelling”) may require large human annotation budgets.
• Dynamic concepts: Many real‑world concepts evolve over time (e.g., slang, market trends). Future work must address how ε tolerances adapt to shifting exemplars.
• Theoretical bounds: Formal connections between ε‑intelligence and classic learning theory (VC‑dimension, PAC‑learnability) are still unexplored.
• Cross‑modal concepts: Extending the framework to concepts that span modalities—such as “a video of a cat chasing a laser pointer”—will test the limits of current distinguisher design.

Addressing these gaps will likely involve tighter integration with data‑centric platforms. The Chroma DB integration can store multi‑modal exemplars and enable rapid retrieval for on‑the‑fly distinguisher updates. Moreover, the Enterprise AI platform by UBOS provides the compute elasticity needed to run large‑scale fidelity evaluations in production.

In the longer term, researchers envision a “concept marketplace” where organizations publish vetted exemplar‑distinguisher bundles, allowing any AI system to instantly benchmark against a shared intelligence standard. Such an ecosystem could accelerate alignment research, democratize safety testing, and bring the field closer to a universally accepted definition of intelligence.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.