Andrii Bidochko

• Updated: June 14, 2026
• 6 min read

From Learning Resources to Competencies: LLM-Based Tagging with Evidence and Graph Constraints

Direct Answer

The paper introduces an end‑to‑end pipeline that uses a large language model (LLM) to automatically tag learning‑resource fragments with competency labels, while simultaneously generating textual evidence and enforcing graph‑based constraints. This matters because it delivers transparent, auditable competency mapping at scale, reducing manual effort and improving curriculum analytics in modern Learning Management Systems (LMS).

Background: Why This Problem Is Hard

Competency‑based education relies on a structured framework that links every instructional artifact—lecture slides, videos, quizzes—to a set of predefined skills or outcomes. In practice, LMS administrators spend countless hours manually assigning these tags, a process that is:

• Labor‑intensive: Large course catalogs contain thousands of resources, each requiring expert judgment.
• Inconsistent: Human annotators differ in terminology, leading to noisy data that hampers downstream analytics.
• Opaque: Fully automatic classifiers often produce a label without justification, making it hard for educators to trust or correct the output.

Existing automated approaches typically fall into two camps. Retrieval‑based methods (e.g., BM25, dense embeddings) retrieve similar competency descriptions but lack contextual reasoning. Supervised classifiers can learn from labeled data but require extensive annotation and still provide no traceable evidence. Neither solution satisfies the dual need for scalability and transparency that modern LMS platforms demand.

What the Researchers Propose

The authors present a three‑stage framework—dubbed LBG (LLM + BM25 + Graph)—that blends retrieval, generative reasoning, and graph‑theoretic refinement:

• Fragmentation: Raw learning materials are split into pedagogically meaningful fragments (e.g., a slide, a paragraph, a quiz item).
• Candidate Retrieval: For each fragment, a BM25 search over a competency repository returns a short list of plausible competencies, enriched with neighboring nodes from the competency graph.
• LLM Tagging with Evidence: An LLM selects the most appropriate competencies from the candidate set and extracts supporting text spans from the fragment as evidence.
• Graph‑Based Refinement: The selected tags are validated against the competency graph to enforce hierarchical consistency and are then aggregated back to the full resource level.

Key agents in the pipeline are the segmenter, the retriever, the LLM tagger, and the graph validator. Each plays a distinct role, yet they cooperate through a shared data structure that carries both candidate competencies and evidence spans.

How It Works in Practice

Step‑by‑Step Workflow

1. Input Ingestion: Course materials (PDFs, HTML, video transcripts) are uploaded to the LMS.
2. Pedagogical Segmentation: A rule‑based or neural segmenter breaks the content into fragments roughly aligned with learning objectives.
3. BM25 Retrieval: Each fragment is queried against a pre‑indexed competency catalog. The top‑k results (typically 5‑10) are returned along with their graph neighborhoods (prerequisite and successor competencies).
4. LLM Reasoning: The fragment text, candidate list, and graph context are fed to a prompting template. The LLM outputs:
   • The selected competency label(s).
   • One or more evidence spans—exact substrings from the fragment that justify the choice.
5. Graph Constraint Enforcement: A post‑processor checks that selected competencies respect the directed acyclic graph (e.g., a “Data Structures” tag cannot appear without its prerequisite “Programming Basics”). Conflicts trigger a fallback to the next best candidate.
6. Aggregation: Fragment‑level tags are merged to produce resource‑level competency profiles, which can be stored in the LMS metadata.
7. Human Auditing: Educators review the evidence spans directly in the LMS UI, allowing quick verification or correction.

What Sets This Approach Apart

• Evidence Generation: Unlike black‑box classifiers, the LLM supplies concrete text excerpts that explain each tag.
• Graph‑Aware Consistency: The competency hierarchy is not an afterthought; it actively shapes retrieval and final selection.
• Hybrid Retrieval + Generation: BM25 narrows the search space, reducing hallucination risk while preserving the LLM’s reasoning power.

Evaluation & Results

The authors built a benchmark from the Université de Technologie de Compiègne (UTC) Computer Science department. The dataset comprises 22 competencies mapped across lecture notes, lab manuals, and assessment items. Evaluation focused on two granularities: fragment‑level tagging and resource‑level aggregation.

Metrics Used

• Micro‑F1 (0.57): Weighted average across all instances, reflecting overall precision‑recall balance.
• Macro‑F1 (0.50 fragment, 0.51 resource): Treats each competency equally, highlighting performance on less‑frequent tags.
• Mean Reciprocal Rank (MRR = 0.82): Measures how high the correct competency appears in the LLM’s ranked output.

Baseline Comparisons

The LBG pipeline outperformed several strong baselines:

• Zero‑shot LLM prompting without retrieval (lower F1, higher hallucination).
• Few‑shot LLM with manually crafted examples (modest gains but still lacked evidence).
• Pure BM25 similarity ranking (good recall but poor precision).
• Supervised classifiers trained on the same data (competitive F1 but no traceable evidence).

Beyond raw scores, qualitative analysis showed that 78 % of generated evidence spans were directly aligned with the competency definition, enabling educators to validate tags in seconds rather than minutes.

Why This Matters for AI Systems and Agents

For AI practitioners building educational agents or curriculum‑analytics platforms, the LBG pipeline offers a blueprint for combining retrieval, generative reasoning, and domain‑specific constraints. The implications include:

• Transparent Decision‑Making: Agents can surface the exact text that led to a competency assignment, satisfying audit requirements and building trust with instructors.
• Modular Architecture: Each component (segmenter, retriever, LLM, graph validator) can be swapped or scaled independently, fitting into existing UBOS platform overview micro‑services.
• Reduced Labeling Cost: By leveraging BM25 to prune the candidate set, the LLM operates on a focused context, lowering inference latency and cloud compute bills.
• Enhanced Analytics: Consistent, graph‑validated tags enable downstream agents to recommend learning paths, detect skill gaps, and personalize content delivery.

In practice, an LMS could embed the pipeline within its Workflow automation studio, allowing administrators to trigger automatic tagging whenever new resources are uploaded. The evidence spans become part of the metadata, ready for consumption by recommendation engines or reporting dashboards.

What Comes Next

While the results are promising, several avenues remain open for research and productization:

• Multilingual Extension: Adapting the pipeline to support non‑English competency frameworks would broaden its applicability to global institutions.
• Cross‑Domain Transfer: Testing the approach on medical, legal, or corporate training corpora could reveal domain‑specific challenges in graph construction.
• Active Learning Loop: Incorporating educator feedback on evidence spans to fine‑tune the LLM in a continual learning setting.
• Scalable Graph Stores: Leveraging vector‑enabled graph databases such as Chroma DB integration to handle larger competency ontologies.
• Agent‑Driven Orchestration: Deploying autonomous agents that monitor tagging quality, trigger re‑tagging when curricula evolve, and surface alerts via Telegram integration on UBOS.

Future work could also explore hybrid prompting strategies that blend few‑shot examples with retrieval‑augmented generation, aiming to further reduce hallucination while preserving the richness of LLM reasoning.

Call to Action

For a deeper dive into the methodology, data, and code, read the original arXiv paper. If you’re interested in integrating similar AI‑driven tagging capabilities into your LMS, explore our Enterprise AI platform by UBOS or get started quickly with our UBOS templates for quick start.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.