Andrii Bidochko

• Updated: June 17, 2026
• 7 min read

FD-RAG: Federated Dual-System Retrieval-Augmented Generation

Direct Answer

FD‑RAG introduces a federated, dual‑system Retrieval‑Augmented Generation framework that separates lightweight memory lookup from on‑demand large‑language‑model (LLM) reasoning, enabling accurate, low‑latency question answering on edge devices without sharing raw documents. It matters because it tackles the twin challenges of knowledge fragmentation and prohibitive inference costs that have limited the deployment of RAG‑style AI in privacy‑sensitive, compute‑constrained environments.

Figure 1: High‑level view of the FD‑RAG dual‑system architecture.

Background: Why This Problem Is Hard

Retrieval‑augmented generation (RAG) has become the de‑facto method for grounding LLMs in up‑to‑date factual knowledge. Traditional RAG pipelines assume a single, centrally hosted knowledge base and unlimited compute for repeated LLM calls. In practice, three hard constraints break that assumption:

• Edge‑centric data silos: Sensors, smartphones, and IoT gateways each hold proprietary corpora that cannot be moved to a cloud server due to privacy regulations (e.g., GDPR, HIPAA) or bandwidth limits.
• Cost of LLM inference: State‑of‑the‑art LLMs consume dozens of GPU seconds per query, making real‑time responses on battery‑powered devices economically infeasible.
• Knowledge fragmentation: When relevant facts are scattered across many devices, a single local index cannot answer a query that spans multiple sources, leading to reduced recall.

Existing solutions either replicate the entire knowledge base on each device—wasting storage and exposing data—or rely on a central server that defeats the purpose of edge deployment. Consequently, developers lack a principled way to combine privacy‑preserving data sharing with the expressive power of LLMs.

What the Researchers Propose

The authors present FD‑RAG (Federated Dual‑System Retrieval‑Augmented Generation), a two‑layer architecture that decouples “memory matching” from “LLM reasoning.” The core ideas are:

• Semantic‑aware adaptive hypergraphs: Each device builds a hypergraph over its local corpus, where nodes represent passages and hyperedges capture multi‑way semantic relationships. The hypergraph is distilled into a compact “QA memory” that stores question‑answer pairs with provenance links back to the hypergraph.
• Dual‑system inference: At query time, the system first attempts a direct match against the local QA memory. If confidence exceeds a learned threshold, the answer is returned instantly. Otherwise, the query is escalated to an on‑demand LLM call, which can also consult the hypergraph for evidence.
• Federated memory aggregation: Devices anonymously share only the distilled QA memories (not raw documents). A central coordinator merges these memories into a global index, enabling cross‑device knowledge without exposing proprietary text.

In short, FD‑RAG creates a lightweight, privacy‑preserving knowledge layer that handles the majority of queries locally, while reserving expensive LLM reasoning for the hard cases.

How It Works in Practice

The operational workflow can be broken into three stages: local hypergraph construction, federated memory sharing, and dual‑system inference.

1. Local Hypergraph Construction

Each edge device ingests its raw documents and runs a semantic encoder (e.g., a sentence‑transformer) to embed passages. Passages that share high cosine similarity are linked via hyperedges, forming a multi‑dimensional graph that captures topics, entities, and logical relations. The hypergraph is then pruned using an adaptive budget that respects device memory limits.

2. QA Memory Distillation

From the hypergraph, the system generates synthetic question‑answer pairs by sampling high‑centrality nodes and prompting a small, on‑device LLM. Each pair is stored with a pointer to the originating hyperedge, creating a traceable “memory slot.” This slot is what other devices will later query.

3. Federated Aggregation

Periodically, devices upload encrypted QA memories to a federated server. The server performs a privacy‑preserving merge (e.g., secure aggregation) to produce a global memory bank. No raw text ever leaves the device, satisfying data‑ownership constraints.

4. Dual‑System Inference

1. Memory Matching: When a user asks a question, the device first searches its local QA memory using a fast vector similarity lookup. If the top‑k matches exceed a confidence threshold (learned during training), the answer is returned instantly, and the associated hypergraph evidence is attached for transparency.
2. LLM Reasoning: If the confidence is low, the query is forwarded to a remote LLM (or a more powerful on‑device model). The LLM can request additional context from the hypergraph or the global memory bank, ensuring it has access to the most relevant facts before generating a response.

This bifurcated approach reduces the number of costly LLM calls dramatically while preserving the ability to answer complex, multi‑source questions.

Evaluation & Results

The authors benchmarked FD‑RAG on three widely used QA datasets (Natural Questions, TriviaQA, and a custom edge‑device corpus). They compared against two baselines:

• Local RAG: Each device runs a conventional RAG pipeline with its own index.
• Federated RAG (non‑dual): Devices share raw embeddings but still invoke the LLM for every query.

Key findings include:

• Accuracy boost: FD‑RAG achieved up to a 7.8 % absolute improvement in exact‑match scores over the strongest baseline, demonstrating that hypergraph‑grounded memories capture richer semantics.
• Latency reduction: Average query latency dropped by 8.4× because 73 % of queries were satisfied by memory matching alone.
• Communication savings: Federated memory exchange required 12 % of the bandwidth of raw embedding sharing, confirming the privacy‑preserving advantage.
• Scalability: Experiments with up to 1,000 simulated edge devices showed linear convergence of the global memory quality, matching the theoretical O(1/ε²) rate proved in the paper.

These results illustrate that FD‑RAG not only preserves answer quality but also delivers tangible performance and privacy benefits in realistic edge scenarios.

Why This Matters for AI Systems and Agents

For practitioners building AI‑driven agents, FD‑RAG offers a concrete blueprint to reconcile three often‑conflicting goals:

• Low‑cost inference: By handling the majority of queries locally, developers can deploy agents on commodity hardware, extending AI capabilities to smartphones, wearables, and industrial sensors.
• Data sovereignty: The federated memory model aligns with emerging regulations that forbid raw data export, making compliance a built‑in feature rather than an afterthought.
• Explainability: Each answer is traceable to a hypergraph node, providing agents with provenance that can be displayed to end‑users or fed into audit pipelines.

These advantages translate directly into product‑level benefits. For example, a customer‑support chatbot that runs on a retailer’s point‑of‑sale terminals can answer inventory questions instantly from its local memory, while still being able to consult a central LLM for rare product specifications. Similarly, autonomous drones can retrieve navigation cues from on‑board memories without streaming video to the cloud, preserving bandwidth and mission‑critical latency.

Developers looking to integrate FD‑RAG concepts can start with existing UBOS tools. The Chroma DB integration provides a vector store that can host the distilled QA memories, while the Workflow automation studio can orchestrate the periodic federated aggregation. For voice‑enabled agents, pairing FD‑RAG with the ElevenLabs AI voice integration creates a seamless spoken‑question experience on edge devices.

What Comes Next

While FD‑RAG marks a significant step forward, several open challenges remain:

• Dynamic corpora: Edge devices frequently add or delete documents. Efficiently updating hypergraphs without full recomputation is an active research area.
• Heterogeneous LLMs: The current design assumes a single remote LLM. Future work could explore model‑selection policies that route queries to specialized experts (e.g., medical vs. legal).
• Robustness to adversarial memories: Since devices contribute synthetic QA pairs, mechanisms to detect and mitigate poisoned memories are needed for secure deployments.

Potential extensions include integrating FD‑RAG with UBOS’s enterprise AI platform to provide out‑of‑the‑box federated orchestration, or leveraging the UBOS templates for quick start to prototype domain‑specific agents in weeks rather than months.

Researchers are also encouraged to build on the theoretical foundation presented in the paper. The O(1/ε²) convergence guarantee suggests that hypergraph learning can be scaled to millions of devices with bounded error, opening the door to continent‑wide AI ecosystems that respect local data ownership.

For a deeper dive into the methodology and to explore the codebase, see the original FD‑RAG paper. Companies interested in piloting federated RAG solutions can contact the UBOS team via the About UBOS page.

Ready to bring decentralized, low‑latency AI to your products? Visit the UBOS homepage to learn how our platform can accelerate your edge‑AI initiatives.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.