Andrii Bidochko

• Updated: July 23, 2026
• 8 min read

HERO: A Heterogeneity‑Aware Benchmark Library for Federated Continual Learning

Direct Answer

The paper introduces HERO (Heterogeneity‑Aware Benchmark Library for Federated Continual Learning), a modular suite that isolates the three major sources of variability in federated continual learning (FCL): task splits, client data splits, and client task sequences. By decoupling these factors, HERO enables reproducible, apples‑to‑apples comparisons of FCL algorithms across easy and highly heterogeneous settings, which is essential for building reliable AI agents that learn continuously from distributed data streams.

Background: Why This Problem Is Hard

Federated continual learning sits at the intersection of two demanding research fronts. On one side, federated learning (FL) must train models across many devices or silos without sharing raw data, coping with communication constraints, privacy regulations, and wildly different data distributions (statistical heterogeneity). On the other side, continual learning (CL) requires a model to assimilate a stream of tasks while preserving knowledge from earlier tasks (catastrophic forgetting). When these challenges are combined, the evaluation landscape becomes a maze of intertwined variables:

• Dataset choice: Researchers often swap CIFAR‑100 for TinyImageNet or switch to graph‑based domains, making direct performance comparison impossible.
• Task split definition: Some works treat each class as a task, others group classes arbitrarily, which changes the difficulty of the forgetting problem.
• Client data split: The degree of data skew (e.g., some clients see only a few classes) dramatically influences convergence speed and fairness.
• Task order per client: Real‑world devices rarely receive tasks in a synchronized order; mismatched sequences can favor different algorithmic strategies.
• Memory assumptions: Some methods assume a replay buffer, others rely on regularization only, yet papers rarely state these constraints uniformly.

Because most prior benchmarks conflate several of these dimensions, a reported improvement may stem from a more favorable data split rather than a genuine algorithmic advance. This lack of standardization hampers reproducibility, slows progress, and makes it difficult for product teams to translate research breakthroughs into production AI agents.

What the Researchers Propose

HERO tackles the reproducibility crisis by providing a heterogeneity‑aware benchmark library that explicitly separates three orthogonal choices:

1. Task Split: Defines how the global dataset is partitioned into sequential learning tasks (e.g., 20 tasks of 5 classes each on CIFAR‑100).
2. Client Data Split: Controls the distribution of data among federated clients, parameterized by α to adjust the level of class‑wise skew.
3. Client Task Sequence: Determines the order in which each client encounters the tasks, governed by ρ to introduce task‑order mismatch across the federation.

By exposing these knobs as independent configuration parameters, HERO lets researchers construct a family of benchmark streams that range from “ideal” (identical task order, balanced data) to “realistic” (highly skewed data, divergent task orders). The library ships with:

• Pre‑defined benchmark streams for image classification (CIFAR‑100, TinyImageNet) and graph‑based domain‑incremental learning (OGB‑MolPCBA).
• Reference implementations of representative FCL methods (e.g., FedAvg‑CL, Replay‑based, Regularization‑based).
• Standardized reporting scripts that compute final average accuracy, average forgetting, and bottom‑10 % client accuracy.

In short, HERO is not a new algorithm; it is a research infrastructure that makes the “heterogeneity” dimension first‑class, enabling fair, setting‑aware evaluation of any FCL technique.

How It Works in Practice

The HERO workflow can be visualized as a three‑stage pipeline:

1. Stream Generation: Users select a base dataset and specify the desired α (data skew) and ρ (task‑order mismatch). HERO then automatically partitions the data into K clients, each receiving a personalized task sequence.
2. Algorithm Plug‑In: Any federated continual learning algorithm can be registered via a simple Python interface. The library handles model broadcasting, local updates, and optional replay buffer synchronization, abstracting away the networking details.
3. Evaluation & Reporting: After the last task, HERO aggregates metrics across all clients. It reports both aggregate statistics (average accuracy) and distribution‑aware figures (bottom‑10 % client accuracy) to surface fairness concerns.

What distinguishes HERO from earlier benchmark suites is the explicit decoupling of the three variability sources. For example, two experiments can share the exact same task split but differ only in α, allowing a clean study of how data skew alone impacts a method’s forgetting behavior. Similarly, by fixing α and varying ρ, researchers can isolate the effect of asynchronous task ordering.

Evaluation & Results

The authors evaluated HERO on two image classification streams (CIFAR‑100 and TinyImageNet) and a graph‑based domain‑incremental learning (Domain‑IL) case study using OGB‑MolPCBA. The experiments compared four representative FCL strategies:

• Federated Averaging with Elastic Weight Consolidation (FedAvg‑EWC)
• Replay‑based method with a shared buffer (FedReplay)
• Regularization‑only approach (FedLwF)
• Baseline federated training without continual mechanisms (FedAvg)

Key findings include:

In the OGB‑MolPCBA case study, the authors swapped the scaffold‑based domain granularity while keeping the prediction task constant. HERO revealed that even when the label space does not change, domain shift alone can cause severe accuracy drops, underscoring the need for domain‑aware continual strategies beyond image classification.

Overall, the results demonstrate that:

• Average accuracy alone can mask poor performance on the most disadvantaged clients.
• Task‑order mismatch (high ρ) favors distillation‑based methods, whereas data skew (high α) benefits replay mechanisms.
• The same algorithm can behave differently across heterogeneous settings, highlighting the importance of a benchmark like HERO for comprehensive evaluation.

For a deeper dive into the experimental setup, see the original arXiv paper.

Why This Matters for AI Systems and Agents

Practitioners building AI agents that operate on edge devices, mobile phones, or enterprise silos face the exact heterogeneity HERO models. The benchmark provides actionable insights:

• Algorithm selection: By matching the heterogeneity profile of a production fleet (e.g., high data skew in retail stores) to the HERO configuration that favors replay, engineers can choose the most suitable FCL method without costly A/B tests.
• Fairness auditing: Bottom‑10 % client accuracy surfaces inequities that would be invisible in a global average, enabling teams to enforce service‑level agreements across all devices.
• Continuous deployment pipelines: HERO’s modular API can be integrated into CI/CD workflows, automatically re‑evaluating new model updates under varied heterogeneity scenarios before rollout.
• Cross‑domain portability: The Domain‑IL case study shows that HERO is not limited to vision; it can assess graph‑based or language‑centric federated streams, making it a universal tool for multi‑modal agents.

Organizations looking to operationalize federated continual learning can leverage HERO alongside existing UBOS capabilities. For instance, the UBOS platform overview offers a managed environment for deploying federated workloads, while the Workflow automation studio can orchestrate the data‑split generation and metric collection steps defined by HERO.

What Comes Next

While HERO marks a significant step toward reproducible FCL research, several open challenges remain:

• Scalability to thousands of clients: Current experiments cap at a few hundred participants; extending the library to massive federations will require more efficient communication simulators.
• Privacy‑preserving metrics: Reporting bottom‑client performance may conflict with differential privacy guarantees; future versions could integrate privacy‑aware aggregation.
• Dynamic client participation: Real‑world federations see clients drop in and out. Incorporating churn models into HERO streams would better reflect production environments.
• Automated hyper‑parameter tuning: The interaction between α, ρ, and algorithmic hyper‑parameters is complex. Meta‑learning or Bayesian optimization wrappers could help discover robust settings.

Researchers interested in extending HERO can start by cloning the public repository, customizing the alpha and rho generators, and contributing new task split definitions. The library’s open‑source nature encourages community‑driven expansion into domains such as natural language processing, reinforcement learning, and multimodal sensor fusion.

For teams ready to experiment, the Enterprise AI platform by UBOS provides pre‑configured compute clusters and secure data pipelines that align with HERO’s federated simulation requirements. Pairing HERO with UBOS’s AI marketing agents can illustrate how continual learning improves personalization over time while respecting client‑level data privacy.

In summary, HERO equips the AI community with a rigorous, heterogeneity‑aware lens to evaluate federated continual learning. By standardizing the experimental variables that have long confounded reproducibility, it paves the way for more reliable, fair, and scalable AI agents that learn continuously across the globe.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.