REDI Framework: Automated Data Readiness for Scientific AI

Direct Answer

REDI is an open‑source, five‑stage pipeline that automatically converts raw scientific datasets into AI‑ready formats while guaranteeing reproducibility, provenance, and FAIR compliance. By exposing each stage as an agent‑callable skill, REDI turns a traditionally manual, error‑prone data‑preparation bottleneck into a reusable, scalable service for climate, proteomics, materials science, and nuclear‑fusion research.

Background: Why This Problem Is Hard

Leadership‑class supercomputing facilities host petabytes of heterogeneous data—satellite imagery, mass‑spectrometry files, molecular simulations, and plasma diagnostics. Before these assets can fuel large‑scale AI models, they must undergo:

• Format conversion (e.g., NetCDF → TFRecord)
• Cleaning and normalization (removing corrupt records, aligning timestamps)
• Metadata enrichment (adding provenance, units, and domain ontologies)
• Structuring for downstream pipelines (sharding, indexing, batching)

Current workflows are handcrafted scripts stitched together with ad‑hoc glue code. They suffer from three systemic issues:

1. Fragmentation: Each research group builds its own pipeline, leading to duplicated effort and divergent standards.
2. Lack of provenance: Without systematic tracking, reproducing a model’s training data becomes impossible, violating FAIR principles.
3. Scalability gaps: Manual pipelines cannot exploit the parallelism of modern HPC systems, causing costly I/O bottlenecks.

These challenges impede rapid AI experimentation, delay scientific discovery, and increase the total cost of ownership for data‑intensive projects.

What the Researchers Propose

The authors introduce REDI (Reusable, Extensible Data‑Readiness Infrastructure), a unified framework that codifies the entire data‑preparation lifecycle into five deterministic stages:

1. Ingest: Securely acquire raw files from storage tiers, object stores, or streaming sources.
2. Preprocess: Perform sanity checks, de‑duplication, and basic cleaning.
3. Transform: Convert formats, apply domain‑specific calibrations, and generate intermediate representations.
4. Structure: Organize data into sharded, indexed collections optimized for AI training (e.g., Parquet, TFRecord).
5. Output: Publish the final AI‑ready dataset to a catalog, attach provenance metadata, and expose a callable skill for downstream agents.

Each stage is instrumented with lightweight profiling hooks that capture execution time, I/O volume, and resource utilization. The framework also ships a companion tool called SetGo, which automates FAIR compliance checks and registers the dataset in a searchable catalog.

How It Works in Practice

Figure 1 (below) visualizes the end‑to‑end flow. A user or autonomous agent triggers REDI via a simple API call; the framework then orchestrates the five stages on the target HPC environment.

Component Interaction

• Orchestrator: A lightweight daemon that schedules stage containers on available compute nodes, respecting data locality.
• Stage Workers: Containerized micro‑services (Docker/Singularity) that implement the logic for ingest, preprocess, etc. Workers report metrics to the orchestrator.
• Provenance Store: A versioned metadata repository (based on JSON‑LD) that records every transformation, input checksum, and software environment.
• Agent Interface: A JSON‑RPC endpoint that lets external AI agents invoke REDI as a “skill,” enabling on‑demand data preparation within larger workflows.
• SetGo CLI: Runs after the Output stage to validate FAIR criteria, generate a DOI‑like identifier, and push the dataset record to the central catalog.

What sets REDI apart is the tight coupling of instrumentation and agent‑native exposure. Traditional ETL pipelines log only success/failure; REDI logs fine‑grained performance counters that agents can query to decide whether to request a different format or to parallelize further.

Evaluation & Results

The authors benchmarked REDI on four distinct scientific domains, each with its own data characteristics and AI use cases.

Test Scenarios

• Climate Science: 12 PB of satellite radiance and reanalysis fields, targeting a convolutional climate‑forecast model.
• Proteomics: 3 TB of mass‑spectrometry raw files, feeding a transformer‑based protein‑structure predictor.
• Materials Science: 800 GB of electron‑diffraction images, used for a graph‑neural‑network alloy discovery pipeline.
• Nuclear Fusion: 1.5 PB of high‑speed camera and sensor streams, supporting a reinforcement‑learning controller for tokamak plasma.

Key Findings

• Full‑stack readiness: REDI successfully transformed every raw dataset into a validated AI‑ready collection without manual intervention.
• Scalability: On the Frontier supercomputer, the climate case achieved near‑linear speed‑up up to 100 nodes, reducing end‑to‑end latency from 48 h (hand‑crafted scripts) to 3.2 h.
• Cost drivers: Profiling revealed that file I/O accounted for ~62 % of total runtime. Switching from HDF5 to columnar Parquet cut I/O time by 28 % and halved storage costs.
• Provenance impact: The embedded provenance layer added only 3 % overhead but enabled instant reproducibility checks, which saved weeks of debugging in downstream model training.
• FAIR compliance: SetGo automatically generated rich metadata (schema.org, DataCite) and published the datasets to a public catalog, meeting all four FAIR principles.

Collectively, these results demonstrate that REDI is not a niche prototype but a cross‑domain platform capable of handling the most demanding scientific AI workloads.

Why This Matters for AI Systems and Agents

For AI practitioners, REDI eliminates the “data‑wrangling” phase that traditionally consumes 70‑80 % of project time. By exposing a standardized skill interface, autonomous agents can request precisely the data shape they need—whether it’s a sharded TFRecord for a GPU trainer or a streaming Parquet feed for an online inference service.

System designers gain several concrete benefits:

• Deterministic pipelines: Every transformation is versioned, enabling reproducible experiments and compliance audits.
• Dynamic orchestration: Agents can query the provenance store to decide whether to reuse an existing transformed artifact or trigger a fresh run, optimizing compute budgets.
• Scalable I/O strategies: REDI’s format‑selection heuristics guide agents toward the most efficient storage layout for their hardware, reducing network traffic on shared HPC fabrics.
• FAIR‑first mindset: By integrating SetGo, agents automatically inherit metadata that downstream discovery services (e.g., data marketplaces) can index.

Enterprises looking to embed AI into scientific workflows can therefore accelerate time‑to‑value, lower operational risk, and meet regulatory data‑management mandates.

Explore how REDI‑style automation can be combined with existing UBOS platform overview capabilities for end‑to‑end AI orchestration.

What Comes Next

While REDI marks a significant step forward, the authors acknowledge several open challenges:

• Domain‑specific extensions: Adding plug‑ins for emerging data types (e.g., quantum‑simulation outputs) will require community‑driven schema evolution.
• Real‑time streaming: Current stages assume batch ingestion; extending the pipeline to handle low‑latency sensor streams is an active research direction.
• Cost‑aware scheduling: Integrating spot‑instance pricing models could further reduce HPC spend, but would need robust fault‑tolerance mechanisms.

Future research may also explore tighter coupling with Workflow automation studio to let non‑technical users compose REDI‑based data‑preparation graphs via a visual interface.

Potential applications extend beyond scientific AI. Industries such as finance, healthcare, and autonomous transportation can adopt the same five‑stage model to enforce data governance while scaling model training on cloud or edge clusters.

For teams ready to experiment, the REDI codebase is hosted on GitHub under an Apache‑2.0 license, and the Automated Data Readiness for Scientific AI (arXiv) paper provides a detailed implementation guide.