Andrii Bidochko

• Updated: July 7, 2026
• 7 min read

Automated Data Readiness for Scientific AI – REDI Framework Overview

Direct Answer

REDI is an open‑source, five‑stage framework that automates the end‑to‑end preparation of massive scientific datasets for AI training, while guaranteeing FAIR compliance, full provenance, and agent‑native deployment. It matters because it turns a historically manual, error‑prone bottleneck into a reproducible, scalable service that can feed next‑generation AI models across climate science, proteomics, materials discovery, and nuclear fusion.

Background: Why This Problem Is Hard

Leadership‑class supercomputing facilities host petabytes of raw experimental and simulation data. Before these data can be used to train modern foundation models, they must undergo a cascade of transformations: format conversion, cleaning, feature extraction, metadata enrichment, and finally structuring into tensors or shards that AI pipelines understand. Each step typically requires domain‑specific scripts, ad‑hoc glue code, and manual validation by scientists.

Existing solutions address fragments of this workflow. Data‑ingestion tools handle transfer, while separate ETL platforms perform cleaning, and yet another suite manages metadata cataloging. The lack of a unified, instrumented pipeline leads to:

• Reproducibility gaps: provenance is lost when scripts are swapped or run on different clusters.
• Scalability limits: hand‑crafted pipelines cannot exploit thousands of nodes without extensive re‑engineering.
• FAIR compliance drift: ensuring that datasets remain Findable, Accessible, Interoperable, and Reusable is an after‑thought rather than a built‑in guarantee.
• Agent integration friction: emerging AI agents (e.g., autonomous data‑curation bots) cannot invoke legacy scripts reliably because the interfaces are not standardized.

These challenges are magnified today as AI research shifts from narrow, curated benchmarks to domain‑specific, high‑resolution scientific corpora. The cost of data preparation now dominates project budgets, and any inconsistency propagates into model bias or training failures.

What the Researchers Propose

The authors introduce REDI (Readiness Engine for Data Integration), a modular framework that unifies ingestion, preprocessing, transformation, structuring, and output into a single, instrumented pipeline. REDI’s design rests on three pillars:

1. Stage‑level instrumentation: each of the five stages emits detailed provenance records (file hashes, timestamps, configuration snapshots) that are automatically stored in a searchable catalog.
2. Agent‑native skill exposure: the entire pipeline can be invoked as a callable skill from autonomous agents, enabling “data‑ready‑on‑demand” behavior without human intervention.
3. FAIR‑first companion tool (SetGo): SetGo automates metadata generation, validates compliance, and publishes datasets to community registries, turning raw files into reusable assets.

Key components include a lightweight orchestration engine, pluggable adapters for file formats (NetCDF, HDF5, CSV, etc.), and a provenance database that can be queried by downstream agents or audit tools.

How It Works in Practice

Conceptually, REDI follows a linear yet highly parallelizable flow:

1. Ingest: Data are pulled from storage systems (parallel file systems, object stores, or streaming APIs) using adapters that verify checksum integrity.
2. Preprocess: Basic cleaning (missing‑value imputation, unit conversion) and lightweight statistical summaries are applied. This stage can be distributed across compute nodes using MPI or Dask.
3. Transform: Domain‑specific feature extraction occurs here—e.g., converting raw climate model outputs into gridded tensors, or extracting peptide spectra into fixed‑length vectors. The framework supplies a library of common transforms while allowing custom plug‑ins.
4. Structure: Processed data are reorganized into AI‑friendly containers (TFRecord, Arrow, or custom shard formats). Metadata is attached, and the dataset is partitioned for training/validation splits.
5. Output: Final artifacts are written to a target location, registered in the provenance store, and optionally published via SetGo to a FAIR catalog.

What distinguishes REDI from traditional pipelines is its built‑in observability. At each stage, timers, I/O counters, and resource usage metrics are captured without requiring developers to instrument code manually. The provenance graph can be visualized to pinpoint bottlenecks—often file I/O or suboptimal format choices—enabling data engineers to apply first‑order optimizations automatically.

Because the pipeline is exposed as an agent‑callable skill, an autonomous “data curator” bot can request a new dataset, specify transformation parameters, and receive a ready‑to‑train shard set—all through a single API call. This eliminates the need for bespoke scripting each time a scientist wants to experiment with a new data source.

Evaluation & Results

The REDI team validated the framework on four distinct scientific domains:

• Climate modeling: Terabytes of CMIP6 output were ingested, normalized, and reshaped into 3‑D tensors for a convolutional climate‑forecast model.
• Proteomics: Mass‑spectrometry raw files were converted into peptide‑level embeddings for a protein‑function predictor.
• Materials science: High‑throughput DFT calculations were transformed into graph representations for a crystal‑property network.
• Nuclear fusion: Time‑resolved diagnostic images were structured into sequences for a plasma‑stability classifier.

Across all cases, REDI succeeded in converting raw archives into AI‑ready datasets that matched or exceeded domain‑expert reference pipelines. Crucially, the authors reported near‑ideal parallel scaling on the Frontier supercomputer: when processing the climate case, runtime decreased proportionally up to 100 nodes, confirming that the framework’s design does not become a bottleneck at scale.

Profiling revealed that file I/O accounted for roughly 60 % of total execution time, while format conversion contributed the next largest share. By swapping to a columnar format (Apache Arrow) in the Structure stage, the team cut overall runtime by 15 % without altering downstream model performance, illustrating the practical value of REDI’s instrumentation.

Why This Matters for AI Systems and Agents

For AI practitioners, REDI offers a turnkey solution that eliminates the “data‑laundering” phase that traditionally consumes weeks of engineering effort. By guaranteeing FAIR compliance through SetGo, organizations can share datasets internally and externally without re‑curating metadata, accelerating collaboration across research groups.

Agent builders gain a reliable skill that can be composed into larger workflows. An autonomous research assistant could, for example, trigger REDI to refresh a climate dataset nightly, validate provenance, and then hand the new shards to a training orchestrator—all without human oversight. This level of automation is essential for continuous‑learning systems that must adapt to evolving scientific data streams.

System designers also benefit from the built‑in profiling. Knowing that I/O dominates execution lets architects provision high‑throughput storage tiers or adopt in‑situ compression strategies, directly improving cost‑efficiency on cloud or on‑prem HPC environments.

For teams looking to embed AI into their products, the UBOS platform overview demonstrates how REDI‑style skills can be integrated into broader enterprise AI stacks, providing a seamless bridge between raw scientific data and downstream analytics.

What Comes Next

While REDI marks a significant step forward, several open challenges remain:

• Dynamic data sources: Real‑time sensor feeds and streaming simulations require incremental ingestion capabilities that are not yet fully supported.
• Cross‑domain schema harmonization: Automating the alignment of metadata vocabularies across disciplines would further reduce manual effort.
• Security and access control: Fine‑grained policies for multi‑tenant HPC environments need tighter integration with the provenance store.
• Agent feedback loops: Enabling agents to suggest optimizations (e.g., alternative transforms) based on downstream model performance is an exciting research direction.

Future work could extend REDI with a plug‑in for Chroma DB integration, allowing vector‑searchable embeddings to be stored alongside raw files, thereby supporting similarity‑based retrieval directly from the pipeline.

In the longer term, we anticipate a ecosystem where REDI‑style pipelines are cataloged as reusable services, discoverable via FAIR registries, and orchestrated by autonomous AI agents that negotiate data contracts, monitor provenance, and trigger re‑training cycles on demand.

References

Automated Data Readiness for Scientific AI

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.