In-Context Reinforcement Learning From Suboptimal Historical Data

Direct Answer

The paper introduces the Decision Importance Transformer (DIT), a novel in‑context reinforcement learning (ICRL) framework that can learn effective policies directly from suboptimal historical trajectories without any additional fine‑tuning. By treating the transformer as a conditional generator of actions and values, DIT extracts the latent decision‑importance signal embedded in imperfect data, enabling agents to act near‑optimally even when only noisy, biased logs are available.

Background: Why This Problem Is Hard

In many real‑world domains—online advertising, recommendation systems, autonomous robotics—organizations accumulate massive logs of past decisions. These logs are rarely generated by optimal policies; they reflect exploration, human heuristics, or legacy systems. Traditional reinforcement learning (RL) pipelines rely on either:

• Online interaction: agents collect fresh data by exploring the environment, which can be costly, risky, or infeasible.
• Offline RL / Imitation Learning: algorithms assume the logged data is near‑optimal or at least covers the optimal policy’s support. When the data is suboptimal, value estimates become biased, and policies may inherit the same shortcomings.

Transformer‑based in‑context learning has shown that large language models can perform few‑shot reasoning by conditioning on examples in the prompt. Extending this capability to RL suggests that a model could “read” a batch of past trajectories and instantly infer a policy for a new task. However, two technical obstacles have prevented this vision from scaling:

1. Decision‑importance estimation: The model must differentiate high‑value actions from low‑value ones within noisy logs, a problem akin to off‑policy evaluation.
2. Training signal mismatch: Standard next‑token prediction treats all demonstrated actions as equally correct, which is inappropriate when many actions are suboptimal.

Consequently, existing ICRL approaches either require curated expert demonstrations or suffer severe performance degradation when faced with realistic, imperfect datasets.

What the Researchers Propose

DIT tackles the above challenges by introducing two tightly coupled transformer modules:

• Value‑Function Transformer (V‑Transformer): Trained to predict the return‑to‑go for each state‑action pair in a trajectory, it learns a latent representation of “how important” each decision was for the eventual outcome.
• Policy Transformer (π‑Transformer): Conditioned on the same context, it generates the next action using a weighted maximum‑likelihood objective, where the weight for each demonstrated action is proportional to the estimated decision importance from the V‑Transformer.

By jointly optimizing these components, DIT converts suboptimal logs into a rich supervision signal: actions that contributed positively to the final reward receive higher weights, while detrimental actions are down‑weighted. The framework operates entirely in‑context: at inference time, a user supplies a prompt consisting of a few recent trajectory snippets, and the π‑Transformer emits the next action without any gradient updates.

How It Works in Practice

The end‑to‑end workflow can be broken down into three conceptual stages:

1. Data Preparation: Historical logs are segmented into overlapping windows (e.g., 10‑step sequences). Each window includes state, action, and reward information.
2. Dual‑Transformer Training:
   • The V‑Transformer receives the window and learns to predict the cumulative future reward for each timestep, effectively estimating a per‑step advantage.
   • The π‑Transformer receives the same window and learns to predict the next action. Its loss is scaled by the advantage estimates from the V‑Transformer, ensuring that high‑impact decisions dominate the learning signal.
3. Inference (In‑Context Decision Making):
   • A user provides a prompt containing a few recent (state, action, reward) tuples from the current episode.
   • The V‑Transformer quickly computes importance scores for the prompt, which are fed to the π‑Transformer.
   • The π‑Transformer outputs the next action, ready to be executed in the environment.

Key differentiators of DIT include:

• Explicit modeling of decision importance rather than treating all demonstrations equally.
• Seamless integration of value estimation and policy generation within a single transformer‑based architecture.
• Zero‑shot adaptation to new tasks via prompt engineering, eliminating the need for costly fine‑tuning.

Evaluation & Results

The authors benchmarked DIT on two families of environments:

Bandit Experiments

In synthetic contextual bandit settings, they generated logs with varying degrees of suboptimality (e.g., 30 % random actions, 70 % greedy). DIT consistently outperformed:

• Standard behavior cloning (BC), which treats all actions equally.
• Weighted behavior cloning (WBC) that uses simple reward‑based weights.
• Offline RL baselines such as Conservative Q‑Learning (CQL).

Even when the historical data contained only 10 % optimal actions, DIT achieved near‑optimal regret, demonstrating its ability to extract useful signal from heavily noisy logs.

MDP Benchmarks

For more complex sequential decision problems, the paper evaluated DIT on classic control tasks (CartPole, MountainCar) and a suite of Atari games. The historical datasets were deliberately corrupted by injecting suboptimal policies and stochastic perturbations. Results showed:

• On CartPole, DIT reached 95 % of the optimal episode reward, whereas BC plateaued at 70 %.
• On Atari “Breakout”, DIT surpassed the best offline RL baseline by 12 % in average score, despite using only 40 % of the data generated by an expert policy.
• Training time was comparable to a single transformer pass, highlighting the efficiency of the in‑context approach.

These findings confirm that DIT not only tolerates suboptimal data but also leverages it to learn robust policies without any environment interaction.

Why This Matters for AI Systems and Agents

From a systems‑building perspective, DIT opens several practical pathways:

• Rapid Prototyping: Engineers can spin up a decision‑making agent by simply feeding it recent logs, bypassing the lengthy data‑collection cycles typical of RL pipelines.
• Safety‑Critical Domains: In fields like healthcare or autonomous driving, exploring the environment is risky. DIT enables policy improvement using only historical records, reducing exposure to unsafe actions.
• Orchestration of Heterogeneous Agents: When multiple subsystems generate logs of varying quality, DIT’s importance weighting naturally fuses them, allowing a central orchestrator to derive a coherent policy.
• Scalable Offline Learning: Because the model operates in‑context, the same pretrained transformer can serve countless downstream tasks, lowering compute costs for large enterprises.

For teams building AI‑driven products, DIT offers a plug‑and‑play component that can be integrated into existing data pipelines. For example, a recommendation engine could ingest click‑through logs (which are inherently suboptimal) and instantly generate a policy that better balances exploration and exploitation. More details on integrating such components can be found in our research hub.

What Comes Next

While DIT marks a significant step forward, several open challenges remain:

• Long‑Horizon Credit Assignment: In very long episodes, the V‑Transformer’s return‑to‑go estimates may become noisy. Future work could incorporate hierarchical prompting or memory‑augmented transformers to improve credit assignment.
• Multi‑Agent Extensions: Extending the importance‑weighting mechanism to settings where multiple agents interact (e.g., traffic control) will require new ways to disentangle individual contributions.
• Robustness to Distribution Shift: When the deployment environment diverges from the logged data distribution, DIT may need additional calibration, perhaps via lightweight online fine‑tuning.
• Interpretability: The decision‑importance scores provide a natural interpretability signal, but systematic tools for visualizing and auditing these scores are still lacking.

Addressing these topics will broaden DIT’s applicability to domains such as finance, robotics, and large‑scale personalization. Researchers interested in building on this work are encouraged to explore the open‑source implementation released alongside the paper and to contribute to the community discussion on our blog platform.

For a complete technical description, see the original preprint: In‑Context Reinforcement Learning From Suboptimal Historical Data.