Andrii Bidochko

• Updated: June 13, 2026
• 7 min read

From Knowing to Doing: A Memory‑Controlled Benchmark for LLM Trading Agents on Stock Markets

Direct Answer

The paper introduces KTD‑Fin (Knowing‑To‑Doing Financial Benchmark), a rigorously controlled end‑to‑end evaluation suite for large‑language‑model (LLM) trading agents that eliminates memorization leakage and decomposes returns into market, style, and genuine stock‑selection alpha. It matters because it reveals that current “profit‑making” benchmarks often overstate LLM skill, and it provides a reproducible template for measuring transferable investment intelligence.

Background: Why This Problem Is Hard

Financial markets are a natural proving ground for autonomous AI agents: they demand real‑time data ingestion, sequential decision making, and risk‑aware execution. Yet, evaluating whether an LLM truly “understands” market dynamics faces two intertwined obstacles.

• Knowledge‑cutoff leakage. State‑of‑the‑art LLMs are trained on data up to a fixed date (often late 2023). When backtesting over historical periods that overlap this cutoff, agents can simply recall ticker symbols, price levels, or headline events instead of reasoning from fundamentals.
• Noisy performance signals. Raw portfolio returns conflate pure alpha with exposure to market beta, sector tilts, or macro‑style factors. An agent that “makes money” might be riding a bullish market wave rather than demonstrating selective stock‑picking skill.

Existing benchmarks typically feed agents raw historical price series and then report cumulative returns. Without masking identifiers or attributing performance, such setups cannot distinguish memorized shortcuts from genuine analytical capability. As a result, research claims about “LLM‑driven alpha” remain questionable, and product teams lack reliable metrics for risk‑adjusted deployment.

What the Researchers Propose

The authors design a two‑pronged framework that directly tackles the leakage and attribution problems.

Data‑Side Masking Protocol

All market‑specific identifiers—ticker symbols, dates, and price levels—are systematically replaced with anonymized placeholders before they ever reach the LLM. The same masking is applied to any external tools (e.g., price lookup APIs) the agent may invoke, ensuring a consistent “knowledge‑blind” environment.

Barra‑Style Attribution Engine

After an agent completes a trading episode, the resulting portfolio is fed into a factor‑model decomposition that isolates three components:

1. Market exposure (beta to the overall index).
2. Style exposure (sector, size, value, momentum factors).
3. Stock‑selection alpha (residual return not explained by the first two).

This attribution mirrors industry‑standard performance analysis, allowing researchers to quantify how much of the profit is attributable to genuine decision making versus passive factor rides.

How It Works in Practice

The KTD‑Fin workflow can be visualized as a pipeline of four interacting modules:

1. Market Simulator. Supplies a chronological stream of anonymized market data (prices, volumes, corporate actions) for the target universe—in the paper’s case, the Chinese CSI300 index from 2024‑2026.
2. LLM Agent Core. A large language model (e.g., GPT‑4, Claude‑2) that receives masked prompts, decides on actions (buy, sell, hold), and optionally calls external “tool” APIs such as sentiment analyzers or factor calculators.
3. Tool Wrapper. Enforces the same masking rules on any data the agent requests, guaranteeing that the agent never sees raw identifiers.
4. Attribution Analyzer. After the simulation, this component runs the Barra‑style factor model on the agent’s trade log, outputting a detailed performance breakdown.

The key differentiator is the knowledge‑controlled nature of the data flow: every piece of market information is filtered through the masking layer before reaching the LLM or any auxiliary tool. This eliminates the possibility that the model “remembers” a price that happened to be in its pre‑training corpus.

Below is a schematic illustration of the KTD‑Fin benchmark architecture:

In practice, a researcher configures the simulator with a chosen time window, selects an LLM checkpoint, and runs multiple episodes to capture variance. The attribution analyzer then aggregates results across runs, producing both aggregate statistics and per‑episode insights.

Evaluation & Results

The authors evaluated ten frontier LLM agents—ranging from open‑source models fine‑tuned on finance data to proprietary APIs—on the masked CSI300 backtest. Two experimental conditions were compared:

• Unmasked baseline. Agents received raw tickers and dates, replicating typical backtesting setups.
• Masked KTD‑Fin. Identifiers were anonymized according to the protocol.

Impact of Masking on Agent Rationales

When identifiers were hidden, agents shifted from narrative‑driven justifications (“Apple announced a new iPhone”) to factor‑centric reasoning (“Increase exposure to high‑momentum, large‑cap stocks”). Qualitative analysis of generated rationales showed a marked reduction in references to specific events that could be memorized.

Attribution Decomposition Findings

Across the masked experiments, the attribution engine revealed that:

• Average market beta accounted for roughly 70 % of total returns, indicating that most profit came from riding overall market movements.
• Style exposures (size, value, momentum) explained an additional 20 %, suggesting agents learned to tilt toward well‑known factor premiums.
• The residual stock‑selection alpha was modest—often statistically indistinguishable from zero—implying limited evidence of genuine, transferable stock‑picking skill.

In contrast, the unmasked baseline reported higher cumulative returns, but attribution (performed post‑hoc) showed that the apparent outperformance was largely driven by memorized price spikes rather than systematic decision making.

These results collectively demonstrate that without leakage control, benchmark scores can be misleading, and that even sophisticated LLM agents currently rely heavily on passive factor exposure rather than true alpha generation.

Why This Matters for AI Systems and Agents

For practitioners building AI‑driven trading platforms, KTD‑Fin offers a reality‑check that aligns evaluation with industry‑grade risk metrics. The implications are threefold:

1. Design of Robust Agents. Knowing that raw returns are insufficient, developers will prioritize architectures that explicitly model factor exposures, risk constraints, and causal reasoning rather than relying on memorized market narratives.
2. Benchmark‑Driven Development. By integrating the masking protocol into continuous‑integration pipelines, teams can detect when model updates inadvertently re‑introduce leakage, ensuring that performance gains are genuine.
3. Regulatory and Compliance Readiness. Attribution reports satisfy many internal audit requirements, providing transparent evidence of how an AI system derives its trades—a prerequisite for deploying LLM agents in regulated environments.

These practical benefits map directly onto existing UBOS capabilities. For example, the OpenAI ChatGPT integration can be wrapped with the KTD‑Fin masking layer, while the Chroma DB integration can store anonymized market embeddings for rapid retrieval during simulation. Moreover, the Workflow automation studio enables orchestration of the four‑module pipeline without custom code, accelerating research cycles.

What Comes Next

While KTD‑Fin sets a new standard, several open challenges remain:

• Extending to Multi‑Asset Classes. The current benchmark focuses on equities; adapting the masking and attribution framework to bonds, commodities, or crypto will test LLM versatility across market structures.
• Dynamic Factor Models. Real‑world factor exposures evolve over time. Incorporating time‑varying factor loadings could sharpen the alpha signal and better reflect live trading conditions.
• Human‑in‑the‑Loop Evaluation. Combining LLM decisions with expert oversight may yield hybrid systems that outperform pure AI agents while retaining interpretability.

Future research could also explore counterfactual masking—systematically varying which identifiers are hidden—to quantify how much each piece of information contributes to decision quality. Such granular analysis would guide data‑curation strategies for next‑generation financial LLMs.

For organizations ready to experiment, the UBOS platform overview provides a sandbox where the KTD‑Fin pipeline can be instantiated, customized, and scaled. Startups can leverage the UBOS for startups program to prototype masked agents without heavy infrastructure overhead, while enterprises may adopt the Enterprise AI platform by UBOS for production‑grade deployment.

In sum, KTD‑Fin reframes the question from “Can LLMs make money?” to “Can LLMs generate reproducible, risk‑adjusted alpha?” Answering that question will shape the next wave of AI‑augmented finance.

References

Zhu, T., Zhao, W., Sun, R., Luan, B., Lu, J., Wang, S., Li, J., Jiang, D., He, Y., & Bai, Z. (2026). From Knowing to Doing: A Memory‑Controlled Benchmark for LLM Trading Agents on Stock Markets. arXiv preprint arXiv:2605.28359v1.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.