Andrii Bidochko

• Updated: July 4, 2026
• 7 min read

Learning What Not to Forget: Long-Horizon Agent Memory from a Few Kilobytes of Learning

Direct Answer

Learned Relevance Eviction (LRE) is a lightweight, CPU‑only scorer that decides, in a few kilobytes of memory, which pieces of an LLM agent’s interaction history are essential enough to keep verbatim. By preserving only the load‑bearing tokens, LRE lets long‑running agents stay within their context window without sacrificing task accuracy.

This matters because it offers a deploy‑ready, model‑free solution to the “memory‑overflow” problem that plagues every production‑grade LLM‑based assistant, chatbot, or autonomous workflow.

Background: Why This Problem Is Hard

Large language models (LLMs) process inputs through a fixed‑size context window—typically a few thousand tokens. When an autonomous agent runs for hours or days, the cumulative conversation, tool calls, and system messages quickly exceed that window. The agent must therefore evict older tokens, but naïve eviction (e.g., dropping the earliest tokens) can discard critical information such as authentication credentials, task identifiers, or a previously computed plan.

Existing mitigation strategies fall into three broad categories:

• Full‑history retention. Storing the entire transcript guarantees fidelity but forces the model to compress or truncate the context, leading to higher latency and cost.
• Neural summarization. Encoder‑decoder models compress history into dense vectors. While they reduce token count, they introduce inference overhead, require additional GPU resources, and inevitably lose exact token‑level details.
• Heuristic pruning. Simple rules (e.g., keep only the last N turns) are cheap but brittle; they cannot anticipate which older tokens will become relevant for a future query.

All three approaches either consume too much compute, sacrifice exactness, or rely on hand‑crafted heuristics that do not generalize across domains. As LLM agents become core components of enterprise automation, a more principled, low‑cost eviction policy is essential.

What the Researchers Propose

The authors introduce Learned Relevance Eviction (LRE), a tiny, language‑model‑free relevance scorer that learns to flag “load‑bearing” units of history. LRE treats each token (or small token group) as a candidate for eviction and assigns a relevance score based on two signals:

• Historical usage patterns. How often a token has been referenced in subsequent system calls.
• Task‑level importance. Whether the token appears in a known critical field (e.g., API keys, file paths, or plan identifiers).

During a brief offline training phase, LRE observes the agent’s own execution traces and learns a binary classifier that predicts whether dropping a token would cause a failure. The resulting model occupies only a few kilobytes, runs on CPU, and can be queried in constant time.

Key components of the LRE framework are:

1. Trace Collector. Captures every interaction—user messages, tool outputs, system prompts—and logs token‑level references.
2. Relevance Learner. Trains a lightweight logistic model (or decision tree) on the collected traces, using the observed success/failure of downstream calls as supervision.
3. Eviction Engine. At runtime, queries the learner for each token and evicts the lowest‑scoring ones until the context fits the window budget.

How It Works in Practice

Below is a conceptual workflow that illustrates LRE in a production LLM agent:

1. Interaction Begins. The user sends a request; the agent appends the new user message to its history buffer.
2. Relevance Scoring. Before the next model call, the Eviction Engine runs the relevance learner over the entire buffer, producing a score for each token.
3. Budget Enforcement. If the total token count exceeds the model’s context limit, the engine removes the lowest‑scoring tokens, preserving high‑relevance items verbatim.
4. Model Invocation. The trimmed buffer is fed to the LLM, which generates a response or decides on a tool call.
5. Feedback Loop. After the tool call returns, the system records whether the call succeeded. If a failure can be traced to a missing token, that token is labeled “essential” for future training.

What sets LRE apart is its proactive nature: it makes eviction decisions *without* knowing the future query, yet it can still retain exact tokens that later prove decisive. Because the scorer is model‑free, it adds virtually no latency and requires no GPU, making it suitable for edge deployments or cost‑sensitive SaaS platforms.

Evaluation & Results

The researchers benchmarked LRE against three baselines across a suite of synthetic and real‑world tasks:

• No‑Eviction. Keep the entire history (upper bound on accuracy, highest cost).
• Neural Summarizer. Use a small encoder to compress history into dense vectors.
• Heuristic Pruner. Retain only the most recent N turns.

Key experimental settings included:

• Agents performing multi‑step web navigation, API orchestration, and conversational memory retrieval.
• Context budgets ranging from 25 % to 75 % of the original token count.
• Metrics: task success rate, number of model calls, and peak context size.

Findings:

• On the simplest tasks, LRE outperformed the no‑eviction baseline by 27 % while eliminating all compressor calls and cutting peak context size by up to 52 %.
• In a controlled trace study, LRE completed a long‑horizon navigation task 37 % faster (fewer model calls) than the full‑history approach and succeeded on 14 tasks where every other policy failed.
• For conversational memory, LRE matched dense encoder performance at zero neural cost, delivering higher answer fidelity when exact token recall mattered (e.g., recalling a previously issued access token).
• When trained only on the system’s own behavior (annotation‑free supervision), LRE recovered 95 % of the effectiveness of a fully supervised scorer, demonstrating that costly human labeling is unnecessary.

Overall, LRE occupied a sweet spot on the accuracy‑cost plane: it never dominated every baseline, but it consistently delivered the best trade‑off across diverse budgets and task complexities.

Why This Matters for AI Systems and Agents

For practitioners building production‑grade agents, LRE offers three immediate benefits:

1. Cost Efficiency. By shrinking the context window without sacrificing critical information, organizations can run larger models on cheaper hardware, reducing inference spend.
2. Reliability. Exact token preservation eliminates the “forgotten credential” failure mode that currently forces developers to add redundant prompts or manual state‑tracking.
3. Scalability. Because LRE runs on CPU and occupies only a few kilobytes, it can be embedded in serverless functions, edge devices, or any environment where GPU resources are scarce.

These advantages translate directly into better agent orchestration on platforms like the UBOS platform overview, where developers often chain multiple LLM calls into a single workflow. By integrating LRE, a workflow can maintain a compact, high‑fidelity memory store, enabling more complex multi‑step automations without hitting context limits.

Moreover, LRE’s annotation‑free training aligns with the growing demand for self‑supervised AI components. Teams can deploy the scorer, let it observe live traffic, and automatically improve relevance predictions—no extra labeling pipeline required.

In concrete terms, a customer building an AI marketing agents solution could use LRE to keep campaign identifiers, audience segments, and API keys in memory across dozens of follow‑up calls, ensuring that each personalized email generation step has the exact context it needs without bloating the prompt.

What Comes Next

While LRE demonstrates strong performance, several open challenges remain:

• Granularity of Tokens. The current implementation scores individual tokens; future work could explore scoring at the phrase or entity level to capture higher‑order dependencies.
• Dynamic Budgeting. Adapting the eviction threshold in real time based on model latency or downstream SLA requirements could further optimize resource usage.
• Cross‑Model Compatibility. Extending LRE to work seamlessly with multimodal models (e.g., vision‑language agents) will require new relevance signals beyond plain text.

Potential applications are already emerging. For instance, integrating LRE with the ChatGPT and Telegram integration would let a chatbot retain user authentication tokens across days of conversation without overwhelming the Telegram bot’s payload limits. Similarly, pairing LRE with the Chroma DB integration could enable hybrid memory stores where high‑relevance tokens are kept in the prompt while older context lives in a vector database.

Developers interested in experimenting with LRE can start by cloning the open‑source reference implementation, feeding it traces from their own Workflow automation studio, and observing the reduction in token usage. As the community contributes more domain‑specific relevance features, LRE could become a standard component of any LLM‑based agent stack.

References

For a deep dive into the methodology and full experimental details, see the original arXiv paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.