Beyond Retrieval: Analytic Memory for Multimodal Agents

Direct Answer

The paper introduces AdaMM, a unified framework that augments traditional retrieval‑based long‑term memory with an analytic memory layer capable of filtering, aggregating, ranking, and temporally comparing multimodal observations. This matters because it lets autonomous agents not only fetch past facts but also compute over them, enabling richer reasoning across images, dialogue, and contextual metadata.

Background: Why This Problem Is Hard

Modern AI assistants are expected to remember interactions that span days, weeks, or even months. In practice, “memory” has been implemented as a retrieval system: a vector store or an indexed log that returns the most relevant snippet when the agent receives a query. While retrieval excels at locating raw records, it falls short when the agent must answer questions that require analysis of the stored data—such as “Which product images showed a price drop over the last three weeks?” or “What is the average sentiment of user feedback about feature X across all sessions?”

Two intertwined challenges make this problem especially hard:

• Multimodal heterogeneity: Agents ingest text, images, audio, and structured metadata. Aligning these disparate signals into a single searchable index is non‑trivial, and most retrieval pipelines flatten everything into a single embedding space, losing the fine‑grained attribute information needed for analytics.
• Temporal and relational reasoning: Long‑term interactions generate recurring patterns (e.g., daily sales figures, weekly status reports). Pure retrieval cannot efficiently compute aggregates or detect trends without pulling massive amounts of raw data into the prompt, which quickly exceeds token limits and hurts latency.

Existing approaches—such as hierarchical summarization, memory‑augmented language models, or external databases—attempt to mitigate these issues but still rely on a retrieval‑first mindset. They either require hand‑crafted schemas that limit flexibility, or they push the analytical burden back onto the language model, leading to hallucinations and inconsistent results.

What the Researchers Propose

The authors formalize analytic memory as a complementary abstraction to retrieval memory. Analytic memory automatically discovers recurring attribute‑value structures from raw multimodal streams and materializes them into queryable tables that retain provenance links to the original observations. In this view, the memory system becomes a hybrid of a document store and a lightweight relational engine.

Key components of the proposal include:

• Provenance‑linked extraction: A multimodal parser extracts key‑value pairs (e.g., {image_id: 123, object: "car", color: "red", timestamp: "2026‑07‑01"}) from dialogues, images, and metadata, while preserving a pointer back to the source.
• Schema discovery engine: Instead of requiring developers to pre‑define tables, the system clusters similar attribute sets across the interaction history, inferring field names and data types on the fly.
• Joint retrieval‑analytic planner: At inference time, a planner decomposes a user query into sub‑tasks—some best served by plain retrieval, others by analytic operations (filter, aggregate, rank, temporal compare)—and routes each sub‑task to the appropriate backend.

By treating analytics as a first‑class memory operation, AdaMM enables agents to answer “how” and “why” questions without resorting to prompt‑engineering tricks or external ETL pipelines.

How It Works in Practice

The AdaMM workflow can be visualized as a three‑stage pipeline:

1. Ingestion Layer: Every interaction—whether a user utterance, an uploaded image, or a system‑generated event—is passed through a multimodal encoder. The encoder produces both a dense embedding (for retrieval) and a set of candidate attribute‑value triples (for analytics). Provenance metadata (source ID, timestamp, modality) is attached to each triple.
2. Memory Engine:
   • Retrieval Store: Dense embeddings are indexed in a vector database (e.g., Chroma DB) for fast nearest‑neighbor lookup.
   • Analytic Store: Extracted triples are streamed into a dynamic schema manager. The manager clusters similar triples, creates or updates tables, and writes rows with provenance links. This store supports SQL‑like operations: SELECT … WHERE … GROUP BY … ORDER BY … and temporal functions such as DIFF or WINDOW.
3. Memory‑Aware Planner: When the agent receives a user query, a lightweight LLM first classifies the intent (retrieval vs. analytics). For analytic intents, the planner translates the natural language request into a structured query (e.g., “Show the top‑3 most frequent objects in images from the last 30 days” becomes SELECT object, COUNT(*) FROM images WHERE timestamp > now() - 30d GROUP BY object ORDER BY COUNT DESC LIMIT 3). The planner then dispatches the retrieval sub‑query to the vector store and the analytic sub‑query to the analytic store, merges the results, and feeds them back to the LLM for final response generation.

This separation of concerns yields two practical benefits:

• Scalability: Analytic queries run on a compact, indexed table rather than scanning millions of raw embeddings.
• Interpretability: Because each analytic row retains a provenance pointer, the agent can surface the original image or dialogue snippet that justified a numeric answer, reducing hallucination risk.

Evaluation & Results

To validate AdaMM, the authors built two long‑term multimodal benchmarks:

• MemEye: A synthetic environment where agents interact with a stream of images annotated with objects, timestamps, and user comments. Queries require temporal aggregation (e.g., “How many times did a blue car appear in the last week?”).
• MemGallery: A real‑world dataset of e‑commerce product images, price histories, and customer reviews. Queries involve ranking and filtering (e.g., “List the top‑5 products whose price dropped by more than 20% in the past month.”).

Across both benchmarks, AdaMM’s hybrid approach outperformed a strong retrieval‑only baseline:

• On MemEye, overall task accuracy improved by 11.3 %, with the most pronounced gains on temporal comparison queries.
• On MemGallery, the system achieved a 7.3 % lift in precision‑at‑5 for ranking‑heavy questions, while maintaining comparable latency.

These results demonstrate that analytic memory not only boosts raw performance but also expands the class of questions agents can answer reliably—especially those that require computation over many observations rather than simple lookup.

Why This Matters for AI Systems and Agents

For practitioners building enterprise‑grade AI assistants, AdaMM offers a concrete pathway to move beyond “memory as cache” toward “memory as calculator.” The implications are threefold:

1. Richer user experiences: Agents can now answer trend‑based queries (“What’s the average sentiment for feature X over the last quarter?”) without external data pipelines, keeping the conversation fluid.
2. Reduced engineering overhead: Automatic schema discovery eliminates the need for developers to pre‑define relational tables for every new data source, accelerating integration of novel modalities.
3. Improved compliance and auditability: Provenance links let organizations trace every analytic answer back to its source, satisfying regulatory requirements for explainability.

These capabilities align closely with the UBOS platform overview, which emphasizes modular memory components and low‑code orchestration. By plugging AdaMM‑style analytic stores into the Workflow automation studio, teams can construct end‑to‑end agents that reason over weeks of multimodal data without writing custom ETL code.

What Comes Next

While AdaMM marks a significant step forward, several open challenges remain:

• Scalable schema evolution: As new attribute patterns emerge, the system must merge or split tables without disrupting ongoing queries.
• Cross‑modal joins: Current implementation treats each modality’s attributes in isolation; future work could enable joins between image‑derived tables and text‑derived tables (e.g., linking detected objects to sentiment scores).
• Privacy‑preserving analytics: Embedding differential privacy mechanisms into the analytic store would allow agents to compute aggregates over user data while respecting confidentiality.

Researchers are already exploring extensions such as “semantic‑aware aggregation,” where the system groups similar but not identical values (e.g., “red” and “crimson”) using ontology‑driven similarity. From an industry perspective, integrating AdaMM with existing vector stores like Chroma DB integration could provide a seamless path for customers to upgrade their memory stacks.

Developers interested in experimenting with analytic memory can start by reviewing the original arXiv paper and exploring the open‑source components released alongside the work. As the community builds tooling around this paradigm, we can expect a new generation of agents that not only remember but also reason, compare, and summarize their own histories.

Ready to prototype an analytic‑memory‑enabled assistant? Visit the UBOS homepage for starter kits, or join the UBOS partner program to collaborate on next‑generation AI workflows.