Andrii Bidochko

• Updated: August 20, 2026
• 7 min read

FM-LLM: A Frequency-Enhanced Mixture-of-Experts Framework for Adapting LLMs to Time Series Forecasting

Direct Answer

FM-LLM introduces a frequency‑enhanced mixture‑of‑experts (MoE) framework that adapts frozen large language models (LLMs) for high‑precision time‑series forecasting without relying on textual prompts. By injecting Fourier‑derived spectral tokens directly into the LLM and separating periodic from residual dynamics via an asymmetric MoE decoder, the method delivers state‑of‑the‑art accuracy across diverse benchmarks while remaining lightweight enough for few‑shot and zero‑shot deployment.

Background: Why This Problem Is Hard

Time‑series forecasting underpins critical operations in finance, supply‑chain management, energy grids, and IoT monitoring. Traditional statistical models (ARIMA, ETS) excel at linear trends but falter when faced with complex, non‑stationary patterns. Deep learning approaches—RNNs, CNNs, and Transformers—have narrowed the gap, yet they still require extensive task‑specific training and often ignore the intrinsic frequency structure of the data.

Recent attempts to repurpose LLMs for cross‑modal forecasting treat the series as a textual prompt, asking the model to “write the next values.” This strategy introduces two major bottlenecks:

• Prompt overhead: Converting numeric sequences into natural‑language tokens inflates input length, consumes token budgets, and adds latency.
• Spectral blindness: LLMs trained on language lack built‑in awareness of harmonic components (seasonality, cycles), leading to drift in long‑horizon rollouts.

Because many real‑world series exhibit strong periodicity (daily demand cycles, weekly traffic patterns), ignoring frequency information wastes a rich source of predictive signal. Moreover, enterprises demand rapid adaptation—often with only a handful of labeled points—making prompt‑heavy pipelines impractical.

What the Researchers Propose

The authors present FM-LLM (Frequency‑Enhanced Mixture‑of‑Experts for LLM‑based forecasting), a three‑layered architecture that bridges the gap between spectral analysis and language modeling:

1. Fourier Analysis Network (FAN) Token Aligner: A lightweight module that performs a short‑time Fourier transform on the raw series, extracts dominant harmonics, and converts them into dense “spectral tokens.” These tokens are numerically compatible with the frozen LLM’s embedding space, allowing direct injection without textual conversion.
2. Asymmetric Mixture‑of‑Experts (MoE) Decoder: The decoder splits responsibilities:
   • Shared experts—augmented with tiny FAN layers—reconstruct the global periodic backbone captured by the spectral tokens.
   • Routed experts—standard feed‑forward networks (FFNs) gated by a learned router—focus on modeling the irregular, non‑periodic residuals that remain after the periodic component is removed.
3. Time‑Frequency Hybrid Loss: A dual objective that simultaneously minimizes traditional temporal error (e.g., MSE) and enforces spectral consistency by penalizing deviations between predicted and true frequency spectra.

This design respects the “frozen” nature of the LLM (no gradient updates to the language backbone) while still granting the model access to domain‑specific frequency cues.

How It Works in Practice

The end‑to‑end workflow can be broken down into four logical stages:

1. Data Ingestion & Spectral Token Generation

Raw numeric observations are first windowed (e.g., 96‑hour sliding windows). Each window undergoes a discrete Fourier transform; the top‑K frequency components (amplitude, phase) are selected and projected into a fixed‑dimensional token vector via a learned linear map. These vectors are concatenated with the original numeric embeddings, forming a hybrid token sequence.

2. Injection into the Frozen LLM

The hybrid sequence is fed into the frozen LLM’s embedding layer. Because the LLM’s positional encodings remain unchanged, the spectral tokens occupy the same positional slots as the original time steps, preserving temporal order while enriching each position with harmonic context.

3. Asymmetric MoE Decoding

A router examines each token’s hidden state and decides whether it should be processed by a “periodic expert” (shared across all tokens) or a “residual expert” (selected from a pool of specialized FFNs). The periodic expert incorporates a miniature FAN sub‑layer that refines the harmonic representation, effectively reconstructing the smooth, repeating pattern. Residual experts capture spikes, anomalies, and trend shifts that the periodic backbone cannot explain.

4. Autoregressive Forecast Generation & Loss Computation

The decoder emits the next time step’s numeric value, which is fed back as input for the subsequent step (standard autoregressive rollout). During training, the hybrid loss evaluates both the point‑wise error and the distance between the predicted and true Fourier spectra, encouraging the model to stay faithful to both time‑domain and frequency‑domain targets.

What sets FM‑LLM apart is the constrained asymmetric coupling—the periodic experts are shared (ensuring a consistent backbone) while the residual experts remain private (allowing specialization). This separation prevents the model from “over‑fitting” periodicity and reduces error accumulation in long‑horizon predictions.

Evaluation & Results

The authors benchmarked FM‑LLM on eleven public time‑series datasets spanning electricity demand, traffic flow, weather, and financial indicators. Evaluation covered 78 distinct metrics (MSE, MAE, sMAPE, etc.) under three regimes:

• Full‑data training: Standard supervised setting with ample historical data.
• Few‑shot (10% of training data): Simulates rapid adaptation scenarios.
• Zero‑shot: Directly forecasts unseen series without any fine‑tuning.

Key takeaways:

• FM‑LLM achieved top‑rank performance on 59 out of 78 metrics, surpassing the strongest autoregressive LLM baseline by an average of 5.3 % in MSE and 5.6 % in MAE.
• Maximum gains reached 8.0 % (MSE) and 8.4 % (MAE) on highly seasonal datasets, confirming the advantage of explicit frequency modeling.
• In few‑shot experiments, the hybrid loss enabled the model to retain 92 % of its full‑data accuracy, whereas baseline LLMs dropped below 80 %.
• Zero‑shot forecasts remained competitive, often beating classical statistical baselines, demonstrating the transferability of the spectral token space.

These results illustrate that FM‑LLM not only improves raw predictive accuracy but also stabilizes long‑term rollouts—a critical factor for production forecasting pipelines where error propagation can be costly.

Why This Matters for AI Systems and Agents

For AI practitioners building autonomous agents or decision‑support systems, FM‑LLM offers several practical benefits:

• Prompt‑free integration: Agents can ingest raw sensor streams directly, bypassing costly prompt engineering and token‑budget constraints.
• Modular compatibility: Because the LLM remains frozen, existing language‑centric agents (e.g., chat‑based planners, report generators) can be extended with a forecasting plug‑in without retraining the entire model.
• Robust few‑shot adaptation: Enterprises can deploy new forecasting tasks (e.g., a newly launched product line) with minimal historical data, accelerating time‑to‑value.
• Scalable orchestration: The asymmetric MoE decoder can be distributed across GPU clusters, allowing large‑scale batch forecasts while keeping the periodic expert lightweight.

These capabilities align with emerging AI‑orchestrated workflows where language models act as the “brain” and specialized modules (like FM‑LLM) provide domain‑specific expertise. For example, a supply‑chain agent could query FM‑LLM for demand forecasts, then feed the results into a routing optimizer, all within a single conversational loop.

Developers can readily embed FM‑LLM into existing pipelines using the UBOS platform overview, which supports custom model components and offers a Workflow automation studio for chaining forecasting outputs with downstream actions.

What Comes Next

While FM‑LLM marks a significant step forward, several open challenges remain:

• Dynamic frequency selection: Current FAN layers use a fixed K‑top frequency set. Adaptive mechanisms that adjust K based on series volatility could further improve accuracy.
• Multi‑modal extensions: Integrating exogenous variables (e.g., weather maps, textual news) alongside spectral tokens may boost performance on highly volatile domains.
• Resource‑aware routing: The MoE router could incorporate latency or memory constraints, enabling real‑time edge deployment.
• Explainability: Mapping residual expert activations back to specific anomalies would provide actionable insights for domain experts.

Future research may also explore coupling FM‑LLM with OpenAI ChatGPT integration to create conversational forecasting assistants that can answer “what‑if” queries in natural language while leveraging the underlying frequency‑aware model.

From a product perspective, the Enterprise AI platform by UBOS could host FM‑LLM as a managed service, offering pre‑configured pipelines for industries such as energy, finance, and retail. Startups looking for rapid prototyping might experiment with the UBOS templates for quick start, which already include a skeleton for time‑series ingestion and forecast visualization.

In summary, FM‑LLM demonstrates that marrying spectral signal processing with the expressive power of frozen LLMs yields a versatile, high‑performing forecasting engine—one that can be plugged into the next generation of AI‑driven business agents.

References

FM-LLM paper on arXiv

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.