Andrii Bidochko

• Updated: July 4, 2026
• 7 min read

Short-Term Electricity Demand Forecasting for New England Using a Hybrid Transformer-XGBoost Framework with Weather, Calendar, and COVID-19 Indicators

Direct Answer

The paper introduces a hybrid forecasting framework that couples a Transformer encoder with XGBoost regression to predict short‑term electricity demand across New England, weaving together weather, calendar, and COVID‑19 variables. By blending deep temporal feature extraction with gradient‑boosted trees, the approach delivers sub‑2 % MAPE accuracy while exposing how pandemic‑era patterns decay once behavior normalizes.

Background: Why This Problem Is Hard

Electric utilities rely on precise demand forecasts to balance generation, schedule market trades, and avoid costly load‑shedding. Short‑term horizons (hour‑to‑day ahead) are especially volatile because they sit at the intersection of three rapidly shifting forces:

• Weather dynamics: Temperature, humidity, and wind speed can swing load by tens of megawatts within minutes.
• Human activity cycles: Calendar effects (weekends, holidays) and socio‑economic rhythms create predictable yet non‑linear demand spikes.
• Exogenous shocks: The COVID‑19 pandemic introduced unprecedented behavioral changes—remote work, altered commercial consumption, and policy‑driven lockdowns—that traditional models struggled to capture.

Conventional statistical methods (ARIMA, exponential smoothing) assume stationarity and linearity, making them brittle when faced with abrupt regime shifts. Machine‑learning alternatives such as plain feed‑forward networks or tree ensembles improve non‑linearity handling but often ignore temporal ordering, leading to information loss about sequential dependencies. Moreover, most published solutions evaluate on static test splits, inadvertently leaking future information and inflating performance claims.

What the Researchers Propose

The authors present a two‑stage hybrid architecture designed to respect both temporal structure and feature heterogeneity:

• Transformer encoder: Acts as a temporal feature extractor, ingesting lagged demand, weather, and calendar signals across a sliding window. Its self‑attention mechanism learns which past hours most influence the target horizon, without imposing a fixed decay pattern.
• XGBoost regressor: Receives the Transformer‑derived embeddings alongside raw tabular inputs (e.g., COVID‑19 case counts, policy indices) and produces the final demand forecast. Gradient‑boosted trees excel at handling mixed data types and capturing high‑order interactions.

By delegating sequence modeling to the Transformer and letting XGBoost specialize in non‑sequential regression, the framework leverages the strengths of each component while mitigating their individual weaknesses.

How It Works in Practice

Conceptual Workflow

1. Data ingestion: Hourly observations from six weather stations (one per New England state) are merged with calendar flags (weekday, holiday) and daily COVID‑19 metrics (new cases, hospitalizations, policy stringency).
2. Windowing: For each prediction timestamp, the most recent 24‑hour slice of demand and auxiliary variables is assembled into a sequence.
3. Transformer encoding: The sequence passes through a multi‑head self‑attention block, producing a fixed‑size embedding that summarizes temporal relevance.
4. Feature concatenation: The embedding is concatenated with the day‑level tabular features (e.g., average temperature, COVID‑19 indicators) to form a comprehensive input vector.
5. XGBoost regression: The vector feeds into a pre‑tuned XGBoost model, which outputs the forecasted megawatt‑hour (MWh) demand for the next day.
6. Post‑processing: Predicted values are clipped to realistic bounds and optionally adjusted for known operational constraints (e.g., scheduled maintenance).

Interaction Between Components

The Transformer and XGBoost modules communicate through a single embedding vector. This design keeps the pipeline lightweight—no back‑propagation through the tree model is required—while still allowing the gradient‑boosted learner to exploit the rich temporal context distilled by the attention layers. The separation also simplifies hyperparameter tuning: the Transformer’s depth and head count are optimized independently from XGBoost’s tree depth and learning rate.

What Sets This Approach Apart

• Leakage‑free chronological split: Training, validation, and test sets follow a strict time order (70 %/15 %/15 %), ensuring that future information never contaminates model learning.
• Automated hyperparameter search: Optuna’s multivariate Tree‑structured Parzen Estimator explores 500 trials, balancing model complexity against over‑fitting risk.
• Explainability via SHAP: Post‑hoc SHAP analysis reveals how pandemic‑related features rank over time, exposing temporal validity decay that many black‑box forecasts hide.

Evaluation & Results

Test Scenarios

The authors evaluated the hybrid model on a held‑out test period covering the transition from pandemic‑driven demand patterns (2020‑2021) to a more normalized 2022‑2023 regime. Two baselines were compared:

• Pure XGBoost (tabular only): Uses the same raw inputs but without any temporal embeddings.
• Statistical benchmark (ARIMA): Serves as a traditional industry reference.

Key Findings

• Accuracy: The hybrid achieved a test RMSE of 8,876 MWh, a MAPE of 2.05 %, and an R² of 0.906, outperforming the pure XGBoost baseline (RMSE = 9,304 MWh, MAPE = 2.21 %).
• Statistical significance: A Diebold‑Mariano test (Harvey‑Leybourne‑Newbold correction) yielded DM = ‑1.126 (p = 0.262), indicating that the observed improvement is not statistically distinguishable from random noise at the 5 % level.
• Ablation of COVID‑19 features: Removing pandemic variables degraded the hybrid’s RMSE by 3.2 % while marginally improving the XGBoost‑only model by 1.2 %, highlighting that pandemic signals help the Transformer‑XGBoost combo during training but become detrimental when the underlying behavior has shifted.
• SHAP temporal decay: Five of eight COVID‑19 features rose in importance on the post‑acute test set, suggesting the model over‑applies learned pandemic patterns—a classic case of temporal validity decay.

Interpretation of Results

These outcomes demonstrate that the hybrid framework can extract meaningful temporal cues that pure tree models miss, yet it also warns practitioners that features tied to transient events (e.g., a pandemic) can become sources of over‑fitting once the event subsides. The statistical insignificance of the RMSE gain underscores the importance of rigorous testing before claiming superiority.

Why This Matters for AI Systems and Agents

For AI practitioners building decision‑support agents in the energy sector, the study offers several actionable insights:

• Modular design for agents: By separating sequence learning (Transformer) from regression (XGBoost), developers can swap or upgrade components without retraining the entire pipeline, aligning with micro‑service architectures used in modern AI platforms.
• Feature‑drift awareness: The SHAP‑based temporal analysis provides a template for agents to monitor feature importance in real time, automatically flagging when pandemic‑related inputs start behaving like noise.
• Hyperparameter automation: Leveraging tools like Optuna demonstrates how agents can self‑tune in production, reducing human‑in‑the‑loop overhead.
• Explainability for regulators: Utilities must justify forecasts to regulators; SHAP explanations satisfy transparency requirements while still delivering high accuracy.

Integrating such a hybrid model into an UBOS platform overview could enable energy analysts to orchestrate data pipelines, run automated hyperparameter searches, and expose SHAP dashboards—all from a single interface. Moreover, the framework’s modularity makes it a natural fit for Workflow automation studio, where agents can trigger re‑training whenever a drift detector flags a shift in COVID‑19 feature relevance.

What Comes Next

While the hybrid model sets a new benchmark for New England demand forecasting, several limitations invite further research:

• Geographic generalization: Extending the approach to larger interconnections (e.g., ERCOT, PJM) will test scalability and the need for region‑specific weather embeddings.
• Real‑time adaptation: Implementing online learning or continual‑training loops could mitigate temporal decay by updating the Transformer’s attention weights as new data arrives.
• Richer exogenous signals: Incorporating mobility data, social media sentiment, or renewable generation forecasts may capture emerging demand drivers beyond weather and calendar effects.
• Model compression: Deploying the Transformer component on edge devices or low‑latency cloud functions may require quantization or knowledge distillation techniques.

Future work could also explore a fully end‑to‑end differentiable pipeline where the XGBoost stage is replaced by a neural regression head, enabling joint optimization of temporal and tabular representations. Such a direction aligns with the growing interest in “Neural‑Tree” hybrids that blend gradient‑boosted decision trees with deep learning.

Practitioners interested in rapid prototyping can experiment with the OpenAI ChatGPT integration to generate synthetic demand scenarios, then feed them into the hybrid model for stress testing. For field deployments, the ChatGPT and Telegram integration offers a low‑friction channel to deliver forecast alerts to operators on mobile devices.

Finally, the broader AI community can benefit from the paper’s methodological rigor—particularly its leakage‑free split and Diebold‑Mariano significance testing—as a template for evaluating any time‑series forecasting model, whether in energy, finance, or supply‑chain domains.

References

For a complete technical description, see the original arXiv paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.