Andrii Bidochko

• Updated: July 7, 2026
• 6 min read

A Sliding-Window-Based Reinforcement Learning for Dynamic Assembly Flow Shop Scheduling with Multi-Product Delivery

Direct Answer

The paper introduces Sliding‑Window‑Based Reinforcement Learning (SWRL), a novel end‑to‑end framework that tackles real‑time, multi‑product kitting delivery in hybrid manufacturing environments. By combining a sliding‑window filter, spatiotemporal graph encoding, and a dynamic action‑mapping strategy, SWRL consistently reduces order tardiness compared with traditional dispatching rules and prior deep‑RL approaches.

Background: Why This Problem Is Hard

Hybrid manufacturing systems—where machining, sub‑assembly, and final assembly coexist—must constantly re‑schedule jobs as new orders arrive. Multi‑product kitting adds a second layer of complexity: components for a single final product may be sourced from different lines, creating inter‑dependent supply chains that shift the bottleneck from one workstation to another in milliseconds.

Existing solutions fall into two camps:

• Static dispatching heuristics (e.g., earliest‑due‑date, shortest‑processing‑time) assume a fixed job set and cannot adapt to sudden order spikes.
• Deep reinforcement learning (DRL) schedulers treat the problem as a monolithic Markov decision process (MDP). They struggle with the sparse reward landscape caused by rare tardiness events and with the rapidly changing action space when machines become idle or new jobs appear.

Consequently, manufacturers experience high late‑delivery penalties, under‑utilized equipment, and difficulty scaling to variable order loads—issues that directly impact profitability and customer satisfaction.

What the Researchers Propose

SWRL reframes dynamic assembly flow shop scheduling as a heterogeneous graph‑based MDP that mirrors the dual‑layer kitting structure:

• Node types represent machines, sub‑assemblies, and final‑product kits.
• Edges encode precedence and material‑flow constraints.

The framework consists of three tightly coupled components:

1. Sliding‑Window Filtering: At each decision epoch, a temporal window discards inactive nodes (e.g., completed sub‑assemblies) and highlights kitting‑critical operations, shrinking the state space.
2. Spatiotemporal Graph Encoder: A graph neural network (GNN) processes the filtered graph, preserving both structural relationships and the evolution of bottlenecks across consecutive windows.
3. Dynamic Action Mapping with Constrained Waiting: The policy outputs a set of feasible machine‑job assignments; a waiting strategy ensures that actions respect the current topology, preventing illegal allocations when resources appear or disappear.

Collectively, these modules enable the RL agent to learn from a dense, informative representation of the shop floor while remaining responsive to real‑time changes.

How It Works in Practice

The operational workflow of SWRL can be visualized as a loop that repeats every few seconds:

1. State Capture: Sensors and the manufacturing execution system (MES) stream the latest job arrivals, machine statuses, and inventory levels into a heterogeneous graph.
2. Sliding‑Window Filter: The system applies a configurable time window (e.g., the next 5‑10 minutes) to prune nodes that will not affect immediate decisions, focusing computational effort on active kitting paths.
3. Graph Encoding: The spatiotemporal GNN aggregates node features (processing time, due date, setup cost) and edge attributes (precedence, transport time) to produce a compact embedding that captures the current bottleneck dynamics.
4. Policy Inference: The RL policy, trained offline with a reward that penalizes tardiness and idle time, maps the embedding to a set of admissible actions—specific machine‑job pairings.
5. Dynamic Action Mapping: Before execution, the action set is reconciled with the live shop floor topology. If a machine becomes unavailable, the constrained waiting module defers the assignment until a feasible slot opens.
6. Execution & Feedback: Chosen actions are dispatched to the MES, which updates the shop floor. The resulting schedule performance (e.g., completed jobs, lateness) feeds back as a reward signal for the next learning iteration.

What sets SWRL apart is its ability to adapt the decision space on the fly without retraining, thanks to the sliding‑window filter and dynamic action mapping. Traditional DRL agents would need to recompute the entire action matrix whenever the topology changes, leading to latency that is unacceptable for real‑time scheduling.

Evaluation & Results

The authors validated SWRL on a suite of real‑world instances supplied by a leading home‑appliance manufacturer. Test scenarios varied across three dimensions:

• Resource configurations: 5 to 12 machines with heterogeneous capabilities.
• Order loads: Low (30 orders/hour) to high (120 orders/hour) arrival rates.
• Arrival concentration: Uniformly spread versus bursty spikes.

Key findings include:

• Across all configurations, SWRL reduced average tardiness by **12‑18%** compared with the best classical dispatching rule (Weighted Shortest Processing Time).
• When benchmarked against two state‑of‑the‑art deep‑RL schedulers, SWRL achieved **15‑22%** lower lateness while requiring **30% less inference time** per decision epoch.
• Performance remained stable under bursty order arrivals, demonstrating robustness to the “shock” events that typically cripple static heuristics.

These results indicate that the sliding‑window and dynamic action mechanisms not only improve solution quality but also keep computational overhead within the sub‑second range required for online deployment.

Why This Matters for AI Systems and Agents

For AI practitioners building autonomous agents in manufacturing, SWRL offers a blueprint for marrying reinforcement learning with graph‑structured, time‑varying environments. The framework’s modular design—filter, encoder, policy—can be repurposed for other domains where resources appear and disappear, such as cloud‑edge orchestration or logistics networks.

From an operational standpoint, integrating SWRL into an Enterprise AI platform by UBOS would enable:

• Real‑time decision making without sacrificing model fidelity.
• Scalable deployment across multiple factories, each with its own topology.
• Continuous learning loops that incorporate live performance metrics, aligning with modern AI‑ops practices.

Moreover, the graph‑based MDP formulation aligns with emerging standards for digital twins, allowing simulation‑to‑real transfer with minimal domain adaptation.

What Comes Next

While SWRL marks a significant step forward, several avenues remain open for exploration:

• Multi‑objective optimization: Extending the reward function to balance energy consumption, wear‑and‑tear, and workforce constraints.
• Transfer learning across plants: Pre‑training the graph encoder on a fleet of factories and fine‑tuning for site‑specific nuances.
• Human‑in‑the‑loop interfaces: Allowing operators to inject priority overrides that the policy can respect without destabilizing learning.

Future research could also investigate hybridizing SWRL with Workflow automation studio tools to orchestrate downstream logistics, creating a seamless end‑to‑end supply‑chain AI stack.

For readers interested in the technical details, the full methodology and code snippets are available in the original arXiv paper.

Conclusion

Sliding‑Window‑Based Reinforcement Learning redefines dynamic assembly flow shop scheduling by addressing the twin challenges of sparse rewards and volatile action spaces. Its graph‑centric perspective, combined with a real‑time filtering mechanism, delivers measurable tardiness reductions while staying computationally lightweight. As manufacturers accelerate toward Industry 4.0, frameworks like SWRL will be essential for turning complex, multi‑product kitting constraints into competitive advantages.

Ready to explore how advanced AI scheduling can transform your operations? Visit the UBOS homepage for more resources, or dive into our platform overview to see SWRL‑style capabilities in action.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.