Andrii Bidochko

• Updated: July 8, 2026
• 6 min read

A Sliding-Window-Based Reinforcement Learning for Dynamic Assembly Flow Shop Scheduling with Multi-Product Delivery

Direct Answer

The paper introduces Sliding‑Window‑Based Reinforcement Learning (SWRL), a novel end‑to‑end framework that dynamically schedules multi‑product kitting deliveries in hybrid manufacturing environments. By treating the problem as a heterogeneous graph‑based Markov decision process and applying a sliding‑window filter, SWRL consistently reduces order tardiness compared with traditional dispatching rules and prior deep‑RL baselines.

Background: Why This Problem Is Hard

Modern factories increasingly blend machining, sub‑assembly, and final‑assembly stations into a single, flexible production line. When a customer orders a bundled product—say a home‑appliance set that includes a refrigerator, a washing machine, and a set of accessories—the factory must:

• Synchronize the availability of disparate components (kitting).
• React to stochastic order arrivals that can appear at any moment.
• Allocate limited machines to jobs whose processing sequences differ.
• Prevent bottlenecks, especially when a “tail product” (the last component in a bundle) becomes a choke point.

These constraints create a combinatorial explosion: the feasible job‑machine assignments shift with each new order, and the reward signal (e.g., on‑time delivery) is extremely sparse because only a few decisions directly affect tardiness. Classic heuristics—such as earliest‑due‑date or shortest‑processing‑time—ignore the inter‑product dependencies, while existing deep‑RL schedulers assume static job sets and homogeneous graph representations, leading to poor scalability and unstable performance in real‑time settings.

What the Researchers Propose

SWRL tackles the dynamic assembly flow shop scheduling problem through three tightly coupled mechanisms:

1. Sliding‑Window Filtering: At each decision epoch, the algorithm discards inactive nodes (machines or jobs that are idle or completed) and highlights “kitting‑critical” operations that directly influence bundle completion.
2. Spatiotemporal Graph Encoding: A graph neural network (GNN) encodes both the current shop floor topology and its evolution over recent windows, allowing the agent to detect shifting bottlenecks and anticipate future conflicts.
3. Dynamic Action Mapping with Constrained Waiting: The action space is rebuilt on‑the‑fly to reflect the current feasible assignments, while a waiting strategy prevents the agent from forcing premature dispatches that would violate kitting precedence.

Collectively, these components let the reinforcement learner operate on a heterogeneous graph that mirrors the dual‑layer kitting structure (product‑level and component‑level) and adapt instantly as orders pour in.

How It Works in Practice

The SWRL workflow can be visualized as a loop that repeats every few minutes on the shop floor:

1. State Capture: Sensors and the Manufacturing Execution System (MES) feed real‑time data—job statuses, machine availability, and newly arrived orders—into a graph builder.
2. Sliding‑Window Pruning: The builder applies a time‑bounded window (e.g., the last 10 decision steps) to filter out nodes that have not changed, reducing graph size and focusing computation on active, kitting‑relevant elements.
3. Graph Encoding: A spatiotemporal GNN processes the pruned graph, producing a latent representation that captures both current constraints and recent dynamics (e.g., a sudden surge in refrigerator orders).
4. Policy Inference: The RL policy network consumes the latent vector and outputs a probability distribution over feasible job‑machine pairs.
5. Dynamic Action Mapping: A mapping module translates the distribution into concrete dispatch commands, respecting the constrained waiting rules that hold a job until its prerequisite components are ready.
6. Execution & Feedback: Dispatches are sent to the shop floor, and the resulting makespan, tardiness, and resource utilization are logged as reward signals for the next learning iteration.

This loop runs continuously, enabling the system to re‑optimize whenever a new order arrives or a machine breaks down.

Evaluation & Results

The authors validated SWRL on a dataset supplied by a leading home‑appliance manufacturer. The testbed comprised:

• Four production lines with mixed machining and assembly stations.
• Three product families (refrigerators, washing machines, and kitchen sets) each requiring a distinct kitting configuration.
• Variable order arrival patterns ranging from uniform to bursty spikes.

Four baselines were compared:

1. Earliest Due Date (EDD) dispatching rule.
2. Shortest Processing Time (SPT) rule.
3. Standard Deep Q‑Network (DQN) without graph encoding.
4. Graph‑based RL (GNN‑RL) that lacks the sliding‑window filter.

Key findings include:

• Consistent tardiness reduction: SWRL cut average tardiness by 12‑18% relative to the best heuristic and by 7‑10% compared with the plain GNN‑RL.
• Robustness to load spikes: Under bursty arrival scenarios, SWRL maintained stable performance, whereas DQN and GNN‑RL exhibited sharp degradation.
• Scalability: The sliding‑window filter trimmed graph size by up to 45%, lowering inference latency from 250 ms to under 120 ms per decision step—critical for real‑time deployment.
• Resource utilization: Machine idle time dropped by 9% compared with EDD, indicating better load balancing across the shop floor.

These results demonstrate that SWRL not only outperforms traditional dispatching logic but also addresses the instability issues that have plagued prior deep‑RL attempts in dynamic manufacturing contexts.

Why This Matters for AI Systems and Agents

From an AI‑engineer’s perspective, SWRL offers a concrete blueprint for building agents that must operate under:

• Non‑stationary environments: The sliding‑window mechanism provides a systematic way to keep the state representation current without overwhelming the model.
• Heterogeneous action spaces: Dynamic action mapping shows how to reconcile a changing set of feasible actions with a stable policy network.
• Sparse reward landscapes: By focusing on kitting‑critical nodes, the agent receives more informative feedback, accelerating convergence.

Practically, manufacturers can embed SWRL into existing UBOS platform overview to orchestrate production lines, integrate with Workflow automation studio, and even expose scheduling decisions to downstream AI marketing agents for demand‑driven planning. The approach also aligns with emerging standards for AI‑driven shop‑floor orchestration, where agents must negotiate resources in real time while respecting business constraints.

What Comes Next

While SWRL marks a significant step forward, several avenues remain open for exploration:

• Multi‑objective optimization: Extending the reward function to balance energy consumption, maintenance costs, and carbon footprint alongside tardiness.
• Transfer learning across factories: Investigating whether a model trained on one product line can be fine‑tuned for another with minimal data.
• Human‑in‑the‑loop interfaces: Designing dashboards that let supervisors override or guide the agent during exceptional events.
• Integration with broader AI ecosystems: Connecting SWRL to OpenAI ChatGPT integration could enable natural‑language queries about schedule health, while a Telegram integration on UBOS might push real‑time alerts to floor managers.

Future research could also explore hierarchical reinforcement learning, where a high‑level planner decides on batch formation and a low‑level SWRL agent handles intra‑batch dispatching. Such a hierarchy would further improve scalability for factories with hundreds of concurrent orders.

References

For a complete technical description, see the original pre‑print: A Sliding‑Window‑Based Reinforcement Learning for Dynamic Assembly Flow Shop Scheduling with Multi‑Product Delivery.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.