Zero‑Downtime Payments Network Migrations: American Express Case Study

American Express successfully migrated its global Payments Network twice—first from a legacy monolith to a micro‑services platform and later from an older Kubernetes cluster to a modern, cloud‑native Kubernetes environment—both migrations were executed with zero customer‑impacting downtime by leveraging a Global Transaction Router, shadow traffic replay, and canary routing techniques.

Why the American Express Payments Network Matters

The Payments Network is the beating heart of American Express’s global commerce ecosystem. It routes live card‑authorization requests between merchants, acquirers, and issuers, handling millions of transactions per second with sub‑millisecond latency requirements. Any interruption would directly affect merchants’ ability to accept payments, erode consumer trust, and trigger costly SLA penalties. Consequently, the network must remain continuously available, highly performant, and resilient—constraints that shaped every decision during the two migrations.

Fintech developers, payment engineers, and technology decision‑makers can draw valuable lessons from how American Express tackled these challenges while maintaining a zero‑downtime posture.

Migration #1: From Legacy Monolith to a Micro‑services Platform

In 2018, American Express launched a multi‑year modernization effort to replace its decades‑old monolithic payments stack with a cloud‑native micro‑services architecture. The migration had three non‑negotiable constraints:

• No planned or unplanned downtime.
• Exact functional parity with the legacy system.
• Latency, throughput, and resiliency must stay equal or improve.

Three‑Stage Migration Strategy

The team broke the migration into three incremental stages, each designed to minimize risk while providing continuous validation.

Stage 1 – Connection Migration

The first step introduced the UBOS platform overview of a new routing layer called the Global Transaction Router (GTR). The GTR sat at the network edge, terminating long‑lived ISO‑8583 TCP connections from acquirers and issuers. Initially, every new connection was routed to the legacy backend, while the GTR simply forwarded traffic without any business logic. This “pass‑through” mode allowed the team to insert a controllable choke point without affecting transaction processing.

Stage 2 – Shadow Traffic Replay

With the GTR in place, a parallel instance of the new micro‑services platform was deployed. Live production traffic was duplicated—shadow traffic—and sent to the new stack for real‑time validation. Any discrepancy between the legacy and new responses surfaced instantly, enabling engineers to fine‑tune business logic before any live traffic ever reached the new platform.

Stage 3 – Canary Routing

Finally, the GTR was enhanced with lightweight routing rules that could direct a configurable percentage of transactions to the new platform based on transaction attributes (e.g., card type, merchant region). Starting at 1 % and gradually scaling to 100 %, the team monitored latency, error rates, and business metrics. If an anomaly appeared, the GTR instantly reverted all traffic to the legacy system—making rollback a first‑class capability.

Throughout the migration, observability dashboards powered by AI marketing agents provided real‑time insights into request latency, error bursts, and system health, ensuring rapid response to any deviation.

Migration #2: Shifting the Kubernetes Infrastructure

After the micro‑services platform proved stable, the next challenge was to move the entire payments stack from a legacy Kubernetes cluster to a purpose‑built, cloud‑native Kubernetes environment. The new environment offered improved networking, stronger security policies, and better resource isolation, but required a full rebuild of the infrastructure.

Preparing the New Cluster

The team leveraged Kubernetes migration best practices: infrastructure‑as‑code (IaC) definitions for pods, services, and network policies were stored in Git, enabling repeatable deployments across regions. Load‑testing against a performance baseline confirmed that the new cluster could meet or exceed the legacy cluster’s latency and throughput targets.

Reusing the Proven Traffic‑Control Pattern

Just as in the first migration, the Global Transaction Router (now running on Envoy Proxy with a custom control plane) acted as the traffic‑shaping gateway. However, this time the canary routing operated at the internal gRPC level between the GTR and the payments micro‑services, allowing the team to shift traffic between the old and new Kubernetes clusters without touching the external ISO‑8583 edge.

Multi‑Region Canary Routing

To further reduce risk, the team introduced a multi‑region canary: traffic from one geographic region was first routed to the new cluster in a different region, validated, and then gradually switched back to the original region now running the upgraded Kubernetes stack. This approach provided an additional safety net and demonstrated the power of Workflow automation studio for orchestrating complex traffic‑shifts.

Shadow Traffic Re‑used

Shadow traffic continued to play a critical role, replaying live transactions to the new cluster for final verification before any production traffic was switched.

Technical Deep‑Dive: Global Transaction Router, Shadow Traffic, and Canary Routing

The three pillars that enabled both migrations are worth examining in detail.

Global Transaction Router (GTR)

The GTR is a high‑performance, low‑latency gateway that terminates ISO‑8583 TCP connections and forwards messages to downstream services. Its responsibilities include:

• Connection pooling and health‑checking of backend services.
• Dynamic routing based on configurable rules (e.g., card brand, merchant ID).
• Failover handling to automatically reroute traffic on backend degradation.

Implemented with ElevenLabs AI voice integration for real‑time monitoring alerts, the GTR became the single source of truth for traffic control.

Shadow Traffic Replay

Shadow traffic is a production‑grade replay of live requests sent to a parallel environment. It differs from traditional testing because it runs on real‑world traffic patterns, exposing edge‑case bugs that unit tests miss. In American Express’s case, shadow traffic was used twice:

1. During the micro‑services migration to validate business logic.
2. During the Kubernetes migration to verify networking and resource limits.

The replay pipeline leveraged OpenAI ChatGPT integration to automatically flag anomalous responses for engineer review.

Canary Routing

Canary routing allowed incremental traffic shifts. Key attributes of the implementation:

• Configurable percentages (1 % → 100 %).
• Attribute‑based routing (card type, transaction amount).
• Instant rollback via GTR rule change.
• Metrics‑driven promotion: only after latency < 2 ms and error rate < 0.01 % did the team increase traffic.

These canary controls are now exposed as reusable components in the Web app editor on UBOS, enabling other teams to adopt the pattern quickly.

Lessons Learned & Best Practices for Zero‑Downtime Migrations

Both migrations reinforced a set of timeless principles that fintech teams should embed into their own roadmaps.

Applying these lessons can help your organization achieve the same level of reliability that American Express demands from a global payments processor.

For teams looking for ready‑made templates to accelerate similar projects, the UBOS templates for quick start include pre‑configured canary routing and shadow traffic modules.

About the Authors

Conclusion

American Express’s double migration showcases that even the most mission‑critical, high‑throughput payment systems can evolve without ever missing a beat. By investing in a robust routing layer, leveraging shadow traffic for real‑world validation, and applying disciplined canary routing, the company set a new benchmark for zero‑downtime migrations in the fintech space.

For a deeper dive into the original presentation and technical details, read the full article on the American Express technology blog: American Express Payments Network Migration.

Ready to modernize your own payment infrastructure? Start with the UBOS homepage and discover how our platform can accelerate zero‑downtime migrations for your organization.