Step‑By‑Step Guide to Testing Failover for OpenClaw Rating API

1. Introduction

Modern SaaS products rely on distributed edge networks to keep latency low and availability high. The OpenClaw Rating API is a prime example: it serves real‑time rating data from edge nodes located in North America, Europe, and Asia‑Pacific. While UBOS automatically provisions these nodes, developers must still validate that failover mechanisms work as expected before a production rollout.

This guide walks developers and DevOps engineers through a step‑by‑step process to test failover, from prerequisite setup to traffic verification. All scripts are written in Bash and Python, and they leverage UBOS’s UBOS platform overview services such as the Workflow Automation Studio and the Web App Editor.

2. Overview of OpenClaw Rating API and Edge Regions

The OpenClaw Rating API exposes three core endpoints:

• /v1/rate – Submit a new rating.
• /v1/score – Retrieve aggregated scores.
• /v1/health – Health‑check endpoint used by UBOS for automatic routing.

Each endpoint is replicated across three edge regions:

UBOS’s global load balancer continuously probes /v1/health from each region. If a probe fails, traffic is automatically shifted to the next healthy region. The failover logic is transparent to API consumers, but you must verify it under controlled outage conditions.

3. Prerequisites

Before you begin, ensure the following items are in place:

1. Access to a UBOS account with About UBOS admin rights.
2. Deployed OpenClaw instance reachable via https://api.openclaw.ubos.io.
3. CLI tools installed on your workstation:
   • curl (v7.68+)
   • jq for JSON parsing
   • python3 (v3.9+)
   • pip and the requests library
4. Terraform or UBOS’s Workflow automation studio to spin up temporary test pods in each edge region.
5. Monitoring stack (Prometheus + Grafana) already configured for the OpenClaw service. If you need a quick start, see the UBOS templates for quick start.

4. Health‑Check Scripts

The first step is to verify that each edge node reports a healthy status. Below are two scripts: a Bash version for quick CLI checks and a Python version for integration into CI pipelines.

4.1 Bash Health‑Check

#!/usr/bin/env bash
REGIONS=("us-east-1" "eu-west-1" "ap-southeast-1")
BASE_URL="https://api.openclaw.ubos.io"

for REGION in "${REGIONS[@]}"; do
  echo "Checking health for $REGION..."
  RESPONSE=$(curl -s -w "%{http_code}" "$BASE_URL/v1/health?region=$REGION")
  HTTP_CODE="${RESPONSE: -3}"
  BODY="${RESPONSE::-3}"
  if [[ $HTTP_CODE -eq 200 && $(echo $BODY | jq -r .status) == "healthy" ]]; then
    echo "✅ $REGION is healthy"
  else
    echo "❌ $REGION health check failed (HTTP $HTTP_CODE)"
  fi
done

4.2 Python Health‑Check (CI/CD Friendly)

import requests
import json

REGIONS = ["us-east-1", "eu-west-1", "ap-southeast-1"]
BASE_URL = "https://api.openclaw.ubos.io/v1/health"

def check_region(region):
    try:
        r = requests.get(BASE_URL, params={"region": region}, timeout=5)
        r.raise_for_status()
        data = r.json()
        return data.get("status") == "healthy"
    except Exception as e:
        print(f"Error checking {region}: {e}")
        return False

if __name__ == "__main__":
    for region in REGIONS:
        healthy = check_region(region)
        print(f"{region}: {'✅ Healthy' if healthy else '❌ Unhealthy'}")

Run the Bash script on a local machine or CI runner. The Python script can be added as a pytest test to enforce health before each deployment.

5. Simulating a Region Outage

To validate failover, you must temporarily block traffic to one edge node. UBOS provides two safe methods:

• Network ACL toggle: Use the UBOS console to disable inbound rules for the target region.
• iptables rule on a test pod: Deploy a lightweight pod in the region and drop outbound packets to the API gateway.

5.1 Using UBOS Console (Recommended)

1. Navigate to the UBOS partner program dashboard.
2. Select “Network Settings” → “Edge Regions”.
3. Click “Disable” next to the region you want to test (e.g., eu-west-1).
4. Confirm the action; UBOS will return a 503 Service Unavailable for health probes from that region.

5.2 Using iptables on a Test Pod

If you prefer a programmatic approach, spin up a test pod in the target region via the Web app editor on UBOS and run:

# Inside the test pod
sudo iptables -A OUTPUT -p tcp --dport 443 -j DROP
echo "Traffic to OpenClaw blocked in this pod"

This rule only affects the pod, leaving other services untouched, which is ideal for isolated testing.

6. Monitoring Key Metrics

While the outage is in effect, you must watch three categories of metrics:

1. Latency: Average response time per region (Prometheus query histogram_quantile(0.95, rate(openclaw_http_duration_seconds_bucket[5m]))).
2. Error Rate: HTTP 5xx counts (increase(openclaw_http_requests_total{status=~"5.."}[5m])).
3. Routing Logs: UBOS edge router logs showing which region served each request (available via the Enterprise AI platform by UBOS log aggregation).

6.1 Grafana Dashboard Example

Import the following JSON into Grafana to get a real‑time view:

{
  "dashboard": {
    "title": "OpenClaw Failover Monitoring",
    "panels": [
      {
        "type": "graph",
        "title": "Latency by Region",
        "targets": [
          {"expr": "histogram_quantile(0.95, rate(openclaw_http_duration_seconds_bucket{region=~\"us-east-1|eu-west-1|ap-southeast-1\"}[5m]))"}
        ]
      },
      {
        "type": "graph",
        "title": "5xx Error Rate",
        "targets": [
          {"expr": "increase(openclaw_http_requests_total{status=~\"5..\"}[5m])"}
        ]
      },
      {
        "type": "table",
        "title": "Routing Log Snapshot",
        "targets": [
          {"expr": "topk(10, openclaw_router_requests_total{region!=\"disabled\"})"}
        ]
      }
    ]
  }
}

When the EU region is disabled, you should see latency for EU drop to 0 and a corresponding rise in NA or APAC latency, while error rates stay below 1%.

7. Verifying Traffic Rerouting

After the outage simulation, confirm that the OpenClaw Rating API continues to serve requests without interruption. Follow these verification steps:

1. Send a burst of requests: Use hey or wrk from a client outside UBOS to generate load.
2. Check response consistency: All 200 OK responses must contain a valid rating_id field.
3. Validate region header: The API returns an X-Edge-Region header indicating which edge served the request. Ensure the header never reports the disabled region.

7.1 Bash Load Test Example

#!/usr/bin/env bash
# Install hey: go get -u github.com/rakyll/hey
ENDPOINT="https://api.openclaw.ubos.io/v1/rate"
PAYLOAD='{"user_id":"test123","item_id":"movie456","score":4}'

hey -n 500 -c 50 -m POST -H "Content-Type: application/json" \
    -d "$PAYLOAD" $ENDPOINT | tee hey_output.txt

# Verify no 5xx responses
if grep -q "5xx" hey_output.txt; then
  echo "❌ Detected server errors during failover"
else
  echo "✅ All requests succeeded"
fi

# Extract X-Edge-Region values
grep "X-Edge-Region" hey_output.txt | sort | uniq -c

The final uniq -c output should list only us-east-1 and/or ap-southeast-1 when the EU region is down.

7.2 Python Verification Script

import requests, concurrent.futures, json

URL = "https://api.openclaw.ubos.io/v1/rate"
PAYLOAD = {"user_id":"devops","item_id":"song789","score":5}
HEADERS = {"Content-Type":"application/json"}

def send_one():
    r = requests.post(URL, json=PAYLOAD, headers=HEADERS, timeout=3)
    return r.status_code, r.headers.get("X-Edge-Region")

def main():
    successes = 0
    regions = {}
    with concurrent.futures.ThreadPoolExecutor(max_workers=50) as executor:
        futures = [executor.submit(send_one) for _ in range(200)]
        for f in concurrent.futures.as_completed(futures):
            code, region = f.result()
            if code == 200:
                successes += 1
                regions[region] = regions.get(region, 0) + 1
    print(f"Successful calls: {successes}/200")
    print("Region distribution:", json.dumps(regions, indent=2))

if __name__ == "__main__":
    main()

After the test, the printed region distribution should not contain the disabled region. If it does, revisit your network ACL settings.