Edu/Supportk8s_redis_sentinel_java_app_2
Redis-Sentinel Java App on Kubernetes ②
Failover test
🖧 Redis-Sentinel Pod diagram: Redis-0 master
1. Start the failover test
Open two SSH terminals. On Terminal-1 create the App Pod and take Redis Pods down; on Terminal-2 watch the App logs.
Create the Java (Spring) App Pod
Run on the server (Terminal-1 [~/kube/app/sentinel])
$ kubectl apply -f app-deployment.yaml -> create Pod
💻 App log
Run on the server (Terminal-2 [~/kube/app/sentinel])
$ kubectl logs -f -l app=spring-redis-sentinel-app
You should see 10 writes per second ([SUCCESS]).
💥 1. Take Redis-0 Pod down
Terminal-1) $ date; kubectl delete pod redis-0 -> 15:47:39
💻 Redis-0 LOG
① 39s Redis down (+0s)
② 41s Redis restarts (+2s)
③ 54s master -> replica (+15s)
Redis-0 full log
💻 Redis-1 LOG
① 43s replica -> master (+4s)
App log
Terminal-2) $ kubectl logs -f -l app=spring-redis-sentinel-app
① 39s Redis connection lost -> reconnect (+0s)
② 44s one write failed: keyA-00000147 (+5s)
③ From 45s writes succeed again (+6s)
App full log
🖧 Redis-Sentinel Pod diagram: after failover (Redis-1 master)
2. Failback test
1) Redis-1 Pod 💥 down (delete)
Terminal-1) $ date; kubectl delete pod redis-1 -> 15:48:05
💻 Redis-1 log
① 05s Redis down (+0s)
② 08s Redis restarts (+3s)
③ 20s master -> replica (+15s)
💻 Redis-0 log
① 09s replica -> master (+4s)
💻 App log
① 06s Redis-1 connection lost -> reconnect (+1s)
② 10s writes succeed (Redis-0 became master at 09s) (+5s)
App full log
🖧 Redis-Sentinel Pod diagram: failback (Redis-0 master)
🧑💻 Review and key takeaways
This failover/failback test verified real integration between Redis-Sentinel on Kubernetes and a Java (Spring Boot) application.
⏱️ Failover timeline
| Phase | Time (relative) |
Redis-Sentinel | Java (Spring) app |
|---|---|---|---|
| Failure | 15:47:39 (+0s) |
Master (Redis-0) Pod deleted | Detects disconnect and reconnects |
| Failover | 15:47:43 (+4s) |
Sentinel detects failure and promotes Redis-1 to master | Receives the new master IP from Sentinel |
| Recovered | 15:47:45 (+6s) |
New master (Redis-1) ready for writes | [SUCCESS] writes resume (only 1 request retried) |
| Old master restart | 15:47:54 (+15s) |
Restarted Redis-0 joins Redis-1 as a replica | No impact (service stays up) |
💡 Key takeaways for production
-
1. Sentinel failover time (about 4–5s):
After the master Pod dies, Sentinel marks it down (down-after-milliseconds 3000), votes (quorum), and promotes a new master — about 4 seconds of downtime. -
2. Spring/Lettuce client retry matters:
Almost no data was lost mainly because of app-side retry (5 attempts / 5s). The one failed request during promotion (keyA-00000147) was retried in the background to the new master, so service stayed nearly uninterrupted. -
3. UX and timeout:
From the client, failover adds about 4–5 seconds of latency. For web/app API calls, combine a front-end timeout and a circuit breaker (e.g. Resilience4j) so the UI does not freeze. -
4. Automatic failback is not supported (this is normal):
When the old master (Redis-0) comes back, Sentinel does not fail back; it becomes a replica of the new master (Redis-1). That is normal. (The failback in test 2 was forced by deleting Redis-1.)
🛡️ Front-end timeout and circuit breaker
🛡️ Safety nets when Redis fails
The 4–5s latency during Sentinel failover can cascade to the backend and the user UI (cascading failure). In production, combine these two safety nets.
1. Front-end timeout
The max time a browser or mobile app waits for a backend API before giving up instead of waiting forever.
- Without a timeout: Redis stalling for 4s makes users mash refresh/buttons. Tomcat/Spring threads exhaust and the whole web server can go down.
- Fix (2–3s timeout): abort after 2s and show "Temporary network delay. Please try again shortly" to stop a backend stampede.
2. Circuit breaker (Resilience4j)
Like a household breaker, it lives in the Spring app and, on Redis failure, opens so Redis calls stop immediately.
- 🟢 CLOSED (healthy): all cache/data requests go to Redis.
- 🔴 OPEN (blocked): if Redis latency/error rate exceeds the threshold, open the breaker skip Redis entirely, and run fallback (e.g. query the main RDB) immediately.
- 🟡 HALF-OPEN (probe): send a few requests to see if Redis recovered; if they succeed, return to 🟢 CLOSED.
🔄 How the two techniques work together on failure
- 0s (Redis master down): Sentinel starts failover; latency appears
- 1–2s (circuit breaker): Resilience4j sees errors/latency and switches to 🔴 OPEN ➔ query RDB or return a default (fallback) instead of Redis
- 2s (front-end timeout): the frontend drops the request so infinite loading and mash-refresh cannot flood the WAS
- 4–5s (failover done & auto-recover): after Sentinel promotes a new master, Resilience4j returns to 🟢 CLOSED and cache service recovers
| << K8s Redis-Sentinel App ① | K8s Sidecar Pattern Prep ① >> |
|---|
