Edu/Support

Online Periodic Inspection
Service Now Open

Redis-Cluster on Kubernetes: Hands-on Guide ③

Failover test

🖧 1. Redis-Cluster Pod diagram (initial state)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟢Redis-0(231) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Master-1 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ │ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟡Redis-4(239) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Replica-2 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

1. Failover test

Take Redis-0 Pod down and confirm Redis-4 switches (promotes) from Replica to Master
The redis.conf replica-priority value is not used in Redis-Cluster. It is used in Redis-Sentinel. Instead, the replica that received the newest data becomes master. Here each master has one replica, so Redis-4 (replica of Redis-0) becomes master.
💥 After a Pod is taken down and restarted, its IP changes.

Redis-0 💥 down (delete)

💻 Terminal

$ kubectl delete pod redis-cluster-0
  pod "redis-cluster-0" deleted

💻 Redis-0 LOG: Master -> Replica

① 24s received kill signal, Redis down (+0s)
② 26s Redis auto-restart (+2s)
③ 35s switched Master -> Replica(Slave) (+11s)

💻 Terminal

1:M 14:42:24.187 #📍received kill signal and went down: Redis is now ready to exit, bye bye...
1:C 14:42:26.706 *📍restarted after 2 seconds: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 
1:S 14:42:35.038 *📍switched Master -> Replica(Slave): Before turning into a replica, ...

Redis-0 full log ▼

📄 redis.log

$ kubectl exec -it redis-cluster-0 -- tail -f /data/redis.log
1:signal-handler (1786167744) Received SIGTERM scheduling shutdown... 
1:M 14:42:24.182 * User requested shutdown...
1:M 14:42:24.187 #📍received kill signal and went down: Redis is now ready to exit, bye bye...
1:C 14:42:26.706 *📍restarted after 2 seconds: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 
1:M 14:42:26.708 * Node configuration loaded, I'm 66390566e7f48211e8bed3a6a0e49155cbe8b54b
1:M 14:42:26.709 * Server initialized
1:M 14:42:26.709 * Reading RDB base file on AOF loading...
1:M 14:42:26.709 * Loading RDB produced by version 8.8.1
1:M 14:42:26.709 * RDB age 3615 seconds
1:M 14:42:26.709 * RDB memory usage when created 2.16 Mb
1:M 14:42:26.710 * RDB is base AOF
1:M 14:42:26.710 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 14:42:26.710 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 14:42:35.027 * DB loaded from incr file aof.1.incr.aof: 8.317 seconds
1:M 14:42:35.027 *➡took 8.3 seconds: DB loaded from append only file: 8.318 seconds	
1:M 14:42:35.031 * Opening AOF incr file aof.1.incr.aof on server start
1:M 14:42:35.031 * Ready to accept connections tcp
1:M 14:42:35.038 * Configuration change detected. Reconfiguring myself as a replica of ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 14:42:35.038 *📍switched Master -> Replica(Slave): Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 14:42:35.038 *➡connecting to the new master: Connecting to MASTER 10.42.0.239:18504 
1:S 14:42:35.038 *➡started receiving data from the new master: MASTER <-> REPLICA sync started	
1:S 14:42:35.039 * Cluster state changed: ok
1:S 14:42:35.042 * Non blocking connect for SYNC fired the event.
1:S 14:42:35.046 * Master replied to PING, replication can continue...
1:S 14:42:35.046 * Trying a partial resynchronization (request b95c76e4720380a0cdced96e236842d144aeb5b8:918944394).
1:S 14:42:35.047 * PSYNC is not possible, initialize RDB channel.
1:S 14:42:38.638 * Starting to receive RDB and replication stream in parallel.
1:S 14:42:38.676 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 14:42:42.179 * Discarding previously cached master state.
1:S 14:42:42.179 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 14:42:42.188 * MASTER <-> REPLICA sync: Flushing old data
1:S 14:42:43.300 * Loading RDB produced by version 8.8.1
1:S 14:42:43.300 * RDB age 5 seconds
1:S 14:42:43.300 * RDB memory usage when created 650.33 Mb
1:S 14:42:45.600 * Done loading RDB, keys loaded: 3053053, keys expired: 0.
1:S 14:42:45.600 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 14:42:45.600 *➡finished loading data from the new master: MASTER <-> REPLICA sync: Finished with success 
1:S 14:42:45.600 *➡AOF rewrite started: MASTER <-> REPLICA sync: Starting AOF after a successful sync  
1:S 14:42:45.600 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 14:42:45.613 * Background append only file rewriting started by pid 19
1:S 14:42:45.613 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (0 bytes).
1:S 14:42:45.613 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (0 bytes in total)
19:C 14:42:48.921 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717496 bytes written.
19:C 14:42:48.927 * Successfully created the temporary AOF base file temp-rewriteaof-bg-19.aof
19:C 14:42:48.933 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 1 MB
1:S 14:42:49.027 * Background AOF rewrite terminated with success
1:S 14:42:49.027 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-19.aof into aof.2.base.rdb
1:S 14:42:49.027 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.2.incr.aof
1:S 14:42:49.031 * Removing the history file aof.1.incr.aof in the background
1:S 14:42:49.031 * Removing the history file aof.1.base.rdb in the background
1:S 14:42:49.036 *➡AOF rewrite finished: Background AOF rewrite finished successfully

💻 Redis-1 LOG: Master

① 28s Redis-0 confirmed down (Fail/odown) (+4s) -> cluster-node-timeout 3000ms
② 29s Redis-4 promoted to new master (+5s)

📄 redis.log

$ kubectl exec -it redis-cluster-1 -- tail -f /data/redis.log
1:M 14:42:28.507 *📍Redis-0 confirmed down: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached).
1:M 14:42:29.608 *📍Redis-4 promoted to new master: Failover auth granted to ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) for epoch 7	
1:M 14:42:35.033 *➡Redis-0 Pod restarted; new IP (10.42.0.242) detected: Address updated for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0), now 10.42.0.242:18504
1:M 14:42:35.078 *➡Redis-0 reconnected as replica; FAIL state cleared: Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):replica is reachable again.

💻 Redis-4 LOG: Replica -> Master

① 24s Redis-0 down detected (+0s)
② 29s promoted to master (+5s)

📄 redis.log

$ kubectl exec -it redis-cluster-4 -- tail -f /data/redis.log
1:S 14:42:24.235 *📍Redis-0 down detected: Connection with master lost.	 
1:M 14:42:29.613 *📍promoted to master: Discarding previously cached master state.

Redis-4 full log ▼

📄 redis.log

$ kubectl exec -it redis-cluster-4 -- tail -f /data/redis.log
1:S 14:42:24.235 *📍Redis-0 down detected: Connection with master lost.	 
1:S 14:42:24.235 * Caching the disconnected master state.
1:S 14:42:24.235 *➡reconnect attempt Reconnecting to MASTER 10.42.0.231:18504 
1:S 14:42:24.235 * MASTER <-> REPLICA sync started
1:S 14:42:24.235 #➡connect failed Error condition on socket for SYNC: Connection refused	
1:S 14:42:24.579 * Connecting to MASTER 10.42.0.231:18504
1:S 14:42:24.579 * MASTER <-> REPLICA sync started
1:S 14:42:28.507 * FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 14:42:28.598 * Start of election delayed for 972 milliseconds (rank #0, offset 918944393).
1:S 14:42:29.602 * Starting a failover election for epoch 7.
1:S 14:42:29.613 * Failover election won: I'm the new master.
1:S 14:42:29.613 * configEpoch set to 7 after successful failover					
1:M 14:42:29.613 *📍promoted to master: Discarding previously cached master state.	
1:M 14:42:29.613 * Setting secondary replication ID to c76d2514db200f6d13ba070d8ef349efe33bc2cf, valid up to offset: 918944394. New replication ID is caab310287582352de716fc92f8542ef435b125d
1:M 14:42:35.033 * Address updated for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0), now 10.42.0.242:18504
1:M 14:42:35.042 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node 66390566e7f48211e8bed3a6a0e49155cbe8b54b () lost 0 slot(s) to node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) with a config epoch of 7
1:M 14:42:35.047 *➡old master (now replica) requests sync: Replica 10.42.0.242:18504 asks for synchronization 
1:M 14:42:35.047 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'b95c76e4720380a0cdced96e236842d144aeb5b8', my replication IDs are 'caab310287582352de716fc92f8542ef435b125d' and 'c76d2514db200f6d13ba070d8ef349efe33bc2cf')
1:M 14:42:35.047 * Replica 10.42.0.242:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 14:42:35.047 * Replica 10.42.0.242:18504 asks for synchronization
1:M 14:42:35.047 * Full resync requested by replica 10.42.0.242:18504 (rdb-channel)
1:M 14:42:35.047 * Delay next BGSAVE for diskless SYNC
1:M 14:42:35.124 * Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):replica is reachable again.
1:M 14:42:38.638 *➡started building data to send: Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 14:42:38.638 * Starting to deliver RDB and replication stream to replica: 10.42.0.242:18504
1:M 14:42:38.652 * Background RDB transfer started by pid 20 to replica socket
20:C 14:42:42.174 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717629 bytes written.
20:C 14:42:42.183 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 1 MB
1:M 14:42:42.267 * Background RDB transfer terminated with success
1:M 14:42:42.267 *➡sync to the new replica finished: Synchronization with replica 10.42.0.242:18504 succeeded 
1:M 14:42:42.267 * Streamed RDB transfer with replica 10.42.0.242:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 14:42:42.267 * RDB transfer completed, rdb only replica (10.42.0.242:18504) should be disconnected asap
1:M 14:42:42.267 * Connection with replica (rdbchannel) 10.42.0.242:18504 lost.

🧮 State after failover

Redis-0 IP changed from '10.42.0.231' to '10.42.0.242'.

💻 Terminal

[redis@Rocky9 kube]$ k get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE    IP            NODE  
redis-cluster-0   1/1     Running   0          3m3s   10.42.0.242   rocky9	IP:231->242
redis-cluster-1   1/1     Running   0          63m    10.42.0.233   rocky9
redis-cluster-2   1/1     Running   0          63m    10.42.0.235   rocky9
redis-cluster-3   1/1     Running   0          63m    10.42.0.237   rocky9
redis-cluster-4   1/1     Running   0          63m    10.42.0.239   rocky9
redis-cluster-5   1/1     Running   0          63m    10.42.0.241   rocky9

🖧 Redis-Cluster Pod diagram (after failover)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Replica-2 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ ▲ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ │ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟢Redis-4(239) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Master-1 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

2. Failback test

Take current master Redis-4 down and confirm Redis-0 switches (promotes) from Replica to Master
📚 'Failback test' is used here to mean making the Redis-0 Pod master again.
(Used to distinguish it from the failover test)

💻 Redis-4 💥 down (delete)

💻 Terminal

$ kubectl delete pod redis-cluster-4
  pod "redis-cluster-4" deleted

💻 Redis-0 LOG: Replica -> Master

① 52s connection to Redis-4 lost (+0s)
② 56s Redis-4 confirmed down (Fail/odown) (+4s)
③ 57s switched Replica -> Master (promoted) (+5s)

📄 redis.log

1:S 15:19:52.289 *📍connection to Redis-4 lost: Connection with master lost.	
1:S 15:19:56.549 *📍Redis-4 confirmed down: Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached). 
1:M 15:19:57.276 *📍switched Replica -> Master (promoted): Discarding previously cached master state.

Redis-0 full log ▼

📄 redis.log

1:S 15:19:52.289 *📍connection to Redis-4 lost: Connection with master lost.	
1:S 15:19:52.289 * Caching the disconnected master state.
1:S 15:19:52.289 *➡trying to reconnect to Redis-4: Reconnecting to MASTER 10.42.0.239:18504	
1:S 15:19:52.289 * MASTER <-> REPLICA sync started
1:S 15:19:52.289 # Error condition on socket for SYNC: Connection refused
1:S 15:19:52.644 * Connecting to MASTER 10.42.0.239:18504
1:S 15:19:52.644 * MASTER <-> REPLICA sync started
1:S 15:19:56.549 *📍Redis-4 confirmed down (odown): Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached).
1:S 15:19:56.560 * Start of election delayed for 639 milliseconds (rank #0, offset 918947529).
1:S 15:19:57.267 * Starting a failover election for epoch 8.
1:S 15:19:57.276 * Failover election won: I'm the new master.
1:S 15:19:57.276 * configEpoch set to 8 after successful failover                              
1:M 15:19:57.276 *📍switched Replica(Slave) -> Master (promoted): Discarding previously cached master state.
1:M 15:19:57.276 * Setting secondary replication ID to caab310287582352de716fc92f8542ef435b125d, valid up to offset: 918947530. New replication ID is 2e46928d6c500b1c71c1038b815e61b9c059cc82
1:M 15:20:02.615 * Address updated for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4), now 10.42.0.243:18504
1:M 15:20:02.632 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 () lost 0 slot(s) to node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) with a config epoch of 8
1:M 15:20:02.637 * Replica 10.42.0.243:18504 asks for synchronization
1:M 15:20:02.637 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'f0e297826ed9c693bb4b5abccd5c415d8319ac10', my replication IDs are '2e46928d6c500b1c71c1038b815e61b9c059cc82' and 'caab310287582352de716fc92f8542ef435b125d')
1:M 15:20:02.637 * Replica 10.42.0.243:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 15:20:02.638 * Replica 10.42.0.243:18504 asks for synchronization
1:M 15:20:02.638 * Full resync requested by replica 10.42.0.243:18504 (rdb-channel)
1:M 15:20:02.638 * Delay next BGSAVE for diskless SYNC
1:M 15:20:02.689 *➡Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):replica is reachable again.
1:M 15:20:05.701 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 15:20:05.701 * Starting to deliver RDB and replication stream to replica: 10.42.0.243:18504
1:M 15:20:05.708 * Background RDB transfer started by pid 28 to replica socket
28:C 15:20:08.931 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717629 bytes written.
28:C 15:20:08.938 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 1 MB
1:M 15:20:09.019 * Background RDB transfer terminated with success
1:M 15:20:09.019 * Synchronization with replica 10.42.0.243:18504 succeeded
1:M 15:20:09.019 * Streamed RDB transfer with replica 10.42.0.243:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 15:20:09.019 * RDB transfer completed, rdb only replica (10.42.0.243:18504) should be disconnected asap
1:M 15:20:09.020 * Connection with replica (rdbchannel) 10.42.0.243:18504 lost.

💻 Redis-1 LOG: Master

① 56s Redis-4 confirmed down (+4s)
② 57s Redis-0 promoted to new master (+5s)

📄 redis.log

1:M 15:19:56.549 *📍Redis-4 confirmed down: FAIL message received from 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:M 15:19:57.271 *📍Redis-0 promoted to new master: Failover auth granted to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) for epoch 8
1:M 15:20:02.615 *➡Redis-4 Pod restarted -> new IP (10.42.0.243) detected: Address updated for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4), now 10.42.0.243:18504
1:M 15:20:02.692 *➡Redis-4 reconnected as replica; FAIL state cleared: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):replica is reachable again.

💻 Redis-4 LOG: Master -> Replica

① 19m 52s received kill signal and went down (+0s)
② 19m 54s Pod (Redis) restarted (+2s)
③ 20m 02s switched Master -> Replica(Slave) (+10s)

📄 redis.log

1:M 15:19:52.237 #📍received kill signal and went down: Redis is now ready to exit, bye bye...
1:M 15:19:54.305 *📍Pod (Redis) restarted: Server initialized
1:S 15:20:02.629 *📍switched Master -> Replica(Slave): Before turning into a replica, ...

Redis-4 full log ▼

📄 redis.log

1:signal-handler (1786169992) Received SIGTERM scheduling shutdown...
1:M 15:19:52.232 * User requested shutdown...
1:M 15:19:52.237 #📍received kill signal and went down: Redis is now ready to exit, bye bye...
1:M 15:19:54.305 *📍Pod (Redis) restarted: Server initialized
1:M 15:19:54.306 * Reading RDB base file on AOF loading...
1:M 15:19:54.306 * Loading RDB produced by version 8.8.1
1:M 15:19:54.306 * RDB age 5681 seconds
1:M 15:19:54.306 * RDB memory usage when created 2.39 Mb
1:M 15:19:54.306 * RDB is base AOF
1:M 15:19:54.306 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 15:19:54.306 * DB loaded from base file aof.2.base.rdb: 0.001 seconds
1:M 15:20:02.608 * DB loaded from incr file aof.2.incr.aof: 8.302 seconds
1:M 15:20:02.609 *➡Data Load took 8.3 seconds: DB loaded from append only file: 8.303 seconds 
1:M 15:20:02.613 * Opening AOF incr file aof.2.incr.aof on server start
1:M 15:20:02.613 * Ready to accept connections tcp
1:M 15:20:02.629 * Configuration change detected. Reconfiguring myself as a replica of 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 15:20:02.629 *📍switched Master -> Replica(Slave): Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 15:20:02.629 *➡connecting to the new master: Connecting to MASTER 10.42.0.242:18504	
1:S 15:20:02.629 * MASTER <-> REPLICA sync started
1:S 15:20:02.629 * Cluster state changed: ok
1:S 15:20:02.632 * Non blocking connect for SYNC fired the event.
1:S 15:20:02.637 * Master replied to PING, replication can continue...
1:S 15:20:02.637 * Trying a partial resynchronization (request f0e297826ed9c693bb4b5abccd5c415d8319ac10:918947530).
1:S 15:20:02.637 * PSYNC is not possible, initialize RDB channel.
1:S 15:20:05.701 * Starting to receive RDB and replication stream in parallel.
1:S 15:20:05.716 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 15:20:08.938 * Discarding previously cached master state.
1:S 15:20:08.938 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 15:20:08.947 * MASTER <-> REPLICA sync: Flushing old data
1:S 15:20:10.042 * Loading RDB produced by version 8.8.1
1:S 15:20:10.042 * RDB age 5 seconds
1:S 15:20:10.042 * RDB memory usage when created 522.06 Mb
1:S 15:20:12.351 * Done loading RDB, keys loaded: 3053053, keys expired: 0.
1:S 15:20:12.351 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 15:20:12.351 *➡sync finished: MASTER <-> REPLICA sync: Finished with success  
1:S 15:20:12.351 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 15:20:12.351 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 15:20:12.370 * Background append only file rewriting started by pid 19
1:S 15:20:12.370 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (0 bytes).
1:S 15:20:12.370 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (0 bytes in total)
19:C 15:20:15.668 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717496 bytes written.
19:C 15:20:15.673 * Successfully created the temporary AOF base file temp-rewriteaof-bg-19.aof
19:C 15:20:15.682 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 1 MB
1:S 15:20:15.790 * Background AOF rewrite terminated with success
1:S 15:20:15.790 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-19.aof into aof.3.base.rdb
1:S 15:20:15.790 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.3.incr.aof
1:S 15:20:15.795 * Removing the history file aof.2.incr.aof in the background
1:S 15:20:15.795 * Removing the history file aof.2.base.rdb in the background
1:S 15:20:15.798 *➡AOF rewrite finished: Background AOF rewrite finished successfully

🧮 State after failback

Redis-4 IP changed from '10.42.0.239' to '10.42.0.243'.

💻 Terminal

$ k get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-cluster-0   1/1     Running   0          47m     10.42.0.242   rocky9	📍IP:231->242
redis-cluster-1   1/1     Running   0          107m    10.42.0.233   rocky9
redis-cluster-2   1/1     Running   0          107m    10.42.0.235   rocky9
redis-cluster-3   1/1     Running   0          107m    10.42.0.237   rocky9
redis-cluster-4   1/1     Running   0          9m34s   10.42.0.243   rocky9 📍IP:239->243
redis-cluster-5   1/1     Running   0          106m    10.42.0.241   rocky9

🧮 Check cluster info

💻 Terminal

$ ./cluster_info.sh
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384	➡📍all slots OK
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3

🧮 Check cluster nodes

💻 Terminal

$ ./cluster_nodes.sh
66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local master - 0 1786170813036 8 connected 0-5460
3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,master - 0 0 2 connected 5461-10922
02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local master - 0 1786170813539 3 connected 10923-16383
3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786170813237 3 connected
ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local slave 66390566e7f48211e8bed3a6a0e49155cbe8b54b 0 1786170814041 8 connected
541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786170813539 2 connected

🖧 Redis-Cluster Pod diagram (after failback)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟢Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Master-1 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ │ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟡Redis-4(243) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Replica-2 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

🧑‍💻 Conclusion

Redis-Cluster failover and failback tests succeeded
Taking a Pod down and restarting it changes the IP, and it is handled without issues.

Failover test: Redis-0 SLEEP (pause)

The SLEEP test creates a situation where Redis cannot respond for any reason, and checks whether failover still works correctly.
💥 'Sleep 10' pauses Redis for 10 seconds, so the IP does not change.

1. Failover test

• Failover test: Redis-0 Sleep 10 -> confirm Redis-4 switches (promotes) from Replica to Master

Redis-0 SLEEP 10

💻 Terminal

$ kubectl exec -it redis-cluster-0 -- redis-cli -p 18504 -a redisgate DEBUG sleep 10

💻 Redis-0 LOG: Master -> Replica

① 17s SLEEP started (+0s)
② 27s SLEEP ended (+10s)
③ 27s switched Master -> Replica (+10s)

📄 redis.log

1:M 17:08:27.199 *📍SLEEP ended: Connection with replica 10.42.0.243:18504 lost.
1:S 17:08:27.200 *📍switched Master -> Replica: Before turning into a replica, ...

Redis-0 full log ▼

📄 redis.log

1:M 17:08:27.199 *📍SLEEP ended: Connection with replica 10.42.0.243:18504 lost.
1:M 17:08:27.200 * Configuration change detected. Reconfiguring myself as a replica of ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 17:08:27.200 *📍 switched Master -> Replica: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 17:08:27.200 * Connecting to MASTER 10.42.0.243:18504
1:S 17:08:27.200 * MASTER <-> REPLICA sync started
1:S 17:08:27.205 * Non blocking connect for SYNC fired the event.
1:S 17:08:27.210 * Master replied to PING, replication can continue...
1:S 17:08:27.211 * Trying a partial resynchronization (request 2e46928d6c500b1c71c1038b815e61b9c059cc82:918956630).
1:S 17:08:27.211 * Successful partial resynchronization with master.
1:S 17:08:27.211 * Master replication ID changed to 8d3577deeb6b8113718c642303195a34c4b78d09
1:S 17:08:27.211 *➡partial sync finished: MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Redis-1 LOG: Master

① 21s Redis-0 confirmed down (Fail/odown) (+4s) -> cluster-node-timeout 3000ms
② 22s Redis-4 switched Replica -> Master (+5s)
③ 27s Redis-0 restarted (+10s)

📄 redis.log

💻 Redis-4 LOG: Replica -> Master

① 21s Redis-0 confirmed down (Fail/odown) (+4s)
② 22s switched Replica -> Master (+5s)

📄 redis.log

1:S 17:08:21.073 *📍Redis-0 confirmed down: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached).
1:S 17:08:22.144 *📍switched Replica(Slave) -> Master: Failover election won: I'm the new master.

Redis-4 full log ▼

📄 redis.log

1:S 17:08:21.073 *📍Redis-0 confirmed down: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached).
1:S 17:08:21.131 * Start of election delayed for 944 milliseconds (rank #0, offset 918956629).
1:S 17:08:22.134 * Starting a failover election for epoch 9.
1:S 17:08:22.144 *📍switched Replica(Slave) -> Master: Failover election won: I'm the new master.
1:S 17:08:22.144 * configEpoch set to 9 after successful failover                               
1:M 17:08:22.144 * Connection with master lost.
1:M 17:08:22.144 * Caching the disconnected master state.
1:M 17:08:22.144 * Discarding previously cached master state.
1:M 17:08:22.144 * Setting secondary replication ID to 2e46928d6c500b1c71c1038b815e61b9c059cc82, valid up to offset: 918956630. New replication ID is 8d3577deeb6b8113718c642303195a34c4b78d09
1:M 17:08:27.205 * Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):master without slots is reachable again.
1:M 17:08:27.205 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) lost 0 slot(s) to node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) with a config epoch of 9
1:M 17:08:27.211 * Replica 10.42.0.242:18504 asks for synchronization
1:M 17:08:27.211 * Partial resynchronization request from 10.42.0.242:18504 accepted. Sending 0 bytes of backlog starting from offset 918956630.

🧮 State after failover

The IP does not change. The Pod itself stays alive. See the 'AGE' column.

💻 Terminal

$ k get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-cluster-0   1/1     Running   0          146m    10.42.0.242   rocky9
redis-cluster-1   1/1     Running   0          3h26m   10.42.0.233   rocky9
redis-cluster-2   1/1     Running   0          3h26m   10.42.0.235   rocky9
redis-cluster-3   1/1     Running   0          3h26m   10.42.0.237   rocky9
redis-cluster-4   1/1     Running   0          108m    10.42.0.243   rocky9
redis-cluster-5   1/1     Running   0          3h26m   10.42.0.241   rocky9

🧮 cluster info

💻 Terminal

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3

🧮 cluster nodes

💻 Terminal

66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local 🟡slave ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 0 1786176794190 9 connected
3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,🟢master - 0 0 2 connected 5461-10922
02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786176794089 3 connected 10923-16383
3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local 🟡slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786176793184 3 connected
ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786176794000 9 connected 0-5460
541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local 🟡slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786176793586 2 connected

🖧 Redis-Cluster Pod diagram (after failover)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Replica-2 │ │ Master-2 │ │ Master-3 │ └────────────────┘ └──────┬─────────┘ └──────┬────────┘ ▲ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ │ ▼ ┌────────────────┐ ┌────┴───────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟢Redis-4(243) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Master-1 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

2. Failback test

• Failback test: Redis-4 Sleep 10 -> confirm Redis-0 switches (promotes) from Replica to Master

💻 Redis-4 SLEEP 10

💻 Terminal

$ kubectl exec -it redis-cluster-4 -- redis-cli -p 18504 -a redisgate DEBUG sleep 10

💻 Redis-0 LOG: Replica -> Master

① 26s SLEEP command executed (+0s)
② 30s Redis-4 confirmed down (Fail/odown) (+4s)
③ 31s Redis-0 switched Replica -> Master (+5s)
④ 36s Redis-4 restarted (+10s)

📄 redis.log

1:S 20:19:30.481 *📍Redis-4 confirmed down: FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 20:19:31.588 *📍switched Replica(Slave) -> Master: Failover election won: I'm the new master.
1:M 20:19:36.349 *📍Redis-4 restarted: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.

Redis-0 full log ▼

📄 redis.log

1:S 20:19:30.481 *📍Redis-4 confirmed down: FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 20:19:30.568 * Start of election delayed for 971 milliseconds (rank #0, offset 918972673).
1:S 20:19:31.573 * Starting a failover election for epoch 10.
1:S 20:19:31.588 *📍switched Replica(Slave) -> Master: Failover election won: I'm the new master.
1:S 20:19:31.588 * configEpoch set to 10 after successful failover
1:M 20:19:31.588 * Connection with master lost.
1:M 20:19:31.588 * Caching the disconnected master state.
1:M 20:19:31.588 * Discarding previously cached master state.
1:M 20:19:31.588 * Setting secondary replication ID to 8d3577deeb6b8113718c642303195a34c4b78d09, valid up to offset: 918972674. New replication ID is 806c6073fffb5fe2ac6d318f6aceb9d2aa090c18
1:M 20:19:36.349 *📍Redis-4 restarted: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.
1:M 20:19:36.349 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) lost 0 slot(s) to node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) with a config epoch of 10
1:M 20:19:36.354 * Replica 10.42.0.243:18504 asks for synchronization
1:M 20:19:36.354 * Partial resynchronization request from 10.42.0.243:18504 accepted. Sending 0 bytes of backlog starting from offset 918972674.

💻 Redis-1 LOG: Master

① 30s Redis-4 confirmed down (Fail/odown) (+4s)
② 31s Redis-0 switched Replica -> Master (promoted) (+5s)
③ 36s Redis-4 restarted (+10s)

📄 redis.log

1:M 20:19:30.481 *📍Redis-4 confirmed down (FAIL): Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached).
1:M 20:19:31.577 *📍switched Replica -> Master (promoted): Failover auth granted to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) for epoch 10
1:M 20:19:36.349 *📍Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.

💻 Redis-2 LOG: same as Redis-1 log, omitted

💻 Redis-4 LOG: Master -> Replica

① 26s SLEEP command executed (+0s)
① 36s SLEEP command ended -> connection with replica (Redis-0) lost (+10s)
② 36s switched Master -> Replica (+10s)

📄 redis.log

1:M 20:19:36.342 *📍connection with replica (Redis-0) lost: Connection with replica 10.42.0.242:18504 lost.
1:S 20:19:36.343 *📍switched Master -> Replica(Slave): Before turning into a replica, ...

Redis-4 full log ▼

📄 redis.log

1:M 20:19:36.342 *📍connection with replica (Redis-0) lost: Connection with replica 10.42.0.242:18504 lost.
1:M 20:19:36.342 # Failover auth denied to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0): its master is up
1:M 20:19:36.343 * Configuration change detected. Reconfiguring myself as a replica of 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 20:19:36.343 *📍switched Master -> Replica(Slave): Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 20:19:36.343 * Connecting to MASTER 10.42.0.242:18504
1:S 20:19:36.343 * MASTER <-> REPLICA sync started
1:S 20:19:36.349 * Non blocking connect for SYNC fired the event.
1:S 20:19:36.354 * Master replied to PING, replication can continue...
1:S 20:19:36.354 * Trying a partial resynchronization (request 8d3577deeb6b8113718c642303195a34c4b78d09:918972674).
1:S 20:19:36.354 * Successful partial resynchronization with master.
1:S 20:19:36.354 * Master replication ID changed to 806c6073fffb5fe2ac6d318f6aceb9d2aa090c18
1:S 20:19:36.354 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

🧮 State after failback

💻 Terminal

$ k get pods -o wide	-> IP unchanged, not restarted
NAME              READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-cluster-0   1/1     Running   0          5h37m   10.42.0.242   rocky9
redis-cluster-1   1/1     Running   0          6h38m   10.42.0.233   rocky9
redis-cluster-2   1/1     Running   0          6h37m   10.42.0.235   rocky9
redis-cluster-3   1/1     Running   0          6h37m   10.42.0.237   rocky9
redis-cluster-4   1/1     Running   0          5h      10.42.0.243   rocky9
redis-cluster-5   1/1     Running   0          6h37m   10.42.0.241   rocky9

🧮 cluster info

💻 Terminal

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3

🧮 cluster nodes

💻 Terminal

66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786188079000 10 connected 0-5460
3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,🟢master - 0 0 2 connected 5461-10922
02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786188078548 3 connected 10923-16383
3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local 🟡slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786188079552 3 connected
ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local 🟡slave 66390566e7f48211e8bed3a6a0e49155cbe8b54b 0 1786188079552 10 connected
541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local 🟡slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786188079000 2 connected

🖧 Redis-Cluster Pod diagram (after failback)

🧑‍💻 Conclusion

Redis-Cluster failover and failback tests using SLEEP succeeded
With SLEEP, Redis does not go down and the IP does not change, but if there is no response for more than 3 seconds, failover runs.

➡ So far we tested Redis-Cluster by itself.
Click here to see it working with a Java (Spring) App.

<< K8s Redis-Cluster Setup ②	K8s Redis-Cluster App ① >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.