Redis-Sentinel Sidecar Pattern Hands-on Guide on Kubernetes ②

Setup (create Pods)

1. Apply YAML files in order

1. Create Config

💻 Terminal

$ k apply -f redis-sentinel-sidecar-config.yaml	➡ apply (create)
  configmap/redis-sentinel-sidecar-config created	➡ created
  configmap/redis-sentinel-sidecar-config configured	➡ changed
  configmap/redis-sentinel-sidecar-config unchanged	➡ unchanged
$ k get configmap redis-sentinel-sidecar-config	➡ check
  NAME                                      DATA   AGE
  redis-sentinel-sidecar-config   3      19s
  * DATA 3 means redis.conf, users.acl, and sentinel.conf.

2. Create Service

💻 Terminal

$ k apply -f redis-sentinel-sidecar-service.yaml	➡ apply (create)
  service/redis-sentinel-service created
$ kubectl get service redis-sentinel-service	➡ check
  NAME                     TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)               AGE
  redis-sentinel-service   ClusterIP   None         <none>        18504/TCP,28504/TCP   3m1s

3. Create StatefulSet

💻 Terminal

$ k apply -f redis-sentinel-sidecar-statefulset.yaml	➡ apply (create)
  statefulset.apps/redis-sentinel created
$ k get pods -o wide	➡ list Pods 🧮
  NAME               READY   STATUS    RESTARTS   AGE     IP            NODE  
  redis-sentinel-0   2/2     Running   0          29s     10.42.0.106   rocky9
  redis-sentinel-1   2/2     Running   0          22s     10.42.0.108   rocky9
  redis-sentinel-2   2/2     Running   0          18s     10.42.0.110   rocky9

2. Verify Redis-Sentinel topology

1) Redis role check: ROLE

💻 Terminal

$ kubectl exec -it redis-sentinel-0 -c redis -- redis-cli -p 18504 -a redisgate role
  master ➡ Redis-0 is master with two replicas below.
    1) slave:redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
    2) slave:redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504
$ kubectl exec -it redis-sentinel-1 -c redis -- redis-cli -p 18504 -a redisgate role
   slave:redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
$ kubectl exec -it redis-sentinel-2 -c redis -- redis-cli -p 18504 -a redisgate role
   slave:redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504

2) Check redis.conf

📄 redis.conf

💻 Redis-0		
  replica-priority 100
  replica-announce-ip redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local  
💻 Redis-1
  replica-priority 110
  replica-announce-ip redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local
  replicaof redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 ➡ points at Redis-0 master 
💻 Redis-2
  replica-priority 120
  replica-announce-ip redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local
  replicaof redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 ➡ points at Redis-0 master

3) Check sentinel.conf

📄 sentinel.conf

💻 Sentinel-0
📍sentinel resolve-hostnames yes
📍sentinel announce-hostnames yes
📍sentinel monitor mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 2
sentinel down-after-milliseconds mymaster 3000
📍sentinel announce-ip "redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local"
sentinel announce-port 28504
# Generated by CONFIG REWRITE
📍sentinel known-replica mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504
📍sentinel known-sentinel mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 28504 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd
sentinel known-sentinel mymaster redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 28504 4d41e0ad06a5e2b48c87b09cf7136473569efe40

💻 Sentinel-1
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel monitor mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 2
sentinel down-after-milliseconds mymaster 3000
sentinel announce-ip "redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local"
sentinel announce-port 28504
# Generated by CONFIG REWRITE
sentinel known-replica mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504
sentinel known-sentinel mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 28504 1d4285d6f0c8027fb97375d2ce4ae7168a8b74ca
sentinel known-sentinel mymaster redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 28504 4d41e0ad06a5e2b48c87b09cf7136473569efe40

💻 Sentinel-2
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel monitor mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 2
sentinel down-after-milliseconds mymaster 3000
sentinel announce-ip "redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local"
sentinel announce-port 28504
# Generated by CONFIG REWRITE
sentinel known-replica mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504
sentinel known-sentinel mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 28504 1d4285d6f0c8027fb97375d2ce4ae7168a8b74ca
sentinel known-sentinel mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 28504 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd

4) Check sentinel monitoring

Run `info sentinel` on Sentinel-0,1,2. Summary of the result:
kubectl exec -it redis-sentinel-0 -c sentinel -- redis-cli -p 28504 -a password info sentinel

💻 Terminal

# Sentinel
sentinel_masters:1
Sentinel-0: master0:name=📍mymaster,status=ok,address=redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504,📍slaves=2,📍sentinels=3 
Sentinel-1: master0:name=mymaster,status=ok,address=redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504,slaves=2,sentinels=3 
Sentinel-2: master0:name=mymaster,status=ok,address=redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504,slaves=2,sentinels=3

5) 🖧 Redis-Sentinel node diagram

┌───────────────┐ │ Redis-0(106) │ │ 🟢Master │ │ Sentinel-0 │ └──────┬────────┘ ┌──────────└───────────┐ ▼ ▼ ┌───────────────┐ ┌───────────────┐ │ Redis-1(108) │ │ Redis-2(110) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ │ Sentinel-1 │ │ Sentinel-2 │ └───────────────┘ └───────────────┘

3. 💽 Load data

If you delete Redis-0 for a failover test, Kubernetes recreates it within about 2 seconds. down-after-milliseconds is 3 seconds, so Sentinel may not notice the outage. Load about 3 million keys so restart takes 5+ seconds (AOF load ~5s).

💻 Terminal

$ kubectl exec -it redis-sentinel-0 -- redis-benchmark -p 18504 -a password -t set -d 100 -r 100000000 -n 3000000 
====== SET ======                                                     
  3000000 requests completed in 30.97 seconds
Summary:
  throughput summary: 96871.06 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.427     0.120     0.407     0.639     0.823    12.599

redis-benchmark generates random keys, so dbsize is a bit under 3 million.
- Actual keys: 2955238
- AOF size: aof.1.incr.aof -> 412M
- Pod memory: kubectl top pod -> 625Mi

➡ Pods are up and data is loaded. Next: failover testing.

<< K8s Sidecar Pattern Prep ①	K8s Sidecar Pattern Test ③ >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.