redis education kubernetes k8s_redis_scaleout

Redis Scale-Out on Kubernetes

Scaling Redis on Kubernetes

1. Advice on scaling Redis-Cluster

Only increasing Pod count with kubectl scale --replicas does not scale Redis-Cluster.
Below is the standard procedure to actually scale Redis-Cluster.

• ① Scale Redis Pods with Kubernetes (e.g. 6 ➔ 10 Pods)
• ② Join new nodes with redis-cli --cluster add-node
• ③ Move slots and keys with cluster rebalance
• Only after these steps is Redis-Cluster actually scaled.

💻 Why live scale-out is hard at 100% CPU

• Step ③ cluster rebalance moves slots and data and adds significant load. Moving large datasets from Redis-0,1,2 to new nodes uses network and CPU. If CPU is already 100%, rebalance can spike latency and cause an outage.
• Slowdown during rebalance: While keys move, apps hit frequent ASK / MOVED redirections between old and new nodes, so latency rises and CPU goes even higher.
• Topology complexity: You must keep an odd number of masters (3 ➔ 5) and 1:1 replicas, so scaling 6 Pods to 10 at once makes rebalance much more complex.
• nodes.conf mismatch risk: If 1–2 Pods die during rebalance, the whole cluster can stall. If a node dies mid-slot-move, nodes.conf slot ownership can disagree across Pods and auto-recovery may fail.
• [Ops guide] If load is expected to grow, scale early while CPU is under 60%. If a spike already happened, wait until load settles, then scale.

🧑‍💻 Auto-scale with Redis-Operator

• You can automate Redis-Cluster scale-out with a Redis-Operator (separate product).
• The operator runs add-node and rebalance for you.
• Even then, do not set a high cpu averageUtilization in hpa.yaml.

🌐 Redis-Operator sources (URL)

• Redis-Operator (OT-CONTAINER-KIT)
  Description: One of the most actively used open-source Redis Operators. Define a `RedisCluster` CRD and it handles add/remove, failover, and auto-scaling.
  Docs / GitHub: 🔗 redis-operator   🔗 getting-started
  
• Spotahome Redis Operator
  Description: Operator from Spotahome (Spain) to manage Redis-Sentinel and Redis-Cluster as Kubernetes controllers.
  Docs / GitHub: 🔗 Spotahome Redis Operator
• Redis Enterprise K8s Operator (official Redis)
  Description: Official operator from Redis Ltd. Integrates with HPA and custom metrics to automate node add and shard rebalancing.
  Official guide: 🔗 Redis Enterprise K8s Operator

2. Advice on scaling Redis-Sentinel

In Redis-Sentinel, 100% CPU almost always happens on the master. A single-master topology funnels all writes (set/update/delete) to one node.

• Limits of adding replicas: Two replicas already share read traffic, so replica CPU bottlenecks are rare. Adding a Pod only adds another replica and does not fix write load or CPU on the master.
• Read-split caveat: A new replica does a full resync from the master. That adds RDB generation and network I/O on the master and can make master CPU worse.

💡 Conclusion and better alternative

When Sentinel is limited by master load, adding replica Pods is not the fix.   The real scale-out is an architecture change: if write traffic will keep growing, leave single-master Sentinel and move to Redis-Cluster so writes spread across multiple masters.

How to scale Redis-Cluster

1. Add Pods

1) Check current state 🧮

redis-cluster-0~5: six Pods.

2) Add Pods: 6 -> 10 ⚡

3) Verify after adding Pods 🧮

redis-cluster-6~9: four extra Pods created

2. Add Redis-Cluster nodes

Added as masters, but no slots yet, so they do not receive data.

1) Add redis-cluster-6: add-node

kubectl exec -it redis-cluster-0 -- redis-cli -a redisgate --cluster add-node
redis-cluster-6.redis-cluster-service.default.svc.cluster.local:18504
redis-cluster-0.redis-cluster-service.default.svc.cluster.local:18504
Connect to Redis-0 and add Redis-6 to the cluster.

2) Add redis-cluster-7: add-node

kubectl exec -it redis-cluster-0 -- redis-cli -a redisgate --cluster add-node
redis-cluster-7.redis-cluster-service.default.svc.cluster.local:18504
redis-cluster-0.redis-cluster-service.default.svc.cluster.local:18504
Connect to Redis-0 and add Redis-7 to the cluster. ⇒ output omitted

3. Assign slots: rebalance

1) Run rebalance ⚡

Assign slots to new nodes: run rebalance
kubectl exec -it redis-cluster-0 -- redis-cli -a redisgate
--cluster rebalance redis-cluster-0.redis-cluster-service.default.svc.cluster.local:18504
--cluster-use-empty-masters
* One '#' per moved slot. Shown in full, not truncated.
* Time depends on how many keys move.

2) Verify slot assignment

3) Redis-Cluster status: cluster info

4) Redis-Cluster status: cluster nodes

4. Add nodes 8 and 9

Add nodes 8 and 9 as replicas.

1) Add node 8: add-node --cluster-slave

kubectl exec -it redis-cluster-0 -- redis-cli -a redisgate --cluster add-node
redis-cluster-8.redis-cluster-service.default.svc.cluster.local:18504
redis-cluster-6.redis-cluster-service.default.svc.cluster.local:18504 --cluster-slave
Add node 8 as a replica (slave) of node 6.
Use `--cluster-slave`: 🚨 `--cluster-replica` is not valid (confirmed in Redis 8.8.2 redis-cli.c)

2) Add node 9: add-node --cluster-slave

kubectl exec -it redis-cluster-0 -- redis-cli -a redisgate --cluster add-node
redis-cluster-9.redis-cluster-service.default.svc.cluster.local:18504
redis-cluster-7.redis-cluster-service.default.svc.cluster.local:18504 --cluster-slave
Add node 9 as a replica (slave) of node 7. ⇒ output omitted

3) Verify added replica nodes

Confirm 8 and 9 are slaves: nodes 199 and 201 went from 0 slaves to 1.

4) Verify added replica nodes

Output rearranged for readability.

5) Final Pod list 🧮

6) Check memory 🧮

7) 🖧 Final Redis-Cluster Pod diagram

🧑‍💻 Wrap-up

Redis-Cluster Pod scale-out on Kubernetes is complete.