redis education kubernetes k8s_intro_3

Redis on Kubernetes Deployment Guide ③

  Preparing Redis on Kubernetes ③  

☸️ How to prepare a Redis Docker image (online vs air-gapped)

1) Internet-connected environment (Online)

If you set 'image: redis:8.8.2' in the Kubernetes manifest (YAML) and 'imagePullPolicy: IfNotPresent', Kubernetes automatically downloads the image from Docker Hub.

2) Air-gapped environment — Rocky Linux 9.5 approach

On an air-gapped server with no external internet, you must package the image and load it into K3s by hand.

• [Step 1] Extract the image on a PC/server with internet
  # Save with Docker
  $ docker pull redis:8.8.2
  $ docker save -o redis-8.8.2.tar redis:8.8.2
• [Step 2] Move the file to the air-gapped Rocky Linux 9.5 server and load it into K3s
  # Method A: import directly with k3s ctr (containerd CLI) (recommended)
  $ sudo k3s ctr images import redis-8.8.2.tar
  # Method B: copy into the K3s agent auto-load images directory
  $ cp redis-8.8.2.tar /var/lib/rancher/k3s/agent/images/
• ⚠️ Caution: when writing the StatefulSet YAML, set 'imagePullPolicy: IfNotPresent' so Kubernetes does not try to pull the image from the internet again and fail with 'ErrImagePull'.

Build and test environment

Every test in this guide series was verified on the environment below.

• OS: Rocky Linux 9.5 (Blue Onyx) - 64bit
• Kubernetes: K3s (v1.x, CNCF Certified Lightweight Kubernetes)
• Redis Version: Redis 8.8.2 (Latest Open Source Release)
• StorageClass: local-path (K3s built-in Persistent Volume)
• Service Ports: Redis Port: 18504, Sentinel Port: 28504, Cluster Bus Port: 28504
• Network: Headless Service Dynamic DNS Resolution

Kubernetes Redis topologies and test outline

This guide continues across three detailed topology pages and one summary/conclusion page. Use the links below to see how each architecture is built, the YAML, and real failover verification logs.

💻 Redis-Cluster (6pods) setup and tests

• Overview: a high-performance sharding architecture with 3 Master + 3 Replica.
• Key points: StatefulSet + Headless Service, IP-change handling with 'cluster-announce-hostname', and the 'create-redis-cluster.sh' automation script.
• 🔗 [Go to the Redis-Cluster (6pods) detailed deployment guide]

💻 Redis-Sentinel (6pods) setup and tests

• Overview: HA with 3 Redis-only Pods and 3 Sentinel-only Pods, fully separated.
• Key points: automatic failover via Pub/Sub among Sentinels, and priority-based master promotion with 'replica-priority'.
• 🔗 [Go to the Redis-Sentinel (6pods) detailed deployment guide]

💻 Redis-Sentinel sidecar pattern (3pods) setup and tests

• Overview: [Redis + Sentinel] containers in one Pod, implemented as 3 Pods in total
• Key points: fewer resources, shared Pod IP/volume, and verification of full process isolation between main Redis and Sentinel.
• 🔗 [Go to the Redis-Sentinel sidecar pattern (3pods) detailed deployment guide]

💻 Java (Spring) app for Redis-Cluster: implementation/tests

• Overview: which settings to use when implementing an app for a Kubernetes Redis-Cluster environment
• 🔗 [Go to Java (Spring) app implementation/tests for Redis-Cluster]

💻 Java (Spring) app for Redis-Sentinel: implementation/tests

• Overview: which settings to use when implementing an app for a Kubernetes Redis-Sentinel environment
• 🔗 [Go to Java (Spring) app implementation/tests for Redis-Sentinel]

💻 Closing: which architecture should you choose?

• Overview: a final architecture-selection guide based on project size, resource limits (Pod count), data volume, and read/write load-balancing needs.
• 🔗 [Go to the Kubernetes Redis architecture selection guide]

⚠️ A must-know for Spring (Java): JVM DNS caching 📍

Here is a very important Java/Spring trait that Redis developers must know. For performance, the Java Virtual Machine (JVM) caches a DNS lookup result in memory for a while by default.

💥 Possible failure scenario

• 1. A Redis/Sentinel Pod dies and restarts, and the IP changes from '10.244.1.20' to '10.244.2.35'.
• 2. K8s CoreDNS knows the new IP ('10.244.2.35').
• 3. But if the Spring app's JVM still caches the old IP ('10.244.1.20'), it never asks CoreDNS and keeps trying the old IP until a connection error ('UnknownHostException' or 'Connection Refused') occurs.

💡 Fix (JVM DNS TTL)

When the Spring app starts, set the JVM DNS cache TTL short (for example 1–3 seconds).
Here we set it to 1 second. Is 1 second too short? Does asking again every second put load on the JVM?   You might think so. But considering total failover time, it is not too short.
① Redis's own failover time: even with 'cluster-node-timeout' at 3 seconds, it takes about 4–5 seconds.
② Java (Spring Lettuce): Redis-Cluster topology update time: 2 seconds
③ If the JVM DNS cache TTL is 10 seconds, that is 5 + 2 + 10 = 17 seconds.
Failing to process commands for 17 seconds is not a small problem.   So set the JVM DNS cache TTL to 1 second and fully recover within about 8 seconds.

• Add a Java option (JVM option) in deployment.yaml: -Dsun.net.inetaddr.ttl=1 📍
  (Cache a successful DNS lookup for only 1 second, then ask CoreDNS again.)
• Add this to the Spring app startup code (Java):
  java.security.Security.setProperty("networkaddress.cache.ttl", "1"); 📍