RocketMQ 5.x 安装教程 (基于 Kubernetes)

有参考 https://juejin.cn/post/7277395050508746809
有参考 https://rocketmq.apache.org/docs/deploymentOperations/01deploy#start-proxy

本文的目的是基于kubernetes搭建一套 rocketmq 5.1.4 (2 nameserver 2 broker 2 proxy )环境.
有以下功能点

  1. 命名空间隔离. 所有资源均创建于 rocketmq namespace 下
  2. 基于 nfs 做数据存储/日志存储. (基于 default StorageClass PV 生成的 pvc)
  3. 用到的镜像均已做 aliyun 镜像源缓存

关于 RocketMQ 5

官方介绍 RocketMQ 5 https://rocketmq.apache.org/version

RocketMQ 4 和 5 的对比

RocketMQ 5 最大特定就是引入了 Proxy 组件
Proxy 组件是无状态的 Broker. 对Broker职责进行拆分，对于客户端协议适配、权限管理、消费管理等计算逻辑进行抽离，独立无状态的代理角色提供服务

Broker则继续专注于存储能力的持续优化。

这套模式可以更好地实现在云环境(Kubernetes)的资源弹性调度

image.png

准备工作

创建命名空间 rocketmq

后续所有 rocketmq 相关资源均处于该命名空间下, 方便管理和清理

kubectl create namespace rocketmq

创建存储 PVC

用来存储日志和数据

如何创建 NFS PV 可参考 https://cloud.docs.jansora.app/cloud-native/install-k8s/kubernetes-pv-config

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-pvc
  namespace: rocketmq
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

创建自定义配置

为了方便后续对参数的调整, 对配置进行维护是必要的

共有以下配置

  • broker.conf: 创建 broker 和 proxy 使用
  • proxy-config.json: 创建 proxy 会用到的 json 配置
  • runbroker.sh: 创建 broker会用到的启动脚本 (对官方的配置进行略微调整, 增加了可调整内存的配置)
  • runserver.sh: 创建 nameserver 和 proxy 会用到的启动脚本 (对官方的配置进行略微调整, 增加了可调整内存的配置)
# 创建 configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: rocketmq-config
  namespace: rocketmq
data:
  broker.conf: |
    brokerClusterName = DefaultCluster
    brokerName = broker-a
    brokerId = 0
    deleteWhen = 04
    fileReservedTime = 48
    brokerRole = ASYNC_MASTER
    flushDiskType = ASYNC_FLUSH
    autoCreateTopicEnable=true
    autoCreateSubscriptionGroup=true
    
  proxy-config.json: |
    {
      "rocketMQClusterName": "DefaultCluster"
    }
    
  runbroker.sh: |
    error_exit ()
    {
    echo "ERROR: $1 !!"
    exit 1
    }
      
      find_java_home()
      {
        case "`uname`" in
        Darwin)
        JAVA_HOME=$(/usr/libexec/java_home)
        ;;
        *)
        JAVA_HOME=$(dirname $(dirname $(readlink -f $(which javac))))
        ;;
        esac
      }
      
      find_java_home
      
      [ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
      [ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
      [ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOME variable in your environment, We need java(x64)!"
      
      export JAVA_HOME
      export JAVA="$JAVA_HOME/bin/java"
      export BASE_DIR=$(dirname $0)/..
      export CLASSPATH=.:${BASE_DIR}/conf:${CLASSPATH}
      
      #===========================================================================================
      # JVM Configuration
      #===========================================================================================
      calculate_heap_sizes()
      {
      case "`uname`" in
      Linux)
      system_memory_in_mb=`free -m| sed -n '2p' | awk '{print $2}'`
      system_cpu_cores=`egrep -c 'processor([[:space:]]+):.*' /proc/cpuinfo`
      ;;
      FreeBSD)
      system_memory_in_bytes=`sysctl hw.physmem | awk '{print $2}'`
      system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
      system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
      ;;
      SunOS)
      system_memory_in_mb=`prtconf | awk '/Memory size:/ {print $3}'`
      system_cpu_cores=`psrinfo | wc -l`
      ;;
      Darwin)
      system_memory_in_bytes=`sysctl hw.memsize | awk '{print $2}'`
      system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
      system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
      ;;
      *)
      # assume reasonable defaults for e.g. a modern desktop or
      # cheap server
      system_memory_in_mb="2048"
      system_cpu_cores="2"
      ;;
      esac
      
      # some systems like the raspberry pi don't report cores, use at least 1
      if [ "$system_cpu_cores" -lt "1" ]
      then
      system_cpu_cores="1"
      fi
      
      # set max heap size based on the following
      # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
      # calculate 1/2 ram and cap to 1024MB
      # calculate 1/4 ram and cap to 8192MB
      # pick the max
      half_system_memory_in_mb=`expr $system_memory_in_mb / 2`
      quarter_system_memory_in_mb=`expr $half_system_memory_in_mb / 2`
      if [ "$half_system_memory_in_mb" -gt "1024" ]
      then
      half_system_memory_in_mb="1024"
      fi
      if [ "$quarter_system_memory_in_mb" -gt "8192" ]
      then
      quarter_system_memory_in_mb="8192"
      fi
      if [ "$half_system_memory_in_mb" -gt "$quarter_system_memory_in_mb" ]
      then
      max_heap_size_in_mb="$half_system_memory_in_mb"
      else
      max_heap_size_in_mb="$quarter_system_memory_in_mb"
      fi
      MAX_HEAP_SIZE="${max_heap_size_in_mb}M"
      
      # Young gen: min(max_sensible_per_modern_cpu_core * num_cores, 1/4 * heap size)
      max_sensible_yg_per_core_in_mb="100"
      max_sensible_yg_in_mb=`expr $max_sensible_yg_per_core_in_mb "*" $system_cpu_cores`
      
      desired_yg_in_mb=`expr $max_heap_size_in_mb / 4`
      
      if [ "$desired_yg_in_mb" -gt "$max_sensible_yg_in_mb" ]
      then
      HEAP_NEWSIZE="${max_sensible_yg_in_mb}M"
      else
      HEAP_NEWSIZE="${desired_yg_in_mb}M"
      fi
      }
      
      calculate_heap_sizes
      
      # Dynamically calculate parameters, for reference.
      Xms=$MAX_HEAP_SIZE
      Xmx=$MAX_HEAP_SIZE
      Xmn=$HEAP_NEWSIZE
      MaxDirectMemorySize=$MAX_HEAP_SIZE
      # Set for `JAVA_OPT`.
      JAVA_OPT="${JAVA_OPT} -server ${JAVA_VM_OPTS}"
      JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:SurvivorRatio=8"
      JAVA_OPT="${JAVA_OPT} -verbose:gc -Xloggc:/dev/shm/mq_gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy"
      JAVA_OPT="${JAVA_OPT} -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m"
      JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow"
      JAVA_OPT="${JAVA_OPT} -XX:+AlwaysPreTouch"
      #  JAVA_OPT="${JAVA_OPT} -XX:MaxDirectMemorySize=${MaxDirectMemorySize}"
      JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages -XX:-UseBiasedLocking"
      JAVA_OPT="${JAVA_OPT} -Djava.ext.dirs=${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib"
      #JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n"
      JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT}"
      JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"
      
      numactl --interleave=all pwd > /dev/null 2>&1
      if [ $? -eq 0 ]
      then
      if [ -z "$RMQ_NUMA_NODE" ] ; then
      numactl --interleave=all $JAVA ${JAVA_OPT} $@
      else
      numactl --cpunodebind=$RMQ_NUMA_NODE --membind=$RMQ_NUMA_NODE $JAVA ${JAVA_OPT} $@
      fi
      else
      $JAVA ${JAVA_OPT} $@
      fi

  runserver.sh: |
    #!/bin/bash

    # Licensed to the Apache Software Foundation (ASF) under one or more
    # contributor license agreements.  See the NOTICE file distributed with
    # this work for additional information regarding copyright ownership.
    # The ASF licenses this file to You under the Apache License, Version 2.0
    # (the "License"); you may not use this file except in compliance with
    # the License.  You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.

    #===========================================================================================
    # Java Environment Setting
    #===========================================================================================
    error_exit ()
    {
        echo "ERROR: $1 !!"
        exit 1
    }

    find_java_home()
    {
        case "`uname`" in
            Darwin)
                JAVA_HOME=$(/usr/libexec/java_home)
            ;;
            *)
                JAVA_HOME=$(dirname $(dirname $(readlink -f $(which javac))))
            ;;
        esac
    }

    find_java_home

    [ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
    [ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
    [ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOME variable in your environment, We need java(x64)!"

    export JAVA_HOME
    export JAVA="$JAVA_HOME/bin/java"
    export BASE_DIR=$(dirname $0)/..
    export CLASSPATH=.:${BASE_DIR}/conf:${CLASSPATH}

    #===========================================================================================
    # JVM Configuration
    #===========================================================================================
    calculate_heap_sizes()
    {
        case "`uname`" in
            Linux)
                system_memory_in_mb=`free -m| sed -n '2p' | awk '{print $2}'`
                system_cpu_cores=`egrep -c 'processor([[:space:]]+):.*' /proc/cpuinfo`
            ;;
            FreeBSD)
                system_memory_in_bytes=`sysctl hw.physmem | awk '{print $2}'`
                system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
                system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
            ;;
            SunOS)
                system_memory_in_mb=`prtconf | awk '/Memory size:/ {print $3}'`
                system_cpu_cores=`psrinfo | wc -l`
            ;;
            Darwin)
                system_memory_in_bytes=`sysctl hw.memsize | awk '{print $2}'`
                system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
                system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
            ;;
            *)
                # assume reasonable defaults for e.g. a modern desktop or
                # cheap server
                system_memory_in_mb="2048"
                system_cpu_cores="2"
            ;;
        esac

        # some systems like the raspberry pi don't report cores, use at least 1
        if [ "$system_cpu_cores" -lt "1" ]
        then
            system_cpu_cores="1"
        fi

        # set max heap size based on the following
        # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
        # calculate 1/2 ram and cap to 1024MB
        # calculate 1/4 ram and cap to 8192MB
        # pick the max
        half_system_memory_in_mb=`expr $system_memory_in_mb / 2`
        quarter_system_memory_in_mb=`expr $half_system_memory_in_mb / 2`
        if [ "$half_system_memory_in_mb" -gt "1024" ]
        then
            half_system_memory_in_mb="1024"
        fi
        if [ "$quarter_system_memory_in_mb" -gt "8192" ]
        then
            quarter_system_memory_in_mb="8192"
        fi
        if [ "$half_system_memory_in_mb" -gt "$quarter_system_memory_in_mb" ]
        then
            max_heap_size_in_mb="$half_system_memory_in_mb"
        else
            max_heap_size_in_mb="$quarter_system_memory_in_mb"
        fi
        MAX_HEAP_SIZE="${max_heap_size_in_mb}M"

        # Young gen: min(max_sensible_per_modern_cpu_core * num_cores, 1/4 * heap size)
        max_sensible_yg_per_core_in_mb="100"
        max_sensible_yg_in_mb=`expr $max_sensible_yg_per_core_in_mb "*" $system_cpu_cores`

        desired_yg_in_mb=`expr $max_heap_size_in_mb / 4`

        if [ "$desired_yg_in_mb" -gt "$max_sensible_yg_in_mb" ]
        then
            HEAP_NEWSIZE="${max_sensible_yg_in_mb}M"
        else
            HEAP_NEWSIZE="${desired_yg_in_mb}M"
        fi
    }

    calculate_heap_sizes

    # Dynamically calculate parameters, for reference.
    Xms=$MAX_HEAP_SIZE
    Xmx=$MAX_HEAP_SIZE
    Xmn=$HEAP_NEWSIZE
    # Set for `JAVA_OPT`.
    JAVA_OPT="${JAVA_OPT} -server ${JAVA_VM_OPTS}"
    JAVA_OPT="${JAVA_OPT} -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:SurvivorRatio=8  -XX:-UseParNewGC"
    JAVA_OPT="${JAVA_OPT} -verbose:gc -Xloggc:/dev/shm/rmq_srv_gc.log -XX:+PrintGCDetails"
    JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow"
    JAVA_OPT="${JAVA_OPT}  -XX:-UseLargePages"
    JAVA_OPT="${JAVA_OPT} -Djava.ext.dirs=${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib"
    #JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n"
    JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT}"
    JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"

    $JAVA ${JAVA_OPT} $@

部署核心组件

  • nameserver : 2 个 Pod 节点(StatefulSets) + 1 个 Service (负载均衡)
  • broker : 2个 Pod 节点(StatefulSets) + 1 个 Service (负载均衡)
  • proxy : 2个 Pod 节点(StatefulSets) + 1 个 Service (负载均衡)

根据实际情况, 可分别对 nameserver broker proxy 的 Pod 进行动态的扩容

部署 nameserver

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq-nameserver-1
  namespace: rocketmq
spec:
  selector:
    matchLabels:
      app: rocketmq-nameserver
  replicas: 1
  template:
    metadata:
      labels:
        app: rocketmq-nameserver
    spec:
      containers:
        - name: rocketmq-nameserver
          image: registry.cn-hongkong.aliyuncs.com/jansora/rocketmq:5.1.4
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_VM_OPTS
              value: "-Xms512M -Xmx512M -Xmn256M "
          resources:
            requests:
              memory: "200Mi"
              cpu: "500m"
            limits:
              memory: "512Mi"
              cpu: "1000m"
          ports:
            - containerPort: 9876
          command: ["sh", "mqnamesrv"]
          volumeMounts:
            - name: nfs-pvc
              subPath: 'rocketmq/rocketmq-nameserver-1'
              mountPath: "/home/rocketmq/logs"
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/bin/runserver.sh # 具体路径请参考官方文档
              subPath: runserver.sh
      volumes:
        - name: config-volume
          configMap:
            name: rocketmq-config
        - name: nfs-pvc
          persistentVolumeClaim:
            claimName: nfs-pvc
  serviceName: rocketmq-nameserver-service


---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq-nameserver-2
  namespace: rocketmq
spec:
  selector:
    matchLabels:
      app: rocketmq-nameserver
  replicas: 1
  template:
    metadata:
      labels:
        app: rocketmq-nameserver
    spec:
      containers:
        - name: rocketmq-nameserver
          image: registry.cn-hongkong.aliyuncs.com/jansora/rocketmq:5.1.4
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_VM_OPTS
              value: "-Xms512M -Xmx512M -Xmn256M "
          ports:
            - containerPort: 9876
          volumeMounts:
            - name: nfs-pvc
              subPath: 'rocketmq/rocketmq-nameserver-2'
              mountPath: "/home/rocketmq/logs"
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/bin/runserver.sh # 具体路径请参考官方文档
              subPath: runserver.sh
          command: ["sh", "mqnamesrv"]
      volumes:
        - name: config-volume
          configMap:
            name: rocketmq-config
        - name: nfs-pvc
          persistentVolumeClaim:
            claimName: nfs-pvc
  serviceName: rocketmq-nameserver-service

---

# 创建 Service 由 kubernetes 来对多个 nameserver 做负载均衡
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmq-nameserver
  name: rocketmq-nameserver-service
  namespace: rocketmq
spec:
  ports:
    - port: 9876
      protocol: TCP
  selector:
    app: rocketmq-nameserver

部署 broker

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq-broker-1
  namespace: rocketmq
spec:
  selector:
    matchLabels:
      app: rocketmq-broker
  replicas: 1
  template:
    metadata:
      labels:
        app: rocketmq-broker
    spec:
      containers:
        - name: rocketmq-broker
          image: registry.cn-hongkong.aliyuncs.com/jansora/rocketmq:5.1.4
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_VM_OPTS
              value: "-Xms512M -Xmx512M -Xmn512M "
#          resources:
#            requests:
#              memory: "512Mi"
#              cpu: "500m"
#            limits:
#              memory: "512Mi"
#              cpu: "1000m"
          ports:
            - containerPort: 10911
            - containerPort: 10909
          volumeMounts:
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/conf/broker.conf # 具体路径请参考官方文档
              subPath: broker.conf
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/bin/runbroker.sh # 具体路径请参考官方文档
              subPath: runbroker.sh
            - name: nfs-pvc
              subPath: 'rocketmq/rocketmq-broker-1/log'
              mountPath: '/home/rocketmq/logs'
            - name: nfs-pvc
              subPath: 'rocketmq/rocketmq-broker-1/store'
              mountPath: "/home/rocketmq/store"
          command: ["sh", "mqbroker", "-n", "rocketmq-nameserver-service.rocketmq.svc.cluster.local:9876", "-c", "/home/rocketmq/rocketmq-5.1.4/conf/broker.conf"]
      volumes:
        - name: config-volume
          configMap:
            name: rocketmq-config
        - name: nfs-pvc
          persistentVolumeClaim:
            claimName: nfs-pvc
  serviceName: rocketmq-broker-service


---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq-broker-2
  namespace: rocketmq
spec:
  selector:
    matchLabels:
      app: rocketmq-broker
  replicas: 1
  template:
    metadata:
      labels:
        app: rocketmq-broker
    spec:
      containers:
        - name: rocketmq-broker
          image: registry.cn-hongkong.aliyuncs.com/jansora/rocketmq:5.1.4
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_VM_OPTS
              value: "-Xms512M -Xmx512M -Xmn512M "
          ports:
            - containerPort: 10911
            - containerPort: 10909
          volumeMounts:
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/conf/broker.conf # 具体路径请参考官方文档
              subPath: broker.conf
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/bin/runbroker.sh # 具体路径请参考官方文档
              subPath: runbroker.sh

            - name: nfs-pvc
              subPath: 'rocketmq/rocketmq-broker-2/logs'
              mountPath: "/home/rocketmq/logs"
            - name: nfs-pvc
              subPath: 'rocketmq/rocketmq-broker-2/store'
              mountPath: "/home/rocketmq/store"
          command: ["sh", "mqbroker", "-n", "rocketmq-nameserver-service.rocketmq.svc.cluster.local:9876", "-c", "/home/rocketmq/rocketmq-5.1.4/conf/broker.conf"]
      volumes:
        - name: config-volume
          configMap:
            name: rocketmq-config
        - name: nfs-pvc
          persistentVolumeClaim:
            claimName: nfs-pvc
  serviceName: rocketmq-broker-service


---

# 创建 Service 由 kubernetes 来对多个 broker 做负载均衡
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmq-broker
  name: rocketmq-broker-service
  namespace: rocketmq
spec:
  ports:
    - port: 9876
      protocol: TCP
  selector:
    app: rocketmq-broker

部署 proxy

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq-proxy-1
  namespace: rocketmq
spec:
  selector:
    matchLabels:
      app: rocketmq-proxy
  replicas: 1
  template:
    metadata:
      labels:
        app: rocketmq-proxy
    spec:
      containers:
        - name: rocketmq-proxy
          image: registry.cn-hongkong.aliyuncs.com/jansora/rocketmq:5.1.4
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_VM_OPTS
              value: "-Xms512M -Xmx512M -Xmn256M "
          ports:
            - containerPort: 10911
            - containerPort: 10909
          volumeMounts:
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/conf/rmq-proxy.json # 具体路径请参考官方文档
              subPath: proxy-config.json
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/bin/runserver.sh # 具体路径请参考官方文档
              subPath: runserver.sh
            - name: nfs-pvc
              subPath: 'rocketmq/rocketmq-proxy-1'
              mountPath: '/home/rocketmq/logs'
          command: ["sh", "mqproxy", "-n", "rocketmq-nameserver-service.rocketmq.svc.cluster.local:9876", "-pc", "/home/rocketmq/rocketmq-5.1.4/conf/rmq-proxy.json"]
      volumes:
        - name: config-volume
          configMap:
            name: rocketmq-config
        - name: nfs-pvc
          persistentVolumeClaim:
            claimName: nfs-pvc
  serviceName: rocketmq-proxy-service


---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq-proxy-2
  namespace: rocketmq
spec:
  selector:
    matchLabels:
      app: rocketmq-proxy
  replicas: 1
  template:
    metadata:
      labels:
        app: rocketmq-proxy
    spec:
      containers:
        - name: rocketmq-proxy
          image: registry.cn-hongkong.aliyuncs.com/jansora/rocketmq:5.1.4
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_VM_OPTS
              value: "-Xms512M -Xmx512M -Xmn256M "
          ports:
            - containerPort: 10911
            - containerPort: 10909
          volumeMounts:
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/conf/rmq-proxy.json # 具体路径请参考官方文档
              subPath: proxy-config.json
            - name: config-volume
              mountPath: /home/rocketmq/rocketmq-5.1.4/bin/runserver.sh # 具体路径请参考官方文档
              subPath: runserver.sh
            - name: nfs-pvc
              subPath: 'rocketmq/rocketmq-proxy-2'
              mountPath: '/home/rocketmq/logs'
          command: ["sh", "mqproxy", "-n", "rocketmq-nameserver-service.rocketmq.svc.cluster.local:9876", "-pc", "/home/rocketmq/rocketmq-5.1.4/conf/rmq-proxy.json"]
      volumes:
        - name: config-volume
          configMap:
            name: rocketmq-config
        - name: nfs-pvc
          persistentVolumeClaim:
            claimName: nfs-pvc
  serviceName: rocketmq-proxy-service



---

# 创建 Service 由 kubernetes 来对多个 broker 做负载均衡
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmq-proxy
  name: rocketmq-proxy-service
  namespace: rocketmq
spec:
  ports:
    - port: 9876
      protocol: TCP
  selector:
    app: rocketmq-proxy

部署 Console

1 个 pod 和 1 个 service (供外部访问)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rocketmq-dashboard
  namespace: rocketmq
spec:
  selector:
    matchLabels:
      app:  rocketmq-dashboard
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 20
      maxUnavailable: 20
  replicas: 1
  template:
    metadata:
      labels:
        app: rocketmq-dashboard
    spec:
      containers:
        - name: rocketmq-dashboard
          image: registry.cn-hongkong.aliyuncs.com/jansora/rocketmq-dashboard:1.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: JAVA_OPTS
              value: ' -Drocketmq.namesrv.addr=rocketmq-nameserver-service.rocketmq.svc.cluster.local:9876 -Dserver.port=8080  -Xms512m -Xmx512m -Xmn256m '

---

apiVersion: v1
kind: Service
metadata:
  name: rocketmq-dashboard-service
  namespace: rocketmq
  labels:
    app: rocketmq-dashboard-service
spec:
  type: ClusterIP
  selector:
    app: rocketmq-dashboard
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      name: http

部署 console 的 ingress

可选项: 部署 basic 认证 以及 tls 证书

创建 basic 认证

用户名:密码 (jansora

)

export PASSWORD=$(openssl passwd -apr1 pwd) && kubectl create secret generic default-basic-auth --from-literal=auth="jansora:$PASSWORD"

部署 tls 证书

kubectl create secret tls wildcard.jansora.com \
  --key /etc/openresty/certs/lets-encrypt-jansora.com/jansora.com.key \
  --cert /etc/openresty/certs/lets-encrypt-jansora.com/jansora.com.crt \
  --namespace rocketmq \
  --dry-run=client -o yaml | kubectl apply -f -
'

部署 ingress


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rocketmq-ingress
  namespace: rocketmq
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true" # 80 -> 443 http 强制跳转 https
    nginx.ingress.kubernetes.io/auth-type: basic  # 添加 basic 认证
    nginx.ingress.kubernetes.io/auth-secret: default-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
    nginx.ingress.kubernetes.io/hsts: "false"
spec:
  ingressClassName: nginx
  tls:
    - secretName: wildcard.jansora.com
  rules:
    - host: rocketmq.kubernetes.jansora.com
      http:
        paths:
          - backend:
              service:
                name: rocketmq-dashboard-service
                port:
                  number: 8080
            pathType: Prefix
            path: /

验证

https://rocketmq.kubernetes.jansora.com

评论栏