基于K3S构建高可用Rancher集群-运维开发故事

亚细亚的孤儿

如果你是条船，漂泊就是你的命运，可别靠岸

北岛

K3S简述：

K3s (轻量级 Kubernetes)：和 RKE 类似，也是经过认证的 Kubernetes 发行版。它比 RKE 更新，更易用且更轻量化，全部组件都在一个小于 100 MB 的二进制文件中。从 Rancher v2.4 开始，Rancher 可以安装在 K3s 集群上。

详情见：https://rancher2.docs.rancher.cn/docs/installation/_index

Rancher简述：

Rancher 是为使用容器的公司打造的容器管理平台。Rancher 简化了使用 Kubernetes 的流程，开发者可以随处运行 Kubernetes（Run Kubernetes Everywhere），满足 IT 需求规范，赋能 DevOps 团队。

详情见：https://rancher2.docs.rancher.cn/docs/overview/_index

使用环境：

操作系统	主机名	IP地址	节点	配置
CentOS 7 1810	nginx-master	192.168.111.21	Nginx主服务器	2C4G
CentOS 7 1810	nginx-backup	192.168.111.22	Nginx备服务器	2C4G
ubuntu-18.04.3-live-server	k3s-node1	192.168.111.50	k3s节点1	4C8G
ubuntu-18.04.3-live-server	k3s-node2	192.168.111.51	k3s节点2	4C8G
CentOS 7 1810	k3s-mysql	192.168.111.52	mysql	4C8G

部署前系统环境准备：

关闭防火墙和SeLinux

为防止因端口问题造成集群组建失败，我们在这里提前关闭防火墙以及selinux

centos ：

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

Ubuntu：
```
sudo ufw disable
```

节点及Docker功能调优

https://rancher2.docs.rancher.cn/docs/best-practices/optimize/os/_index

配置host文件：

192.168.111.21 nginx-master
192.168.111.22 nginx-backup
192.168.111.50 k3s-node1
192.168.111.51 k3s-node2
192.168.111.52 k3s-mysql

配置host文件，并确保每台机器上都可以通过主机名互通

需要用到的工具：

此安装需要以下 CLI 工具。请确保这些工具已经安装并在$PATH中可用

CLI工具的安装在k3s节点上进行

kubectl - Kubernetes 命令行工具.
helm - Kubernetes 的软件包管理工具。

请参阅Helm 版本要求选择 Helm 的版本来安装 Rancher。

开始部署：

安装 Kubectl：

安装参考K8S官网，由于某些特殊原因，此处我们使用snap

sudo apt-get install snapd
sudo snap install kubectl --classic # 此处安装较慢，请耐心等待
# 验证安装
kubectl help

安装 Helm：

安装参考Helm官网，Helm是Kubernetes的包管理器，Helm的版本需要高于v3

# 下载安装包
wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
# 解压
tar zxvf helm-v3.2.1-linux-amd64.tar.gz
# 将二进制文件移动至/usr/local/bin/
sudo mv linux-amd64/helm /usr/local/bin/helm
# 验证安装
helm help

创建 Nginx+Keepalived 集群：

此处在CentOS节点上进行

安装 Nginx

# 下载Nginx安装包
wget http://nginx.org/download/nginx-1.17.10.tar.gz
# 解压安装包
tar zxvf nginx-1.17.10.tar.gz
# 安装编译时必备的软件包
yum install -y gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel libnl3-devel
# 进入nginx目录，此处我们需要使用https，所有在编译时选择 --with-http_ssl_module 模块
cd nginx-1.17.10
mkdir -p /usr/local/nginx
./configure --prefix=/usr/local/nginx --with-stream
# 安装nginx
make && make install
# 创建nginx命令软连接
ln -s /usr/local/nginx/sbin/nginx /usr/local/bin/nginx
# 验证安装
nginx -V
# 启动nginx
nginx

安装 Keepalived

# 下载安装包
wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz
# 解压安装包
tar zxvf keepalived-2.0.20.tar.gz
# 编译安装keepalived
cd keepalived-2.0.20
mkdir /usr/local/keepalived
./configure --prefix=/usr/local/keepalived/
make && make install
# 配置 keepalived 为系统服务
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
touch /etc/init.d/keepalived
chmod +x /etc/init.d/keepalived # keepalived 中的内容见下文
vim /etc/init.d/keepalived
# 配置 keepalived
mkdir /etc/keepalived/
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
vim /etc/keepalived/keepalived.conf #keepalived.conf 中的内容见下文
# 启动keepalived
systemctl start keepalived
systemctl enable keepalived
# 验证
systemctl status keepalived
# 此时keepalived应该是运行，一个为master，一个为backup， master上执行 ip addr 命令时，应该存在一个虚拟ip地址，backup上不应该有
# 访问 https://192.168.111.20 验证配置

# /etc/init.d/keepalived文件内容
#!/bin/sh
#
# Startup script for the Keepalived daemon
#
# processname: keepalived
# pidfile: /var/run/keepalived.pid
# config: /etc/keepalived/keepalived.conf
# chkconfig: - 21 79
# description: Start and stop Keepalived

# Source function library
. /etc/rc.d/init.d/functions

# Source configuration file (we set KEEPALIVED_OPTIONS there)
. /etc/sysconfig/keepalived

RETVAL=0

prog="keepalived"

start() {
    echo -n $"Starting $prog: "
    daemon keepalived ${KEEPALIVED_OPTIONS}
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
}

stop() {
    echo -n $"Stopping $prog: "
    killproc keepalived
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog
}

reload() {
    echo -n $"Reloading $prog: "
    killproc keepalived -1
    RETVAL=$?
    echo
}

# See how we were called.
case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    reload)
        reload
        ;;
    restart)
        stop
        start
        ;;
    condrestart)
        if [ -f /var/lock/subsys/$prog ]; then
            stop
            start
        fi
        ;;
    status)
        status keepalived
        RETVAL=$?
        ;;
    *)
        echo "Usage: $0 {start|stop|reload|restart|condrestart|status}"
        RETVAL=1
esac

exit $RETVAL

# /etc/keepalived/keepalived.conf 中的内容
! Configuration File for keepalived

global_defs {
   router_id 192.168.111.21 # 此id在网络中有且只有一个，不应有重复的id
}

vrrp_script chk_nginx {     #因为要检测nginx服务状态，所以创建一个检查脚本
    script "/usr/local/keepalived/check_ng.sh"
    interval 3
}

vrrp_instance VI_1 {
    state MASTER    # 配置此节点为master，备机上设置为BACKUP
    interface ens33    # 设置绑定的网卡
    virtual_router_id 51    # vrrp 组， 主备的vrrp组应该一样
    priority 120    # 优先级，优先级大的为主
    advert_int 1    # 检查间隔
    authentication { # 认证
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress { # 虚拟IP
        192.168.111.20
    }
    track_script {    # 执行脚本
        chk_nginx
    }
}

# /usr/local/keepalived/check_ng.sh 中的内容
#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
n=`ps -C nginx --no-heading|wc -l`
if [ $n -eq "0" ]; then
        nginx
        n2=`ps -C nginx --no-heading|wc -l`
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                systemctl stop keepalived
        fi
fi

安装 docker-ce ：

此处在RKE节点上进行

# 移除旧版本Docker
sudo apt-get remove docker docker-engine docker.io containerd runc
# 安装工具包
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
# 添加 Docker官方 GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# 添加 stable apt 源
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
# 安装 Docker-ce
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
# 验证安装
docker info
# 将当前用户加入"docker"用户组，加入到该用户组的账号在随后安装过程会用到。用于节点访问的SSH用户必须是节点上docker组的成员
sudo usermod -aG docker $USER

配置四层负载均衡

此处在Nginx集群操作

# 更新nginx配置文件
# vim /usr/local/nginx/conf/nginx.conf

#user  nobody;
worker_processes  4;
worker_rlimit_nofile 40000;

events {
    worker_connections  8192;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server 192.168.111.50:80 max_fails=3 fail_timeout=5s;
        server 192.168.111.51:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     80;
        proxy_pass rancher_servers_http;
    }

    upstream rancher_servers_https {
        least_conn;
        server 192.168.111.50:443 max_fails=3 fail_timeout=5s;
        server 192.168.111.51:443 max_fails=3 fail_timeout=5s;
    }

    server {
        listen     443;
        proxy_pass rancher_servers_https;
    }
}

部署 MySQL 5.7

# 下载地址：https://dev.mysql.com/downloads/mysql/5.7.html#downloads
# 创建运行MySQL数据库的用户和用户组
groupadd -r mysql
useradd -r -g mysql mysql
# 解压安装包，更改目录权限，创建数据库目录
tar zxvf mysql-5.7.30-linux-glibc2.12-x86_64.tar.gz
mkdir -p /app/mysql/data
mv mysql-5.7.30-linux-glibc2.12-x86_64/* /app/mysql/
chown -R mysql:mysql /app/mysql
# 初始化数据库
cd /app/mysql
./bin/mysqld --initialize \
--user=mysql --basedir=/app/mysql/ \
--datadir=/app/mysql/data/
# !!注意最后一行的初始密码
7Jlhi:gg?rE0
# 创建RSA private key
./bin/mysql_ssl_rsa_setup --datadir=/app/mysql/data/
# 添加 MySQL 到开机启动，修改/etc/init.d/mysqld中的basedir和datadir
cp support-files/mysql.server /etc/init.d/mysqld

basedir=/app/mysql
datadir=/app/mysql/data

chkconfig mysqld on
# 修改环境变量
vim /etc/profile
# 添加
export PATH=/app/mysql/bin:$PATH
# 使环境变量生效
source /etc/profile

# 备份系统自带的/etc/my.cnf，在/app/mysql/目录新建my.cnf,并且将文件属性调整为mysql:mysql
mv /etc/my.cnf /etc/my.cnf.bak
touch /app/mysql/my.cnf    # 具体内容见下文

# 启动mysql
/etc/init.d/mysqld start
# 创建mysql.sock软链接
ln -s /app/mysql/mysql.sock /tmp/mysql.sock
# 使用初始密码登陆
mysql -uroot -p

# 登陆成功后修改密码
alter user 'root'@'localhost' identified by "12345678";
flush privileges;

# 配置数据库远程登录
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '12345678' WITH GRANT OPTION;
flush privileges;
# 验证 略

# my.cnf
[mysqld]
character-set-server=utf8
datadir=/app/mysql/data
socket=/app/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[client]
character-set-server=utf8
socket=/app/mysql/mysql.sock

[mysql]
character-set-server=utf8
socket=/app/mysql/mysql.sock

部署k3s：

# 启动 k3s Server
# ！注意，所有k3s节点上都要运行此命令
curl -sfL https://docs.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - server \
--datastore-endpoint="mysql://root:12345678@tcp(192.168.111.52:3306)/k3s"
# 验证
sudo k3s kubectl get nodes

# 在每个 Rancher Server 节点上安装 K3s 时，会在节点上/etc/rancher/k3s/k3s.yaml位置创建一个kubeconfig文件。该文件包含用于完全访问集群的凭据。# 复制 k3s.yaml 到 ~/.kube/config
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
# 验证 kubectl
sudo kubectl get pods --all-namespaces

kube-system   coredns-8655855d6-c26h8                  1/1     Running     0          11m
kube-system   metrics-server-7566d596c8-v65fd          1/1     Running     0          11m
kube-system   helm-install-traefik-ttrfg               0/1     Completed   0          11m
kube-system   svclb-traefik-hxmzw                      2/2     Running     0          8m16s
kube-system   svclb-traefik-zxmg2                      2/2     Running     0          8m16s
kube-system   traefik-758cd5fc85-xsxbm                 1/1     Running     0          8m16s
kube-system   local-path-provisioner-6d59f47c7-497rl   1/1     Running     0          11m

部署 Rancher：

添加 Helm Chart 仓库

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

为 Rancher 创建 Namespace

sudo kubectl create namespace cattle-system

生成证书

mkdir certs
cd certs
touch ~/.rnd
cp /usr/lib/ssl/openssl.cnf ./ # openssl.cnf内容有改动，详情见下文
vim openssl.cnf

openssl genrsa -out cakey.pem 2048

openssl req -x509 -new -nodes -key cakey.pem \
-days 36500 \
-out cacerts.pem \
-extensions v3_ca \
-subj "/CN=rancher.local.com" \
-config ./openssl.cnf

openssl genrsa -out server.key 2048

openssl req -new -key server.key \
-out server.csr \
-subj "/CN=rancher.local.com" \
-config ./openssl.cnf

openssl x509 -req -in server.csr \
-CA cacerts.pem \
-CAkey cakey.pem \
-CAcreateserial -out server.crt \
-days  36500 -extensions v3_req \
-extfile ./openssl.cnf

openssl x509 -noout -in server.crt -text | grep DNS

cp server.crt tls.crt
cp server.key tls.key

openssl修改部分

[req]
distinguished_name      = req_distinguished_name
req_extetions           = v3_req
x509_extensions         = v3_ca

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = rancher.local.com

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer:always
basicConstraints = critical,CA:true
subjectAltName = @alt_names

ca证书密文

sudo kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=./tls.crt --key=./tls.key
sudo kubectl -n cattle-system create secret generic tls-ca \
--from-file=cacerts.pem

部署 Rancher 集群

 sudo helm install rancher rancher-stable/rancher \
 --namespace cattle-system \
 --set hostname=rancher.local.com \
 --set ingress.tls.source=secret \
 --set privateCA=true

等待 Rancher 集群运行

sudo kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

如果看到以下错误：error: deployment "rancher" exceeded its progress deadline, 可以通过运行以下命令来检查 deployment 的状态
```
sudo kubectl -n cattle-system get deploy rancher
```
搭建完成，在你的hosts文件中，把域名解析到负载均衡器，访问 https://rancher.local.com