目錄
Part.01 Kubernets與docker
Part.02 Docker版本
Part.03 Kubernetes原理
Part.04 資源規(guī)劃
Part.05 基礎(chǔ)環(huán)境準(zhǔn)備
Part.06 Docker安裝
Part.07 Harbor搭建
Part.08 K8s環(huán)境安裝
Part.09 K8s集群構(gòu)建
Part.10 容器回退

第五章 基礎(chǔ)環(huán)境準(zhǔn)備

5.1.SSH免密登錄

在master01、master02、master03上生成公鑰，配置免密登錄到其他節(jié)點(diǎn)

ssh-keygen -t rsa -f ~/.ssh/id_rsa -C username_root
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.1
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.2
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.3
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.11
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.12
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.20

5.2.ansbile配置

在外網(wǎng)服務(wù)器上，下載ansible及相關(guān)依賴包

yum install -y epel-release
yumdownloader --resolve --destdir /opt/ansible/ ansible

上傳至master01上，并進(jìn)行安裝

rpm -ivh /opt/ansible/*

安裝完成后查詢版本

[root@localhost ~]# ansible --version
ansible 2.9.27config file = /etc/ansible/ansible.cfgconfigured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']ansible python module location = /usr/lib/python2.7/site-packages/ansibleexecutable location = /usr/bin/ansiblepython version = 2.7.5 (default, Aug  7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

配置ansible和rhel-system-roles，創(chuàng)建配置文件

mkdir /root/ansible
cd /root/ansible
cp /etc/ansible/ansible.cfg /root/ansible/

修改配置文件，/root/ansible/ansible.cfg

[defaults]
inventory      = /root/ansible/inventory
ask_pass      = false
remote_user = root

配置inventory文件，/root/ansible/inventory

[k8s:children]
master
worker
harbor
[master]
192.168.111.1 hostname=master01
192.168.111.2 hostname=master02
192.168.111.3 hostname=master03
[worker]
192.168.111.11 hostname=worker01
192.168.111.12 hostname=worker02
[harbor]
192.168.111.20 hostname=harbor01

測(cè)試

[root@master01 ansible]# ansible all -m ping
192.168.111.3 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"},"changed": false,"ping": "pong"
}
192.168.111.12 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"},"changed": false,"ping": "pong"
}
192.168.111.11 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"},"changed": false,"ping": "pong"
}
192.168.111.1 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"},"changed": false,"ping": "pong"
}
192.168.111.2 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"},"changed": false,"ping": "pong"
}
192.168.111.20 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"},"changed": false,"ping": "pong"
}

5.3.修改主機(jī)名

創(chuàng)建playbook，/root/ansible/hostname.yml

---
- name: modify hostnamehosts: alltasks:- name: modify hostname permanentlyraw: "echo {{ hostname | quote }} > /etc/hostname"- name: modify hostname temporarilyshell: hostname {{ hostname | quote }}

執(zhí)行并確認(rèn)

[root@master01 ansible]# ansible-playbook hostname.ymlPLAY [modify hostname] ****************************************************************************************************************************************TASK [Gathering Facts] ****************************************************************************************************************************************
ok: [192.168.111.11]
ok: [192.168.111.12]
ok: [192.168.111.1]
ok: [192.168.111.2]
ok: [192.168.111.3]
ok: [192.168.111.20]TASK [modify hostname permanently] ****************************************************************************************************************************
changed: [192.168.111.2]
changed: [192.168.111.1]
changed: [192.168.111.11]
changed: [192.168.111.3]
changed: [192.168.111.12]
changed: [192.168.111.20]TASK [modify hostname temporarily] ****************************************************************************************************************************
changed: [192.168.111.3]
changed: [192.168.111.11]
changed: [192.168.111.1]
changed: [192.168.111.2]
changed: [192.168.111.12]
changed: [192.168.111.20]PLAY RECAP ****************************************************************************************************************************************************
192.168.111.1              : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.11             : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.12             : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.2              : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.20             : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.3              : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0[root@master01 ansible]# ansible all -m shell -a 'hostname'
192.168.111.3 | CHANGED | rc=0 >>
master03
192.168.111.11 | CHANGED | rc=0 >>
worker01
192.168.111.1 | CHANGED | rc=0 >>
master01
192.168.111.2 | CHANGED | rc=0 >>
master02
192.168.111.12 | CHANGED | rc=0 >>
worker02
192.168.111.20 | CHANGED | rc=0 >>
harbor01

5.4.修改hosts列表

在master01上修改主機(jī)列表，/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.111.1 master01.k8s.local   master01
192.168.111.2 master02.k8s.local   master02
192.168.111.3 master03.k8s.local   master03
192.168.111.11 worker01.k8s.local   worker01
192.168.111.12 worker02.k8s.local   worker02
192.168.111.20 harbor01.k8s.local   harbor01

分發(fā)至其他節(jié)點(diǎn)

ansible all -m template -a 'src=/etc/hosts dest=/etc/hosts'

5.5.關(guān)閉firewall和SELinux

關(guān)閉firewall

ansible all -m service -a 'name=firewalld state=stopped enabled=no'

確認(rèn)狀態(tài)

[root@master01 ansible]# ansible all -m shell -a 'systemctl status firewalld | grep Active'
192.168.111.11 | CHANGED | rc=0 >>Active: inactive (dead)
192.168.111.12 | CHANGED | rc=0 >>Active: inactive (dead)
192.168.111.1 | CHANGED | rc=0 >>Active: inactive (dead)
192.168.111.3 | CHANGED | rc=0 >>Active: inactive (dead)
192.168.111.2 | CHANGED | rc=0 >>Active: inactive (dead)
192.168.111.20 | CHANGED | rc=0 >>Active: inactive (dead)

關(guān)閉SELinux

ansible all -m selinux -a 'policy=targeted state=disabled'

確認(rèn)狀態(tài)

[root@localhost ansible]# ansible all -m shell -a 'getenforce'
192.168.111.1 | CHANGED | rc=0 >>
Permissive
192.168.111.11 | CHANGED | rc=0 >>
Permissive
192.168.111.3 | CHANGED | rc=0 >>
Permissive
192.168.111.2 | CHANGED | rc=0 >>
Permissive
192.168.111.12 | CHANGED | rc=0 >>
Permissive
192.168.111.20 | CHANGED | rc=0 >>
Permissive

5.6.配置系統(tǒng)Yum源

【master01】配置CentOS鏡像Yum源

mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom/
rm -f /etc/yum.repos.d/*

創(chuàng)建repo文件，/etc/yum.repos.d/local.repo

[centos]
name=centos
baseurl=file:///mnt/cdrom
gpgcheck=0
enabled=1

更新yum源

yum clean all
yum makecache fast

安裝httpd服務(wù)

yum install -y httpd
systemctl enable --now httpd

配置http服務(wù)指向CentOS源

mkdir /var/www/html/centos
umount /mnt/cdrom/
mount /dev/cdrom /var/www/html/centos/

刪除原有repo文件

ansible all -m shell -a 'rm -f /etc/yum.repos.d/*.repo'

配置所有節(jié)點(diǎn)的系統(tǒng)Yum源

ansible all -m yum_repository -a 'name="centos" description="centos" baseurl="http://master01.k8s.local/centos" enabled=yes gpgcheck=no'
ansible all -m shell -a 'yum clean all'
ansible all -m shell -a 'yum makecache fast'
ansible all -m shell -a 'yum update -y'

5.7.安裝基礎(chǔ)軟件

安裝vim等基礎(chǔ)軟件，/root/ansible/packages.yml

---
- hosts: alltasks:- name: install packagesyum:name:- pciutils- bash-completion- vim- chrony- net-toolsstate: present

執(zhí)行并確認(rèn)

[root@master01 ansible]# ansible-playbook packages.ymlPLAY [all] ****************************************************************************************************************************************************TASK [Gathering Facts] ****************************************************************************************************************************************
ok: [192.168.111.3]
ok: [192.168.111.1]
ok: [192.168.111.12]
ok: [192.168.111.11]
ok: [192.168.111.2]
ok: [192.168.111.20]TASK [install packages] ***************************************************************************************************************************************
ok: [192.168.111.2]
ok: [192.168.111.11]
ok: [192.168.111.1]
ok: [192.168.111.12]
ok: [192.168.111.20]
changed: [192.168.111.3]PLAY RECAP ****************************************************************************************************************************************************
192.168.111.1              : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.11             : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.12             : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.2              : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.20             : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
192.168.111.3              : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

5.8.NTP時(shí)鐘

以master01為時(shí)鐘源，其余節(jié)點(diǎn)從master01進(jìn)行時(shí)鐘同步
服務(wù)端（master01）
修改配置文件，/etc/chrony.conf

# 不指定外部NTP源
# 允許本網(wǎng)段其節(jié)點(diǎn)作為客戶端訪問
allow 192.168.111.0/24
# 如果時(shí)間服務(wù)可不用，則使用本地時(shí)間作為標(biāo)準(zhǔn)時(shí)間授權(quán)，層數(shù)為10
local stratum 10

重啟服務(wù)

systemctl restart chronyd

客戶端（mster02/worker01/worker02/harbor01）
在外網(wǎng)服務(wù)器上下載ansible system role的安裝包

yumdownloader --resolve rhel-system-roles

將安裝包上傳至master01的/opt/ansible/下，并進(jìn)行安裝

[root@localhost ~]# rpm -ivh /opt/ansible/python-netaddr-0.7.5-9.el7.noarch.rpm
warning: /opt/ansible/python-netaddr-0.7.5-9.el7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...1:python-netaddr-0.7.5-9.el7       ################################# [100%]
[root@localhost ~]# rpm -ivh /opt/ansible/rhel-system-roles-1.7.3-4.el7_9.noarch.rpm
warning: /opt/ansible/rhel-system-roles-1.7.3-4.el7_9.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...1:rhel-system-roles-1.7.3-4.el7_9  ################################# [100%]

安裝ntp時(shí)鐘，/root/ansible/timesync.yml

---
- hosts: 192.168.111.2,192.168.111.3,worker,harborvars:timesync_ntp_servers:- hostname: 192.168.111.1iburst: yesroles:- rhel-system-roles.timesync

執(zhí)行

ansible-playbook /root/ansible/timesync.yml

確認(rèn)時(shí)鐘同步情況

[root@master01 ansible]# ansible 192.168.111.2,192.168.111.3,worker,harbor -m shell -a 'chronyc sources -v'
192.168.111.12 | CHANGED | rc=0 >>
210 Number of sources = 1.-- Source mode  '^' = server, '=' = peer, '#' = local clock./ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local           10   6   377    46  +5212ns[  +19us] +/-   73us
192.168.111.3 | CHANGED | rc=0 >>
210 Number of sources = 1.-- Source mode  '^' = server, '=' = peer, '#' = local clock./ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local           10   6    17    30   -261ns[  -62us] +/-  966us
192.168.111.11 | CHANGED | rc=0 >>
210 Number of sources = 1.-- Source mode  '^' = server, '=' = peer, '#' = local clock./ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local           10   6   377    35    -17us[  -20us] +/-  130us
192.168.111.20 | CHANGED | rc=0 >>
210 Number of sources = 1.-- Source mode  '^' = server, '=' = peer, '#' = local clock./ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local           10   6   377    25  -4152ns[-7463ns] +/-   96us
192.168.111.2 | CHANGED | rc=0 >>
210 Number of sources = 1.-- Source mode  '^' = server, '=' = peer, '#' = local clock./ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local           10   6   377    27    -52us[  -50us] +/-  191us

5.9.關(guān)閉swap

臨時(shí)關(guān)閉：

ansible all -m shell -a 'swapoff -a'

永久關(guān)閉：

ansible all -m shell -a 'sed -ri "s/.*swap.*/#&/" /etc/fstab'

5.10.啟用ipvs轉(zhuǎn)發(fā)

在kubernetes中service有兩種代理模型，一種是基于iptables的，一種是基于ipvs的；ipvs轉(zhuǎn)發(fā)性能更好。
在master01-03上開啟ipvs轉(zhuǎn)發(fā)

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

賦予執(zhí)行權(quán)限并執(zhí)行

chmod +x /etc/sysconfig/modules/ipvs.modules
/bin/bash /etc/sysconfig/modules/ipvs.modules

5.11.啟用網(wǎng)橋過濾及內(nèi)核轉(zhuǎn)發(fā)

bridge-nf-call-iptables這個(gè)內(nèi)核參數(shù)，表示bridge設(shè)備在二層轉(zhuǎn)發(fā)時(shí)也去調(diào)用iptables配置的三層規(guī)則（包含conntrack），所以開啟這個(gè)參數(shù)就能夠解決Service同節(jié)點(diǎn)通信問題。
在master01上創(chuàng)建/etc/sysctl.d/k8s.conf，添加網(wǎng)橋過濾及內(nèi)核轉(zhuǎn)發(fā)配置

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

分發(fā)至其他節(jié)點(diǎn)

ansible all -m template -a 'src=/etc/sysctl.d/k8s.conf dest=/etc/sysctl.d/'
ansible all -m shell -a 'modprobe br_netfilter'

驗(yàn)證是否生效

[root@master01 ansible]# ansible all -m shell -a 'sysctl --system | grep -A3 k8s'
192.168.111.3 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.1 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.12 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.11 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.2 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.20 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1