HAproxy实现nginx+php负载均衡，后端数据库读写分离

? 202.106.0.6: as client ? 202.106.0.17：as firewall ? 202.106.0.147: as powerDNS ? 192.168.205.27: as NFS server ? 192.168.205.37: as NFS Backup server(inotify+rsync) ? 192.168.205.47: as proxysql1 ? 192.168.205.57: as porxysql2 ? 192.168.205.67: as MHA for manager mysql master/slave ? 192. 168.205.77: as mysql primary ? 192.168.205.87: as mysql secondary ? 192.168.205.97: as secondary 2 ? 192.168.205.107: as HAproxy1 ? 192.168.205.117: as HAproxy2 ? 192.168.205.127: as web1 (nginx +php+wordpress) ? 192.168.205.137: as web2 (nginx +php+wordpress) 注：所有操作系统默认停用firewalld,iptable为空,关闭selinux

版本：

• OS: centos 7 1810 with mini install
• poweradmin-2.1.7.tgz
• mariadb-5.5.60（主要测试主从自动切换和读写分离没有使用高版本）
• nginx-1.16.1.tar.gz
• php-7.3.7.tar.xz
• wordpress-5.0.4-zh_CN.tar.gz
• mha4mysql-manager-0.56-0.el6.noarch.rpm
• mha4mysql-node-0.56-0.el6.noarch.rpm
  注：其它未注明均为yum 安装

目地：

? 实现nginx编译安装和支持fast-cgi, 编译安装php7.3支持最新的wordpress, ? 通过MHA实现主从的自动切换，通过proxysql实现读写的分离，并实瑞proxy的高可用性 ? web静态页面存在nfs共享文件中，通地inotify 和rsync进行实时的备份网站的数据。 ? 前端通过haproxy实现负载均衡

步骤：

1. 安装mariadb并配置主从 2. 配置半同步复制 3. 实现MHA管理 4. 实现proxysql 5. 实现proxysql的keepalive 6. 安装rsync server做为nfs backup服务器 7. 安装nginx和php 8. 安装haproxy 9. 实现firewall DNAT 10. 测试

安装mariadb,实现主从

1. 通过脚本yum真接安装mariadb在77，87，97上，安装完成后自动重启

   [root@master data]#vi maridb_yum.sh #!/bin/bash ID=`ip a show dev eth0 | sed -r ‘3!d;s@(.*inet)(.*)(/.*)@\2@‘ | cut -d. -f4` rpm -q mariadb-server ||yum install -y mariadb-server [ -d /data/mysql ] || mkdir -p /data/mysql [ -d /data/logs ] || mkdir -p /data/logs chown mysql:mysql /data/{mysql,logs} sed -i ‘s@datadir=/var/lib/mysql@datadir=/data/mysql@‘ /etc/my.cnf grep "log-bin" /etc/my.cnf || sed -i ‘/\[mysqld\]/a log-bin=/data/logs/bin‘ /etc/my.cnf rep "innodb_file_per_table" /etc/my.cnf || sed -i ‘/\[mysqld\]/a innodb_file_per_table = on‘ /etc/my.cnf grep "skip_name_resolve" /etc/my.cnf || sed -i ‘/\[mysqld\]/a skip_name_resolve = on‘ /etc/my.cnf grep "server-id" /etc/my.cnf || sed -i "/\[mysqld\]/a server-id=$ID" /etc/my.cnf service mariadb restart

2. 修改slave数据库的配置

   [root@slave1 ~]#vi /etc/my.cnf [mysqld] read_only relay_log_purge=0 [root@slave1 ~]#systemctl restart mariadb [root@slave2 ~]#vi /etc/my.cnf [mysqld] read_only [root@slave2 ~]#systemctl restart mariadb

3. 在主服务器上记录复制位置

   MariaDB [(none)]> show master logs;

4. 在主服务器创建同步帐号

   MariaDB [(none)]> grant replication slave on *.* to repluser‘192.168.205.%‘ identified by ‘centos‘;

5. 分别在从服务器上修改change master to

   CHANGE MASTER TO MASTER_HOST=‘192.168.205.77‘, MASTER_USER=‘repluser‘, MASTER_PASSWORD=‘centos‘, MASTER_PORT=3306, MASTER_LOG_FILE=‘bin.000003‘, #此文件跟据主上的show master logs MASTER_LOG_POS=245; #此位置跟据主上的show master logs

6. 在所有的从节点上启动i/o线和relay线程

   MariaDB [(none)]> satar slave MariaDB [(none)]> show slave status;

   配置半同步复制

7. 记录半同步插件的文件名称

   [root@master ~]#rpm -ql mariadb-server /usr/lib64/mysql/plugin/semisync_master.so /usr/lib64/mysql/plugin/semisync_slave.so

8. 在主服务器和从服务器上安装半同步插件

   MariaDB [(none)]> install plugin rpl_semi_sync_master soname ‘semisync_master.so‘;

9. 查看半同步的状态

   MariaDB [(none)]> show global variables like ‘%semi%‘ ; +------------------------------------+-------+ | Variable_name | Value | +------------------------------------+-------+ | rpl_semi_sync_master_enabled | OFF | #半同步默认off | rpl_semi_sync_master_timeout | 10000 | #超时毫秒，10秒 | rpl_semi_sync_master_trace_level | 32 | | rpl_semi_sync_master_wait_no_slave | ON | +------------------------------------+-------+ 4 rows in set (0.00 sec)

10. 启用半同步

    MariaDB [(none)]> set global rpl_semi_sync_master_enabled=on; Query OK, 0 rows affected (0.00 sec)

11. 查看半同步状态信息

    MariaDB [(none)]> show global status like ‘%semi%‘; +--------------------------------------------+-------+ | Variable_name | Value | +--------------------------------------------+-------+ | Rpl_semi_sync_master_clients | 0 | | Rpl_semi_sync_master_net_avg_wait_time | 0 | | Rpl_semi_sync_master_net_wait_time | 0 | | Rpl_semi_sync_master_net_waits | 0 | | Rpl_semi_sync_master_no_times | 0 | | Rpl_semi_sync_master_no_tx | 0 | | Rpl_semi_sync_master_status | ON | | Rpl_semi_sync_master_timefunc_failures | 0 | | Rpl_semi_sync_master_tx_avg_wait_time | 0 | | Rpl_semi_sync_master_tx_wait_time | 0 | | Rpl_semi_sync_master_tx_waits | 0 | | Rpl_semi_sync_master_wait_pos_backtraverse | 0 | | Rpl_semi_sync_master_wait_sessions | 0 | | Rpl_semi_sync_master_yes_tx | 0 | +--------------------------------------------+-------+ 14 rows in set (0.00 sec)

12. 在所有的从节点安装slave semi插件；

    MariaDB [(none)]> install plugin rpl_semi_sync_slave soname ‘semisync_slave.so‘; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> show plugins; … | rpl_semi_sync_slave | ACTIVE | REPLICATION | semisync_slave.so | GPL | +--------------------------------+----------+--------------------+-------------------+---------+ 43 rows in set (0.00 sec)

13. 查看所有从节点的半同步状态，并启用半同步状态，需要重新启动线程，再查看半同步状态是on才可以

    MariaDB [(none)]> show global variables like ‘%semi%‘; +---------------------------------+-------+ | Variable_name | Value | +---------------------------------+-------+ | rpl_semi_sync_slave_enabled | OFF | | rpl_semi_sync_slave_trace_level | 32 | +---------------------------------+-------+ 2 rows in set (0.00 sec) MariaDB [(none)]> set global rpl_semi_sync_slave_enabled=on; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> show global variables like ‘%semi%‘; +---------------------------------+-------+ | Variable_name | Value | +---------------------------------+-------+ | rpl_semi_sync_slave_enabled | ON | | rpl_semi_sync_slave_trace_level | 32 | +---------------------------------+-------+ 2 rows in set (0.00 sec) MariaDB [(none)]> stop slave; Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show global status like ‘%semi%‘;
+----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| Rpl_semi_sync_slave_status | ON |
+----------------------------+-------+
1 row in set (0.00 sec)

14. 此时查看主节点的状态

MariaDB [(none)]> show global status like ‘%semi%‘;
+--------------------------------------------+-------+
| Variable_name | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients | 2 | #表示已经有两个客户端，说明成功
| Rpl_semi_sync_master_net_avg_wait_time | 0 |
| Rpl_semi_sync_master_net_wait_time | 0 |
| Rpl_semi_sync_master_net_waits | 0 |
| Rpl_semi_sync_master_no_times | 0 |
| Rpl_semi_sync_master_no_tx | 0 |
| Rpl_semi_sync_master_status | ON | #状态是on表半同步打开
| Rpl_semi_sync_master_timefunc_failures | 0 |
| Rpl_semi_sync_master_tx_avg_wait_time | 0 |
| Rpl_semi_sync_master_tx_wait_time | 0 |
| Rpl_semi_sync_master_tx_waits | 0 |
| Rpl_semi_sync_master_wait_pos_backtraverse | 0 |
| Rpl_semi_sync_master_wait_sessions | 0 |
| Rpl_semi_sync_master_yes_tx | 0 |
+--------------------------------------------+-------+
14 rows in set (0.00 sec)

#### 实现MHA管理 15. 安装从mha网站下载的mha rpm软件包，依赖包要去epel下载，所以要启用epel源

[root@MHA ~]#yum install mha4mysql-manager-0.56-0.el6.noarch.rpm mha4mysql-node-0.56-0.el6.noarch.rpm

16. 所有节点上安装node包，无论主不是从，mha都会当做一个node来看

[root@master ~]#yum install mha4mysql-node-0.56-0.el6.noarch.rpm
[root@slave1 data]#yum install mha4mysql-node-0.56-0.el6.noarch.rpm
[root@slave2 data]#yum install mha4mysql-node-0.56-0.el6.noarch.rpm

17. 在主服务器上创建帐号做为mha的监控帐号使用

MariaDB [(none)]> grant all on . to mhauser@‘192.168.205.%‘ identified by ‘centos‘;

18. 由于主从在切换时，MHA要修改配置文件等，所以需要ssh key验证，我们采用速的方法，直接在本地产生key和authorized文件一并复制到所有节点

[root@MHA ~]#ssh-keygen
[root@MHA ~]#ssh-copy-id 192.168.205.67
[root@MHA ~]#cat .ssh/authorized_keys
[root@MHA ~]#scp -r .ssh 192.168.205.77:/root/
[root@MHA ~]#scp -r .ssh 192.168.205.87:/root/
[root@MHA ~]#scp -r .ssh 192.168.205.97:/root/

19. MHA配置文件没有，我们直接按照如下创建

[root@MHA ~]#mkdir /etc/mha
[root@MHA ~]#vim /etc/mha/app1.cnf
[server default]
master_binlog_dir=/data/logs/
user=mhauser #这个帐号为mha监控mysql的帐号
password=centos #帐号的密码
manager_workdir=/data/mastermha/app1/ #配置文件的存放位置
manager_log=/data/mastermha/app1/manager.log #日志的存放位置
remote_workdir=/data/mastermha/app1/
ssh_user=root #SSH key连接的用户名
repl_user=repluser #复制帐号，也就是我们上面创建的复制帐号名
repl_password=centos #复制帐号的密码
ping_interval=1 #检测间隔为每一秒
[server1] #定议节点服务器
hostname=192.168.205.77
candidate_master=1
[server2]
hostname=192.168.205.87
[server3]
hostname=192.168.205.97
candidate_master=1 #表示当主不可用时优先提升为主的从服务器

20. 启动之前进行检查ssh，repl复制是否准备好, 如果successful 我们可以进行下一步。

[root@MHA ~]#masterha_check_ssh --cOnf=/etc/mha/app1.cnf
[root@MHA ~]#masterha_check_repl --cOnf=/etc/mha/app1.cnf

21. 起动进程，此进程前台运行，当主节点失败时切换完成后它会终止，所以要想持续要重启进程

[root@MHA ~]#masterha_manager --cOnf=/etc/mha/app1.cnf
Mon Aug 12 23:33:22 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Aug 12 23:33:22 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Mon Aug 12 23:33:22 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..

#### 实现proxysql 22. 我们直接在两个proxysql服务器上创建yum源来安装proxysql

cat <[proxysql_repo]
name= ProxySQL YUM repository
#baseurl=https://repo.proxysql.com/ProxySQL/proxysql-2.0.x/centos/\$releasever
baseurl=https://repo.proxysql.com/ProxySQL/proxysql-1.4.x/centos/\$releasever
gpgcheck=1
gpgkey=https://repo.proxysql.com/ProxySQL/repo_pub_key
EOF

23. 因为proxysql是一个基于轻量的数据库配置的，所以我们需要一个sql客户端来连接设置proxysql

[root@proxysql1 ~]#yum install proxysql mariadb
[root@proxysql2 ~]#yum install proxysql mariadb

23. 启动服务器查看端口，其中6032为proxysql的数据库端口，6033为proxysql的用户连接端口

[root@proxysql1 ~]#service proxysql start
[root@proxysql2 ~]#service proxysql start
[root@proxysql1 ~]#ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 :6032 :
LISTEN 0 128 :6033 :
LISTEN 0 128 :6033 :
LISTEN 0 128 :6033 :
LISTEN 0 128 :6033 :*

24. proxysql默认的用户名和密码是admin admin， 监听端口为6032,

[root@proxysql1 ~]#mysql -uadmin -padmin -P6032 -h127.0.0.1

25. 在所有的proxysql注册sql server的节点，包括所有的主从节点，使有和insert插入记录到mysql_servers库用

MySQL [(none)]> insert into mysql_servers(hostgroup_id,hostname,port) values(10,‘192.168.205.77‘,3306);
MySQL [(none)]> insert into mysql_servers(hostgroup_id,hostname,port) values(10,‘192.168.205.87‘,3306);
MySQL [(none)]> insert into mysql_servers(hostgroup_id,hostname,port) values(10,‘192.168.205.97‘,3306);
MySQL [(none)]> load mysql servers to runtime;
MySQL [(none)]> save mysql servers to disk;

26. 查看一下我们刚才添加的主机记录是否在库中

MySQL [(none)]> select * from mysql_servers;

27. 由于proxysql是查看主和从的数据的read_only来判读谁是主谁是从的，所以建立一账号用来连接到主和从服务器上，我们要在主节点上建立这个帐号，它会复制到从节点上

MariaDB [(none)]> grant replication client on . to monitor@‘192.168.205.%‘ identified by ‘centos‘;

28. 在所有的proxysql服务器设置监控账号，保存状态

MySQL [(none)]> set mysql-monitor_username=‘monitor‘;
MySQL [(none)]> set mysql-monitor_password=‘centos‘;
MySQL [(none)]> load mysql variables to runtime;
MySQL [(none)]> save mysql variables to disk;

29. 查看一下相关的日志，以前出错的原因是因为默认没使用monitor密码是monitor进行连接(在proxysql.cnf中可以看到)，所以会出错，当你添加完帐号就成功了,

MySQL [(none)]> select from mysql_server_connect_log;
MySQL [(none)]> select from mysql_server_ping_log;

30. 需要修改的是main库中的mysql_replication_hostgroups表，该表有3个字段：writer_hostgroup, reader_hostgroup，comment, 指定写组的id为10，读组的id为20

MySQL [(none)]> insert into mysql_replication_hostgroups values(10,20,"test");
MySQL [(none)]> load mysql servers to runtime;
MySQL [(none)]> save mysql servers to disk;

31. proxySQL会跟据刚才连接帐号判断read_only并自动的把三个服务器按读写组加到这个表中了

MySQL [(none)]> select hostgroup_id,hostname,port,status,weight from mysql_servers;
MySQL [(none)]> select * from mysql_server_read_only_log;

32. 在主服务器上设置一个帐号用来测试

MySQL [(none)]> grant all on . to sqluser@‘192.168.205.%‘ identified by ‘centos‘;

33. 在proxysql服务器上设置这个帐号的缺省组为10

MySQL [(none)]> insert into mysql_users(username,password,default_hostgroup) values(‘sqluser‘,‘centos‘,10);
MySQL [(none)]> load mysql users to runtime;
MySQL [(none)]> save mysql users to disk;

34. 此时proxysql还是不知道那些sql语句算读，那些为写，我们要定义好，让它来区别并发送到不同的组服务器上,其中10为写，20组为读

MySQL [(none)]>insert into mysql_query_rules
(rule_id,active,match_digest,destination_hostgroup,apply) VALUES
(1,1,‘^SELECT.*FOR UPDATE$‘,10,1),(2,1,‘^SELECT‘,20,1);
MySQL [(none)]>load mysql query rules to runtime;
MySQL [(none)]>save mysql query rules to disk;

35. 查看一下你添加的规则

MySQL [(none)]>select rule_id,active,match_digest,destination_hostgroup,apply from mysql_query_rules;

36. 测试连接并实现读，可以看到一会调度到87，一会调度到97

mysql -usqluser -pcentos -P6033 -h127.0.0.1 -e ‘select @@server_id‘

37. 如果使用事务不能发送到读服务器，只会发送到主服务器

mysql -usqluser -pcentos -P6033 -h127.0.0.1 -e ‘begin;select @@server_id;commit‘
mysql -usqluser -pcentos -P6033 -h127.0.0.1 -e ‘create database testdb‘
mysql -usqluser -pcentos testdb -P6033 -h127.0.0.1 -e ‘create table t1(id int)‘

39. 我们可以用下列查询看调度是否成功，并调度到那台服务器上

select hostgroup hg,sum_time,count_star,digest_text from stats_mysql_query_digest order by sum_time desc;

#### 实现proxysql的keepalive 40. 使用yum安装keepalvie，psmisc，psmisc中有个命令为killall可以检测到进程的状态，keepalive会使用这个做为脚本检测进行是否运行

[root@proxysql1 ~]#yum install keepalived ipvsadm psmisc
[root@proxysql2 ~]#yum install keepalived ipvsadm psmisc

41. 修改keepalive配置文件

[root@proxysql1 ~]#vi /etc/keepalived/keepalived.conf
global_defs {
notification_email {root@localhost
br/>root@localhost
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_1
vrrp_mcast_group4 224.0.0.100
}

vrrp_script check_proxy { #定义脚本测试proxysql的进程，如果进程down实现主备切换
script "killall -0 proxysql" #监控进程
interval 2 #每二秒监控一次
weight -30 #进程down时priority减30
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 45
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.205.45/24 dev eth0 label eth0:0
}
track_script {
check_proxy
}
}
[root@proxysql1 ~]#systemctl start keepalived

42. 修改proxysql2的keepalive

[root@proxysql2 ~]#vi /etc/keepalived/keepalived.conf
global_defs {
notification_email {root@localhost
br/>root@localhost
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_2
vrrp_mcast_group4 224.0.0.100
}

vrrp_script check_proxy {
script "killall -0 proxysql"
interval 2
weight -30
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 45
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.205.45/24 dev eth0 label eth0:0
}
track_script {
check_proxy
}
}
[root@proxysql2 ~]#systemctl start keepalived

43. 在proxysql1上查看IP，可以看到vip 192.168.205.45, 停掉服务，看一下IP

[root@proxysql1 ~]#ip a
2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:37:f9:93 brd ff:ff:ff:ff:ff:ff
inet 192.168.205.47/24 brd 192.168.205.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 192.168.205.45/24 scope global secondary eth0:0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe37:f993/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@proxysql1 ~]#systemctl stop proxysql

44. 在proxysql2上查看可以看到vip 192.168.205.45已经飘到了proxysql2上

[root@proxysql2 ~]#ip a
2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:cf:e5:bb brd ff:ff:ff:ff:ff:ff
inet 192.168.205.57/24 brd 192.168.205.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 192.168.205.45/24 scope global secondary eth0:0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fecf:e5bb/64 scope link noprefixroute
valid_lft forever preferred_lft forever

45. 我们测试一下能否通过web server通过连接vip连接到后端的服务器，从而实现读写分离

[root@web1 data]#mysql -uwordpress -pcentos -P6033 -h192.168.205.45

#### 安装rsync server做为nfs backup服务器 46. 安装rsync服务

[root@nfs2 data]#yum install rsync

47. 编辑rsyncd.conf文件，让rsyncc以daemon的方式运行

[root@nfs2 data]#vi /etc/rsyncd.conf
uid = root #以什么身份开启服务
gid = root
use chroot = no
max cOnnections= 0 #不限制连接
ignore errors #忽略错误
exclude = lost+found/
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
reverse lookup = no 反向解析名称与ip
hosts allow = 192.168.205.0/24 充许连接的主机列表
[backup] 起名子
path = /data/www/
comment = webserver www backup
read Only= no 可以写
auth users = rsync
secrets file = /etc/rsync.pass 密码文件

48. 服务器端生成验证文件

[root@nfs2 data]#echo "rsync:centos" > /etc/rsync.pass
[root@nfs2 data]#chmod 600 /etc/rsync.pass

49. 服务器端准备目录

[root@nfs2 data]#mkdir /data/www

50. 服务器端启动rsync服务

[root@nfs2 data]#systemctl start rsyncd

#### 启用NFS服务器 51. 启用epel

[root@nfs1 data]#yum install inotify-tools nfs-utils rsync

52. 创建目录

[root@nfs1 data]#mkdir /data/www

53. 创建用户nginx用户

[root@nfs1 data]# useradd -s /sbin/nologin nginx -u 2000
[root@nfs1 data]# id nginx
uid=2000(nginx) gid=2000(nginx) groups=2000(nginx)

54. 修改NFS配置文件，共享www目录

[root@nfs1 data]# vi /etc/exports
/data/www 192.168.205.0/24(rw,all_squash,anOnuid=2000,anOngid=2000)
[root@nfs1 data]# systemctl restart nfs-server

55. 将NFS服务器配置为rsync的客户端，先配置密码文件

[root@nfs1 data]#echo "centos" > /etc/rsync.pass
[root@nfs1 data]#chmod 600 /etc/rsync.pass

56. 安装inotify实现实时同步数据，客户端创建inotify_rsync.sh脚本

[root@nfs1 data]#cat inotify_rsync.sh #!/bin/bash
SRC=‘/data/www/‘
DEST=‘rsync@192.168.205.37::backup‘
br/>#!/bin/bash
SRC=‘/data/www/‘
DEST=‘rsync@192.168.205.37::backup‘
-e create,delete,moved_to,close_write,attrib ${SRC} | \
while read DATE TIME DIR FILE;do
FILEPATH=${DIR}${FILE}
rsync -az --delete --password-file=/etc/rsync.pass $SRC $DEST && echo "At ${TIME} \
on ${DATE}, file $FILEPATH was backuped up via rsync" >> /var/log/changelist.log
done

57. 将脚本文件存在rc.local中，启动进会自动执行

[root@nfs1 data]#chmod +x inotify_rsync.sh
[root@nfs1 data]#vi /etc/rc.d/rc.local
/data/inotify_rsync.sh &
[root@nfs1 data]#chmod +x /etc/rc.d/rc.local

58. 在web server中实现挂载，安装nfs-utils工具才可以mount NFS文件系统

[root@web1 ~]#yum install nfs-utils
[root@web2 ~]#yum install nfs-utils

59. 测试连接NFS服务器的共享文件

[root@web1 ~]#showmount -e 192.168.205.27
Export list for 192.168.205.27:
/data/www 192.168.205.0/24
[root@web1 ~]#mount 192.168.205.27:/data/www /data/www
[root@web2 ~]#mount 192.168.205.27:/data/www /data/www
[root@web1 ~]#df
[root@web2 ~]#df

60. 在两个web server中将mount写入到fstab文件中，实现开机自动mount

[root@web1 ~]##vi /etc/fstab
192.168.205.27:/data/www /app/httpd24/htdocs nfs defaults 0 0

#### 安装nginx和php 61. 复制nginx和php源码文件到一个目录中在两个web服务器上

nginx-1.16.1.tar.gz
php-7.3.7.tar.xz

62. 执行nginx安装脚本在同一个目录中

[root@web2 ~]#cat nginx_install_bin.sh
#!/bin/bash
#########################
#difination variables
#########################
TMP_DIR=pwd
NGINX="nginx-1.16.1.tar.gz"
NGINX_DIR=echo "$NGINX" |rev | cut -d. -f3- | rev
INS_DIR="/apps/nginx"
##########################
#Packges check and install
##########################
pkg(){
for i in $PKGS; do
rpm -q $i &> /dev/null && echo "Packge rpm -q $i is installed" || yum -y install $i
done
}
#####################
#NGINX INSTALLATION
#1.unarchive binary
####################
#install dependency packges,just add packges name to variable PKGS separate by SPACE.
#Example PKGS="libaio gcc glibc"
nginx_ins(){
PKGS="gcc pcre-devel openssl-devel zlib-devel"
pkg
cd $TMP_DIR
[ -e $TMP_DIR/$NGINX ] || wget -c http://nginx.org/download/$NGINX
tar xf $TMP_DIR/$NGINX
cd $NGINX_DIR
./configure \
--prefix=$INS_DIR \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_dav_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-http_realip_module \
--with-pcre \
--with-threads \
--with-file-aio \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module
if [ $? -eq 0 ]; then
make -j 4 && make install
cd $TMP_DIR
rm -rf $HTTPD_DIR
else
echo "please remake and make install"
fi
id nginx || useradd nginx -s /sbin/nologin -u 2000
chown nginx:nginx -R $INS_DIR
echo "$INS_DIR/sbin/nginx" >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local
ln -s $INS_DIR/sbin/nginx /sbin/nginx
nginx
}

###############
#OPTIONS SELECT
###############
case $1 in
install)
nginx_ins
;;
remove)
nginx_rm
;;
*)
echo "Useage $0 $1 install|remove"
;;
esac
[root@web2 ~]#./nginx_install_bin.sh install

63. 执行如下脚本安装php

[root@web2 ~]#cat apache_php_install.sh #!/bin/bash ######################### #difination variables ######################### TMP_DIR=`pwd` APR="apr-1.7.0.tar.bz2" APR_UTIL="apr-util-1.6.1.tar.bz2" HTTPD="httpd-2.4.39.tar.bz2" PHP="php-7.3.7.tar.xz" HTTPD_DIR=`echo "$HTTPD" |rev | cut -d. -f3- | rev` APR_DIR=`echo "$APR" |rev | cut -d. -f3- | rev` APR_UTIL_DIR=`echo "$APR_UTIL" | rev | cut -d. -f3- | rev` PHP_DIR=`echo "$PHP" | rev | cut -d. -f3- | rev` INS_HTTPD_DIR=/app/httpd24 INS_PHP_DIR=/app/php ########################## #Packges check and install ########################## pkg(){ for i in $PKGS; do rpm -q $i &> /dev/null && echo "Packge `rpm -q $i` is installed" || yum -y install $i done } ##################### #APACHE INSTALLATION #1.unarchive binary #################### #install dependency packges,just add packges name to variable PKGS separate by SPACE. #Example PKGS="libaio gcc glibc" httpd_ins(){ PKGS="gcc prce-devel openssl-devel expat-devel lbzip2" pkg cd $TMP_DIR if [ -e $TMP_DIR/$HTTPD ]; then tar xf $HTTPD else echo "file $HTTPD does not exist, pleases download it" exit fi if [ -e $TMP_DIR/$APR ]; then tar xf $APR mv $APR_DIR $HTTPD_DIR/srclib/apr else echo "file $APR does not exist, pleases download it" exit fi if [ -e $TMP_DIR/$APR_UTIL ]; then tar xf $APR_UTIL mv $APR_UTIL_DIR $HTTPD_DIR/srclib/apr-util else echo "file $APR_UTIL does not exist, pleases download it" exit fi ######################## #2.make and make install ######################## id apache||useradd -r -s /sbin/nologin apache cd $HTTPD_DIR ./configure --prefix=$INS_HTTPD_DIR --enable-so --enable-ssl --enable-cgi --enable-rewrite --with-zlib --with-pcre --with-included-apr --enable-modules=most --enable-mpms-shared=all --with-mpm=prefork if [ $? -eq 0 ]; then make -j 4 && make install cd $TMP_DIR rm -rf $HTTPD_DIR else echo "please remake and make install" fi ##################### #3. modify config files ###################### sed -ri ‘s@^(.*) daemon$@\1 apache@‘ $INS_HTTPD_DIR/conf/httpd.conf sed -ri ‘s@DirectoryIndex@DirectoryIndex index.php @‘ $INS_HTTPD_DIR/conf/httpd.conf sed -ri ‘s@#(LoadModule proxy_module modules/mod_proxy.so)@\1@‘ $INS_HTTPD_DIR/conf/httpd.conf sed -ri ‘s@#(LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so)@\1@‘ $INS_HTTPD_DIR/conf/httpd.conf sed -ri ‘s@#(LoadModule proxy_fdpass_module modules/mod_proxy_fdpass.so)@\1@‘ $INS_HTTPD_DIR/conf/httpd.conf cat >> $INS_HTTPD_DIR/conf/httpd.conf <<-EOF AddType application/x-httpd-php .php AddType application/x-httpd-php-source .phps ProxyRequests Off ProxyPassMatch "^/.*\.php(/.*)?$" "fcgi://localhost:9000/app/httpd24/htdocs/" EOF echo "PATH=$INS_HTTPD_DIR/bin:\$PATH" > /etc/profile.d/httpd.sh source /etc/profile.d/httpd.sh echo "$INS_HTTPD_DIR/bin/apachectl start" >> /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local apachectl start } ######################### #PHP INSALLATION #1. unarchive install packge ######################### php_ins(){ PKGS="libxml2-devel bzip2-devel libmcrypt-devel gd-devel" pkg cd $TMP_DIR echo $TMP_DIR if [ -e $TMP_DIR/$PHP ]; then tar xvf $PHP else echo "files $PHP does not exist, pleases download it" exit fi ####################### #2. make and make install ####################### cd $PHP_DIR/ ./configure --prefix=$INS_PHP_DIR --enable-mysqlnd --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd --with-openssl --with-freetype-dir --with-jpeg-dir --with-png-dir --with-zlib --with-libxml-dir=/usr --with-config-file-path=/etc --with-config-file-scan-dir=/etc/php.d --enable-mbstring --enable-xml --enable-sockets --enable-fpm --enable-maintainer-zts --disable-fileinfo --with-gd --with-imap--with-ldap--with-odbcver--with-iodbc--with-pear--with-libxml-dir--with-xmlrpc--enable-mbstring--with-mhash--with-gettext if [ $? -eq 0 ]; then make && make install else echo "please re make and make install" fi ###################### #3. modify config file ###################### cp php.ini-production /etc/php.ini cp sapi/fpm/init.d.php-fpm /etc/init.d/php-fpm chmod +x /etc/init.d/php-fpm cd $INS_PHP_DIR/etc cp php-fpm.conf.default php-fpm.conf cd php-fpm.d/ cp www.conf.default www.conf sed -ri ‘s@(^.*) = nobody@\1 = apache@‘ $INS_PHP_DIR/etc/php-fpm.d/www.conf chkconfig --add php-fpm service php-fpm start cd $TMP_DIR rm -rf $PHP_DIR echo ‘‘ > /app/httpd24/htdocs/index.php } ################## #4. remove PHP ################# rmphp(){ service php-rpm stop rm /app/php -rf rm /etc/php.ini -f chkconfig --del php-fpm rm /etc/init.d/php-fpm -f } ################ #5. remove HTTPD ################ rmhttpd(){ apachectl stop rm /app/httpd24 -rf sed -i ‘/\/app\/http24\/bin\/apachectl start/d‘ /etc/rc.d/rc.local rm /etc/profile.d/httpd.sh } ############### #OPTIONS SELECT ############### case $1 in install) case $2 in httpd) httpd_ins ;; php) php_ins ;; *) echo "Useage $0 $1 httpd|php" ;; esac ;; remove) case $2 in php) rmphp ;; httpd) rmhttpd ;; *) echo "Useage $0 $1 httpd|php" ;; esac ;; *) echo "Useage $0

" ;; esac [root@web2 ~]#./apache_php_install.sh install php 64. 修改nginx配置文件 [root@web2 ~]#vi /etc/nginx/nginx.conf server { listen 80; server_name www.zhaoli.com; location / { root /data/www; index index.php index.html index.htm; } location ~ \.php$ { root /data/www; fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi_params; } }

65. 修改php配置文件

[root@web2 ~]#vi /app/php/etc/php-fpm.d/www.conf user = nginx group = nginx listen = 127.0.0.1:9000 listen.owner = nginx listen.group = nginx listen.mode = 0660

66. 将所有配置文件复制到所有的web server中并启动服务

[root@web2 ~]#nginx -s reload [root@web2 ~]#service php-fpm restart

67. 解压缩文件 wordpress-5.0.4-zh_CN.tar.gz

[root@web1 data]#tar xf wordpress-5.0.4-zh_CN.tar.gz -C www

68. 在主服务器上创建wordpress数据库及用户名和密码

MariaDB [(none)]> CREATE DATABASE wordpress; MariaDB [(none)]> GRANT ALL PRIVILEGES ON wordpress.* TO "wordpress"@"192.168.205.%" IDENTIFIED BY "centos"; MariaDB [(none)]> FLUSH PRIVILEGES;

69. 同样我也需要将这个帐号在所有的proxysql服务器授权

[root@proxysql1 ~]#mysql -uadmin -padmin -P6032 -h127.0.0.1 [root@proxysql2 ~]#mysql -uadmin -padmin -P6032 -h127.0.0.1 所有proxysql中执行如下： MySQL [(none)]> insert into mysql_users(username,password,default_hostgroup) values(‘wordpress‘,‘centos‘,10); MySQL [(none)]> load mysql users to runtime; MySQL [(none)]> save mysql users to disk;

70. 修改wordpress配置文件

[root@web1 www]#cp wp-config-sample.php wp-config.php [root@web1 www]#vi wp-config.php define(‘DB_NAME‘, ‘wordpress‘); define(‘DB_USER‘, ‘wordpress‘); define(‘DB_PASSWORD‘, ‘centos‘); define(‘DB_HOST‘, ‘192.168.205.45:6033‘); #注意这里要添加proxysql vip的地址，端口为6033

71. 生成密钥，可以使用网站https://api.wordpress.org/secret-key/1.1/salt/ 自动生成，然后直接替换即可

[root@web1 www]#vi wp-config.php define(‘AUTH_KEY‘, ‘]xRUezwud7/sl9n{5Qv-=VM|uoqaFauAuc3|6wyo+HfjC!I+‘); define(‘SECURE_AUTH_KEY‘, ‘=e[P3g1~S|:+J@I)f-(:MTf3~h+;hQCg?wuk50NMP)Dgoj3X kL@BDDk%&;zed^`‘); define(‘LOGGED_IN_KEY‘, ‘f,B`O^3qW20-,`k>dHdW8Bt^/]HZ5 -sA1rz$x:|x3R3~!j*}^mw?0|N)YTOHkX#B3:fWbQ;[n^5FD)-4r9C!/+Swwv:k~~HZ|-l‘); define(‘SECURE_AUTH_SALT‘, ‘;=3HS/eY&DRN0p1_->e#]%h#x=*Q?Zj]A*tC=@*H$9_T%+SF+!w0?b}f/`#K&[h&‘); define(‘LOGGED_IN_SALT‘, ‘iVWA_K4+X&guJiXc90L4UnQ-#E7+q--rH1_`nhdbSzlC2X.}}R11aua{>8 $n7tJAq]?12ngpfi(]Cl{zfKs>!.Y?9|4@59{R*Q

72. 此时我们可以直接访问web1或web2来测试 #### 安装haproxy 73. 在所有的proxy server上安装HAproxy和keepalived

[root@haproxy1 ~]#yum install haproxy keepalived [root@haproxy2 ~]#yum install haproxy keepalived

74. 修改proxysql1的keepalive配置

[root@haproxy1 ~]#vi /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_1 vrrp_skip_check_adv_addr vrrp_strict vrrp_iptables vrrp_garp_interval 0 vrrp_gna_interval 0 } # Script used to check if HAProxy is running vrrp_script check_haproxy { script "killall -0 haproxy" interval 2 weight -30 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 111 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.205.111/24 dev eth0 label eth0:0 } track_script { check_haproxy } }

75. 修改proxysql2的keepalived配置

[root@haproxy2 ~]#vi /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_2 vrrp_skip_check_adv_addr vrrp_strict vrrp_iptables vrrp_garp_interval 0 vrrp_gna_interval 0 } # Script used to check if HAProxy is running vrrp_script check_haproxy { script "killall -0 haproxy" interval 2 weight -30 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 111 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.205.111/24 dev eth0 label eth0:0 } track_script { check_haproxy } }

76. 启动服务

[root@haproxy1 ~]#systemctl enable keepalived [root@haproxy1 ~]#systemctl start keepalived [root@haproxy2 ~]#systemctl enable keepalived [root@haproxy2 ~]#systemctl start keepalived

77. 修改haproxy配置文件在所有的haproxy server上

[root@haproxy1 ~]#vi /etc/haproxy/haproxy.cfg defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen web-80 bind 192.168.205.111:80 server web1 192.168.205.127:80 check inter 3s fall 3 rise 5 server web1 192.168.205.137:80 check inter 3s fall 3 rise 5

78. 启动服务

[root@haproxy1 ~]#systemctl enable haproxy [root@haproxy1 ~]#systemctl start haproxy [root@haproxy1 ~]#ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 192.168.205.111:80 *:*

78. 将haprox配置文件复制到haproxy2上并启动服务

[root@haproxy1 ~]#scp /etc/haproxy/haproxy.cfg 192.168.205.117:/etc/haproxy/ [root@haproxy2 ~]#systemctl enable haproxy [root@haproxy2 ~]#systemctl start haproxy

79. 我们发现没有监听192.168.205.111：80端口，因为默认不会监听不存在IP的端口

[root@haproxy2 ~]#ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::*

80. 修改内核参数可以实现监听，重启haproxy, 可以看到haproxy2也实现了监听, 同样我也需要在haproxy1上加上这个选项否则主失效的情况下，再切回来情况下，haproxy会出错，无法启动

[root@haproxy2 ~]#sysctl -a | grep bind net.ipv4.ip_nonlocal_bind = 0 [root@haproxy2 ~]#vi /etc/sysctl.conf net.ipv4.ip_nonlocal_bind = 1 [root@haproxy2 ~]#sysctl -p net.ipv4.ip_nonlocal_bind = 1 [root@haproxy2 ~]#systemctl restart haproxy [root@haproxy2 ~]#ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 192.168.205.111:80 *:*

81. 通过192.168.205.111访问web服务器，发现没有问题 82. 如果停掉haproxy1的keepalive, 再测试

[root@haproxy1 ~]#systemctl stop keepalived

83. 恢复keepalived，尝试停掉haproxy，再测试，发现vip已经切到haproxy2上

[root@haproxy1 ~]#systemctl stop haproxy [root@haproxy2 ~]#ip a 2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:05:be:a7 brd ff:ff:ff:ff:ff:ff inet 192.168.205.117/24 brd 192.168.205.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet 192.168.205.111/24 scope global secondary eth0:0 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe05:bea7/64 scope link noprefixroute valid_lft forever preferred_lft forever

#### 实现firewall 84. 为了简化我们使用一台linux的iptables做为DNAT,确保关闭firewald

[root@centos7 ~]#iptables -t nat -A PREROUTING -s 0/0 -d 202.106.0.17 -p tcp --dport 80 -j DNAT --to-destination 192.168.205.111:80

85. 开启ip转发功能

[root@centos7 network-scripts]#cat /proc/sys/net/ipv4/ip_forward 0 [root@centos7 network-scripts]#sysctl -w net.ipv4.ip_forward=1 net.ipv4.ip_forward = 1 [root@centos7 ~]#vi /etc/sysctl.conf net.ipv4.ip_forward = 1

86. 安装iptables service实现自动保存

[root@centos7 ~]#yum install iptables-services [root@centos7 ~]#iptables-save >/etc/sysconfig/iptables [root@centos7 ~]#systemctl enable iptables.service

87. 注意，Haproxy的默认网关要指向firewall, 两个webserver也默认网关也要指定firewall, 原因在于webserver回包时直接回到网关，实际是不是这样，有待考证，但我加了网站才能够正常访问

[root@haproxy1 ~]#ip r default via 192.168.205.17 dev eth0 proto static metric 102 [root@haproxy2 ~]#ip r default via 192.168.205.17 dev eth0 proto static metric 102 [root@web1 ~]#ip r a default via 192.168.205.17 dev eth0 [root@web1 ~]#ip r default via 192.168.205.17 dev eth0 [root@web2 ~]#ip r a default via 192.168.205.17 dev eth0 [root@web2 ~]#ip r default via 192.168.205.17 dev eth0

#### 实现powerdns 88. 安装包：基于EPEL源

yum install -y pdns pdns-backend-mysql mariadb-server