运维实战负载均衡Nginx

由于其同时具有商业版和社区版,这里我使用社区版.(同时也因为其开源特性有很多衍生版本,比如阿里经过大量修改和自需求优化后的版本叫做Tengine)

由于之前进行过HAProxy的高可用和Fence搭建,因此请先格式化虚拟机或关闭集群,保证Server1不受之前的实验环境干扰.

• Nginx使用源码编译安装因此需要提前安装编译所需的C语言编译器.

yum install -y gcc


• 了解编译所需时间与软件源码大小以及机器CPU性能有关.
• 在配置过程中可能会出现error,根据提示补全所需的依赖后重新配置即可.
• 所需以来的软件包名通常为名称-devel,可以进行优先尝试.(如缺少Openssl则先尝试安装openssl-devel进行解决).
• configure时可以按需增加参数,支持的参数可以通过--help查看.

##安装流程,省略部分安装过程滚动信息.
[root&＃64;Server1 mnt]# ls
nginx-1.18.0.tar.gz
[root&＃64;Server1 mnt]# tar zxf nginx-1.18.0.tar.gz
[root&＃64;Server1 mnt]# cd nginx-1.18.0/
[root&＃64;Server1 nginx-1.18.0]# ls
auto CHANGES.ru configure html man src
CHANGES conf contrib LICENSE README[root&＃64;Server1 nginx-1.18.0]# ./configure --prefix&＃61;/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module
./configure: error: the HTTP rewrite module requires the PCRE library.
You can either disable the module by using --without-http_rewrite_module
option, or install the PCRE library into the system, or build the PCRE library
statically from the source with nginx by using --with-pcre&＃61;<path> option.##缺少PCRE依赖,进行安装##
[root&＃64;Server1 nginx-1.18.0]# yum install -y pcre-devel[root&＃64;Server1 nginx-1.18.0]# ./configure --prefix&＃61;/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module
./configure: error: SSL modules require the OpenSSL library.
You can either do not enable the modules, or install the OpenSSL library
into the system, or build the OpenSSL library statically from the source
with nginx by using --with-openssl&＃61;<path> option.##缺少Openssl依赖,进行安装##
[root&＃64;Server1 nginx-1.18.0]# yum install -y openssl-devel##正确配置
[root&＃64;Server1 nginx-1.18.0]# ./configure --prefix&＃61;/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module##可以看到出现了编译所需的文件Makefile
[root&＃64;Server1 nginx-1.18.0]# ls
auto CHANGES.ru configure html Makefile objs src
CHANGES conf contrib LICENSE man README
##编译并安装
[root&＃64;Server1 nginx-1.18.0]# make
[root&＃64;Server1 nginx-1.18.0]# make install##编译安装到此结束,下文均为安装后目录中操作
##与解压目录无关
[root&＃64;Server1 nginx-1.18.0]# cd /usr/local/nginx/##配置环境变量路径并刷新激活
[root&＃64;Server1 sbin]# vim ~/.bash_profile
[root&＃64;Server1 sbin]# source ~/.bash_profile ##尝试启动nginx,可以看到其运行在80端口
[root&＃64;Server1 sbin]# nginx
[root&＃64;Server1 sbin]# netstat -antlp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 9021/nginx: master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3221/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3391/master
tcp 0 0 172.25.5.1:22 172.25.5.250:47860 ESTABLISHED 3642/sshd: root&＃64;pts
tcp6 0 0 :::22 :::* LISTEN 3221/sshd
tcp6 0 0 ::1:25 :::* LISTEN 3391/master##向nginx发送停止命令,相当于stop
[root&＃64;Server1 sbin]# nginx -s stop


• 通过ps aux可以看到nginx运行后出现两个新进程.

##进程号不一定一致9021 ? Ss 0:00 nginx: master process nginx9022 ? S 0:00 nginx: worker process

修改主配置文件

• Nginx默认没有设置用户因此如果按照默认配置运行则进程用户为nobody,这是我们不想看到的.
• 修改Nginx的进程数和句柄数来提高性能.
• 与HAProxy中的逻辑相同,更改应用的句柄数后一样要更改系统的.
• 在主配置文件的http服务设置部分设置upstream来实现负载均衡,后续的模块相关设置也在这里.
• 在主配置文件的http服务设置部分设置server.
• nginx -t检测语法.
• nginx -s reload重载配置.

##添加供Nginx使用的用户,设置不可用于登录,不自动创建家目录且手动制定家目录
[root&＃64;Server1 conf]# useradd -M -d /usr/local/nginx/ -s /sbin/nologin nginx
[root&＃64;Server1 conf]# id nginx
uid&＃61;1001(nginx) gid&＃61;1001(nginx) groups&＃61;1001(nginx)##配置nginx相关用户设置并重载
##可以看到进程的用户变为nginx
[root&＃64;Server1 conf]# vim nginx.conf
[root&＃64;Server1 conf]# nginx -s reload
[root&＃64;Server1 conf]# ps aux
nginx 9080 0.0 0.1 46420 2024 ? S 10:19 0:00 nginx: worker p


##尝试使用curl访问,可以看到调度
[root&＃64;foundation5 mnt]# curl www.westos.org
Server3
[root&＃64;foundation5 mnt]# curl www.westos.org
Server2
[root&＃64;foundation5 mnt]# curl www.westos.org
Server3


##使用Apache功能进行并发测试,10个用户总共5000条请求.
[root&＃64;foundation5 mnt]# ab -c10 -n5000 http://www.westos.org/index.html
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/Benchmarking www.westos.org (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requestsServer Software: nginx/1.18.0
Server Hostname: www.westos.org
Server Port: 80Document Path: /index.html
Document Length: 8 bytesConcurrency Level: 10
Time taken for tests: 1.996 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 1285000 bytes
HTML transferred: 40000 bytes
Requests per second: 2504.81 [#/sec] (mean)
Time per request: 3.992 [ms] (mean)
Time per request: 0.399 [ms] (mean, across all concurrent requests)
Transfer rate: 628.65 [Kbytes/sec] receivedConnection Times (ms)min mean[&＃43;/-sd] median max
Connect: 0 0 0.4 0 3
Processing: 1 3 1.2 3 15
Waiting: 1 3 1.1 3 15
Total: 1 4 1.2 4 16Percentage of the requests served within a certain time (ms)50% 466% 475% 480% 590% 595% 698% 799% 8100% 16 (longest request)


• 修改过后的nginx.conf内容


user nginx nginx;
worker_processes 2;
worker_cpu_affinity 01 10;#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;#pid logs/nginx.pid;events {worker_connections 65535;
}http {upstream Test{server 172.25.5.2:80;server 172.25.5.3:80;}include mime.types;default_type application/octet-stream;#log_format main &＃39;$remote_addr - $remote_user [$time_local] "$request" &＃39;# &＃39;$status $body_bytes_sent "$http_referer" &＃39;# &＃39;"$http_user_agent" "$http_x_forwarded_for"&＃39;;#access_log logs/access.log main;sendfile on;#tcp_nopush on;#keepalive_timeout 0;keepalive_timeout 65;#gzip on;server {listen 80;server_name localhost;#charset koi8-r;#access_log logs/host.access.log main;location / {root html;index index.html index.htm;}#error_page 404 /404.html;# redirect server error pages to the static page /50x.html#error_page 500 502 503 504 /50x.html;location &＃61; /50x.html {root html;}# proxy the PHP scripts to Apache listening on 127.0.0.1:80##location ~ \.php$ {# proxy_pass http://127.0.0.1;#}# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000##location ~ \.php$ {# root html;# fastcgi_pass 127.0.0.1:9000;# fastcgi_index index.php;# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;# include fastcgi_params;#}# deny access to .htaccess files, if Apache&＃39;s document root# concurs with nginx&＃39;s one##location ~ /\.ht {# deny all;#}}# another virtual host using mix of IP-, name-, and port-based configuration##server {# listen 8000;# listen somename:8080;# server_name somename alias another.alias;# location / {# root html;# index index.html index.htm;# }#}# HTTPS server##server {# listen 443 ssl;# server_name localhost;# ssl_certificate cert.pem;# ssl_certificate_key cert.key;# ssl_session_cache shared:SSL:1m;# ssl_session_timeout 5m;# ssl_ciphers HIGH:!aNULL:!MD5;# ssl_prefer_server_ciphers on;# location / {# root html;# index index.html index.htm;# }#}server {listen 80;server_name www.westos.org;location / {proxy_pass http://Test;}
}
}

vim /etc/security/limit.conf# End of file
nginx - nofile 65535

基于COOKIEs的会话保持

由于sticky COOKIEs模式只有商业付费版本的Nginx Plus才能使用,这里使用第三方模块nginx-goodies-nginx-sticky-module来进行基于COOKIEs的会话保持.

需求场景

请求经过类似CDN之类的反向代理后,对后端RealServer的请求IP变更为反响代理服务器的IP.

如果使用基于IP的哈希验证,则相当于根本没有做负载均衡.

而基于COOKIEs进行验证则不会出现这个问题.

##解压第三方模块并在配置/编译时加入
[root&＃64;Server1 mnt]# unzip nginx-goodies-nginx-sticky-module-ng-08a395c66e42.zip
[root&＃64;Server1 nginx-1.18.0]# ./configure --prefix&＃61;/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --add-module&＃61;/mnt/nginx-goodies-nginx-sticky-module-ng-08a395c66e42


对Nginx主配置中的服务部分引入COOKIEs模块并重载配置.


user nginx nginx;
worker_processes auto;#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;#pid logs/nginx.pid;events {worker_connections 65535;
}http {upstream Test {sticky;server 172.25.5.2:80;server 172.25.5.3:80;}include mime.types;default_type application/octet-stream;#log_format main &＃39;$remote_addr - $remote_user [$time_local] "$request" &＃39;# &＃39;$status $body_bytes_sent "$http_referer" &＃39;# &＃39;"$http_user_agent" "$http_x_forwarded_for"&＃39;;#access_log logs/access.log main;sendfile on;#tcp_nopush on;#keepalive_timeout 0;keepalive_timeout 65;#gzip on;server {listen 80;server_name localhost;#charset koi8-r;#access_log logs/host.access.log main;location / {root html;index index.html index.htm;}#error_page 404 /404.html;# redirect server error pages to the static page /50x.html#error_page 500 502 503 504 /50x.html;location &＃61; /50x.html {root html;}# proxy the PHP scripts to Apache listening on 127.0.0.1:80##location ~ \.php$ {# proxy_pass http://127.0.0.1;#}# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000##location ~ \.php$ {# root html;# fastcgi_pass 127.0.0.1:9000;# fastcgi_index index.php;# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;# include fastcgi_params;#}# deny access to .htaccess files, if Apache&＃39;s document root# concurs with nginx&＃39;s one##location ~ /\.ht {# deny all;#}}# another virtual host using mix of IP-, name-, and port-based configuration##server {# listen 8000;# listen somename:8080;# server_name somename alias another.alias;# location / {# root html;# index index.html index.htm;# }#}# HTTPS server##server {# listen 443 ssl;# server_name localhost;# ssl_certificate cert.pem;# ssl_certificate_key cert.key;# ssl_session_cache shared:SSL:1m;# ssl_session_timeout 5m;# ssl_ciphers HIGH:!aNULL:!MD5;# ssl_prefer_server_ciphers on;# location / {# root html;# index index.html index.htm;# }#}server {listen 80;server_name www.westos.org;location / {proxy_pass http://Test;}}
}


分别使用curl和浏览器对www.wesos.org进行访问.

• 由于curl没有COOKIEs功能所以以就可以实现轮转
• 而浏览器由于有COOKIEs所以会始终保持第一次访问的RS

[root&＃64;foundation5 nginx-1.18.0]# curl www.westos.org
Server2
[root&＃64;foundation5 nginx-1.18.0]# curl www.westos.org
Server2
[root&＃64;foundation5 nginx-1.18.0]# curl www.westos.org
Server3
[root&＃64;foundation5 nginx-1.18.0]# curl www.westos.org
Server2