业务背景
主要需求是系统更新期间实现热部署，提供不间断服务。其次是为多个应用服务器（Tomcat）实现负载均衡的效果。另外还需要开启https访问，同事支持http与https同时访问，并根据不同的访问请求转发到Tomcat不同的端口。
软硬件需求
Tengine(淘宝开源Nginx服务器) * 1，Tomcat7.53 * 2，服务器IP：192.168.64.166
实现细节
1.编译Nginx
首先下载Nginx服务器，地址：http://tengine.taobao.org
如下编译:
本次应用中需要用到HTTPS服务，所以在编译时需要特殊加入 –with-http_ssl_module。
|
1 2 3 4 5 |
$ tar zxvf tengine2.*.tar.gz $ cd 进入解压完成后的目录 $ ./configure --with-http_ssl_module $ make $ make install |
如果在执行第一行 ./configure –with-http_ssl_module出现缺少pcre的错误，可以通过如下命令安装此组件。内网服务器没有网络连接，无法通过yum在线安装，可以通过寻找相关包进行安装。
|
1 2 |
# yum install pcre # yum install pcre_devel |
在顺利执行安装完成后，进入下一步。配置Tomcat服务器。
2.Tomcat服务器配置
本次共部署2台Tomcat服务器，且均在同一台服务器上所以需要对其中一台Tomcat服务器做端口更改处理，否则同一台服务器不能启动2台Tomcat服务器。当现实场景中不在同一服务器，且每台服务器只有一台Tomcat服务器时，可以不考虑端口更改。
第二台Tomcat服务器的配置文件端口更改如下：对所有默认端口+1，修改文件为Tomcat文件中conf文件夹内的server.xml。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 |
<?xml version='1.0' encoding='utf-8'?> <Server port="8006" shutdown="SHUTDOWN"> <!-- Security listener. Documentation at /docs/config/listeners.html <Listener className="org.apache.catalina.security.SecurityListener" /> --> <!--APR library loader. Documentation at /docs/apr.html --> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html --> <Listener className="org.apache.catalina.core.JasperListener" /> <!-- Prevent memory leaks due to use of particular java/javax APIs--> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> <!-- Global JNDI resources Documentation at /docs/jndi-resources-howto.html --> <GlobalNamingResources> <!-- Editable user database that can also be used by UserDatabaseRealm to authenticate users --> <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" /> </GlobalNamingResources> <!-- A "Service" is a collection of one or more "Connectors" that share a single "Container" Note: A "Service" is not itself a "Container", so you may not define subcomponents such as "Valves" at this level. Documentation at /docs/config/service.html --> <Service name="Catalina"> <!--The connectors can use a shared executor, you can define one or more named thread pools--> <!-- <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" maxThreads="150" minSpareThreads="4"/> --> <!-- A "Connector" represents an endpoint by which requests are received and responses are returned. Documentation at : Java HTTP Connector: /docs/config/http.html (blocking & non-blocking) Java AJP Connector: /docs/config/ajp.html APR (HTTP/AJP) Connector: /docs/apr.html Define a non-SSL HTTP/1.1 Connector on port 8080 --> <Connector port="8081" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8444" /> <!-- Define a SSL HTTP/1.1 Connector on port 8444 This connector uses the JSSE configuration, when using APR, the connector should be using the OpenSSL style configuration described in the APR documentation --> <Connector port="8444" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" keystoreFile="/home/webabc.jks" keystorePass="12345678" /> <!-- Define an AJP 1.3 Connector on port 8009 --> <Connector port="8010" protocol="AJP/1.3" redirectPort="8444" /> <Engine name="Catalina" defaultHost="localhost"> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8"> <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/> <Channel className="org.apache.catalina.tribes.group.GroupChannel"> <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000"/> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="auto" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/> <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> </Sender> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/> </Channel> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/> <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" watchDir="/tmp/war-listen/" watchEnabled="false"/> <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/> <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/> </Cluster> <Realm className="org.apache.catalina.realm.LockOutRealm"> <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/> </Realm> <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true"> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t "%r" %s %b" /> </Host> </Engine> </Service> </Server> |
3.Nginx配置
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
#user nobody; worker_processes 8; //建议根据处理器的核心数开启worker_processes的个数 #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } # load modules compiled as Dynamic Shared Object (DSO) # #dso { # load ngx_http_fastcgi_module.so; # load ngx_http_rewrite_module.so; #} http { include mime.types; default_type application/octet-stream; underscores_in_headers on; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; upstream mytomcats { ip_hash; //此处采用ip_hash负载均衡方案,同一IP,同一session生命周期内在某server不宕机的情况下,都会将HTTP请求转发到该server server 192.168.64.166:8080; //第一台Tomcat服务器的地址及端口,接收HTTP访问请求 server 192.168.64.166:8081; //第二台Tomcat服务器的地址及端口,接收HTTP访问请求 } server { listen 80; server_name 192.168.64.166; #charset koi8-r; rewrite ^/(.*)/api/rest/* https://$server_name/$1/api/rest/ last; //此处为本次部署的特殊需求:在使用HTTP协议访问/api/rest/路径下资源时,自动跳转到HTTPS #access_log logs/host.access.log main; location / { proxy_pass http://mytomcats; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; root html; index index.html index.htm; } #error_page 404 /404.html; error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } upstream myssltomcats { ip_hash; //此处采用ip_hash负载均衡方案,同一IP,同一session生命周期内在某server不宕机的情况下,都会将HTTP请求转发到该server server 192.168.64.166:8444; //第一台Tomcat服务器的地址及端口,接收HTTPS访问请求 server 192.168.64.166:8443; //第二台Tomcat服务器的地址及端口,接收HTTPS访问请求 } # HTTPS server ##############以下是开启SSL即HTTPS访问配置############### server { listen 443; server_name 192.168.64.166; ssl on; ssl_certificate /usr/local/nginx/conf/ssl/localhost.crt; //通过openssl可在本地生成此处所需证书 ssl_certificate_key /usr/local/nginx/conf/ssl/localhost.key; //通过openssl可在本地生成此处所需密钥 ssl_client_certificate /usr/local/nginx/conf/ssl/ca.crt; //通过openssl可在本地生成此处所需证书 ssl_session_timeout 5m; ssl_protocols SSLv2 SSLv3 TLSv1; ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP; ssl_prefer_server_ciphers on; location / { proxy_pass https://myssltomcats; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; root html; index index.html index.htm; } } } |
4.热部署步骤
1.更新应用时，首先停用其中一台Tomcat1服务器，让另一台提供服务。
2.更新首先停用的Tomcat1服务器下的应用，然后启动。
3.启动Tomat1完成后，再停止另外一台Tomcat2服务器，更新其应用。
4.再启动已停止的Tomcat2。
其实以前就写过类似文章：
《Nginx的反向代理负载均衡》 http://chenzhiguo.cn/archives/nginx_load_balancing