热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

Redis4集群failover后客户端未感知拓扑变化排查

排查结论23:18分,30.1这台机器出现异常,其在30.2上的SlaveRedis服务检测到后提出升主请求并获得通过,成

排查结论


  1. 23:18分,30.1这台机器出现异常,其在30.2上的Slave Redis服务检测到后提出升主请求并获得通过,成为Master;
  2. 23:18至23:34之间,Redis client端对于Redis Server的连接检测并未异常,没有触发拓扑更新,因而对于Redis Server访问异常;
  3. 23:34,SRE重启30.1,此时 Redis Client检测到连接断开,触发拓扑更新,Redis访问正常。

解决方案

当前触发Redis Client更新拓扑的方式有两种:1. 向服务端发请求返回MOVE或ASK,2. ConnectionWatchDog检测到连接断开。当时的状况下,服务器状态未知(待与SRE确认),两种条件都没有触发。

所以解决方案为:Redis Client发新版,增加定期更新拓扑。

问题背景

2021-03-11 oncall群里由SRE发现一台redis服务器故障,通知如下

从而引发了帖子,蛋卷,UC,推荐相关服务的报警,redis的failover是在1min内完成的,但是其中推荐的服务的告警持续了近15min恢复,因此以下排查基于推荐服务进行

排查过程


时间线


redis服务器层面


节点描述

30.1

物理机在23:34:59完成了服务启动
30.2这个从节点在23:18:45完成了主从切换

30.1:
11 23:34:59 服务器启动
11 Mar 23:56:25.132 30.1:25701 实例启动;


30.2:
11 Mar 23:18:44 30.2 detect down
11 Mar 23:18:45.417 # configEpoch set to 8 after successful failover
11 Mar 23:56:26 30.1 slave full sync


推荐服务层面

读写操作分析

节点读写报错时间:start读写报错时间:end
160-11623:19:10开始出现读写报错23:30:25读写报错结束(这个时间进行了服务重启)
160-11923:19:10开始出现读写报错23:34:39读写报错结束(这个时间30.1那台redis物理机重新启动)

------------------160-116
app@bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |head
2021-03-11 23:19:10.664|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@4f8d63a2
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy139.zrevrangeWithScores(Unknown Source)

app@bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:30:25.627|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@d6ea39d
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy139.zrevrangeWithScores(Unknown Source)

------------------160-119
app@bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |head
2021-03-11 23:19:10.611|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@6897ac3e
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy134.zrevrangeWithScores(Unknown Source)

app@bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:34:39.158|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@3323f9a4
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy134.zrevrangeWithScores(Unknown Source)

lettuce发现connection异常分析

节点出现connection异常时间:start出现connection异常时间:end
160-11623:35:16出现connection异常提示23:35:16出现connection异常提示,有且仅有这么一次(服务启动时发现拓扑的过程发生的错误)
160-11923:34:08出现connection异常提示(这个时间30.1那台redis物理机重新启动)23:42:35最后出现connection异常(23:42:26服务进行了重启)

------------------160-116
app@bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log|head
2021-03-11 23:35:16.180|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:35:16.230|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

app@bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:35:16.180|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:35:16.230|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

------------------160-119
app@bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log|head
2021-03-11 23:34:08.452|WARN|i.l.c.p.ConnectionWatchdog||Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: /10.10.30.1:25071
2021-03-11 23:34:08.453|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

app@bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:42:34.992|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:42:35.116|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)


监控图层面


节点QPS异常开始时间QPS回复时间
160-11623:19开始发生(与上面的读写操作日志分析对应上了)23:35开始恢复(与上面的lettuce发现connection日志异常分析对应上了,这个是重启的结果不能够说明SDK拓扑更新的逻辑)
160-11923:19开始发生(与上面的读写操作日志分析对应上了)

23:34开始恢复(23:34服务没有重启,QPS回复正常,证明拓扑是更新了。

倾向认为:23:18~23:34 redis-server30.1不响应redis-Command,但是TCP未断连。

23:34由于TCP断连触发拓扑刷新)



对应使用redis2.8的服务监控

帖子服务的2.8的redis client的P999是和上面吻合的

http://matrix.snowballfinance.com/d/ryrAxKXZk/ugc-snowflake-status-basic?orgId=1&from=1615474888737&to=1615478153783&fullscreen&panelId=8

UC服务的2.8的redis client的P999不吻合的(这个需要UC来解释)

http://matrix.snowballfinance.com/d/000000005/uc-snowflake-usercenter?orgId=1&fullscreen&edit&panelId=10&from=1615475760054&to=1615477519942


redis metric监控系统

redis的metric监控系统是golang编写的,原理就是for循环进行监控数据采集

时间描述
在23:18的时候对30.1这个机器的节点进行getinfo的时候,出现了阻塞
在23:34的时候30.1这个机器重新启动,阻塞得到释放(从侧面印证了上述119这台机器监控和日志的解释)

http://matrix.snowballfinance.com/d/000000024/sre-redis-detail?orgId=1&from=1615474845742&to=1615480482137


redis所在物理机监控

30.1物理机处于什么状态?导致18~34没有监控指标?

23:18-23:34监控数据没有,grafana使用了1条直线连接了23:18和23:34

http://matrix.snowballfinance.com/d/000000012/sre-wu-li-ji-detail?orgId=1&from=1615475297782&to=1615478502199&var-Hostname=bjxg-rs-30-1


原因分析


tcp/ip跟本次事件相关的原理背景

TCP 建立连接要进行三次握手(有对应的SYN信号),而断开连接要进行四次(有对应的FIN信号,且都是seq的序列号来进行确认)。这是由于TCP 的半关闭造成的。这也是为什么四次挥手要比三次握手多一次,因为TCP 连接是全双工的(即数据可在两个方向上同时传递)所以进行关闭时每个方向上都要单独进行关闭。单方向的关闭就叫半关闭。

TCP正常创建流程对fd的处理流程,TCP正常close和shutdown的处理流程,那么tcp连接异常终端后对应的socket会做些什么?

TCP/IP可选特性Keepalive(默认关闭)
有3个参数,以linux为例,搜索keepalive相关的系统参数(sysctl -A |grep keepalive),3个参数组合起来表示:

如果TCP连接2小时(7200秒)内没有数据传输,则发送嗅探包,每隔75秒发送一次,共重试9次。9次对方都没响应,则表明此连接已死。(留个问题,这个时候socket会给fd发送什么信号??)

以linux为例:
         recv的man对返回值的介绍如下:
              RETURN VALUE
              These calls return the number of bytes received, or -1 if an error occurred. The return value will be 0 when the peer has performed an orderly shutdown.
          即,recv返回0表示对端关闭。对于send,如果返回值为-1,且errno为ECONNRESET,表示对端关闭。

一般服务器程序不会用到这个机制,而是把Keepalive放到业务层,自己实现Keepalive机制。这样做的好处是可以减少程序对特定协议的依赖,且发送给客户端的请求可以组合起来,在心跳包到来的时候一并返回给客户端。

摘自《TCP/IP详解》卷1第23章,强烈建议点开链接看看,解释的真通俗易懂

主要本次关注以下几点:

1.没有收到FIN信号,没有keepalive,那么该TCP会在两个小时之后尝试嗅探(天塌了它也不管)

2.对端服务器崩溃、关机、正在重启、网络不可达,那么该TCP的上层通常是报告连接超时(留个问题,信号量是多少??)

3.对端服务重启之后,那么该TCP得到的回复是RST

 

 


netty跟本次事件相关的原理背景

netty这部分没啥好说的,看了好多的资料,总结下来关于本次事件相关的连接信息有以下内容:

1.NioEventLoop,reactor线程在自旋中执行细节

accept事件接收连接

2.workEventLoop,上一步的在接收完链接后,会触发相关读事件( fireChannelRead),以及读完成事件( fireChannelReadComplete)对于server端来说,会设置对应selectKeys事件设置为 读事件监听。

由于会去pipeline中调用每个channelHandler进行事件处理,所以在nio 的accept事件之后,会执行 fireChannelRead 方法。

读事件第一个处理为 headContext。而后由 HeadContext 将事件进行向后传播。DefaultChannelPipeline 的 HeadContext。

3.close 与 channelInactive 的关系是 close 是主动关闭 channel 的动作, 而 channelInactive 是关闭成功后收到通知的事件.

关于netty的tcp参数设置

通过Netty配置TCP心跳,或者泛化的说配置TCP协议参数的方式有两种:
1、childOption方式是正确的
2、option方法不起作用,也不能说错吧,就是没啥意义而已

即本质就是一种方式。另外在4.1.36以后版本,如果你是JDK7以上,那么可以直接这样配置客户端连接的协议参数:.childOption(NioChannelOption.of(StandardSocketOptions.SO_KEEPALIVE),true)

网络层复现

背景介绍:使用的64.169/64.15这两台虚拟机用来当做服务器,本地开发机当做client

然后通过IPtable drop来尝试模拟服务节点突然下线

网络层的TCP信号

经历了开始的三次握手(SYN信号) ------→ 数据交互,正常读写(P.信号) ------→ 其中一个节点连接突然中断(R.信号) ------→ 尝试重连(不断的S信号) ------→ 连接恢复(SYN信号) ------→ 恢复正常读写(P.信号) ------→ 最后的链接断开(FIN信号)

20:08:39.861243 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [S], seq 1251904794, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071045 ecr 0,sackOK,eol], length 0
20:08:39.861536 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [S.], seq 4226972078, ack 1251904795, win 28960, options [mss 1460,sackOK,TS val 107727801 ecr 731071045,nop,wscale 7], length 0
20:08:39.861574 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972079, win 2058, options [nop,nop,TS val 731071045 ecr 107727801], length 020:08:39.862538 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [P.], seq 1251904795:1251904861, ack 4226972079, win 2058, options [nop,nop,TS val 731071046 ecr 107727801], length 66
20:08:39.862838 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [.], ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 0
20:08:39.863019 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [P.], seq 4226972079:4226972084, ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 5
20:08:39.863065 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972084, win 2058, options [nop,nop,TS val 731071046 ecr 107727802], length 020:08:39.865550 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [S], seq 4104613940, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071048 ecr 0,sackOK,eol], length 0
20:08:39.865820 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [S.], seq 413298472, ack 4104613941, win 28960, options [mss 1460,sackOK,TS val 107727805 ecr 731071048,nop,wscale 7], length 0
20:08:39.865868 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298473, win 2058, options [nop,nop,TS val 731071049 ecr 107727805], length 020:08:39.866943 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [P.], seq 4104613941:4104614007, ack 413298473, win 2058, options [nop,nop,TS val 731071050 ecr 107727805], length 66
20:08:39.867192 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [.], ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 0
20:08:39.867329 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [P.], seq 413298473:413298478, ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 5
20:08:39.867378 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298478, win 2058, options [nop,nop,TS val 731071050 ecr 107727807], length 014
20:16:00.604659 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509045 ecr 107733081], length 48
20:16:00.754648 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509195 ecr 107733081], length 48
20:16:01.055889 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509495 ecr 107733081], length 48
20:16:01.456264 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509895 ecr 107733081], length 48
20:16:02.062739 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731510495 ecr 107733081], length 48
20:16:03.065437 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731511495 ecr 107733081], length 48
20:16:04.871576 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731513295 ecr 107733081], length 48
20:16:08.287194 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731516695 ecr 107733081], length 48
20:16:14.912828 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731523295 ecr 107733081], length 48
20:16:21.529555 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731529895 ecr 107733081], length 48
20:16:28.147866 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731536495 ecr 107733081], length 48
20:16:34.771964 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731543095 ecr 107733081], length 48
20:16:41.398904 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731549695 ecr 107733081], length 48
20:16:48.025246 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731556295 ecr 107733081], length 48
20:16:54.655115 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [R.], seq 4241911420, ack 1651645875, win 2048, length 07
20:17:17.609293 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731585757 ecr 0,sackOK,eol], length 0
20:17:18.610567 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731586757 ecr 0,sackOK,eol], length 0
20:17:19.613758 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731587757 ecr 0,sackOK,eol], length 0
20:17:20.617512 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731588758 ecr 0,sackOK,eol], length 0
20:17:21.620555 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731589758 ecr 0,sackOK,eol], length 0
20:17:22.625515 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731590758 ecr 0,sackOK,eol], length 0
20:17:24.631276 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731592758 ecr 0,sackOK,eol], length 011
20:17:27.661985 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595780 ecr 0,sackOK,eol], length 0
20:17:27.805463 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595921 ecr 0,sackOK,eol], length 0
20:17:27.906594 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596021 ecr 0,sackOK,eol], length 0
20:17:28.007657 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596122 ecr 0,sackOK,eol], length 0
20:17:28.108495 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596222 ecr 0,sackOK,eol], length 0
20:17:28.211396 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596322 ecr 0,sackOK,eol], length 0
20:17:28.412573 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596523 ecr 0,sackOK,eol], length 0
20:17:28.813800 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596923 ecr 0,sackOK,eol], length 0
20:17:29.617595 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731597723 ecr 0,sackOK,eol], length 0
20:17:31.226228 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731599323 ecr 0,sackOK,eol], length 0
20:17:34.434295 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:37.766336 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605835 ecr 0,sackOK,eol], length 0
20:17:37.907801 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605976 ecr 0,sackOK,eol], length 0
20:17:38.009930 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606076 ecr 0,sackOK,eol], length 0
20:17:38.111023 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606177 ecr 0,sackOK,eol], length 0
20:17:38.212162 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606277 ecr 0,sackOK,eol], length 0
20:17:38.312800 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606377 ecr 0,sackOK,eol], length 0
20:17:38.513278 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606577 ecr 0,sackOK,eol], length 0
20:17:38.913743 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606977 ecr 0,sackOK,eol], length 0
20:17:39.718061 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731607777 ecr 0,sackOK,eol], length 0
20:17:41.323608 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731609377 ecr 0,sackOK,eol], length 0
20:17:44.538196 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:47.867437 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731615897 ecr 0,sackOK,eol], length 0
20:17:48.008751 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616038 ecr 0,sackOK,eol], length 0
20:17:48.109361 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616138 ecr 0,sackOK,eol], length 0
20:17:48.209657 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616238 ecr 0,sackOK,eol], length 0
20:17:48.312000 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616338 ecr 0,sackOK,eol], length 0
20:17:48.413063 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616439 ecr 0,sackOK,eol], length 0
20:17:48.613676 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616639 ecr 0,sackOK,eol], length 0
20:17:49.015418 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617039 ecr 0,sackOK,eol], length 0
20:17:49.818549 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617839 ecr 0,sackOK,eol], length 0
20:17:51.422985 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731619439 ecr 0,sackOK,eol], length 0
20:17:54.631568 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:57.969994 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0
20:17:57.970906 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0
20:17:58.115551 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0
20:17:58.115561 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0
20:17:58.217038 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0
20:17:58.217049 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0
20:17:58.317493 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0
20:17:58.317504 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0
20:17:58.418207 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0
20:17:58.418219 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0
20:17:58.519006 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0
20:17:58.519018 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0
20:17:58.721709 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0
20:17:58.721721 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0
20:17:59.123040 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0
20:17:59.123052 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0
20:17:59.925894 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0
20:17:59.925910 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0
20:18:01.530428 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0
20:18:01.530436 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0
20:18:04.739272 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:04.739280 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,sackOK,eol], length 013
20:18:08.063149 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636018 ecr 0,sackOK,eol], length 0
20:18:08.204712 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636159 ecr 0,sackOK,eol], length 0
20:18:08.305275 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636259 ecr 0,sackOK,eol], length 0
20:18:08.406168 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636359 ecr 0,sackOK,eol], length 0
20:18:08.507152 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636459 ecr 0,sackOK,eol], length 0
20:18:08.607218 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636559 ecr 0,sackOK,eol], length 0
20:18:08.808894 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636759 ecr 0,sackOK,eol], length 0
20:18:09.209305 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637159 ecr 0,sackOK,eol], length 0
20:18:10.011676 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637959 ecr 0,sackOK,eol], length 0
20:18:11.616529 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731639559 ecr 0,sackOK,eol], length 0
20:18:14.828898 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:21.286606 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:27.721777 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 013
20:20:58.899104 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805899 ecr 0,sackOK,eol], length 0
20:20:58.900162 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805900 ecr 0,sackOK,eol], length 0
20:20:59.045755 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806040 ecr 0,sackOK,eol], length 0
20:20:59.047010 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806041 ecr 0,sackOK,eol], length 0
20:20:59.146241 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806140 ecr 0,sackOK,eol], length 0
20:20:59.147300 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806141 ecr 0,sackOK,eol], length 0
20:20:59.247515 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806240 ecr 0,sackOK,eol], length 0
20:20:59.248746 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806241 ecr 0,sackOK,eol], length 0
20:20:59.347871 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806340 ecr 0,sackOK,eol], length 0
20:20:59.348974 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806341 ecr 0,sackOK,eol], length 0
20:20:59.449197 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0
20:20:59.449211 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0
20:20:59.654647 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0
20:20:59.654653 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0
20:21:00.056939 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0
20:21:00.056958 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0
20:21:00.867762 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0
20:21:00.867778 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0
20:21:02.484895 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0
20:21:02.484903 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0
20:21:05.715910 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:05.715916 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:12.139150 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:12.139164 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:18.570932 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:18.570946 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 020:23:49.944343 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [S], seq 1827331675, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976206 ecr 0,sackOK,eol], length 0
20:23:49.944875 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [S], seq 1554881183, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976207 ecr 0,sackOK,eol], length 0
20:23:49.954078 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [S.], seq 2130024761, ack 1827331676, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976206,nop,wscale 7], length 0
20:23:49.954085 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [S.], seq 1062682904, ack 1554881184, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976207,nop,wscale 7], length 0
20:23:49.954203 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024762, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0
20:23:49.954257 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062682905, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0
20:23:49.957101 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [P.], seq 1827331676:1827331724, ack 2130024762, win 2058, options [nop,nop,TS val 731976217 ecr 108637891], length 48
20:23:49.959283 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881184:1554881250, ack 1062682905, win 2058, options [nop,nop,TS val 731976218 ecr 108637891], length 66
20:23:49.999821 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [.], ack 1827331724, win 227, options [nop,nop,TS val 108637944 ecr 731976217], length 0
20:23:49.999822 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881250, win 227, options [nop,nop,TS val 108637944 ecr 731976218], length 0
20:23:50.000112 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881250:1554881278, ack 1062682905, win 2058, options [nop,nop,TS val 731976258 ecr 108637944], length 28
20:23:50.000541 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881278, win 227, options [nop,nop,TS val 108637944 ecr 731976258], length 0
20:23:50.001223 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881278:1554881304, ack 1062682905, win 2058, options [nop,nop,TS val 731976259 ecr 108637944], length 26
20:23:50.001880 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 0
20:23:50.002150 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [P.], seq 1062682905:1062683695, ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 79020:40:12.418726 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [F.], seq 1554881304, ack 1062684496, win 2048, options [nop,nop,TS val 732912682 ecr 108637947], length 0
20:40:12.418803 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [F.], seq 1827331724, ack 2130024795, win 2058, options [nop,nop,TS val 732912682 ecr 108637946], length 0
20:40:12.421643 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [F.], seq 1062684496, ack 1554881305, win 227, options [nop,nop,TS val 109620371 ecr 732912682], length 0
20:40:12.421695 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062684497, win 2048, options [nop,nop,TS val 732912685 ecr 109620371], length 0
20:40:12.422976 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [F.], seq 2130024795, ack 1827331725, win 227, options [nop,nop,TS val 109620372 ecr 732912682], length 0
20:40:12.423014 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024796, win 2058, options [nop,nop,TS val 732912686 ecr 109620372], length 0

中间过程发现在进行topology刷新的时候出现大于1的的TCP连接,这些多余的连接都发生在问题节点所在的ip上,clientName叫做:lettuce#ClusterTopologyRefresh

[root@localhost ~]# /data/server/redis-cluster/redis-5.0.5/src/redis-cli -c -h 192.168.64.15 -p 8052 client list |grep 64.64
id=287 addr=192.168.64.64:64321 fd=33 name=lettuce#ClusterTopologyRefresh age=1292 idle=1282 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=289 addr=192.168.64.64:64523 fd=35 name=lettuce#ClusterTopologyRefresh age=1111 idle=943 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=290 addr=192.168.64.64:64672 fd=36 name=lettuce#ClusterTopologyRefresh age=940 idle=940 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=286 addr=192.168.64.64:63831 fd=13 name= age=1849 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=5 oll=0 omem=0 events=r cmd=setex


Netty层的channel信号

中间过程当第一次出现读写超时事件时,netty是能够收到inactive事件(虽然这个时间点没有接收到TCP的对应信号,那么netty是怎么判断关闭TCP的呢?答: [R.] 就是这个reset信号量,这个就是最后的倔强)

然后就会触发scheduleReconnect方法(后续会阐述这个重连实现)尝试重连,这个过程中redis cluster正在进行主从切换,挡住从切换完成之后client端获取到正常的网络拓扑,读写恢复正常。

但是读写就算恢复正常,失去连接的节点底层还是会触发inactive事件,所以还是一直会报io.netty.channel.ConnectTimeoutException: connection timed out:/192.168.64.15:8052异常

java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:269) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 9 common frames omitted
20:58:19.845 WARN i.l.c.c.t.ClusterTopologyRefresh - Unable to connect to 192.168.64.13:8051
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_201]at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_201]at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 7 common frames omitted
Caused by: java.net.ConnectException: Connection refused... 11 common frames omitted

复现不出来,好痛苦,接下来就看看这个TCP的reset信号啥时候发送,以及是netty触发的还是TCP触发的。。。




推荐阅读
  • Nacos 0.3 数据持久化详解与实践
    本文详细介绍了如何将 Nacos 0.3 的数据持久化到 MySQL 数据库,并提供了具体的步骤和注意事项。 ... [详细]
  • MicrosoftDeploymentToolkit2010部署培训实验手册V1.0目录实验环境说明3实验环境虚拟机使用信息3注意:4实验手册正文说 ... [详细]
  • 思科IOS XE与ISE集成实现TACACS认证配置
    本文详细介绍了如何在思科IOS XE设备上配置TACACS认证,并通过ISE(Identity Services Engine)进行用户管理和授权。配置包括网络拓扑、设备设置和ISE端的具体步骤。 ... [详细]
  • 本文介绍了如何使用Python的Paramiko库批量更新多台服务器的登录密码。通过示例代码展示了具体实现方法,确保了操作的高效性和安全性。Paramiko库提供了强大的SSH2协议支持,使得远程服务器管理变得更加便捷。此外,文章还详细说明了代码的各个部分,帮助读者更好地理解和应用这一技术。 ... [详细]
  • SecureCRT是一款功能强大的终端仿真软件,支持SSH1和SSH2协议,适用于在Windows环境下高效连接和管理Linux服务器。该工具不仅提供了稳定的连接性能,还具备丰富的配置选项,能够满足不同用户的需求。通过SecureCRT,用户可以轻松实现对远程Linux系统的安全访问和操作。 ... [详细]
  • 本文介绍了如何利用Shell脚本高效地部署MHA(MySQL High Availability)高可用集群。通过详细的脚本编写和配置示例,展示了自动化部署过程中的关键步骤和注意事项。该方法不仅简化了集群的部署流程,还提高了系统的稳定性和可用性。 ... [详细]
  • Java Socket 关键参数详解与优化建议
    Java Socket 的 API 虽然被广泛使用,但其关键参数的用途却鲜为人知。本文详细解析了 Java Socket 中的重要参数,如 backlog 参数,它用于控制服务器等待连接请求的队列长度。此外,还探讨了其他参数如 SO_TIMEOUT、SO_REUSEADDR 等的配置方法及其对性能的影响,并提供了优化建议,帮助开发者提升网络通信的稳定性和效率。 ... [详细]
  • 在Cisco IOS XR系统中,存在提供服务的服务器和使用这些服务的客户端。本文深入探讨了进程与线程状态转换机制,分析了其在系统性能优化中的关键作用,并提出了改进措施,以提高系统的响应速度和资源利用率。通过详细研究状态转换的各个环节,本文为开发人员和系统管理员提供了实用的指导,旨在提升整体系统效率和稳定性。 ... [详细]
  • Python 伦理黑客技术:深入探讨后门攻击(第三部分)
    在《Python 伦理黑客技术:深入探讨后门攻击(第三部分)》中,作者详细分析了后门攻击中的Socket问题。由于TCP协议基于流,难以确定消息批次的结束点,这给后门攻击的实现带来了挑战。为了解决这一问题,文章提出了一系列有效的技术方案,包括使用特定的分隔符和长度前缀,以确保数据包的准确传输和解析。这些方法不仅提高了攻击的隐蔽性和可靠性,还为安全研究人员提供了宝贵的参考。 ... [详细]
  • 利用 Python Socket 实现 ICMP 协议下的网络通信
    在计算机网络课程的2.1实验中,学生需要通过Python Socket编程实现一种基于ICMP协议的网络通信功能。与操作系统自带的Ping命令类似,该实验要求学生开发一个简化的、非标准的ICMP通信程序,以加深对ICMP协议及其在网络通信中的应用的理解。通过这一实验,学生将掌握如何使用Python Socket库来构建和解析ICMP数据包,并实现基本的网络探测功能。 ... [详细]
  • 在使用 SQL Server 时,连接故障是用户最常见的问题之一。通常,连接 SQL Server 的方法有两种:一种是通过 SQL Server 自带的客户端工具,例如 SQL Server Management Studio;另一种是通过第三方应用程序或开发工具进行连接。本文将详细分析导致连接故障的常见原因,并提供相应的解决策略,帮助用户有效排除连接问题。 ... [详细]
  • 尽管我们尽最大努力,任何软件开发过程中都难免会出现缺陷。为了更有效地提升对支持部门的协助与支撑,本文探讨了多种策略和最佳实践,旨在通过改进沟通、增强培训和支持流程来减少这些缺陷的影响,并提高整体服务质量和客户满意度。 ... [详细]
  • 本文全面解析了 gRPC 的基础知识与高级应用,从 helloworld.proto 文件入手,详细阐述了如何定义服务接口。例如,`Greeter` 服务中的 `SayHello` 方法,该方法在客户端和服务器端的消息交互中起到了关键作用。通过实例代码,读者可以深入了解 gRPC 的工作原理及其在实际项目中的应用。 ... [详细]
  • 如果程序使用Go语言编写并涉及单向或双向TLS认证,可能会遭受CPU拒绝服务攻击(DoS)。本文深入分析了CVE-2018-16875漏洞,探讨其成因、影响及防范措施,为开发者提供全面的安全指导。 ... [详细]
  • 本文详细介绍了如何在 Linux 系统上安装 JDK 1.8、MySQL 和 Redis,并提供了相应的环境配置和验证步骤。 ... [详细]
author-avatar
mobiledu2502875063
这个家伙很懒,什么也没留下!
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有