首页    技术博客    PHP教程    数据库技术    前端开发   HTML5    Nginx    php论坛
新用户注册 | 会员登录
加入会员，解锁专属内容
取消
热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

Redis4集群failover后客户端未感知拓扑变化排查

作者：mobiledu2502875063 | 来源：互联网 | 2023-08-26 11:34
排查结论23：18分，30.1这台机器出现异常，其在30.2上的SlaveRedis服务检测到后提出升主请求并获得通过，成

排查结论


  1. 23&＃xff1a;18分&＃xff0c;30.1这台机器出现异常&＃xff0c;其在30.2上的Slave Redis服务检测到后提出升主请求并获得通过&＃xff0c;成为Master&＃xff1b;
  2. 23&＃xff1a;18至23&＃xff1a;34之间&＃xff0c;Redis client端对于Redis Server的连接检测并未异常&＃xff0c;没有触发拓扑更新&＃xff0c;因而对于Redis Server访问异常&＃xff1b;
  3. 23&＃xff1a;34&＃xff0c;SRE重启30.1&＃xff0c;此时 Redis Client检测到连接断开&＃xff0c;触发拓扑更新&＃xff0c;Redis访问正常。

解决方案

当前触发Redis Client更新拓扑的方式有两种&＃xff1a;1. 向服务端发请求返回MOVE或ASK&＃xff0c;2. ConnectionWatchDog检测到连接断开。当时的状况下&＃xff0c;服务器状态未知&＃xff08;待与SRE确认&＃xff09;&＃xff0c;两种条件都没有触发。

所以解决方案为&＃xff1a;Redis Client发新版&＃xff0c;增加定期更新拓扑。

问题背景

2021-03-11 oncall群里由SRE发现一台redis服务器故障&＃xff0c;通知如下

从而引发了帖子&＃xff0c;蛋卷&＃xff0c;UC&＃xff0c;推荐相关服务的报警&＃xff0c;redis的failover是在1min内完成的&＃xff0c;但是其中推荐的服务的告警持续了近15min恢复&＃xff0c;因此以下排查基于推荐服务进行

排查过程


时间线


redis服务器层面


节点描述

30.1

物理机在23:34:59完成了服务启动
30.2这个从节点在23:18:45完成了主从切换

30.1:
11 23:34:59 服务器启动
11 Mar 23:56:25.132 30.1:25701 实例启动;


30.2:
11 Mar 23:18:44 30.2 detect down
11 Mar 23:18:45.417 # configEpoch set to 8 after successful failover
11 Mar 23:56:26 30.1 slave full sync


推荐服务层面

读写操作分析

节点读写报错时间&＃xff1a;start读写报错时间&＃xff1a;end
160-11623:19:10开始出现读写报错23:30:25读写报错结束&＃xff08;这个时间进行了服务重启&＃xff09;
160-11923:19:10开始出现读写报错23:34:39读写报错结束&＃xff08;这个时间30.1那台redis物理机重新启动&＃xff09;

------------------160-116
app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |head
2021-03-11 23:19:10.664|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;4f8d63a2
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy139.zrevrangeWithScores(Unknown Source)

app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:30:25.627|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;d6ea39d
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy139.zrevrangeWithScores(Unknown Source)

------------------160-119
app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |head
2021-03-11 23:19:10.611|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;6897ac3e
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy134.zrevrangeWithScores(Unknown Source)

app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:34:39.158|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;3323f9a4
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy134.zrevrangeWithScores(Unknown Source)

lettuce发现connection异常分析

节点出现connection异常时间&＃xff1a;start出现connection异常时间&＃xff1a;end
160-11623:35:16出现connection异常提示23:35:16出现connection异常提示&＃xff0c;有且仅有这么一次&＃xff08;服务启动时发现拓扑的过程发生的错误&＃xff09;
160-11923:34:08出现connection异常提示&＃xff08;这个时间30.1那台redis物理机重新启动&＃xff09;23:42:35最后出现connection异常&＃xff08;23:42:26服务进行了重启&＃xff09;

------------------160-116
app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log|head
2021-03-11 23:35:16.180|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:35:16.230|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:35:16.180|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:35:16.230|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

------------------160-119
app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log|head
2021-03-11 23:34:08.452|WARN|i.l.c.p.ConnectionWatchdog||Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: /10.10.30.1:25071
2021-03-11 23:34:08.453|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:42:34.992|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:42:35.116|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)


监控图层面


节点QPS异常开始时间QPS回复时间
160-11623:19开始发生&＃xff08;与上面的读写操作日志分析对应上了&＃xff09;23:35开始恢复&＃xff08;与上面的lettuce发现connection日志异常分析对应上了&＃xff0c;这个是重启的结果不能够说明SDK拓扑更新的逻辑&＃xff09;
160-11923:19开始发生&＃xff08;与上面的读写操作日志分析对应上了&＃xff09;

23:34开始恢复&＃xff08;23:34服务没有重启&＃xff0c;QPS回复正常&＃xff0c;证明拓扑是更新了。

倾向认为&＃xff1a;23&＃xff1a;18~23&＃xff1a;34 redis-server30.1不响应redis-Command&＃xff0c;但是TCP未断连。

23&＃xff1a;34由于TCP断连触发拓扑刷新&＃xff09;



对应使用redis2.8的服务监控

帖子服务的2.8的redis client的P999是和上面吻合的

http://matrix.snowballfinance.com/d/ryrAxKXZk/ugc-snowflake-status-basic?orgId&＃61;1&from&＃61;1615474888737&to&＃61;1615478153783&fullscreen&panelId&＃61;8

UC服务的2.8的redis client的P999不吻合的&＃xff08;这个需要UC来解释&＃xff09;

http://matrix.snowballfinance.com/d/000000005/uc-snowflake-usercenter?orgId&＃61;1&fullscreen&edit&panelId&＃61;10&from&＃61;1615475760054&to&＃61;1615477519942


redis metric监控系统

redis的metric监控系统是golang编写的&＃xff0c;原理就是for循环进行监控数据采集

时间描述
在23:18的时候对30.1这个机器的节点进行getinfo的时候&＃xff0c;出现了阻塞
在23:34的时候30.1这个机器重新启动&＃xff0c;阻塞得到释放&＃xff08;从侧面印证了上述119这台机器监控和日志的解释&＃xff09;

http://matrix.snowballfinance.com/d/000000024/sre-redis-detail?orgId&＃61;1&from&＃61;1615474845742&to&＃61;1615480482137


redis所在物理机监控

30.1物理机处于什么状态&＃xff1f;导致18~34没有监控指标&＃xff1f;

23&＃xff1a;18-23&＃xff1a;34监控数据没有&＃xff0c;grafana使用了1条直线连接了23:18和23:34

http://matrix.snowballfinance.com/d/000000012/sre-wu-li-ji-detail?orgId&＃61;1&from&＃61;1615475297782&to&＃61;1615478502199&var-Hostname&＃61;bjxg-rs-30-1


原因分析


tcp/ip跟本次事件相关的原理背景

TCP 建立连接要进行三次握手&＃xff08;有对应的SYN信号&＃xff09;&＃xff0c;而断开连接要进行四次&＃xff08;有对应的FIN信号&＃xff0c;且都是seq的序列号来进行确认&＃xff09;。这是由于TCP 的半关闭造成的。这也是为什么四次挥手要比三次握手多一次&＃xff0c;因为TCP 连接是全双工的(即数据可在两个方向上同时传递)所以进行关闭时每个方向上都要单独进行关闭。单方向的关闭就叫半关闭。

TCP正常创建流程对fd的处理流程&＃xff0c;TCP正常close和shutdown的处理流程&＃xff0c;那么tcp连接异常终端后对应的socket会做些什么&＃xff1f;

TCP/IP可选特性Keepalive&＃xff08;默认关闭&＃xff09;
有3个参数&＃xff0c;以linux为例&＃xff0c;搜索keepalive相关的系统参数&＃xff08;sysctl -A |grep keepalive&＃xff09;&＃xff0c;3个参数组合起来表示&＃xff1a;

如果TCP连接2小时&＃xff08;7200秒&＃xff09;内没有数据传输&＃xff0c;则发送嗅探包&＃xff0c;每隔75秒发送一次&＃xff0c;共重试9次。9次对方都没响应&＃xff0c;则表明此连接已死。&＃xff08;留个问题&＃xff0c;这个时候socket会给fd发送什么信号&＃xff1f;&＃xff1f;&＃xff09;

以linux为例&＃xff1a;
         recv的man对返回值的介绍如下&＃xff1a;
              RETURN VALUE
              These calls return the number of bytes received, or -1 if an error occurred. The return value will be 0 when the peer has performed an orderly shutdown.
          即&＃xff0c;recv返回0表示对端关闭。对于send&＃xff0c;如果返回值为-1&＃xff0c;且errno为ECONNRESET&＃xff0c;表示对端关闭。

一般服务器程序不会用到这个机制&＃xff0c;而是把Keepalive放到业务层&＃xff0c;自己实现Keepalive机制。这样做的好处是可以减少程序对特定协议的依赖&＃xff0c;且发送给客户端的请求可以组合起来&＃xff0c;在心跳包到来的时候一并返回给客户端。

摘自《TCP/IP详解》卷1第23章&＃xff0c;强烈建议点开链接看看&＃xff0c;解释的真通俗易懂

主要本次关注以下几点&＃xff1a;

1.没有收到FIN信号&＃xff0c;没有keepalive&＃xff0c;那么该TCP会在两个小时之后尝试嗅探&＃xff08;天塌了它也不管&＃xff09;

2.对端服务器崩溃、关机、正在重启、网络不可达&＃xff0c;那么该TCP的上层通常是报告连接超时&＃xff08;留个问题&＃xff0c;信号量是多少&＃xff1f;&＃xff1f;&＃xff09;

3.对端服务重启之后&＃xff0c;那么该TCP得到的回复是RST

 

 


netty跟本次事件相关的原理背景

netty这部分没啥好说的&＃xff0c;看了好多的资料&＃xff0c;总结下来关于本次事件相关的连接信息有以下内容&＃xff1a;

1.NioEventLoop&＃xff0c;reactor线程在自旋中执行细节

accept事件接收连接

2.workEventLoop&＃xff0c;上一步的在接收完链接后&＃xff0c;会触发相关读事件( fireChannelRead)&＃xff0c;以及读完成事件( fireChannelReadComplete)对于server端来说&＃xff0c;会设置对应selectKeys事件设置为 读事件监听。

由于会去pipeline中调用每个channelHandler进行事件处理&＃xff0c;所以在nio 的accept事件之后&＃xff0c;会执行 fireChannelRead 方法。

读事件第一个处理为 headContext。而后由 HeadContext 将事件进行向后传播。DefaultChannelPipeline 的 HeadContext。

3.close 与 channelInactive 的关系是 close 是主动关闭 channel 的动作, 而 channelInactive 是关闭成功后收到通知的事件.

关于netty的tcp参数设置

通过Netty配置TCP心跳&＃xff0c;或者泛化的说配置TCP协议参数的方式有两种&＃xff1a;
1、childOption方式是正确的
2、option方法不起作用&＃xff0c;也不能说错吧&＃xff0c;就是没啥意义而已

即本质就是一种方式。另外在4.1.36以后版本&＃xff0c;如果你是JDK7以上&＃xff0c;那么可以直接这样配置客户端连接的协议参数&＃xff1a;.childOption(NioChannelOption.of(StandardSocketOptions.SO_KEEPALIVE),true)

网络层复现

背景介绍&＃xff1a;使用的64.169/64.15这两台虚拟机用来当做服务器&＃xff0c;本地开发机当做client

然后通过IPtable drop来尝试模拟服务节点突然下线

网络层的TCP信号

经历了开始的三次握手&＃xff08;SYN信号&＃xff09; ------→ 数据交互&＃xff0c;正常读写&＃xff08;P.信号&＃xff09; ------→ 其中一个节点连接突然中断&＃xff08;R.信号&＃xff09; ------→ 尝试重连&＃xff08;不断的S信号&＃xff09; ------→ 连接恢复&＃xff08;SYN信号&＃xff09; ------→ 恢复正常读写&＃xff08;P.信号&＃xff09; ------→ 最后的链接断开&＃xff08;FIN信号&＃xff09;

20:08:39.861243 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [S], seq 1251904794, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071045 ecr 0,sackOK,eol], length 0
20:08:39.861536 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [S.], seq 4226972078, ack 1251904795, win 28960, options [mss 1460,sackOK,TS val 107727801 ecr 731071045,nop,wscale 7], length 0
20:08:39.861574 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972079, win 2058, options [nop,nop,TS val 731071045 ecr 107727801], length 020:08:39.862538 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [P.], seq 1251904795:1251904861, ack 4226972079, win 2058, options [nop,nop,TS val 731071046 ecr 107727801], length 66
20:08:39.862838 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [.], ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 0
20:08:39.863019 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [P.], seq 4226972079:4226972084, ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 5
20:08:39.863065 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972084, win 2058, options [nop,nop,TS val 731071046 ecr 107727802], length 020:08:39.865550 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [S], seq 4104613940, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071048 ecr 0,sackOK,eol], length 0
20:08:39.865820 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [S.], seq 413298472, ack 4104613941, win 28960, options [mss 1460,sackOK,TS val 107727805 ecr 731071048,nop,wscale 7], length 0
20:08:39.865868 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298473, win 2058, options [nop,nop,TS val 731071049 ecr 107727805], length 020:08:39.866943 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [P.], seq 4104613941:4104614007, ack 413298473, win 2058, options [nop,nop,TS val 731071050 ecr 107727805], length 66
20:08:39.867192 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [.], ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 0
20:08:39.867329 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [P.], seq 413298473:413298478, ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 5
20:08:39.867378 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298478, win 2058, options [nop,nop,TS val 731071050 ecr 107727807], length 014
20:16:00.604659 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509045 ecr 107733081], length 48
20:16:00.754648 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509195 ecr 107733081], length 48
20:16:01.055889 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509495 ecr 107733081], length 48
20:16:01.456264 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509895 ecr 107733081], length 48
20:16:02.062739 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731510495 ecr 107733081], length 48
20:16:03.065437 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731511495 ecr 107733081], length 48
20:16:04.871576 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731513295 ecr 107733081], length 48
20:16:08.287194 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731516695 ecr 107733081], length 48
20:16:14.912828 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731523295 ecr 107733081], length 48
20:16:21.529555 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731529895 ecr 107733081], length 48
20:16:28.147866 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731536495 ecr 107733081], length 48
20:16:34.771964 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731543095 ecr 107733081], length 48
20:16:41.398904 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731549695 ecr 107733081], length 48
20:16:48.025246 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731556295 ecr 107733081], length 48
20:16:54.655115 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [R.], seq 4241911420, ack 1651645875, win 2048, length 07
20:17:17.609293 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731585757 ecr 0,sackOK,eol], length 0
20:17:18.610567 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731586757 ecr 0,sackOK,eol], length 0
20:17:19.613758 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731587757 ecr 0,sackOK,eol], length 0
20:17:20.617512 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731588758 ecr 0,sackOK,eol], length 0
20:17:21.620555 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731589758 ecr 0,sackOK,eol], length 0
20:17:22.625515 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731590758 ecr 0,sackOK,eol], length 0
20:17:24.631276 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731592758 ecr 0,sackOK,eol], length 011
20:17:27.661985 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595780 ecr 0,sackOK,eol], length 0
20:17:27.805463 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595921 ecr 0,sackOK,eol], length 0
20:17:27.906594 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596021 ecr 0,sackOK,eol], length 0
20:17:28.007657 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596122 ecr 0,sackOK,eol], length 0
20:17:28.108495 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596222 ecr 0,sackOK,eol], length 0
20:17:28.211396 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596322 ecr 0,sackOK,eol], length 0
20:17:28.412573 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596523 ecr 0,sackOK,eol], length 0
20:17:28.813800 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596923 ecr 0,sackOK,eol], length 0
20:17:29.617595 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731597723 ecr 0,sackOK,eol], length 0
20:17:31.226228 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731599323 ecr 0,sackOK,eol], length 0
20:17:34.434295 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:37.766336 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605835 ecr 0,sackOK,eol], length 0
20:17:37.907801 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605976 ecr 0,sackOK,eol], length 0
20:17:38.009930 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606076 ecr 0,sackOK,eol], length 0
20:17:38.111023 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606177 ecr 0,sackOK,eol], length 0
20:17:38.212162 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606277 ecr 0,sackOK,eol], length 0
20:17:38.312800 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606377 ecr 0,sackOK,eol], length 0
20:17:38.513278 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606577 ecr 0,sackOK,eol], length 0
20:17:38.913743 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606977 ecr 0,sackOK,eol], length 0
20:17:39.718061 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731607777 ecr 0,sackOK,eol], length 0
20:17:41.323608 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731609377 ecr 0,sackOK,eol], length 0
20:17:44.538196 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:47.867437 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731615897 ecr 0,sackOK,eol], length 0
20:17:48.008751 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616038 ecr 0,sackOK,eol], length 0
20:17:48.109361 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616138 ecr 0,sackOK,eol], length 0
20:17:48.209657 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616238 ecr 0,sackOK,eol], length 0
20:17:48.312000 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616338 ecr 0,sackOK,eol], length 0
20:17:48.413063 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616439 ecr 0,sackOK,eol], length 0
20:17:48.613676 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616639 ecr 0,sackOK,eol], length 0
20:17:49.015418 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617039 ecr 0,sackOK,eol], length 0
20:17:49.818549 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617839 ecr 0,sackOK,eol], length 0
20:17:51.422985 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731619439 ecr 0,sackOK,eol], length 0
20:17:54.631568 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:57.969994 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0
20:17:57.970906 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0
20:17:58.115551 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0
20:17:58.115561 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0
20:17:58.217038 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0
20:17:58.217049 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0
20:17:58.317493 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0
20:17:58.317504 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0
20:17:58.418207 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0
20:17:58.418219 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0
20:17:58.519006 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0
20:17:58.519018 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0
20:17:58.721709 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0
20:17:58.721721 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0
20:17:59.123040 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0
20:17:59.123052 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0
20:17:59.925894 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0
20:17:59.925910 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0
20:18:01.530428 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0
20:18:01.530436 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0
20:18:04.739272 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:04.739280 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,sackOK,eol], length 013
20:18:08.063149 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636018 ecr 0,sackOK,eol], length 0
20:18:08.204712 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636159 ecr 0,sackOK,eol], length 0
20:18:08.305275 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636259 ecr 0,sackOK,eol], length 0
20:18:08.406168 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636359 ecr 0,sackOK,eol], length 0
20:18:08.507152 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636459 ecr 0,sackOK,eol], length 0
20:18:08.607218 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636559 ecr 0,sackOK,eol], length 0
20:18:08.808894 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636759 ecr 0,sackOK,eol], length 0
20:18:09.209305 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637159 ecr 0,sackOK,eol], length 0
20:18:10.011676 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637959 ecr 0,sackOK,eol], length 0
20:18:11.616529 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731639559 ecr 0,sackOK,eol], length 0
20:18:14.828898 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:21.286606 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:27.721777 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 013
20:20:58.899104 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805899 ecr 0,sackOK,eol], length 0
20:20:58.900162 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805900 ecr 0,sackOK,eol], length 0
20:20:59.045755 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806040 ecr 0,sackOK,eol], length 0
20:20:59.047010 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806041 ecr 0,sackOK,eol], length 0
20:20:59.146241 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806140 ecr 0,sackOK,eol], length 0
20:20:59.147300 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806141 ecr 0,sackOK,eol], length 0
20:20:59.247515 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806240 ecr 0,sackOK,eol], length 0
20:20:59.248746 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806241 ecr 0,sackOK,eol], length 0
20:20:59.347871 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806340 ecr 0,sackOK,eol], length 0
20:20:59.348974 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806341 ecr 0,sackOK,eol], length 0
20:20:59.449197 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0
20:20:59.449211 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0
20:20:59.654647 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0
20:20:59.654653 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0
20:21:00.056939 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0
20:21:00.056958 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0
20:21:00.867762 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0
20:21:00.867778 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0
20:21:02.484895 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0
20:21:02.484903 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0
20:21:05.715910 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:05.715916 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:12.139150 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:12.139164 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:18.570932 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:18.570946 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 020:23:49.944343 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [S], seq 1827331675, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976206 ecr 0,sackOK,eol], length 0
20:23:49.944875 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [S], seq 1554881183, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976207 ecr 0,sackOK,eol], length 0
20:23:49.954078 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [S.], seq 2130024761, ack 1827331676, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976206,nop,wscale 7], length 0
20:23:49.954085 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [S.], seq 1062682904, ack 1554881184, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976207,nop,wscale 7], length 0
20:23:49.954203 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024762, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0
20:23:49.954257 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062682905, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0
20:23:49.957101 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [P.], seq 1827331676:1827331724, ack 2130024762, win 2058, options [nop,nop,TS val 731976217 ecr 108637891], length 48
20:23:49.959283 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881184:1554881250, ack 1062682905, win 2058, options [nop,nop,TS val 731976218 ecr 108637891], length 66
20:23:49.999821 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [.], ack 1827331724, win 227, options [nop,nop,TS val 108637944 ecr 731976217], length 0
20:23:49.999822 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881250, win 227, options [nop,nop,TS val 108637944 ecr 731976218], length 0
20:23:50.000112 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881250:1554881278, ack 1062682905, win 2058, options [nop,nop,TS val 731976258 ecr 108637944], length 28
20:23:50.000541 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881278, win 227, options [nop,nop,TS val 108637944 ecr 731976258], length 0
20:23:50.001223 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881278:1554881304, ack 1062682905, win 2058, options [nop,nop,TS val 731976259 ecr 108637944], length 26
20:23:50.001880 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 0
20:23:50.002150 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [P.], seq 1062682905:1062683695, ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 79020:40:12.418726 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [F.], seq 1554881304, ack 1062684496, win 2048, options [nop,nop,TS val 732912682 ecr 108637947], length 0
20:40:12.418803 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [F.], seq 1827331724, ack 2130024795, win 2058, options [nop,nop,TS val 732912682 ecr 108637946], length 0
20:40:12.421643 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [F.], seq 1062684496, ack 1554881305, win 227, options [nop,nop,TS val 109620371 ecr 732912682], length 0
20:40:12.421695 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062684497, win 2048, options [nop,nop,TS val 732912685 ecr 109620371], length 0
20:40:12.422976 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [F.], seq 2130024795, ack 1827331725, win 227, options [nop,nop,TS val 109620372 ecr 732912682], length 0
20:40:12.423014 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024796, win 2058, options [nop,nop,TS val 732912686 ecr 109620372], length 0

中间过程发现在进行topology刷新的时候出现大于1的的TCP连接&＃xff0c;这些多余的连接都发生在问题节点所在的ip上&＃xff0c;clientName叫做&＃xff1a;lettuce#ClusterTopologyRefresh

[root&＃64;localhost ~]# /data/server/redis-cluster/redis-5.0.5/src/redis-cli -c -h 192.168.64.15 -p 8052 client list |grep 64.64
id&＃61;287 addr&＃61;192.168.64.64:64321 fd&＃61;33 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;1292 idle&＃61;1282 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;289 addr&＃61;192.168.64.64:64523 fd&＃61;35 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;1111 idle&＃61;943 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;290 addr&＃61;192.168.64.64:64672 fd&＃61;36 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;940 idle&＃61;940 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;286 addr&＃61;192.168.64.64:63831 fd&＃61;13 name&＃61; age&＃61;1849 idle&＃61;0 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;32768 obl&＃61;5 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;setex


Netty层的channel信号

中间过程当第一次出现读写超时事件时&＃xff0c;netty是能够收到inactive事件&＃xff08;虽然这个时间点没有接收到TCP的对应信号&＃xff0c;那么netty是怎么判断关闭TCP的呢&＃xff1f;答&＃xff1a; [R.] 就是这个reset信号量&＃xff0c;这个就是最后的倔强&＃xff09;

然后就会触发scheduleReconnect方法&＃xff08;后续会阐述这个重连实现&＃xff09;尝试重连&＃xff0c;这个过程中redis cluster正在进行主从切换&＃xff0c;挡住从切换完成之后client端获取到正常的网络拓扑&＃xff0c;读写恢复正常。

但是读写就算恢复正常&＃xff0c;失去连接的节点底层还是会触发inactive事件&＃xff0c;所以还是一直会报io.netty.channel.ConnectTimeoutException: connection timed out:/192.168.64.15:8052异常

java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:269) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 9 common frames omitted
20:58:19.845 WARN i.l.c.c.t.ClusterTopologyRefresh - Unable to connect to 192.168.64.13:8051
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_201]at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_201]at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 7 common frames omitted
Caused by: java.net.ConnectException: Connection refused... 11 common frames omitted

复现不出来&＃xff0c;好痛苦&＃xff0c;接下来就看看这个TCP的reset信号啥时候发送&＃xff0c;以及是netty触发的还是TCP触发的。。。




推荐阅读
  • lua

    电商高并发解决方案详解

    电商高并发解决方案详解
    本文以京东为例，详细探讨了电商中常见的高并发解决方案，包括多级缓存和Nginx限流技术，旨在帮助读者更好地理解和应用这些技术。 ... [详细]
  • get

    深入理解Java多线程与并发机制

    深入理解Java多线程与并发机制
    本文探讨了Java多线程和并发机制的核心概念，包括多线程类的分类、执行器框架、并发容器及控制工具。通过详细解析这些组件，帮助开发者更好地理解和应用多线程技术。 ... [详细]
  • ascii

    Redis 数据类型及其应用场景

    本文详细介绍了 Redis 中的主要数据类型，包括 String、Hash、List、Set、ZSet、Geo 和 HyperLogLog，并提供了每种类型的基本操作命令和应用场景。 ... [详细]
  • get

    使用Java ProcessBuilder类调用外部程序的方法详解

    在Java开发中，如何利用ProcessBuilder类调用外部程序是一个常见的需求。本文将详细介绍ProcessBuilder类的使用方法，并提供示例代码帮助你更好地理解和应用。 ... [详细]
  • uri

    2023年7月7日网络安全动态

    汇总了2023年7月7日最新的网络安全新闻和技术更新，包括最新的漏洞披露、工具发布及安全事件。 ... [详细]
  • object

    LaTeX高级应用技巧

    本文分享了作者在使用LaTeX过程中的几点心得，涵盖了从文档编辑、代码高亮、图形绘制到3D模型展示等多个方面的内容。适合希望深入了解LaTeX高级功能的用户。 ... [详细]
  • input

    2019年独角兽企业招聘Python工程师标准课程概览

    2019年独角兽企业招聘Python工程师标准课程概览
    本文详细介绍了2019年独角兽企业在招聘Python工程师时的标准课程内容，包括Shell脚本中的逻辑判断、文件属性判断、if语句的特殊用法及case语句的应用。 ... [详细]
  • get

    C# 中创建和执行存储过程的方法

    C# 中创建和执行存储过程的方法
    本文详细介绍了如何使用 C# 创建和调用 SQL Server 存储过程，包括连接数据库、定义命令类型、设置参数等步骤。 ... [详细]
  • get

    UE4 中的距离场技术详解

    本文将深入探讨 Unreal Engine 4 (UE4) 中的距离场技术，包括其原理、实现细节以及在渲染中的应用。距离场技术在现代游戏引擎中用于提高光照和阴影的效果，尤其是在处理复杂几何形状时。文章将结合具体代码示例，帮助读者更好地理解和应用这一技术。 ... [详细]
  • string

    Redis：缓存与内存数据库详解

    Redis：缓存与内存数据库详解
    本文介绍了数据库的基本分类，重点探讨了关系型与非关系型数据库的区别，并详细解析了Redis作为非关系型数据库的特点、工作模式、优点及持久化机制。 ... [详细]
  • get

    Linux环境下MySQL的安装与配置

    Linux环境下MySQL的安装与配置
    本文详细介绍了在Linux系统上安装和配置MySQL的步骤，包括安装前的准备工作、下载和解压安装包、初始化数据库、配置文件编辑、启动服务以及设置开机自启动等。 ... [详细]
  • bit

    Mac平台下的MySQL安装与配置指南

    本文详细介绍了在Mac平台上安装和配置MySQL的步骤，包括下载安装包、卸载MySQL以及解决命令行中找不到mysql命令的问题。 ... [详细]
  • callback

    Spring Boot + RabbitMQ 消息确认机制详解

    Spring Boot + RabbitMQ 消息确认机制详解
    本文详细介绍如何在 Spring Boot 项目中使用 RabbitMQ 的消息确认机制，包括消息发送确认和消息接收确认，帮助开发者解决在实际操作中可能遇到的问题。 ... [详细]
  • object

    深入解析 Redis 的数据结构与对象系统

    Redis 是一个高性能的开源键值存储系统，支持多种数据结构。本文将详细介绍 Redis 中的六种底层数据结构及其在对象系统中的应用，包括字符串对象、列表对象、哈希对象、集合对象和有序集合对象。通过12张图解，帮助读者全面理解 Redis 的数据结构和对象系统。 ... [详细]
  • bit

    使用 Jupyter Notebook 实现 Markdown 编写与代码运行

    使用 Jupyter Notebook 实现 Markdown 编写与代码运行
    Jupyter Notebook 是一个开源的基于网页的应用程序，允许用户在同一文档中编写 Markdown 文本和运行多种编程语言的代码，并实时查看运行结果。 ... [详细]
author-avatar
mobiledu2502875063
这个家伙很懒，什么也没留下！
Tags | 热门标签
RankList | 热门文章
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有