首页    技术博客    PHP教程    数据库技术    前端开发   HTML5    Nginx    php论坛
新用户注册 | 会员登录
加入会员，解锁专属内容
取消
热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

Redis4集群failover后客户端未感知拓扑变化排查

作者：mobiledu2502875063 | 来源：互联网 | 2023-08-26 11:34
排查结论23：18分，30.1这台机器出现异常，其在30.2上的SlaveRedis服务检测到后提出升主请求并获得通过，成

排查结论


  1. 23&＃xff1a;18分&＃xff0c;30.1这台机器出现异常&＃xff0c;其在30.2上的Slave Redis服务检测到后提出升主请求并获得通过&＃xff0c;成为Master&＃xff1b;
  2. 23&＃xff1a;18至23&＃xff1a;34之间&＃xff0c;Redis client端对于Redis Server的连接检测并未异常&＃xff0c;没有触发拓扑更新&＃xff0c;因而对于Redis Server访问异常&＃xff1b;
  3. 23&＃xff1a;34&＃xff0c;SRE重启30.1&＃xff0c;此时 Redis Client检测到连接断开&＃xff0c;触发拓扑更新&＃xff0c;Redis访问正常。

解决方案

当前触发Redis Client更新拓扑的方式有两种&＃xff1a;1. 向服务端发请求返回MOVE或ASK&＃xff0c;2. ConnectionWatchDog检测到连接断开。当时的状况下&＃xff0c;服务器状态未知&＃xff08;待与SRE确认&＃xff09;&＃xff0c;两种条件都没有触发。

所以解决方案为&＃xff1a;Redis Client发新版&＃xff0c;增加定期更新拓扑。

问题背景

2021-03-11 oncall群里由SRE发现一台redis服务器故障&＃xff0c;通知如下

从而引发了帖子&＃xff0c;蛋卷&＃xff0c;UC&＃xff0c;推荐相关服务的报警&＃xff0c;redis的failover是在1min内完成的&＃xff0c;但是其中推荐的服务的告警持续了近15min恢复&＃xff0c;因此以下排查基于推荐服务进行

排查过程


时间线


redis服务器层面


节点描述

30.1

物理机在23:34:59完成了服务启动
30.2这个从节点在23:18:45完成了主从切换

30.1:
11 23:34:59 服务器启动
11 Mar 23:56:25.132 30.1:25701 实例启动;


30.2:
11 Mar 23:18:44 30.2 detect down
11 Mar 23:18:45.417 # configEpoch set to 8 after successful failover
11 Mar 23:56:26 30.1 slave full sync


推荐服务层面

读写操作分析

节点读写报错时间&＃xff1a;start读写报错时间&＃xff1a;end
160-11623:19:10开始出现读写报错23:30:25读写报错结束&＃xff08;这个时间进行了服务重启&＃xff09;
160-11923:19:10开始出现读写报错23:34:39读写报错结束&＃xff08;这个时间30.1那台redis物理机重新启动&＃xff09;

------------------160-116
app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |head
2021-03-11 23:19:10.664|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;4f8d63a2
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy139.zrevrangeWithScores(Unknown Source)

app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:30:25.627|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;d6ea39d
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy139.zrevrangeWithScores(Unknown Source)

------------------160-119
app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |head
2021-03-11 23:19:10.611|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;6897ac3e
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy134.zrevrangeWithScores(Unknown Source)

app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:34:39.158|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;3323f9a4
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy134.zrevrangeWithScores(Unknown Source)

lettuce发现connection异常分析

节点出现connection异常时间&＃xff1a;start出现connection异常时间&＃xff1a;end
160-11623:35:16出现connection异常提示23:35:16出现connection异常提示&＃xff0c;有且仅有这么一次&＃xff08;服务启动时发现拓扑的过程发生的错误&＃xff09;
160-11923:34:08出现connection异常提示&＃xff08;这个时间30.1那台redis物理机重新启动&＃xff09;23:42:35最后出现connection异常&＃xff08;23:42:26服务进行了重启&＃xff09;

------------------160-116
app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log|head
2021-03-11 23:35:16.180|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:35:16.230|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:35:16.180|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:35:16.230|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

------------------160-119
app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log|head
2021-03-11 23:34:08.452|WARN|i.l.c.p.ConnectionWatchdog||Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: /10.10.30.1:25071
2021-03-11 23:34:08.453|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:42:34.992|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:42:35.116|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)


监控图层面


节点QPS异常开始时间QPS回复时间
160-11623:19开始发生&＃xff08;与上面的读写操作日志分析对应上了&＃xff09;23:35开始恢复&＃xff08;与上面的lettuce发现connection日志异常分析对应上了&＃xff0c;这个是重启的结果不能够说明SDK拓扑更新的逻辑&＃xff09;
160-11923:19开始发生&＃xff08;与上面的读写操作日志分析对应上了&＃xff09;

23:34开始恢复&＃xff08;23:34服务没有重启&＃xff0c;QPS回复正常&＃xff0c;证明拓扑是更新了。

倾向认为&＃xff1a;23&＃xff1a;18~23&＃xff1a;34 redis-server30.1不响应redis-Command&＃xff0c;但是TCP未断连。

23&＃xff1a;34由于TCP断连触发拓扑刷新&＃xff09;



对应使用redis2.8的服务监控

帖子服务的2.8的redis client的P999是和上面吻合的

http://matrix.snowballfinance.com/d/ryrAxKXZk/ugc-snowflake-status-basic?orgId&＃61;1&from&＃61;1615474888737&to&＃61;1615478153783&fullscreen&panelId&＃61;8

UC服务的2.8的redis client的P999不吻合的&＃xff08;这个需要UC来解释&＃xff09;

http://matrix.snowballfinance.com/d/000000005/uc-snowflake-usercenter?orgId&＃61;1&fullscreen&edit&panelId&＃61;10&from&＃61;1615475760054&to&＃61;1615477519942


redis metric监控系统

redis的metric监控系统是golang编写的&＃xff0c;原理就是for循环进行监控数据采集

时间描述
在23:18的时候对30.1这个机器的节点进行getinfo的时候&＃xff0c;出现了阻塞
在23:34的时候30.1这个机器重新启动&＃xff0c;阻塞得到释放&＃xff08;从侧面印证了上述119这台机器监控和日志的解释&＃xff09;

http://matrix.snowballfinance.com/d/000000024/sre-redis-detail?orgId&＃61;1&from&＃61;1615474845742&to&＃61;1615480482137


redis所在物理机监控

30.1物理机处于什么状态&＃xff1f;导致18~34没有监控指标&＃xff1f;

23&＃xff1a;18-23&＃xff1a;34监控数据没有&＃xff0c;grafana使用了1条直线连接了23:18和23:34

http://matrix.snowballfinance.com/d/000000012/sre-wu-li-ji-detail?orgId&＃61;1&from&＃61;1615475297782&to&＃61;1615478502199&var-Hostname&＃61;bjxg-rs-30-1


原因分析


tcp/ip跟本次事件相关的原理背景

TCP 建立连接要进行三次握手&＃xff08;有对应的SYN信号&＃xff09;&＃xff0c;而断开连接要进行四次&＃xff08;有对应的FIN信号&＃xff0c;且都是seq的序列号来进行确认&＃xff09;。这是由于TCP 的半关闭造成的。这也是为什么四次挥手要比三次握手多一次&＃xff0c;因为TCP 连接是全双工的(即数据可在两个方向上同时传递)所以进行关闭时每个方向上都要单独进行关闭。单方向的关闭就叫半关闭。

TCP正常创建流程对fd的处理流程&＃xff0c;TCP正常close和shutdown的处理流程&＃xff0c;那么tcp连接异常终端后对应的socket会做些什么&＃xff1f;

TCP/IP可选特性Keepalive&＃xff08;默认关闭&＃xff09;
有3个参数&＃xff0c;以linux为例&＃xff0c;搜索keepalive相关的系统参数&＃xff08;sysctl -A |grep keepalive&＃xff09;&＃xff0c;3个参数组合起来表示&＃xff1a;

如果TCP连接2小时&＃xff08;7200秒&＃xff09;内没有数据传输&＃xff0c;则发送嗅探包&＃xff0c;每隔75秒发送一次&＃xff0c;共重试9次。9次对方都没响应&＃xff0c;则表明此连接已死。&＃xff08;留个问题&＃xff0c;这个时候socket会给fd发送什么信号&＃xff1f;&＃xff1f;&＃xff09;

以linux为例&＃xff1a;
         recv的man对返回值的介绍如下&＃xff1a;
              RETURN VALUE
              These calls return the number of bytes received, or -1 if an error occurred. The return value will be 0 when the peer has performed an orderly shutdown.
          即&＃xff0c;recv返回0表示对端关闭。对于send&＃xff0c;如果返回值为-1&＃xff0c;且errno为ECONNRESET&＃xff0c;表示对端关闭。

一般服务器程序不会用到这个机制&＃xff0c;而是把Keepalive放到业务层&＃xff0c;自己实现Keepalive机制。这样做的好处是可以减少程序对特定协议的依赖&＃xff0c;且发送给客户端的请求可以组合起来&＃xff0c;在心跳包到来的时候一并返回给客户端。

摘自《TCP/IP详解》卷1第23章&＃xff0c;强烈建议点开链接看看&＃xff0c;解释的真通俗易懂

主要本次关注以下几点&＃xff1a;

1.没有收到FIN信号&＃xff0c;没有keepalive&＃xff0c;那么该TCP会在两个小时之后尝试嗅探&＃xff08;天塌了它也不管&＃xff09;

2.对端服务器崩溃、关机、正在重启、网络不可达&＃xff0c;那么该TCP的上层通常是报告连接超时&＃xff08;留个问题&＃xff0c;信号量是多少&＃xff1f;&＃xff1f;&＃xff09;

3.对端服务重启之后&＃xff0c;那么该TCP得到的回复是RST

 

 


netty跟本次事件相关的原理背景

netty这部分没啥好说的&＃xff0c;看了好多的资料&＃xff0c;总结下来关于本次事件相关的连接信息有以下内容&＃xff1a;

1.NioEventLoop&＃xff0c;reactor线程在自旋中执行细节

accept事件接收连接

2.workEventLoop&＃xff0c;上一步的在接收完链接后&＃xff0c;会触发相关读事件( fireChannelRead)&＃xff0c;以及读完成事件( fireChannelReadComplete)对于server端来说&＃xff0c;会设置对应selectKeys事件设置为 读事件监听。

由于会去pipeline中调用每个channelHandler进行事件处理&＃xff0c;所以在nio 的accept事件之后&＃xff0c;会执行 fireChannelRead 方法。

读事件第一个处理为 headContext。而后由 HeadContext 将事件进行向后传播。DefaultChannelPipeline 的 HeadContext。

3.close 与 channelInactive 的关系是 close 是主动关闭 channel 的动作, 而 channelInactive 是关闭成功后收到通知的事件.

关于netty的tcp参数设置

通过Netty配置TCP心跳&＃xff0c;或者泛化的说配置TCP协议参数的方式有两种&＃xff1a;
1、childOption方式是正确的
2、option方法不起作用&＃xff0c;也不能说错吧&＃xff0c;就是没啥意义而已

即本质就是一种方式。另外在4.1.36以后版本&＃xff0c;如果你是JDK7以上&＃xff0c;那么可以直接这样配置客户端连接的协议参数&＃xff1a;.childOption(NioChannelOption.of(StandardSocketOptions.SO_KEEPALIVE),true)

网络层复现

背景介绍&＃xff1a;使用的64.169/64.15这两台虚拟机用来当做服务器&＃xff0c;本地开发机当做client

然后通过IPtable drop来尝试模拟服务节点突然下线

网络层的TCP信号

经历了开始的三次握手&＃xff08;SYN信号&＃xff09; ------→ 数据交互&＃xff0c;正常读写&＃xff08;P.信号&＃xff09; ------→ 其中一个节点连接突然中断&＃xff08;R.信号&＃xff09; ------→ 尝试重连&＃xff08;不断的S信号&＃xff09; ------→ 连接恢复&＃xff08;SYN信号&＃xff09; ------→ 恢复正常读写&＃xff08;P.信号&＃xff09; ------→ 最后的链接断开&＃xff08;FIN信号&＃xff09;

20:08:39.861243 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [S], seq 1251904794, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071045 ecr 0,sackOK,eol], length 0
20:08:39.861536 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [S.], seq 4226972078, ack 1251904795, win 28960, options [mss 1460,sackOK,TS val 107727801 ecr 731071045,nop,wscale 7], length 0
20:08:39.861574 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972079, win 2058, options [nop,nop,TS val 731071045 ecr 107727801], length 020:08:39.862538 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [P.], seq 1251904795:1251904861, ack 4226972079, win 2058, options [nop,nop,TS val 731071046 ecr 107727801], length 66
20:08:39.862838 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [.], ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 0
20:08:39.863019 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [P.], seq 4226972079:4226972084, ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 5
20:08:39.863065 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972084, win 2058, options [nop,nop,TS val 731071046 ecr 107727802], length 020:08:39.865550 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [S], seq 4104613940, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071048 ecr 0,sackOK,eol], length 0
20:08:39.865820 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [S.], seq 413298472, ack 4104613941, win 28960, options [mss 1460,sackOK,TS val 107727805 ecr 731071048,nop,wscale 7], length 0
20:08:39.865868 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298473, win 2058, options [nop,nop,TS val 731071049 ecr 107727805], length 020:08:39.866943 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [P.], seq 4104613941:4104614007, ack 413298473, win 2058, options [nop,nop,TS val 731071050 ecr 107727805], length 66
20:08:39.867192 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [.], ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 0
20:08:39.867329 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [P.], seq 413298473:413298478, ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 5
20:08:39.867378 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298478, win 2058, options [nop,nop,TS val 731071050 ecr 107727807], length 014
20:16:00.604659 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509045 ecr 107733081], length 48
20:16:00.754648 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509195 ecr 107733081], length 48
20:16:01.055889 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509495 ecr 107733081], length 48
20:16:01.456264 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509895 ecr 107733081], length 48
20:16:02.062739 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731510495 ecr 107733081], length 48
20:16:03.065437 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731511495 ecr 107733081], length 48
20:16:04.871576 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731513295 ecr 107733081], length 48
20:16:08.287194 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731516695 ecr 107733081], length 48
20:16:14.912828 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731523295 ecr 107733081], length 48
20:16:21.529555 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731529895 ecr 107733081], length 48
20:16:28.147866 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731536495 ecr 107733081], length 48
20:16:34.771964 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731543095 ecr 107733081], length 48
20:16:41.398904 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731549695 ecr 107733081], length 48
20:16:48.025246 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731556295 ecr 107733081], length 48
20:16:54.655115 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [R.], seq 4241911420, ack 1651645875, win 2048, length 07
20:17:17.609293 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731585757 ecr 0,sackOK,eol], length 0
20:17:18.610567 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731586757 ecr 0,sackOK,eol], length 0
20:17:19.613758 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731587757 ecr 0,sackOK,eol], length 0
20:17:20.617512 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731588758 ecr 0,sackOK,eol], length 0
20:17:21.620555 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731589758 ecr 0,sackOK,eol], length 0
20:17:22.625515 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731590758 ecr 0,sackOK,eol], length 0
20:17:24.631276 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731592758 ecr 0,sackOK,eol], length 011
20:17:27.661985 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595780 ecr 0,sackOK,eol], length 0
20:17:27.805463 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595921 ecr 0,sackOK,eol], length 0
20:17:27.906594 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596021 ecr 0,sackOK,eol], length 0
20:17:28.007657 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596122 ecr 0,sackOK,eol], length 0
20:17:28.108495 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596222 ecr 0,sackOK,eol], length 0
20:17:28.211396 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596322 ecr 0,sackOK,eol], length 0
20:17:28.412573 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596523 ecr 0,sackOK,eol], length 0
20:17:28.813800 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596923 ecr 0,sackOK,eol], length 0
20:17:29.617595 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731597723 ecr 0,sackOK,eol], length 0
20:17:31.226228 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731599323 ecr 0,sackOK,eol], length 0
20:17:34.434295 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:37.766336 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605835 ecr 0,sackOK,eol], length 0
20:17:37.907801 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605976 ecr 0,sackOK,eol], length 0
20:17:38.009930 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606076 ecr 0,sackOK,eol], length 0
20:17:38.111023 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606177 ecr 0,sackOK,eol], length 0
20:17:38.212162 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606277 ecr 0,sackOK,eol], length 0
20:17:38.312800 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606377 ecr 0,sackOK,eol], length 0
20:17:38.513278 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606577 ecr 0,sackOK,eol], length 0
20:17:38.913743 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606977 ecr 0,sackOK,eol], length 0
20:17:39.718061 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731607777 ecr 0,sackOK,eol], length 0
20:17:41.323608 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731609377 ecr 0,sackOK,eol], length 0
20:17:44.538196 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:47.867437 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731615897 ecr 0,sackOK,eol], length 0
20:17:48.008751 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616038 ecr 0,sackOK,eol], length 0
20:17:48.109361 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616138 ecr 0,sackOK,eol], length 0
20:17:48.209657 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616238 ecr 0,sackOK,eol], length 0
20:17:48.312000 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616338 ecr 0,sackOK,eol], length 0
20:17:48.413063 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616439 ecr 0,sackOK,eol], length 0
20:17:48.613676 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616639 ecr 0,sackOK,eol], length 0
20:17:49.015418 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617039 ecr 0,sackOK,eol], length 0
20:17:49.818549 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617839 ecr 0,sackOK,eol], length 0
20:17:51.422985 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731619439 ecr 0,sackOK,eol], length 0
20:17:54.631568 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:57.969994 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0
20:17:57.970906 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0
20:17:58.115551 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0
20:17:58.115561 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0
20:17:58.217038 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0
20:17:58.217049 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0
20:17:58.317493 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0
20:17:58.317504 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0
20:17:58.418207 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0
20:17:58.418219 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0
20:17:58.519006 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0
20:17:58.519018 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0
20:17:58.721709 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0
20:17:58.721721 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0
20:17:59.123040 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0
20:17:59.123052 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0
20:17:59.925894 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0
20:17:59.925910 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0
20:18:01.530428 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0
20:18:01.530436 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0
20:18:04.739272 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:04.739280 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,sackOK,eol], length 013
20:18:08.063149 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636018 ecr 0,sackOK,eol], length 0
20:18:08.204712 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636159 ecr 0,sackOK,eol], length 0
20:18:08.305275 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636259 ecr 0,sackOK,eol], length 0
20:18:08.406168 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636359 ecr 0,sackOK,eol], length 0
20:18:08.507152 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636459 ecr 0,sackOK,eol], length 0
20:18:08.607218 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636559 ecr 0,sackOK,eol], length 0
20:18:08.808894 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636759 ecr 0,sackOK,eol], length 0
20:18:09.209305 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637159 ecr 0,sackOK,eol], length 0
20:18:10.011676 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637959 ecr 0,sackOK,eol], length 0
20:18:11.616529 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731639559 ecr 0,sackOK,eol], length 0
20:18:14.828898 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:21.286606 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:27.721777 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 013
20:20:58.899104 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805899 ecr 0,sackOK,eol], length 0
20:20:58.900162 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805900 ecr 0,sackOK,eol], length 0
20:20:59.045755 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806040 ecr 0,sackOK,eol], length 0
20:20:59.047010 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806041 ecr 0,sackOK,eol], length 0
20:20:59.146241 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806140 ecr 0,sackOK,eol], length 0
20:20:59.147300 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806141 ecr 0,sackOK,eol], length 0
20:20:59.247515 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806240 ecr 0,sackOK,eol], length 0
20:20:59.248746 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806241 ecr 0,sackOK,eol], length 0
20:20:59.347871 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806340 ecr 0,sackOK,eol], length 0
20:20:59.348974 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806341 ecr 0,sackOK,eol], length 0
20:20:59.449197 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0
20:20:59.449211 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0
20:20:59.654647 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0
20:20:59.654653 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0
20:21:00.056939 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0
20:21:00.056958 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0
20:21:00.867762 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0
20:21:00.867778 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0
20:21:02.484895 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0
20:21:02.484903 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0
20:21:05.715910 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:05.715916 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:12.139150 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:12.139164 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:18.570932 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:18.570946 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 020:23:49.944343 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [S], seq 1827331675, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976206 ecr 0,sackOK,eol], length 0
20:23:49.944875 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [S], seq 1554881183, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976207 ecr 0,sackOK,eol], length 0
20:23:49.954078 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [S.], seq 2130024761, ack 1827331676, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976206,nop,wscale 7], length 0
20:23:49.954085 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [S.], seq 1062682904, ack 1554881184, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976207,nop,wscale 7], length 0
20:23:49.954203 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024762, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0
20:23:49.954257 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062682905, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0
20:23:49.957101 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [P.], seq 1827331676:1827331724, ack 2130024762, win 2058, options [nop,nop,TS val 731976217 ecr 108637891], length 48
20:23:49.959283 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881184:1554881250, ack 1062682905, win 2058, options [nop,nop,TS val 731976218 ecr 108637891], length 66
20:23:49.999821 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [.], ack 1827331724, win 227, options [nop,nop,TS val 108637944 ecr 731976217], length 0
20:23:49.999822 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881250, win 227, options [nop,nop,TS val 108637944 ecr 731976218], length 0
20:23:50.000112 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881250:1554881278, ack 1062682905, win 2058, options [nop,nop,TS val 731976258 ecr 108637944], length 28
20:23:50.000541 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881278, win 227, options [nop,nop,TS val 108637944 ecr 731976258], length 0
20:23:50.001223 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881278:1554881304, ack 1062682905, win 2058, options [nop,nop,TS val 731976259 ecr 108637944], length 26
20:23:50.001880 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 0
20:23:50.002150 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [P.], seq 1062682905:1062683695, ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 79020:40:12.418726 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [F.], seq 1554881304, ack 1062684496, win 2048, options [nop,nop,TS val 732912682 ecr 108637947], length 0
20:40:12.418803 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [F.], seq 1827331724, ack 2130024795, win 2058, options [nop,nop,TS val 732912682 ecr 108637946], length 0
20:40:12.421643 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [F.], seq 1062684496, ack 1554881305, win 227, options [nop,nop,TS val 109620371 ecr 732912682], length 0
20:40:12.421695 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062684497, win 2048, options [nop,nop,TS val 732912685 ecr 109620371], length 0
20:40:12.422976 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [F.], seq 2130024795, ack 1827331725, win 227, options [nop,nop,TS val 109620372 ecr 732912682], length 0
20:40:12.423014 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024796, win 2058, options [nop,nop,TS val 732912686 ecr 109620372], length 0

中间过程发现在进行topology刷新的时候出现大于1的的TCP连接&＃xff0c;这些多余的连接都发生在问题节点所在的ip上&＃xff0c;clientName叫做&＃xff1a;lettuce#ClusterTopologyRefresh

[root&＃64;localhost ~]# /data/server/redis-cluster/redis-5.0.5/src/redis-cli -c -h 192.168.64.15 -p 8052 client list |grep 64.64
id&＃61;287 addr&＃61;192.168.64.64:64321 fd&＃61;33 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;1292 idle&＃61;1282 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;289 addr&＃61;192.168.64.64:64523 fd&＃61;35 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;1111 idle&＃61;943 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;290 addr&＃61;192.168.64.64:64672 fd&＃61;36 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;940 idle&＃61;940 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;286 addr&＃61;192.168.64.64:63831 fd&＃61;13 name&＃61; age&＃61;1849 idle&＃61;0 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;32768 obl&＃61;5 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;setex


Netty层的channel信号

中间过程当第一次出现读写超时事件时&＃xff0c;netty是能够收到inactive事件&＃xff08;虽然这个时间点没有接收到TCP的对应信号&＃xff0c;那么netty是怎么判断关闭TCP的呢&＃xff1f;答&＃xff1a; [R.] 就是这个reset信号量&＃xff0c;这个就是最后的倔强&＃xff09;

然后就会触发scheduleReconnect方法&＃xff08;后续会阐述这个重连实现&＃xff09;尝试重连&＃xff0c;这个过程中redis cluster正在进行主从切换&＃xff0c;挡住从切换完成之后client端获取到正常的网络拓扑&＃xff0c;读写恢复正常。

但是读写就算恢复正常&＃xff0c;失去连接的节点底层还是会触发inactive事件&＃xff0c;所以还是一直会报io.netty.channel.ConnectTimeoutException: connection timed out:/192.168.64.15:8052异常

java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:269) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 9 common frames omitted
20:58:19.845 WARN i.l.c.c.t.ClusterTopologyRefresh - Unable to connect to 192.168.64.13:8051
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_201]at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_201]at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 7 common frames omitted
Caused by: java.net.ConnectException: Connection refused... 11 common frames omitted

复现不出来&＃xff0c;好痛苦&＃xff0c;接下来就看看这个TCP的reset信号啥时候发送&＃xff0c;以及是netty触发的还是TCP触发的。。。




推荐阅读
  • config

    Nacos 0.3 数据持久化详解与实践

    Nacos 0.3 数据持久化详解与实践
    本文详细介绍了如何将 Nacos 0.3 的数据持久化到 MySQL 数据库，并提供了具体的步骤和注意事项。 ... [详细]
  • command

    MDT2010实验部署手册（一）

    MicrosoftDeploymentToolkit2010部署培训实验手册V1.0目录实验环境说明3实验环境虚拟机使用信息3注意：4实验手册正文说 ... [详细]
  • command

    思科IOS XE与ISE集成实现TACACS认证配置

    思科IOS XE与ISE集成实现TACACS认证配置
    本文详细介绍了如何在思科IOS XE设备上配置TACACS认证，并通过ISE（Identity Services Engine）进行用户管理和授权。配置包括网络拓扑、设备设置和ISE端的具体步骤。 ... [详细]
  • config

    利用Python Paramiko库批量更新多台服务器的登录密码

    本文介绍了如何使用Python的Paramiko库批量更新多台服务器的登录密码。通过示例代码展示了具体实现方法，确保了操作的高效性和安全性。Paramiko库提供了强大的SSH2协议支持，使得远程服务器管理变得更加便捷。此外，文章还详细说明了代码的各个部分，帮助读者更好地理解和应用这一技术。 ... [详细]
  • config

    在Windows环境中使用SecureCRT高效连接Linux服务器

    SecureCRT是一款功能强大的终端仿真软件，支持SSH1和SSH2协议，适用于在Windows环境下高效连接和管理Linux服务器。该工具不仅提供了稳定的连接性能，还具备丰富的配置选项，能够满足不同用户的需求。通过SecureCRT，用户可以轻松实现对远程Linux系统的安全访问和操作。 ... [详细]
  • config

    使用Shell脚本高效部署MHA高可用集群

    本文介绍了如何利用Shell脚本高效地部署MHA（MySQL High Availability）高可用集群。通过详细的脚本编写和配置示例，展示了自动化部署过程中的关键步骤和注意事项。该方法不仅简化了集群的部署流程，还提高了系统的稳定性和可用性。 ... [详细]
  • get

    Java Socket 关键参数详解与优化建议

    Java Socket 关键参数详解与优化建议
    Java Socket 的 API 虽然被广泛使用，但其关键参数的用途却鲜为人知。本文详细解析了 Java Socket 中的重要参数，如 backlog 参数，它用于控制服务器等待连接请求的队列长度。此外，还探讨了其他参数如 SO_TIMEOUT、SO_REUSEADDR 等的配置方法及其对性能的影响，并提供了优化建议，帮助开发者提升网络通信的稳定性和效率。 ... [详细]
  • config

    理解和优化进程与线程状态转换机制

    在Cisco IOS XR系统中，存在提供服务的服务器和使用这些服务的客户端。本文深入探讨了进程与线程状态转换机制，分析了其在系统性能优化中的关键作用，并提出了改进措施，以提高系统的响应速度和资源利用率。通过详细研究状态转换的各个环节，本文为开发人员和系统管理员提供了实用的指导，旨在提升整体系统效率和稳定性。 ... [详细]
  • process

    Python 伦理黑客技术：深入探讨后门攻击（第三部分）

    Python 伦理黑客技术：深入探讨后门攻击（第三部分）
    在《Python 伦理黑客技术：深入探讨后门攻击（第三部分）》中，作者详细分析了后门攻击中的Socket问题。由于TCP协议基于流，难以确定消息批次的结束点，这给后门攻击的实现带来了挑战。为了解决这一问题，文章提出了一系列有效的技术方案，包括使用特定的分隔符和长度前缀，以确保数据包的准确传输和解析。这些方法不仅提高了攻击的隐蔽性和可靠性，还为安全研究人员提供了宝贵的参考。 ... [详细]
  • get

    利用 Python Socket 实现 ICMP 协议下的网络通信

    利用 Python Socket 实现 ICMP 协议下的网络通信
    在计算机网络课程的2.1实验中，学生需要通过Python Socket编程实现一种基于ICMP协议的网络通信功能。与操作系统自带的Ping命令类似，该实验要求学生开发一个简化的、非标准的ICMP通信程序，以加深对ICMP协议及其在网络通信中的应用的理解。通过这一实验，学生将掌握如何使用Python Socket库来构建和解析ICMP数据包，并实现基本的网络探测功能。 ... [详细]
  • command

    SQL Server 连接故障总结与解决方案分析

    在使用 SQL Server 时，连接故障是用户最常见的问题之一。通常，连接 SQL Server 的方法有两种：一种是通过 SQL Server 自带的客户端工具，例如 SQL Server Management Studio；另一种是通过第三方应用程序或开发工具进行连接。本文将详细分析导致连接故障的常见原因，并提供相应的解决策略，帮助用户有效排除连接问题。 ... [详细]
  • process

    如何更有效地提升对支持部门的协助与支撑？ - Enhancing Support for the Support Department: Strategies and Best Practices

    尽管我们尽最大努力，任何软件开发过程中都难免会出现缺陷。为了更有效地提升对支持部门的协助与支撑，本文探讨了多种策略和最佳实践，旨在通过改进沟通、增强培训和支持流程来减少这些缺陷的影响，并提高整体服务质量和客户满意度。 ... [详细]
  • client

    深入理解 gRPC：从基础到进阶的全面指南

    本文全面解析了 gRPC 的基础知识与高级应用，从 helloworld.proto 文件入手，详细阐述了如何定义服务接口。例如，`Greeter` 服务中的 `SayHello` 方法，该方法在客户端和服务器端的消息交互中起到了关键作用。通过实例代码，读者可以深入了解 gRPC 的工作原理及其在实际项目中的应用。 ... [详细]
  • config

    Golang TLS双向认证中的DoS漏洞深度解析（CVE-2018-16875）

    如果程序使用Go语言编写并涉及单向或双向TLS认证，可能会遭受CPU拒绝服务攻击（DoS）。本文深入分析了CVE-2018-16875漏洞，探讨其成因、影响及防范措施，为开发者提供全面的安全指导。 ... [详细]
  • get

    Linux 环境下 Java 及相关软件的安装指南

    本文详细介绍了如何在 Linux 系统上安装 JDK 1.8、MySQL 和 Redis，并提供了相应的环境配置和验证步骤。 ... [详细]
author-avatar
mobiledu2502875063
这个家伙很懒，什么也没留下！
Tags | 热门标签
RankList | 热门文章
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有