Kuboard无法发送邮件和钉钉告警问题解决

照着官方文档已经安装设置成功,也启动了一个busybox容器,让它一直崩溃重启来触发告警邮件和钉钉消息.发现无法收到邮件,也无法收到钉钉消息.

查看kuboard / alertmanager-main日志,可以看出邮件和钉钉消息发送报错了.

level&＃61;warn ts&＃61;2022-05-27T15:32:50.287Z caller&＃61;notify.go:723 component&＃61;dispatcher receiver&＃61;Critical integration&＃61;email[0] msg&＃61;"Notify attempt failed, will retry later" attempts&＃61;1 err&＃61;"&＃39;require_tls&＃39; is true (default) but \"smtp.exmail.qq.com:465\" does not advertise the STARTTLS extension"
level&＃61;error ts&＃61;2022-05-27T15:35:07.518Z caller&＃61;dispatch.go:310 component&＃61;dispatcher msg&＃61;"Notify for alerts failed" num_alerts&＃61;1 err&＃61;"Default/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: http://dingtalk:8060/dingtalk/Default-shzdp/send; Default/email[0]: notify retry canceled after 17 attempts: &＃39;require_tls&＃39; is true (default) but \"smtp.exmail.qq.com:465\" does not advertise the STARTTLS extension"

邮件告警:

我用的QQ企业邮箱,一直发不了告警邮件,smtp.exmail.qq.com:465,分析报错日志后发现需要关闭TLS选项,邮件发送不成功问题就解决了.

 钉钉告警:

level&＃61;error ts&＃61;2022-05-27T15:41:46.569Z caller&＃61;dispatch.go:310 component&＃61;dispatcher msg&＃61;"Notify for alerts failed" num_alerts&＃61;2 err&＃61;"Critical/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: http://dingtalk:8060/dingtalk/Critical-8p8nz/send"

日志中发现钉钉容器服务接口404了,我们在kuboard找到钉钉pod,进去查看日志.

level&＃61;info ts&＃61;2022-05-27T16:31:50.549Z caller&＃61;entry.go:26 component&＃61;web http_scheme&＃61;http http_proto&＃61;HTTP/1.1 http_method&＃61;POST remote_addr&＃61;100.100.196.227:34048 user_agent&＃61;Alertmanager/0.22.2 uri&＃61;http://dingtalk:8060/dingtalk/Default-shzdp/send resp_status&＃61;404 resp_bytes_length&＃61;19 resp_elapsed_ms&＃61;0.068677 msg&＃61;"request complete"
level&＃61;warn ts&＃61;2022-05-27T16:31:50.549Z caller&＃61;dingtalk.go:75 component&＃61;web target&＃61;Default-shzdp msg&＃61;"target not found"

果然有错误日志,看这个容器时哪个镜像,我们找到它的源代码看看啥情况.

 在hub.docker.com找到该镜像,timonwong/prometheus-webhook-dingtalk Tags | Docker Hubhttps://hub.docker.com/r/timonwong/prometheus-webhook-dingtalk/tags在github找到该镜像源代码,定位到想要报错的文件行.

prometheus-webhook-dingtalk/dingtalk.go at v2.0.0 · timonwong/prometheus-webhook-dingtalk (github.com)https://github.com/timonwong/prometheus-webhook-dingtalk/blob/v2.0.0/web/dingtalk/dingtalk.go 

我们看readme.md看看这个东西到底是个啥.发现原来是该容器使用的配置文件,配置了钉钉发送消息的一些参数.

知道了是容器的配置文件,我们就应该进入dingtalk容器查看以下,这个文件对不对.

果然有这个文件,我们查看一下文件内容.

 文件内容正确无误,此时我们该思考,配置文件没问题,为什么代码说找不到targets呢?初步猜测可能是文件没生效,容器先启动,后再kuboard上配置的access_token.

怎么解决呢?

重启dingtalk容器组即可.