作者:zengshiming | 来源:互联网 | 2023-09-04 10:10
What did you do?
Nothing, kubernetes operator restarted by itself. After restarting, it got stuck on a crash loop. I tried to recreate it, but it fails.
What did you expect to see?
kubernetes-operator does not crash loop
What did you see instead? Under which circumstances?
kubernetes-operator crash loops
Environment
Kops AWS v1.8.0
- Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T21:12:46Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Kops AWS v1.8.0
All default except for aws load balanced prometheus service
- Prometheus Operator Logs:
2018-02-12T23:17:47.162572304Z ts=2018-02-12T23:17:47.162243797Z caller=operator.go:156 compOnent=alertmanageroperator msg="connection established" cluster-version=v1.8.0
2018-02-12T23:17:47.162840421Z ts=2018-02-12T23:17:47.162701034Z caller=operator.go:257 compOnent=prometheusoperator msg="connection established" cluster-version=v1.8.0
2018-02-12T23:17:47.164968205Z ts=2018-02-12T23:17:47.164694593Z caller=operator.go:170 compOnent=alertmanageroperator msg="CRD API endpoints ready"
2018-02-12T23:17:47.169490417Z ts=2018-02-12T23:17:47.169204619Z caller=operator.go:303 compOnent=alertmanageroperator msg="Alertmanager added" key=monitoring/main
2018-02-12T23:17:47.247001533Z ts=2018-02-12T23:17:47.24664382Z caller=operator.go:323 compOnent=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
2018-02-12T23:17:47.247942277Z ts=2018-02-12T23:17:47.247754274Z caller=operator.go:379 compOnent=alertmanageroperator msg="sync alertmanager" key=monitoring/main
2018-02-12T23:17:47.346454739Z ts=2018-02-12T23:17:47.346131562Z caller=operator.go:1035 compOnent=prometheusoperator msg="CRD created" crd=Prometheus
2018-02-12T23:17:47.446679243Z ts=2018-02-12T23:17:47.446258093Z caller=operator.go:1035 compOnent=prometheusoperator msg="CRD created" crd=ServiceMonitor
2018-02-12T23:17:47.546456369Z E0212 23:17:47.546096 1 operator.go:255] Sync "monitoring/main" failed: creating statefulset failed: statefulsets.apps "alertmanager-main" already exists
2018-02-12T23:17:47.547239207Z ts=2018-02-12T23:17:47.546963148Z caller=operator.go:379 compOnent=alertmanageroperator msg="sync alertmanager" key=monitoring/main
2018-02-12T23:17:47.647580584Z E0212 23:17:47.647423 1 operator.go:255] Sync "monitoring/main" failed: creating statefulset failed: statefulsets.apps "alertmanager-main" already exists
2018-02-12T23:17:47.647698239Z ts=2018-02-12T23:17:47.647467106Z caller=operator.go:379 compOnent=alertmanageroperator msg="sync alertmanager" key=monitoring/main
2018-02-12T23:17:47.74641215Z E0212 23:17:47.746122 1 operator.go:255] Sync "monitoring/main" failed: creating statefulset failed: statefulsets.apps "alertmanager-main" already exists
2018-02-12T23:17:47.746535052Z ts=2018-02-12T23:17:47.746205004Z caller=operator.go:379 compOnent=alertmanageroperator msg="sync alertmanager" key=monitoring/main
2018-02-12T23:17:47.756038249Z ts=2018-02-12T23:17:47.755795024Z caller=operator.go:379 compOnent=alertmanageroperator msg="sync alertmanager" key=monitoring/main
2018-02-12T23:17:47.766503496Z ts=2018-02-12T23:17:47.766350285Z caller=operator.go:379 compOnent=alertmanageroperator msg="sync alertmanager" key=monitoring/main
2018-02-12T23:17:53.4537421Z ts=2018-02-12T23:17:53.453558451Z caller=main.go:128 msg="Unhandled error received. Exiting..." err="creating CRDs failed: timed out waiting for Custom Resoruce: json: cannot unmarshal string into Go value of type []string"
该提问来源于开源项目:prometheus-operator/prometheus-operator
If possible I recommend to delete the configmaps created in the default namespace and recreate the Promteheus, Alertmanager and ServiceMonitor objects.