kube-prometheus-0.7 部署出错问题求助

2021-08-31 11:27:23 +08:00
 ice2016

系统环境是 CentOS7.9 + K8s v1.19.14 ，根据官方兼容性说明下载安装的是 kube-prometheus-release-0.7 版本，老是报错，安装包是 https://github.com/prometheus-operator/kube-prometheus/archive/refs/heads/release-0.7.zip

cd /root/kube-prometheus-release-0.7/manifests
kubectl create -f setup

[root@k8s-master manifests]# kubectl get pod -o wide -n monitoring
NAME                                   READY   STATUS             RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
prometheus-operator-7649c7454f-q9m29   1/2     CrashLoopBackOff   6          10m   10.201.1.58   k8s-node1   <none>           <none>

查看日志：

[root@k8s-master manifests]# kubectl logs prometheus-operator-7649c7454f-q9m29 --all-containers -n monitoring
level=info ts=2021-08-31T03:16:07.410585761Z caller=main.go:235 msg="Starting Prometheus Operator" version="(version=0.44.1, branch=refs/tags/pkg/apis/monitoring/v0.44.1, revision=d8b7d3766225908d0239fd0d78258892cd0fc384)"
level=info ts=2021-08-31T03:16:07.410684044Z caller=main.go:236 build_context="(go=go1.14.12, user=simonpasquier, date=20201209-17:17:30)"
ts=2021-08-31T03:16:07.415831751Z caller=main.go:107 msg="Starting insecure server on [::]:8080"
ts=2021-08-31T03:16:10.422020132Z caller=main.go:422 msg="Unhandled error received. Exiting..." err="communicating with server failed: Get \"https://10.96.0.1:443/version?timeout=32s\": dial tcp 10.96.0.1:443: connect: connection timed out"
I0831 03:10:09.496650       1 main.go:190] Valid token audiences:
I0831 03:10:09.496764       1 main.go:262] Generating self signed cert as no cert is provided
I0831 03:10:10.447004       1 main.go:311] Starting TCP socket on :8443
I0831 03:10:10.447347       1 main.go:318] Listening securely on :8443

根据报错测试几个节点上 10.96.0.1:443 这个地址都可以访问

[root@k8s-master manifests]# curl -k https://10.96.0.1:443/version?timeout=32s
{
  "major": "1",
  "minor": "19",
  "gitVersion": "v1.19.14",
  "gitCommit": "0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c",
  "gitTreeState": "clean",
  "buildDate": "2021-08-11T18:02:17Z",
  "goVersion": "go1.15.15",
  "compiler": "gc",
  "platform": "linux/amd64"

iptables 信息：

[root@k8s-master manifests]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
KUBE-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER-ISOLATION-STAGE-1  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  10.244.0.0/16        0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            10.244.0.0/16

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP       all  -- !127.0.0.0/8          127.0.0.0/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination

环境信息如下：

[root@k8s-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:07:41Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:02:17Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

[root@k8s-master ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

[root@k8s-master ]# kubectl get node
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   10d   v1.19.14
k8s-node1    Ready    <none>   10d   v1.19.14
k8s-node2    Ready    <none>   10d   v1.19.14
k8s-node3    Ready    <none>   10d   v1.19.14
1801 次点击
所在节点    Kubernetes
6 条回复
defunct9
2021-08-31 11:34:15 +08:00
开 ssh，让我上去看看
vemier
2021-08-31 12:31:07 +08:00
@defunct9 老哥又来了
nino789pzw
2021-08-31 13:39:46 +08:00
10.96.0.1 是什么的 ip ？
你的 curl 是 insecure 的 command，ssh 到 Promethus 的 pod 上 ssh 到 10.96.0.1 试试？
ice2016
2021-08-31 21:10:10 +08:00
@nino789pzw 把网络插件由 flannel 换成 calico 以后，正常部署了，也不报错，但是打开 svc
对外暴露端口以后，一会正常一会就打不开了，过会又正常，，
感觉哪里不对。。
SIGEV13
2021-08-31 22:53:19 +08:00
老哥去开个 issue， 看看社区有没有懂得。
ethangogogo
2021-09-03 08:57:29 +08:00
暴露端口之后有报错吗？

这是一个专为移动设备优化的页面（即为了让你能够在 Google 搜索结果里秒开这个页面），如果你希望参与 V2EX 社区的讨论，你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/799008

V2EX 是创意工作者们的社区，是一个分享自己正在做的有趣事物、交流想法，可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX