请教个 k8s 问题, node 貌似不定期会挂

2019-07-30 13:22:32 +08:00
 snappyone

症状就是 controller_manager 的 log 里面定期会有 nodenotready 的日志

然后 pod 对应的 endpoint ip 被清空,看了下 journalctl -f -u kubelet 输出如下,对这块不太熟,请教懂的同学帮忙看看

journalctl -f -u kubelet 
-- Logs begin at Tue 2019-07-30 00:44:14 CEST. --
Jul 30 03:35:34 server-m1 kubelet[14853]: W0730 03:35:34.257474   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5686_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5686_systemd_test_default.slice: no such file or directory
Jul 30 03:35:44 server-m1 kubelet[14853]: W0730 03:35:44.166927   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/blkio/libcontainer_5704_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/blkio/libcontainer_5704_systemd_test_default.slice: no such file or directory
Jul 30 03:35:44 server-m1 kubelet[14853]: W0730 03:35:44.166982   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/libcontainer_5704_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/libcontainer_5704_systemd_test_default.slice: no such file or directory
Jul 30 03:35:44 server-m1 kubelet[14853]: W0730 03:35:44.167008   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5704_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5704_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.165976   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/libcontainer_5761_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/libcontainer_5761_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.166075   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5761_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5761_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.184573   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/libcontainer_5767_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/libcontainer_5767_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.184612   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5767_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5767_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.253787   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/libcontainer_5809_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/libcontainer_5809_systemd_test_default.slice: no such file or directory
Jul 30 03:35:54 server-m1 kubelet[14853]: W0730 03:35:54.253812   14853 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5809_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5809_systemd_test_default.slice: no such file or directory
3765 次点击
所在节点    程序员
1 条回复
snappyone
2019-07-30 13:49:55 +08:00
貌似找到问题了,参照以下 issue:
https://github.com/kubernetes/kubernetes/issues/76531

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/587469

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX