mongodb 频繁异常退出 errno:24 Too many open files 求助

2019-08-02 10:35:43 +08:00
 comwrg

部分日志

2019-08-01T23:59:02.301+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:02.302+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:03.302+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:03.302+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:04.302+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:04.302+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:05.302+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:05.302+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:06.302+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:06.302+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:07.302+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:07.302+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:08.302+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:08.303+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:09.303+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:09.303+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:10.303+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:10.303+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:11.303+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:11.303+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:12.303+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:12.303+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:13.303+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:13.303+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:14.303+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:14.304+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:15.304+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:15.304+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:16.304+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:16.304+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:17.304+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:17.304+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:18.304+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:18.304+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:19.295+0800 W NETWORK  [HostnameCanonicalizationWorker] Failed to obtain address information for hostname iZuf61zao4uxbprumx45dlZ: System error
2019-08-01T23:59:19.304+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:19.304+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:20.304+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:20.305+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:21.305+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:21.305+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:22.305+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:22.305+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:23.305+0800 I NETWORK  [initandlisten] Listener: accept() returns -1 errno:24 Too many open files
2019-08-01T23:59:23.305+0800 E NETWORK  [initandlisten] Out of file descriptors. Waiting one second before trying to accept more connections.
2019-08-01T23:59:23.631+0800 E STORAGE  [thread2] WiredTiger (24) [1564675163:631372][9783:0x7f4e30730700], file:WiredTiger.wt, WT_SESSION.checkpoint: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Too many open files
2019-08-01T23:59:23.632+0800 E STORAGE  [thread2] WiredTiger (24) [1564675163:632761][9783:0x7f4e30730700], checkpoint-server: checkpoint server error: Too many open files
2019-08-01T23:59:23.632+0800 E STORAGE  [thread2] WiredTiger (-31804) [1564675163:632802][9783:0x7f4e30730700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-08-01T23:59:23.632+0800 I -        [thread2] Fatal Assertion 28558
2019-08-01T23:59:23.632+0800 I -        [thread2] 

***aborting after fassert() failure


2019-08-01T23:59:23.638+0800 F -        [thread2] Got signal: 6 (Aborted).
ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31862
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31862
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

设置了 sysctl.conf fs.file-max = 2097152

每天都会崩溃 实在不清楚问题所在根源

14729 次点击
所在节点    MongoDB
22 条回复
qq1340691923
2020-06-05 13:06:23 +08:00
@comwrg 你倒是分享一下最终结果啊..
comwrg
2020-06-05 13:56:02 +08:00
@qq1340691923
缓解方案,将 ulimit 设置的非常大 之前设置到了 200000 就没有出现那种情况了
暂时还是不清楚是什么原因,不过推断可能是 collection 过多的原因(数量级大于十万)

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/588437

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX