Java 程序内存不足问题求解

2022-10-09 11:12:56 +08:00
 jiobanma

1.问题描述

  1. 服务忽然宕机，查看服务器发现生成了很多 hs_err_pidxxxxxx.log 这样的 log 文件。下面我会提出来文件的一些内容。
  2. 查看服务的日志文件有如下错误提示
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /neworiental/software/bootstrap/hs_err_pid17840.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006c0000000, 2863661056, 0) failed; error='Cannot allocate memory' (errno=12)
  1. JVM 参数
-Xms4G -Xmx4G -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/xxx/software/bootstrap/

2. 问题

  1. 简单的查了查，对 jvm 不是特别的熟悉。不太清楚这个问题是 jvm 分配的内存有点小，服务内存溢出导致的，还是服务器内存资源不够了，导致服务产生异常。
  2. 我加了 HeapDumpOnOutOfMemoryError 这个参数，如果是服务运行期间内存溢出，为什么没有产生 dump 文件。
  3. 希望有了解这块内容的大佬帮忙看看，提供些思路感谢。

3. hs_err_pidxxxxxx.log 文件

Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.172-b11) for linux-amd64 JRE (1.8.0_172-b11), built on Mar 28 2018 21:44:09 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Sun Oct  9 00:28:39 2022
elapsed time: 0 seconds (0d 0h 0m 0s)

^C
[dontovertime@HYY-BMPP-P-sql96d40-09 bootstrap]$ tail -10000f hs_err_pid18191.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=18191, tid=0x00007f6c15c70700
#
# JRE version:  (8.0_172-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

---------------  T H R E A D  ---------------

Current thread (0x00007f6c0c00a800):  JavaThread "Unknown thread" [_thread_in_vm, id=18192, stack(0x00007f6c15b71000,0x00007f6c15c71000)]

Stack: [0x00007f6c15b71000,0x00007f6c15c71000],  sp=0x00007f6c15c6d2b0,  free space=1008k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xad221a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x50185b]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V  [libjvm.so+0x92fee3]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x123
V  [libjvm.so+0x930109]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V  [libjvm.so+0x9273ca]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V  [libjvm.so+0x99d363]  PSVirtualSpace::expand_by(unsigned long)+0x53
V  [libjvm.so+0x98d477]  PSOldGen::initialize(ReservedSpace, unsigned long, char const*, int)+0xb7
V  [libjvm.so+0x2dbb8a]  AdjoiningGenerations::AdjoiningGenerations(ReservedSpace, GenerationSizer*, unsigned long)+0x39a
V  [libjvm.so+0x9514a6]  ParallelScavengeHeap::initialize()+0x1d6
V  [libjvm.so+0xa9a253]  Universe::initialize_heap()+0xf3
V  [libjvm.so+0xa9a7be]  universe_init()+0x3e
V  [libjvm.so+0x643de5]  init_globals()+0x65
V  [libjvm.so+0xa7ef7e]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e
V  [libjvm.so+0x6d8b34]  JNI_CreateJavaVM+0x74
C  [libagentloader.so+0x1146d]  JNI_CreateJavaVM+0x8ed

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x00007f6c0c00a800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=18192, stack(0x00007f6c15b71000,0x00007f6c15c71000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


---------------  S Y S T E M  ---------------

OS:CentOS Linux release 7.7.1908 (Core)

uname:Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 63433, NOFILE 102536, AS infinity
load average:3.00 3.58 2.35

/proc/meminfo:
MemTotal:       16260372 kB
MemFree:          425348 kB
MemAvailable:     367844 kB
Buffers:               0 kB
Cached:           235068 kB
SwapCached:       148856 kB
Active:         13576768 kB
Inactive:        1854056 kB
Active(anon):   13530096 kB
Inactive(anon):  1771152 kB
Active(file):      46672 kB
Inactive(file):    82904 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:         874708 kB
Dirty:               488 kB
Writeback:             0 kB
AnonPages:      15048848 kB
Mapped:            49124 kB
Shmem:            105552 kB
Slab:             162980 kB
SReclaimable:      94124 kB
SUnreclaim:        68856 kB
KernelStack:       27168 kB
PageTables:        65984 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12324484 kB
Committed_AS:   33627812 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      195136 kB
VmallocChunk:   34359326716 kB
HardwareCorrupted:     0 kB
AnonHugePages:   5726208 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       87936 kB
DirectMap2M:     5154816 kB
DirectMap1G:    13631488 kB


CPU:total 8 (initial active 8) (1 cores per cpu, 1 threads per core) family 6 model 79 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, 3dnowpref, tsc, tscinvbit

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 2
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 4
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 6
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 8
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 8
initial apicid	: 8
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 10
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 10
initial apicid	: 10
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 12
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 12
initial apicid	: 12
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 14
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 14
initial apicid	: 14
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.172-b11) for linux-amd64 JRE (1.8.0_172-b11), built on Mar 28 2018 21:44:09 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Sun Oct  9 00:28:39 2022
elapsed time: 0 seconds (0d 0h 0m 0s)
4239 次点击
所在节点    Java
35 条回复
jiobanma
2022-10-09 14:36:30 +08:00
@cloud107202 在服务器里的，没有在容器里跑
@MineDog mmap 啊这 这个不懂诶，没接触过； ulimit -c unlimited 这个我试一下
cloud107202
2022-10-09 14:59:09 +08:00
看样子就是泄露了，可能是托管堆内也可能是线程。找机会用 jmap 主动 dump 内存，最好是做好监控的前提下，在 process memory 跳涨后去 dump, 然后扔 https://visualvm.github.io/ 里分析
mengyxu
2022-10-09 15:26:22 +08:00
宿主级内存不够,调小 Xms 和 Xmx 参数,如果你程序真的需要分配这么多内存的话建议宿主机少跑点进程
urnoob
2022-10-09 17:20:24 +08:00
mmap 2G 内存，有可能是哪个库或者自己的代码加载大文件或者写文件如 log 。jvm 需要扩大 heap 不是用 mmap 。
jvm 设的内存太大了，导致可用堆外内存不足。 如果这个应用的 jvm 确实需要那么多，那 lz 就加内存。
剩下就是内存泄露问题。
imzcg2
2022-10-09 18:09:47 +08:00
Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit

看见这个了吗???? 你这分明是在 64 位主机上运行了 32 位的 jdk 啊! 赶紧回去在主机上运行 java --version 看是不是 32 位的 jdk,是就换 64 位
imzcg2
2022-10-09 18:12:37 +08:00
抱歉没看完,是 64 位的
chainsR
2022-10-09 20:35:48 +08:00
@jiobanma 不过 8 的垃圾回收确实得指定好，我的是跑着跑着把内存跑满了，把这些加上去后一切正常
LeegoYih
2022-10-10 00:26:26 +08:00
把初始堆内存大小和最大内存大小设置成一样的值，这样启动时内存不足报错就说明宿主机内存确实是不够，免得 JVM 后续再去申请内存时才暴露问题。

-Xms4g
-Xmx4g
nothingistrue
2022-10-10 10:11:26 +08:00
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2640), pid=18191, tid=0x00007f6c15c70700
#
# JRE version: (8.0_172-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

这一段已经说明错误现象，可能的原因，和大致的解决方案了。

现象：JVM 向基本系统申请内存的时候失败了，2863661056 bytes 大概 356 MB 。
这里申请的不是 -Xms -Xmx 指定的 JVM 堆内存，而是 JVM 需要占用的其他内存，JVM 堆内存在 JVM 启动的时候就独占了，后面不会再申请。Java 实例化后的对象（实际上是对象的成员变量），方法运行期间的变量，这俩是放到 JVM 堆内存的。但是 Java 类定义本身（类结构、类当中方法的函数定义、类的静态变量这些），是放到额外内存中的。此外 JVM 本身也会占用内存。后两个内存是动态的，现在要申请的可能就是这俩内存，然后因超出物理机或者进程内存上限而失败了。

原因就两个：一，物理机内存，包括实际物理内存和交换区（ Windows 下就是虚拟内存），都满了；二，32 位程序，单进程内存通常上限是 4G 。
nothingistrue
2022-10-10 10:20:38 +08:00
上面 JVM 堆内存描述有误，应该时：启动时只占用 -Xms 指定的大小，后面随需要扩大占用，直到 -Xmx 指定的大小。
yisier
2022-10-10 10:21:01 +08:00
加个定时任务，每隔一段时间查询一下线程总数，如果线程数越来越多，那就是内存泄露了


ThreadMXBean bean = ManagementFactory.getThreadMXBean();
log.info("线程总数为 = " + bean.getThreadCount());
jiobanma
2022-10-10 13:58:03 +08:00
@nothingistrue
@yisier
感谢解惑
cloud107202
2022-10-10 15:08:44 +08:00
Java HotSpot(TM) 64-Bit Server VM warning: 已经说明是 64 位 JVM
32bit 那个提示是在 Possible reasons 这一段下面，只是个内置的 hint 不要搞偏方向
pierswu
2022-10-10 18:38:24 +08:00
5 楼、29 楼才是正解!（虽然 29 楼算错了 2863661056 bytes 是 1731 MB

就是你服务器的剩余内存不够了
pierswu
2022-10-10 18:39:12 +08:00
我也算错了 2863661056 bytes 是 2731 MB

这是一个专为移动设备优化的页面（即为了让你能够在 Google 搜索结果里秒开这个页面），如果你希望参与 V2EX 社区的讨论，你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/885468

V2EX 是创意工作者们的社区，是一个分享自己正在做的有趣事物、交流想法，可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX