Java 程序内存不足问题求解

2022-10-09 11:12:56 +08:00
 jiobanma

1.问题描述

  1. 服务忽然宕机,查看服务器发现生成了很多 hs_err_pidxxxxxx.log 这样的 log 文件。下面我会提出来文件的一些内容。
  2. 查看服务的日志文件有如下错误提示
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /neworiental/software/bootstrap/hs_err_pid17840.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006c0000000, 2863661056, 0) failed; error='Cannot allocate memory' (errno=12)
  1. JVM 参数
-Xms4G -Xmx4G -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/xxx/software/bootstrap/

2. 问题

  1. 简单的查了查,对 jvm 不是特别的熟悉。不太清楚这个问题是 jvm 分配的内存有点小,服务内存溢出导致的,还是服务器内存资源不够了,导致服务产生异常。
  2. 我加了 HeapDumpOnOutOfMemoryError 这个参数,如果是服务运行期间内存溢出,为什么没有产生 dump 文件。
  3. 希望有了解这块内容的大佬帮忙看看,提供些思路感谢。

3. hs_err_pidxxxxxx.log 文件

Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.172-b11) for linux-amd64 JRE (1.8.0_172-b11), built on Mar 28 2018 21:44:09 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Sun Oct  9 00:28:39 2022
elapsed time: 0 seconds (0d 0h 0m 0s)

^C
[dontovertime@HYY-BMPP-P-sql96d40-09 bootstrap]$ tail -10000f hs_err_pid18191.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=18191, tid=0x00007f6c15c70700
#
# JRE version:  (8.0_172-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

---------------  T H R E A D  ---------------

Current thread (0x00007f6c0c00a800):  JavaThread "Unknown thread" [_thread_in_vm, id=18192, stack(0x00007f6c15b71000,0x00007f6c15c71000)]

Stack: [0x00007f6c15b71000,0x00007f6c15c71000],  sp=0x00007f6c15c6d2b0,  free space=1008k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xad221a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x50185b]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V  [libjvm.so+0x92fee3]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x123
V  [libjvm.so+0x930109]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V  [libjvm.so+0x9273ca]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V  [libjvm.so+0x99d363]  PSVirtualSpace::expand_by(unsigned long)+0x53
V  [libjvm.so+0x98d477]  PSOldGen::initialize(ReservedSpace, unsigned long, char const*, int)+0xb7
V  [libjvm.so+0x2dbb8a]  AdjoiningGenerations::AdjoiningGenerations(ReservedSpace, GenerationSizer*, unsigned long)+0x39a
V  [libjvm.so+0x9514a6]  ParallelScavengeHeap::initialize()+0x1d6
V  [libjvm.so+0xa9a253]  Universe::initialize_heap()+0xf3
V  [libjvm.so+0xa9a7be]  universe_init()+0x3e
V  [libjvm.so+0x643de5]  init_globals()+0x65
V  [libjvm.so+0xa7ef7e]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e
V  [libjvm.so+0x6d8b34]  JNI_CreateJavaVM+0x74
C  [libagentloader.so+0x1146d]  JNI_CreateJavaVM+0x8ed

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x00007f6c0c00a800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=18192, stack(0x00007f6c15b71000,0x00007f6c15c71000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


---------------  S Y S T E M  ---------------

OS:CentOS Linux release 7.7.1908 (Core)

uname:Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 63433, NOFILE 102536, AS infinity
load average:3.00 3.58 2.35

/proc/meminfo:
MemTotal:       16260372 kB
MemFree:          425348 kB
MemAvailable:     367844 kB
Buffers:               0 kB
Cached:           235068 kB
SwapCached:       148856 kB
Active:         13576768 kB
Inactive:        1854056 kB
Active(anon):   13530096 kB
Inactive(anon):  1771152 kB
Active(file):      46672 kB
Inactive(file):    82904 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:         874708 kB
Dirty:               488 kB
Writeback:             0 kB
AnonPages:      15048848 kB
Mapped:            49124 kB
Shmem:            105552 kB
Slab:             162980 kB
SReclaimable:      94124 kB
SUnreclaim:        68856 kB
KernelStack:       27168 kB
PageTables:        65984 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12324484 kB
Committed_AS:   33627812 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      195136 kB
VmallocChunk:   34359326716 kB
HardwareCorrupted:     0 kB
AnonHugePages:   5726208 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       87936 kB
DirectMap2M:     5154816 kB
DirectMap1G:    13631488 kB


CPU:total 8 (initial active 8) (1 cores per cpu, 1 threads per core) family 6 model 79 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, 3dnowpref, tsc, tscinvbit

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 2
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 4
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 6
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 8
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 8
initial apicid	: 8
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 10
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 10
initial apicid	: 10
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 12
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 12
initial apicid	: 12
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 79
model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping	: 1
microcode	: 0xb00001b
cpu MHz		: 2199.038
cache size	: 30720 KB
physical id	: 14
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 14
initial apicid	: 14
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
bogomips	: 4399.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.172-b11) for linux-amd64 JRE (1.8.0_172-b11), built on Mar 28 2018 21:44:09 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Sun Oct  9 00:28:39 2022
elapsed time: 0 seconds (0d 0h 0m 0s)
4216 次点击
所在节点    Java
35 条回复
ShuA1
2022-10-09 11:15:13 +08:00
看上去就是 OOM 啊
OctopusGO
2022-10-09 11:21:40 +08:00
调整一下 jvm 每个线程的初始化的时候申请的空间大小再看看
OctopusGO
2022-10-09 11:24:36 +08:00
一般就是代码里面有执行慢的地方,然乎这个地方调用的多,线程太多 OOM 了
WindWarrior
2022-10-09 11:28:41 +08:00
看看如果是 4GB 的服务器 -Xmx 是不能开到 4GB 的 大概开到 70%~ 80%
facelezz
2022-10-09 11:35:22 +08:00
"Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free)"
剩余 425M
"Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory."
申请 2863M

这不是很明显 宿主机内存不够么
larisboy
2022-10-09 11:37:22 +08:00
要不调低程序的所需内存,要不给宿主机加资源
jiobanma
2022-10-09 11:38:22 +08:00
@WindWarrior 服务器是 16g 的
@facelezz 但是服务器是 16g 的 发生这个原因的时候 重启了一下 但是重启前没有查看内存占用情况,现在 free -m 看的话 内存还剩 10 个 g
chainsR
2022-10-09 11:57:36 +08:00
我最近有个项目也遇到过,java 8 是么,可以试试把这些都加上去,就好了 https://www.cnblogs.com/lovelyli/p/14955585.html
asiufasd
2022-10-09 12:03:40 +08:00
@jiobanma
"我加了 HeapDumpOnOutOfMemoryError 这个参数,如果是服务运行期间内存溢出,为什么没有产生 dump 文件。"
因为可能没有发生 OutOfMemoryError ,而是操作系统级别的内存不够用了,可以检查一下是否有别的应用占用了系统的资源
littlewing
2022-10-09 12:09:43 +08:00
HeapDumpOnOutOfMemoryError 产生 dump 文件的条件是你的程序被 JVM OOM 了,但你这个是被系统把 JVM OOM 了
Red998
2022-10-09 12:14:23 +08:00
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2640), pid=18191, tid=0x00007f6c15c70700
#
# JRE version: (8.0_172-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again


Java 运行时环境没有足够的内存来继续。
# 本机内存分配 (mmap) 未能映射 2863661056 字节以提交保留内存。
# 可能的原因:
# 系统的物理 RAM 或交换空间不足
# 在 32 位模式下,达到了进程大小限制
# 可能的解决方案:
# 减少系统内存负载
# 增加物理内存或交换空间
# 检查交换后备存储是否已满
# 在 64 位操作系统上使用 64 位 Java
# 减小 Java 堆大小 (-Xmx/-Xms)
# 减少 Java 线程数
# 减少 Java 线程堆栈大小 (-Xss)
# 使用 -XX:ReservedCodeCacheSize= 设置更大的代码缓存
Jooooooooo
2022-10-09 13:33:25 +08:00
就是内存不够了.

简单解决就是先把堆调大, 然后 dump 堆看看里面都是啥, 有没有可以优化的.
jiobanma
2022-10-09 13:36:14 +08:00
@asiufasd
@littlewing
明白了,之前没有遇到过服务器内存满的情况。
@chainsR 是 java8 都加上有点离谱 线上不敢都加上去试 搞出点别的问题就 gg 了
jiobanma
2022-10-09 13:47:58 +08:00
@Jooooooooo 你的意思是继续加大-Xmx 吗? 现在已经是 4g 了,而且按照 9,10 楼大佬所说的,是服务器内存满了,我继续加大 jvm 的内存依然会出现这个问题的。并且现在没有产生出 dump 文件,所以才会疑惑是操作系统的问题还是代码的问题
lmshl
2022-10-09 13:57:12 +08:00
既然已经 dump heap 了,直接丢进 eclipse memory analyzer 看原因咯
facelezz
2022-10-09 13:59:49 +08:00
@jiobanma 基本的宿主机监控还是要有的 不过你这个显然除了 Java 还有其他进程在跑 吃掉了内存
MineDog
2022-10-09 14:05:30 +08:00
没 dump 的原因不是都写了么,"Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again"
wiix
2022-10-09 14:10:55 +08:00
线程堆栈占的内存和 Xmx 设置的堆内存是分开的,默认每个线程的开销大概是 1M 。是不是什么地方的线程数开太大了?
MineDog
2022-10-09 14:17:21 +08:00
还有就是用了 mmap 直接内存分配,是程序什么功能需要用到呢,这玩意也不是在堆里面分配的啊
cloud107202
2022-10-09 14:34:57 +08:00
程序运行在容器里么?有无设置容器内存 limit

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/885468

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX