java - High cpu usage on openjdk17 zgc

I am trying to use zgc in product environment, so I updated jdk from jdk8 to openjdk 17, and the gc related options, but the cpu usage goes to 100% after a few days later of the jvm start(normal cpu usage should be 5~10%). After a few seconds or ten seconds the cpu goes down.

The jvm options is:

-Xms30G -Xmx30G -server -Xss256K -XX:-OmitStackTraceInFastThrow -XX:MaxDirectMemorySize=256G -DPreloadBufferPool.MaxMemory=70% -Dioty.maxDirectMemory=10474836480 -DPreloadBufferPool.WritePageExtraWeightMs=86400000 -XX:+UseZGC -XX:+PrintGCDetails --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED -Xlog:safepoint=info,classhisto*=trace,age*=info,gc*=info:file=/export/Logs/jmq4-server/gc-%t.log:time,level,tid,tags:filecount=5,filesize=2000M -Xlog:async -XX:+UseCountedLoopSafepoints -XX:ZAllocationSpikeTolerance=5 -XX:ZCollectionInterval=60 -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:-ZProactive -XX:MaxMetaspaceSize=10G -DasmEnable=false

I found that when the CPU is high, it is in the Concurrent Mark stage. And many thread's CPU is high:

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
383297 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9   0:25.49 joyqueue-b+
383376 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9   0:14.65 mqtt-consu+
383386 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9 744:12.38 joyqueue-m+
383394 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9   0:32.39 joyqueue-f+
383406 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9   1:06.33 joyqueue-f+
383492 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9  11:44.16 joyqueue-f+
383587 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9   0:34.88 vertx-bloc+
383604 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9  11:58.13 joyqueue-f+
383723 admin     20   0 1564.3g 161.8g  76.8g R 99.9 85.9  11:39.30 joyqueue-f+
383059 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9 100:36.40 ZWorker#0
383064 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9 103:17.62 ZWorker#5
383070 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9 101:48.11 ZWorker#11
383250 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:13.71 UMP-PROFIL+
383251 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:11.42 UMP-PROFIL+
383252 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:10.69 joyqueue-t+
383256 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:03.53 joyqueue-c+
383286 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:13.46 joyqueue-c+
383289 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:46.84 Thread-2
383303 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:51.21 joyqueue-c+
383324 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:44.98 Thread-20
383333 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:40.58 Monitor-Ca+
383343 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:19.65 LeaderElec+
383374 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:15.80 mqtt-consu+
383375 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:17.89 mqtt-consu+
383378 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:08.01 mqtt-consu+
383379 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:23.00 mqtt-consu+
383381 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:32.15 mqtt-consu+
383382 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:32.46 mqtt-consu+
383411 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:11.54 joyqueue-f+
383438 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  35:09.12 joyqueue-f+
383440 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:26.33 joyqueue-f+
383472 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:31.28 joyqueue-f+
383484 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:19.15 joyqueue-f+
383485 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:44.90 joyqueue-f+
383501 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:37.81 joyqueue-f+
383503 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:25.02 joyqueue-f+
383533 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:54.75 joyqueue-f+
383552 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:47.50 joyqueue-f+
383563 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:35.71 joyqueue-f+
383582 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:27.18 joyqueue-f+
383606 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  20:50.89 joyqueue-f+
383609 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:44.34 joyqueue-f+
383620 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:20.15 joyqueue-f+
383666 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:36.64 joyqueue-f+
383678 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:20.14 joyqueue-f+
383704 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   0:57.91 joyqueue-f+
383724 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:45.75 joyqueue-f+
383752 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:54.81 joyqueue-f+
383768 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:45.60 joyqueue-f+
383797 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:37.44 joyqueue-f+
383804 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:15.88 joyqueue-f+
383807 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:23.07 joyqueue-f+
383816 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:06.54 joyqueue-f+
383840 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:27.39 joyqueue-f+
383875 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9  11:05.33 joyqueue-f+
383975 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:53.26 joyqueue-b+
384128 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   4:53.48 joyqueue-f+
384183 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   4:33.40 vert.x-eve+
384346 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1:19.58 joyqueue-b+
384492 admin     20   0 1564.3g 161.8g  76.8g R 94.7 85.9   1228:23 WriteThrea+
383659 admin     20   0 1564.3g 161.8g  76.8g R 89.5 85.9  11:37.54 joyqueue-f+
383530 admin     20   0 1564.3g 161.8g  76.8g R 84.2 85.9   1:00.60 joyqueue-f+
383321 admin     20   0 1564.3g 161.8g  76.8g R 73.7 85.9   0:57.41 joyqueue-f+
383611 admin     20   0 1564.3g 161.8g  76.8g S  5.3 85.9  11:43.17 joyqueue-f+
383646 admin     20   0 1564.3g 161.8g  76.8g S  5.3 85.9  54:39.73 joyqueue-f+
383658 admin     20   0 1564.3g 161.8g  76.8g S  5.3 85.9  58:43.68 joyqueue-f+
383758 admin     20   0 1564.3g 161.8g  76.8g S  5.3 85.9  11:45.80 joyqueue-f+
383057 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 java
383058 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:12.16 java
383060 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 100:18.21 ZWorker#1
383061 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 101:31.89 ZWorker#2
383062 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 101:16.71 ZWorker#3
383063 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 100:37.58 ZWorker#4
383065 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 100:55.73 ZWorker#6
383066 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 102:40.30 ZWorker#7
383067 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 102:46.46 ZWorker#8
383068 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 103:04.17 ZWorker#9
383069 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 101:48.28 ZWorker#10
383071 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 101:49.37 ZWorker#12
383072 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 100:46.01 ZWorker#13
383073 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 103:17.67 ZWorker#14
383074 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9 100:22.50 ZWorker#15
383075 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 ZUnmapper
383076 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 ZUncommitt+
383175 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   1:17.22 ZDriver
383176 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   1:11.07 ZDirector
383177 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:42.65 ZStat
383178 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 RuntimeWor+
383179 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 RuntimeWor+
383180 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 RuntimeWor+
383181 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 RuntimeWor+
383182 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 RuntimeWor+
383183 admin     20   0 1564.3g 161.8g  76.8g S  0.0 85.9   0:00.00 RuntimeWor+

Another info, The system CPU is high, but the user CPU is not high.

Some thread dumps when CPU is hish:

"joyqueue-frontend-produce-threads - 2" #233 prio=5 os_prio=0 cpu=797067.03ms elapsed=565273.30s tid=0x00007efa90009070 nid=0x59ebb runnable  [0x00007effb19ba000]
java.lang.Thread.State: RUNNABLE
    at java.util.concurrent.ConcurrentSkipListMap.doPut([email protected]/ConcurrentSkipListMap.java:620)
    at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent([email protected]/ConcurrentSkipListMap.java:1787)
    at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:106)
    at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:84)
    at com.codahale.metrics.Histogram.update(Histogram.java:41)
    at .joyqueue.monitor.metrics.Metrics.mark(Metrics.java:53)
    at .joyqueue.monitor.metrics.CachedMetrics.mark(CachedMetrics.java:32)
    at .joyqueue.monitor.cacheWindow.cacheWindowStat.EnQueueStat.mark(EnQueueStat.java:18)
    at .joyqueue.monitor.CacheManager.fromDecodeToStore(CacheManager.java:335)
    at .joyqueue.monitor.CacheManager.fromDecodeToStore(CacheManager.java:328)
    at .joyqueue.monitor.DefaultMonitorService.fromDecodeToStore(DefaultMonitorService.java:215)
    at .joyqueue.broker.kafka.handler.ProduceRequestHandler.handle(ProduceRequestHandler.java:210)
    at .joyqueue.brokerwork.protocol.support.CommandHandlerWrapper.handle(CommandHandlerWrapper.java:42)
    at .joyqueuework.transportmand.handler.filter.CommandHandlerInvocation.invoke(CommandHandlerInvocation.java:50)
    at .joyqueuework.transportmand.support.CommandExecuteTask.run(CommandExecuteTask.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run([email protected]/Thread.java:840)

Locked ownable synchronizers:
    - <0x000004000e13feb0> (a java.util.concurrent.ThreadPoolExecutor$Worker)


"joyqueue-frontend-io-eventLoop - 17" #175 prio=5 os_prio=0 cpu=120746.84ms elapsed=565273.50s tid=0x00007efb5c0163a0 nid=0x59e7e runnable  [0x00007effb0a79000]
java.lang.Thread.State: RUNNABLE
    at ioty.util.internal.InternalThreadLocalMap.get(InternalThreadLocalMap.java:102)
    at ioty.util.concurrent.FastThreadLocal.get(FastThreadLocal.java:136)
    at ioty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:373)
    at ioty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    at ioty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    at ioty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
    at ioty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
    at ioty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
    at ioty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780)
    at ioty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
    at ioty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at ioty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at ioty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run([email protected]/Thread.java:840)

Locked ownable synchronizers:
    - None

"joyqueue-frontend-io-eventLoop - 21" #179 prio=5 os_prio=0 cpu=103325.46ms elapsed=565273.49s tid=0x00007efb5c01ac70 nid=0x59e82 runnable  [0x00007effb0975000]
java.lang.Thread.State: RUNNABLE
    at ioty.buffer.PoolThreadCache.allocate(PoolThreadCache.java:167)
    at ioty.buffer.PoolThreadCache.allocateSmall(PoolThreadCache.java:151)
    at ioty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:150)
    at ioty.buffer.PoolArena.allocate(PoolArena.java:136)
    at ioty.buffer.PoolArena.allocate(PoolArena.java:128)
    at ioty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:378)
    at ioty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    at ioty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    at ioty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
    at ioty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
    at ioty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
    at ioty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780)
    at ioty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
    at ioty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at ioty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at ioty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run([email protected]/Thread.java:840)

Locked ownable synchronizers:
    - None

"joyqueue-frontend-produce-threads - 48" #286 prio=5 os_prio=0 cpu=807454.33ms elapsed=551868.00s tid=0x00007f540d010e70 nid=0x5da33 runnable  [0x00007f5879f37000]
java.lang.Thread.State: RUNNABLE
    at java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss([email protected]/ThreadLocal.java:462)
    at java.lang.ThreadLocal$ThreadLocalMap.getEntry([email protected]/ThreadLocal.java:439)
    at java.lang.ThreadLocal.get([email protected]/ThreadLocal.java:165)
    at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared([email protected]/ReentrantReadWriteLock.java:487)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared([email protected]/AbstractQueuedSynchronizer.java:1026)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock([email protected]/ReentrantReadWriteLock.java:738)
    at com.codahale.metrics.ExponentiallyDecayingReservoir.lockForRegularUsage(ExponentiallyDecayingReservoir.java:201)
    at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:95)
    at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:84)
    at com.codahale.metrics.Histogram.update(Histogram.java:41)
    at .joyqueue.monitor.metrics.Metrics.mark(Metrics.java:53)
    at .joyqueue.monitor.metrics.CachedMetrics.mark(CachedMetrics.java:32)
    at .joyqueue.monitor.cacheWindow.cacheWindowStat.EnQueueStat.mark(EnQueueStat.java:18)
    at .joyqueue.monitor.CacheManager.onStoreEnQueue(CacheManager.java:788)
    at .joyqueue.monitor.DefaultMonitorService.onStoreEnQueue(DefaultMonitorService.java:513)
    at .joyqueue.store.PartitionGroupStoreManager.asyncWrite(PartitionGroupStoreManager.java:1198)
    at .joyqueue.store.QosStore.asyncWrite(QosStore.java:118)
    at .joyqueue.broker.producer.ProduceManager.writeMessagesAsync(ProduceManager.java:386)
    at .joyqueue.broker.producer.ProduceManager.putMessageAsync(ProduceManager.java:244)
    at .joyqueue.broker.producer.ProduceManager.putMessageAsync(ProduceManager.java:226)
    at .joyqueue.broker.kafka.handler.ProduceHandler.produceMessage(ProduceHandler.java:50)
    at .joyqueue.broker.kafka.handler.ProduceRequestHandler.handle(ProduceRequestHandler.java:251)
    at .joyqueue.brokerwork.protocol.support.CommandHandlerWrapper.handle(CommandHandlerWrapper.java:42)
    at .joyqueuework.transportmand.handler.filter.CommandHandlerInvocation.invoke(CommandHandlerInvocation.java:50)
    at .joyqueuework.transportmand.support.CommandExecuteTask.run(CommandExecuteTask.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run([email protected]/Thread.java:840)

Locked ownable synchronizers:
    - <0x0000040011040f98> (a java.util.concurrent.ThreadPoolExecutor$Worker)

科技改变生活-雨落星辰 - 所有的伟大,都源于一个勇敢的开始

java - High cpu usage on openjdk17 zgc - Stack Overflow

与本文相关的文章

评论列表(0)