CPU Load > 99%

Leo3 years ago

Hi there,

I have been running Traccar v 4.3 for a couple of years without problems but recently I began having issues with CPU load > 99%.

In the server log I see lots of:

2021-05-29 00:00:00  WARN: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. - Too many open files - IOException (...)

I took a thread dump and saw lots of:

Error occurred during stack walking:
java.lang.RuntimeException: No type named "VtableBlob" in database

I have recently added a few Computed Attributes. Would it be the reason?

Full thread dump:

Attaching to process ID 1157, please wait...
WARNING: Hotspot VM version 11.0.2-ojdkbuild+9 does not match with SA version 11.0.11+9-Ubuntu-0ubuntu2.18.04. You may see unexpected results. 
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.2-ojdkbuild+9
Deadlock Detection:

No deadlocks found.

"HikariPool-1 connection adder" #8251 daemon prio=5 tid=0x00007f14000bf000 nid=0x2a29 waiting on condition [0x00007f13f2396000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   JavaThread state: _thread_blocked
Error occurred during stack walking:
java.lang.RuntimeException: No type named "VtableBlob" in database
    at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.lookupType(BasicTypeDataBase.java:86)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotTypeDataBase.lookupType(HotSpotTypeDataBase.java:137)
    at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.lookupType(BasicTypeDataBase.java:80)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:75)
    at jdk.hotspot.agent/sun.jvm.hotspot.code.CodeCache.findBlobUnsafe(CodeCache.java:120)
    at jdk.hotspot.agent/sun.jvm.hotspot.code.CodeCache.findBlob(CodeCache.java:93)
...
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.runWithArgs(JStack.java:90)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJSTACK(SALauncher.java:297)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:533)


"Reference Handler" #2 daemon prio=10 tid=0x00007f1428100800 nid=0x496 waiting on condition [0x00007f142cd0f000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_blocked
Error occurred during stack walking:
java.lang.RuntimeException: No type named "VtableBlob" in database
    at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.lookupType(BasicTypeDataBase.java:86)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotTypeDataBase.lookupType(HotSpotTypeDataBase.java:137)
    at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.lookupType(BasicTypeDataBase.java:80)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:75)
    at jdk.hotspot.agent/sun.jvm.hotspot.code.CodeCache.findBlobUnsafe(CodeCache.java:120)
    at jdk.hotspot.agent/sun.jvm.hotspot.code.CodeCache.findBlob(CodeCache.java:93)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.Frame.cb(Frame.java:117)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.x86.X86Frame.adjustUnextendedSP(X86Frame.java:322)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.x86.X86Frame.<init>(X86Frame.java:141)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:99)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:276)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:238)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:81)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.run(JStack.java:67)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.runWithArgs(JStack.java:90)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJSTACK(SALauncher.java:297)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:533)

Thanks in advance,

Anton Tananaev3 years ago

I think the error means that you used incompatible tool to get the thread dump.