-Xss option should be added to Optimization docs

Iván Ávalos 2 years ago

In my Traccar deployment, I have noticed that thread memory is one of the first things to exhaust in large deployments with ~1K device connections, based on the logs:

2024-01-09 21:30:56.405	2024-01-09 13:29:28  WARN: HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m4s441ms686µs53ns).

The default thread memory limit in Java is generally 1MB, which is probably not sufficient, given that Traccar opens a new thread for every connected device. Even if the VM is setup with loads of memory (via -Xms and -Xmx), Traccar will malfunction. This issue has been bugging us for years, forcing us to restart the server every week, and I feel like this might be the solution. That's why I propose adding it to the docs.

Iván Ávalos 2 years ago

The -Xss option in Java affects the stack memory limit. Increasing it (e.g. to 2MB) can be done like this:

java -Xss2M -jar tracker-server.jar conf/traccar.xml
Iván Ávalos 2 years ago

Errata of the first post:

  • There is no thread memory, but rather stack memory.
  • Stack memory gets exhausted because every thread uses stack.
  • 1MB is the default stack memory limit, not thread memory limit.
Anton Tananaev 2 years ago

I don't understand the reasoning. Thread starvation has nothing to do with memory. It points to CPU.

Iván Ávalos 2 years ago

Thanks for clearing it up, I'll check my CPU, then. But still, I feel like increasing the stack memory limit should at least somewhat improve stability or performance in large scale deployments, shouldn't it? >1K threads open should have an impact in stack memory.

Anton Tananaev 2 years ago

No, it shouldn't. If you hit the stack limit, the service will crash with out of memory exception.