Traccar Device Freezing issue

I followed the guidelines and optimised the server. but I notice some of my devices are freezing. CPU and RAM usages are normal. tested using htop command.

The strange is when I restart MariaDB and the traccar service, It fix the issue. not only the issue, Now I see the missed positions as well. I face this issue most of the time. please help me.

I have 3500+ live devices.

Server : AWS r6g.2xlarge

ubuntu@Server1:~$ cat /proc/$(pidof java)/limits | grep "open files"
Max open files            50000                50000                files 
ubuntu@Server1:~$ cat /etc/systemd/system/traccar.service
[Unit]
Description=traccar
After=network.target

[Service]
Type=simple
WorkingDirectory=/opt/traccar
ExecStart=/opt/traccar/jre/bin/java -Djava.io.tmpdir=/mnt/db/tmp -Xmx10G -Xms10G -jar tracker-server.jar conf/traccar.xml
SyslogIdentifier=traccar
SuccessExitStatus=143
WatchdogSec=600
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
ubuntu@Server1:/etc/mysql/mariadb.conf.d$ cat custom.cnf 
[mysqld]
max_connections = 100
max_allowed_packet=64M
innodb_buffer_pool_size = 38G
innodb_log_file_size = 2G  
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 0
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 2
#log_queries_not_using_indexes = 1
ubuntu@Server1:/etc/mysql/mariadb.conf.d$

Config inside "/etc/sysctl.conf"

vm.max_map_count = 250000
fs.file-max = 250000
net.ipv4.ip_local_port_range = 1024 65535

Any luck?

Check if server is accepting device connection at the time when this happens. Check if the port is open.

Thanks for the quick reply. Yes, I tested the port at that time and Its accepting the connections. I used telnet to test the port.

harshana@MacBook-Pro ~ % telnet s1.xxxxxxx.com 5023
Trying XXX.XXX.XXX.XXX...
Connected to s1.xxxxxxx.com.
Escape character is '^]'.
^CConnection closed by foreign host.
harshana@MacBook-Pro ~ % 

Do you see that in the logs?

Yes. here it is.

ubuntu@Server1:/opt/traccar/logs$ grep xxx.134.180.50 tracker-server.log
2024-07-17 13:16:32  INFO: [T999f6c85: gt06 < xxx.134.180.50] 68690d0a
2024-07-17 13:16:34  INFO: [T999f6c85: gt06 < xxx.134.180.50] fff4fffd06

xxx.134.180.50 is my current IP address

And the server is frozen currently?

As I restarted, yes now its working. But the log is from the frozen time.

But I said you have to check when it's frozen. What's the point of checking now? Obviously it's going to work.

Also get a jstack dump once it happens again.

The details and the log is from the frozen time. I will update here once its frozen again.

The server freezed again and htop says the resource usage is still normal. Here is the dump.

Download JStack Dump

And what about port access when it was frozen?

port is working. I tried to access using telnet and I see the connection from the traccar log as well.

harshana@MacBook-Pro ~ % telnet s1.xxxxxxxxx.com 5023
Trying xxx.xxx.xxx.xxx...
Connected to s1.xxxxxxxxx.com.
Escape character is '^]'.
hi
test
Connection closed by foreign host.

2024-07-19 13:55:14  INFO: [Tf6f2bf09: gt06 < xxx.157.135.168] 68690d0a
2024-07-19 13:55:25  INFO: [Tf6f2bf09: gt06 < xxx.157.135.168] 746573740d0a

And this is the port you're using for your devices?

yes. gt06