Watchdog systemd

5of9 6 years ago

The new watchdog for systemd makes some problems so I had to disable it. It restarts always:
Logs:

Jul 06 15:26:34 gps. systemd[1]: traccar.service: Watchdog timeout (limit 1min 40s)!
Jul 06 15:26:34 gps.i systemd[1]: traccar.service: Killing process 21763 (java) with signal SIGABRT.
Jul 06 15:26:34 gps. systemd[1]: traccar.service: Main process exited, code=killed, status=6/ABRT
Jul 06 15:26:34 gps.i systemd[1]: traccar.service: Failed with result 'watchdog'.
Jul 06 15:26:45 gps.i systemd[1]: traccar.service: Scheduled restart job, restart counter is at 13.

Anton Tananaev 6 years ago

Do you mind sharing details about your system?

5of9 6 years ago

It's Ubuntu 20.04, and here start Infos from Traccar:

2020-07-06 15:27:46  INFO: Operating system name: Linux version: 5.4.0-40-generic architecture: amd64
2020-07-06 15:27:46  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: Oracle Corporation version: 11.0.5+10-LTS
2020-07-06 15:27:46  INFO: Memory limit heap: 359mb non-heap: 0mb
2020-07-06 15:27:46  INFO: Character encoding: UTF-8 charset: UTF-8
2020-07-06 15:27:46  INFO: Version: 4.9
2020-07-06 15:27:46  INFO: Starting server...
2020-07-06 15:27:47  INFO: jetty-9.4.30.v20200611; built: 2020-06-11T12:34:51.929Z; git: 271836e4c1f4612f12b7bb13ef5a92a927634b0d; jvm 11.0.5+10-LTS
2020-07-06 15:27:47  INFO: Started o.t.w.@1179fc8c{/,null,AVAILABLE}
2020-07-06 15:27:47  INFO: DefaultSessionIdManager workerName=node0
2020-07-06 15:27:47  INFO: No SessionScavenger set, using defaults
2020-07-06 15:27:47  INFO: node0 Scavenging every 600000ms
2020-07-06 15:27:51  INFO: Started o.e.j.s.ServletContextHandler@65a2e14e{/,null

Anton Tananaev 6 years ago

Can we see a full log fragment from one restart to another?

5of9 6 years ago

2020-07-06 15:26:47 INFO: HikariPool-1 - Starting...
2020-07-06 15:26:48 INFO: HikariPool-1 - Start completed.
2020-07-06 15:26:50 INFO: Clearing database change log checksums
2020-07-06 15:26:50 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOGLOCK
2020-07-06 15:26:50 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOGLOCK
2020-07-06 15:26:50 INFO: SELECT LOCKED FROM gps.DATABASECHANGELOGLOCK WHERE ID=1
2020-07-06 15:26:50 INFO: Successfully acquired change log lock
2020-07-06 15:26:51 INFO: SELECT MD5SUM FROM gps.DATABASECHANGELOG WHERE MD5SUM IS NOT NULL LIMIT 1
2020-07-06 15:26:51 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = NULL
2020-07-06 15:26:51 INFO: Successfully released change log lock
2020-07-06 15:26:51 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOGLOCK
2020-07-06 15:26:51 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOGLOCK
2020-07-06 15:26:51 INFO: SELECT LOCKED FROM gps.DATABASECHANGELOGLOCK WHERE ID=1
2020-07-06 15:26:51 INFO: Successfully acquired change log lock
2020-07-06 15:26:51 INFO: Can not use class liquibase.parser.core.yaml.YamlChangeLogParser as a Liquibase service because org.yaml.snakeyaml.constructor.BaseConstructor is not in the classpath
2020-07-06 15:26:51 INFO: Can not use class liquibase.parser.core.json.JsonChangeLogParser as a Liquibase service because org.yaml.snakeyaml.constructor.BaseConstructor is not in the classpath
2020-07-06 15:26:55 INFO: SELECT MD5SUM FROM gps.DATABASECHANGELOG WHERE MD5SUM IS NOT NULL LIMIT 1
2020-07-06 15:26:55 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOG
2020-07-06 15:26:55 INFO: Reading from gps.DATABASECHANGELOG
2020-07-06 15:26:55 INFO: SELECT * FROM gps.DATABASECHANGELOG ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:ea6f202534dc7845fd1a49e1d8384d9c' WHERE ID = 'changelog-3.3' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.3'
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:194e3ae7ec6d201917136e04a26e237b' WHERE ID = 'changelog-3.5' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.5'
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:1ec7941b03749f24e365bc6126b07e1a' WHERE ID = 'changelog-3.6' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.6'
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:2e444e379f31d0b8e09273ec7ffe87a0' WHERE ID = 'changelog-3.7' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.7'
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:974c33d2fb399ef6477c3897450fb078' WHERE ID = 'changelog-3.7-notmssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.7'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:74d68027951f8d2ae6bb2e24df8365bc' WHERE ID = 'changelog-3.8' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.8'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:34822842d65deb843a7d16f857d52ecc' WHERE ID = 'changelog-3.9' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.9'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:e1ddbe83e1ecf856a912755fc118f82e' WHERE ID = 'changelog-3.10' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.10'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:191c21d8f0f921845cf93bbc9d0639b9' WHERE ID = 'changelog-3.10-notmssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.10'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:ad1f63566e8d08812fbf0b93a118ef6e' WHERE ID = 'changelog-3.10-mssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.10'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:127b36b9d32a9d236df51d19b18c3766' WHERE ID = 'changelog-3.7-mssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.10'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:7800f890b9706a480bd5a79b591b6ca7' WHERE ID = 'changelog-3.11' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.11'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:5ce520811d626ad325a014b9fcbb1a13' WHERE ID = 'changelog-3.12' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.12'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:a3bf7fabcde29e106fe2f89829a76a84' WHERE ID = 'changelog-3.12-notmssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.12'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:cfc881bd2dadb561aa9c1a467bc8cc1c' WHERE ID = 'changelog-3.12-pgsql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.12'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:1be7e6c0520f8be53ef1b099d96afba5' WHERE ID = 'changelog-3.14' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.14'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:ac8ccec176027e98bbf45dc6c15d9b05' WHERE ID = 'changelog-3.15' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.15'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:b59407d70bfe9bf57bc9968adfbbdf1c' WHERE ID = 'changelog-3.16' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.16'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:3f14c3b08068eb7628d0d3e2941eb2d3' WHERE ID = 'changelog-3.3-admin' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.17'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:8aa3e56afe6ba86cd004fa801507bc28' WHERE ID = 'changelog-3.17' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.17'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:90aedfa378aa717f8d8ae541f97b87b2' WHERE ID = 'changelog-4.0-renaming' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:ac63c4153f5b2ee5c7a07056da269571' WHERE ID = 'changelog-4.0' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:d59fa93049c031a8bcfbc02958c1942c' WHERE ID = 'changelog-4.0-clean' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0-clean'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:1adabae3238e02ccc97cc422359a58a0' WHERE ID = 'changelog-4.0-clean-common' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0-clean'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:3974bfe5a2e962c0cd663433c832c16a' WHERE ID = 'changelog-4.0-pre' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:65fb49c5be37693183708351c507dd50' WHERE ID = 'changelog-4.0-common' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:9831511507d8ae1d6759c8ccf506a27a' WHERE ID = 'changelog-4.0-pg' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:b148f52efe9c6a3e74a56e33e257a3e2' WHERE ID = 'changelog-4.1-mssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.1'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:a23040cfc84d4b8021f40ee214a9d78b' WHERE ID = 'changelog-4.7' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.7'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:805ef202791dcc246373ca16f7c19c4f' WHERE ID = 'changelog-4.9' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.9'
2020-07-06 15:26:58 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOG
2020-07-06 15:26:58 INFO: Reading from gps.DATABASECHANGELOG
2020-07-06 15:26:58 INFO: SELECT * FROM gps.DATABASECHANGELOG ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
2020-07-06 15:26:59 INFO: Successfully released change log lock
2020-07-06 15:26:59 INFO: Logging initialized @14570ms to org.eclipse.jetty.util.log.Slf4jLog
2020-07-06 15:27:00 INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2020-07-06 15:27:02 INFO: Operating system name: Linux version: 5.4.0-40-generic architecture: amd64
2020-07-06 15:27:02 INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: Oracle Corporation version: 11.0.5+10-LTS
2020-07-06 15:27:02 INFO: Memory limit heap: 359mb non-heap: 0mb
2020-07-06 15:27:02 INFO: Character encoding: UTF-8 charset: UTF-8
2020-07-06 15:27:02 INFO: Version: 4.9
2020-07-06 15:27:02 INFO: Starting server...
2020-07-06 15:27:03 INFO: jetty-9.4.30.v20200611; built: 2020-06-11T12:34:51.929Z; git: 271836e4c1f4612f12b7bb13ef5a92a927634b0d; jvm 11.0.5+10-LTS
2020-07-06 15:27:03 INFO: Started o.t.w.@1179fc8c{/,null,AVAILABLE}
2020-07-06 15:27:03 INFO: DefaultSessionIdManager workerName=node0
2020-07-06 15:27:03 INFO: No SessionScavenger set, using defaults
2020-07-06 15:27:03 INFO: node0 Scavenging every 600000ms
2020-07-06 15:27:06 INFO: Started o.e.j.s.ServletContextHandler@65a2e14e{/,null,AVAILABLE}
2020-07-06 15:27:06 INFO: Started ServerConnector@2abbd0d9{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
2020-07-06 15:27:06 INFO: Started @21649ms
2020-07-06 15:27:07 INFO: Health check enabled with period 80000
2020-07-06 15:27:29 INFO: Shutting down server...

Anton Tananaev 6 years ago

Maybe you need to increase the timeout? Your server seems to be pretty slow.

thomas 6 years ago

Same problem here on a cheap VM running debian, host operated by netcup.de (the smallest VM for 2,36€/month), if I directly run traccar it keeps runnning. I am just checking out longer timeouts :-)

Anton Tananaev 6 years ago

You can also remove health checks in the systemd config.

5of9 6 years ago

Thank you, maybe this could be a chance. Put the watchdog in systemd config file to 600 at slow machines, worked for the first short test!

Juan Sánchez 6 years ago

Hello,

How can i remove healthcheck on systemd?

Regards,

Mindreader 6 years ago

Where can we remove it ?

thomas 6 years ago

On my Debian it's quite simple, just edit the service-file:

nano /etc/systemd/system/traccar.service

My file looks like that now:

[Unit]
Description=traccar
After=network.target

[Service]
Type=simple
WorkingDirectory=/opt/traccar
ExecStart=/opt/traccar/jre/bin/java -jar tracker-server.jar conf/traccar.xml
SyslogIdentifier=traccar
SuccessExitStatus=143
WatchdogSec=300
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

after editing I once needed to execute "systemctl daemon-reload"

Now with the 300sec no problems with the services for about 3 hours now :-)

5of9 6 years ago

Other possibility is to disable systemd watchdog as you say and build an easy diy watchdog as I did before: Cronjob which checks the last database entry date and if its longer as it should be (you know how often positions should come minimum) than restart the service...

Juan Sánchez 6 years ago

On raspberry pi 3b, 1000000 is not enough

5of9 6 years ago

Other possibility than edit systemd file should be traccar config:

<key='web.healthCheck'>false</entry>

Am I right!? Maybe this should be in the config manual file with text like this:
To disable the systemd watchdog on linux machines (for example on slow machines) this can be set to false