503 Service Unavailable - after upgrade to 4.6

Glenn Miler 6 years ago

Hi Guys

After upgrading i simply can't connect to the server, i know it is working and i have been looking apache logs and traccar logs but nothing seems to help i even tried to reinstall everything, but nothing helps.

Traccar log

2019-10-27 17:39:11  INFO: HikariPool-1 - Starting...
2019-10-27 17:39:11  INFO: HikariPool-1 - Start completed.
2019-10-27 17:39:12  INFO: Clearing database change log checksums
2019-10-27 17:39:12  INFO: SELECT COUNT(*) FROM db1002847.DATABASECHANGELOGLOCK
2019-10-27 17:39:12  INFO: SELECT COUNT(*) FROM db1002847.DATABASECHANGELOGLOCK
2019-10-27 17:39:12  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:42:12  INFO: Waiting for changelog lock....
2019-10-27 17:42:22  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:42:22  INFO: Waiting for changelog lock....
2019-10-27 17:42:32  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:42:32  INFO: Waiting for changelog lock....
2019-10-27 17:42:42  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:42:42  INFO: Waiting for changelog lock....
2019-10-27 17:42:52  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:42:52  INFO: Waiting for changelog lock....
2019-10-27 17:43:02  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:43:02  INFO: Waiting for changelog lock....
2019-10-27 17:43:12  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:43:12  INFO: Waiting for changelog lock....
2019-10-27 17:43:22  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:43:22  INFO: Waiting for changelog lock....
2019-10-27 17:43:32  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:43:32  INFO: Waiting for changelog lock....
2019-10-27 17:43:42  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:43:42  INFO: Waiting for changelog lock....
2019-10-27 17:43:52  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:43:52  INFO: Waiting for changelog lock....
2019-10-27 17:44:02  INFO: SELECT `LOCKED` FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:44:02  INFO: Waiting for changelog lock....
2019-10-27 17:44:12  INFO: SELECT ID,`LOCKED`,LOCKGRANTED,LOCKEDBY FROM db1002847.DATABASECHANGELOGLOCK WHERE ID=1
2019-10-27 17:44:12 ERROR: Main method error - Could not acquire change log lock.  Currently locked by 2a01:4f8:c2c:6fe5:0:0:0:1%eth0 (2a01:4f8:c2c:6fe5:0:0:0:1%eth0) since 10/27/19, 9:34 AM - LockException (... < DataManager:310 < *:90 < Context:292 < Main:110 < ...)

Apache Log

[Sun Oct 27 17:37:37.407080 2019] [proxy:error] [pid 942:tid 139780658497280] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8082 (localhost) failed
[Sun Oct 27 17:37:37.407178 2019] [proxy_http:error] [pid 942:tid 139780658497280] [client 82.103.130.178:8751] AH01114: HTTP: failed to make connection to backend: localhost
[Sun Oct 27 17:37:39.522094 2019] [proxy:error] [pid 942:tid 139780641711872] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8082 (localhost) failed
[Sun Oct 27 17:37:39.522161 2019] [proxy_http:error] [pid 942:tid 139780641711872] [client 82.103.130.178:18111] AH01114: HTTP: failed to make connection to backend: localhost

Traccar Status

root@track:/opt/traccar/logs# sudo systemctl status traccar.service
● traccar.service - traccar
   Loaded: loaded (/etc/systemd/system/traccar.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2019-10-27 17:54:01 CET; 41s ago
  Process: 1785 ExecStart=/opt/traccar/jre/bin/java -jar tracker-server.jar conf/traccar.xml (code=exited, status=1/FAILURE)
 Main PID: 1785 (code=exited, status=1/FAILURE)

Oct 27 17:54:01 track.seginova.com traccar[1785]:         at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:236)
Oct 27 17:54:01 track.seginova.com traccar[1785]:         at liquibase.Liquibase.clearCheckSums(Liquibase.java:1455)
Oct 27 17:54:01 track.seginova.com traccar[1785]:         at org.traccar.database.DataManager.initDatabaseSchema(DataManager.java:310)
Oct 27 17:54:01 track.seginova.com traccar[1785]:         at org.traccar.database.DataManager.<init>(DataManager.java:90)
Oct 27 17:54:01 track.seginova.com traccar[1785]:         at org.traccar.Context.init(Context.java:292)
Oct 27 17:54:01 track.seginova.com traccar[1785]:         at org.traccar.Main.run(Main.java:110)
Oct 27 17:54:01 track.seginova.com traccar[1785]:         ... 1 more
Oct 27 17:54:01 track.seginova.com systemd[1]: traccar.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 17:54:01 track.seginova.com systemd[1]: traccar.service: Unit entered failed state.
Oct 27 17:54:01 track.seginova.com systemd[1]: traccar.service: Failed with result 'exit-code'.

Can any help where i need to look, it has worked before this with out any problems.

Anton Tananaev 6 years ago

Your database is locked. Usually it means that you restarted in the middle of the migration. Search forum. It has been discussed many times.

Glenn Miler 6 years ago

Hi Anton

Thanks for answering me back it was also more like pointing in my right direction instead of looking 1000 places for fixes :)

And you where right.

I open phpmyadmin and in the table DATABASECHANGELOGLOCK i changed LOCKED from 1 to 0 and now everything is working again. thanks.

Hussah Saleh 5 years ago

Hi All,
Well, I am pointing two traccar server to the same database. Old one working fine while the new traccar server always get 503 even if I update the locked to 0 on DATABASECHANGELOGLOCK.

any help how can I make those two traccar servers pointing to the same DB without any issues.

Thank you,

Hussah Saleh 5 years ago

This is my log file:

2020-08-04 02:40:55  INFO: UPDATE traccar.DATABASECHANGELOG SET MD5SUM = '8:805ef202791dcc246373ca16f7c19c4f' WHERE ID = 'changelog-4.9' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.9'
2020-08-04 02:40:56  INFO: SELECT COUNT(*) FROM traccar.DATABASECHANGELOG
2020-08-04 02:40:56  INFO: Reading from traccar.DATABASECHANGELOG
2020-08-04 02:40:56  INFO: SELECT * FROM traccar.DATABASECHANGELOG ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
2020-08-04 02:40:56  INFO: Successfully released change log lock
2020-08-04 02:40:57  INFO: Logging initialized @22431ms to org.eclipse.jetty.util.log.Slf4jLog
2020-08-04 02:40:59  INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2020-08-04 02:41:01  INFO: Operating system name: Linux version: 4.15.0-112-generic architecture: amd64
2020-08-04 02:41:01  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: Oracle Corporation version: 11.0.5+10-LTS
Hussah Saleh 5 years ago

Sorry for non clear log file, here is my log file:

This is my log file:

2020-08-04 02:48:08  INFO: SELECT MD5SUM FROM traccar.DATABASECHANGELOG WHERE MD5SUM IS NOT NULL LIMIT 1
2020-08-04 02:48:08  INFO: SELECT COUNT(*) FROM traccar.DATABASECHANGELOG
2020-08-04 02:48:08  INFO: Reading from traccar.DATABASECHANGELOG
2020-08-04 02:48:08  INFO: SELECT * FROM traccar.DATABASECHANGELOG ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
2020-08-04 02:48:08  INFO: UPDATE traccar.DATABASECHANGELOG SET MD5SUM = '8:ea6f202534dc7845fd1a49e1d8384d9c' WHERE ID = 'changelog-3.3' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.3'
2020-08-04 02:48:08  INFO: UPDATE traccar.DATABASECHANGELOG SET MD5SUM = '8:194e3ae7ec6d201917136e04a26e237b' WHERE ID = 'changelog-3.5' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.5'
.
.
.
2020-08-04 02:48:17  INFO: UPDATE traccar.DATABASECHANGELOG SET MD5SUM = '8:a23040cfc84d4b8021f40ee214a9d78b' WHERE ID = 'changelog-4.7' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.7'
2020-08-04 02:48:17  INFO: UPDATE traccar.DATABASECHANGELOG SET MD5SUM = '8:805ef202791dcc246373ca16f7c19c4f' WHERE ID = 'changelog-4.9' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.9'
2020-08-04 02:48:17  INFO: SELECT COUNT(*) FROM traccar.DATABASECHANGELOG
2020-08-04 02:48:17  INFO: Reading from traccar.DATABASECHANGELOG
2020-08-04 02:48:17  INFO: SELECT * FROM traccar.DATABASECHANGELOG ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
2020-08-04 02:48:18  INFO: Successfully released change log lock
2020-08-04 02:48:19  INFO: Logging initialized @22201ms to org.eclipse.jetty.util.log.Slf4jLog
2020-08-04 02:48:21  INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2020-08-04 02:48:23  INFO: Operating system name: Linux version: 4.15.0-112-generic architecture: amd64
2020-08-04 02:48:23  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: Oracle Corporation version: 11.0.5+10-LTS