unresponsive server while cleaning old data, lock wait timeout

Erfan 4 years ago

Hi ,
I have a latest traccar server running with about 5K devices and 2K active devices , with mysql database size +20GB ,
recently i noticed the server become unresponsive with 100% cpu usage every day at an specific hour for about 1~2 hours and then become normal again ,
Logs shows the following error :

2020-05-14 12:16:36  WARN: Update device status error - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:135 < DataManager:337 < DeviceManager:268 < ...)
2020-05-14 12:16:36  WARN: Event save error - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:140 < DataManager:456 < NotificationManager:65 < ...)
2020-05-14 12:16:45  WARN: Update device status error - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:135 < DataManager:337 < DeviceManager:268 < ...)
2020-05-14 12:16:45  WARN: Update device status error - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:135 < DataManager:337 < DeviceManager:268 < ...)
2020-05-14 12:16:45  WARN: Update device status error - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:135 < DataManager:337 < DeviceManager:268 < ...)
2020-05-14 12:16:46  WARN: Failed to store position - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:140 < DataManager:456 < DefaultDataHandler:40 < ...)
2020-05-14 12:16:46  WARN: Failed to store position - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:140 < DataManager:456 < DefaultDataHandler:40 < ...)
2020-05-14 12:16:46  WARN: Event save error - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:140 < DataManager:456 < NotificationManager:65 < ...)
2020-05-14 12:16:46  WARN: Failed to update device - HikariPool-1 - Connection is not available, request timed out after 30000ms. - SQLTransientConnectionException (... < QueryBuilder:59 < *:135 < DataManager:351 < DeviceManager:296 < ...)

I have a HP G7 DL360 server with dual 2.66 GHz/4-core and 32GB RAM ,
I tracked down the problem and found out at this time the cleanup old data (<entry key='database.historyDays'>90</entry>) process is being done because of this query is being processed : DELETE FROM tc_positions WHERE serverTime < '2020-02-14 10:51:37.654' AND id NOT IN (SELECT position
the CPU usage become 100% at this time by mysql , traccar can not store some data in database,
i also did the steps on https://www.traccar.org/optimization/ but it didnt help.
is there anyway to smooth the clean-up process so we dont lose data everyday at cleanup time ?
i had the same problem with backing up database everyday with mysqldump , that i could fix it by using "mysqldump -u USER -p --single-transaction --quick --lock-tables=false" while backing up database .

Anders Yuran4 years ago

You must really be at the limit what the server can take. And there is some limitations to mysql also

Anton Tananaev4 years ago

You should do cleanup outside of Traccar. Just write a script that would do it in reasonable batches.