Server Stopped Working (Warn: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit)

code_af 3 years ago

Hello to all,

Today I wanted to send a positionSingle command to a watch protocol's device, during watching the log file, suddenly saw that all devices are disconnecting from the server while all them were online, see the logs before:

2022-09-19 14:35:51  INFO: user: 306, action: login, from: 91.97.97.44
2022-09-19 14:37:16  INFO: [Ta3f01520] id: 8800000004, command type: positionSingle sent
2022-09-19 14:37:16  INFO: [Ta3f01520: watch > 20.14.13.210] 5b53472a383830303030303030342a303030322a43525d
2022-09-19 14:37:41  INFO: [Te21d2770] timed out
2022-09-19 14:37:41  INFO: [Te21d2770] disconnected
2022-09-19 14:38:17  INFO: [T54fde7f5] timed out
2022-09-19 14:38:17  INFO: [T54fde7f5] disconnected
2022-09-19 14:38:45  INFO: user: 306, report: events, from: 2022-08-20 14:38, to: 2022-09-19 14:38, devices: [-1], groups: []
2022-09-19 14:38:45  INFO: user: 306, report: events, from: 2022-08-20 14:38, to: 2022-09-19 14:38, devices: [952], groups: []
2022-09-19 14:38:48  INFO: [T89a2c85c] timed out
2022-09-19 14:38:48  INFO: [T89a2c85c] disconnected
2022-09-19 14:38:53  INFO: [Tcc0acd73] timed out
2022-09-19 14:38:53  INFO: [Tcc0acd73] disconnected
2022-09-19 14:38:56  INFO: [Td9794c6e] timed out
2022-09-19 14:38:56  INFO: [Td9794c6e] disconnected
2022-09-19 14:39:05  INFO: [T70eb653e] timed out
2022-09-19 14:39:05  INFO: [T70eb653e] disconnected
2022-09-19 14:39:07  INFO: [Tcba87f48] timed out
2022-09-19 14:39:07  INFO: [Tcba87f48] disconnected
2022-09-19 14:39:08  INFO: [T81c5323e] timed out
2022-09-19 14:39:08  INFO: [T81c5323e] disconnected
2022-09-19 14:39:10  INFO: [T3ecd7613] timed out
2022-09-19 14:39:10  INFO: [T3ecd7613] disconnected
2022-09-19 14:39:26  INFO: user: 306, action: login, from: 91.97.97.44
2022-09-19 14:39:27  INFO: [Td0a5d097] timed out
2022-09-19 14:39:27  INFO: [Td0a5d097] disconnected
2022-09-19 14:39:34  INFO: [Ta3f01520] timed out
2022-09-19 14:39:34  INFO: [Ta3f01520] disconnected
2022-09-19 14:39:49  INFO: [Te1d00c35] timed out
2022-09-19 14:39:49  INFO: [Te1d00c35] disconnected
2022-09-19 14:40:08  INFO: [T769f2e2a] timed out
2022-09-19 14:40:08  INFO: [T769f2e2a] disconnected
2022-09-19 14:40:13  INFO: user: 306, report: events, from: 2022-08-20 14:40, to: 2022-09-19 14:40, devices: [-1], groups: []
2022-09-19 14:40:13  INFO: user: 306, report: events, from: 2022-08-20 14:40, to: 2022-09-19 14:40, devices: [952], groups: []
2022-09-19 14:40:39  INFO: [T1678f300] timed out
2022-09-19 14:40:39  INFO: [T1678f300] disconnected
2022-09-19 14:40:51  INFO: [Ta86b79c4] timed out
2022-09-19 14:40:51  INFO: [Ta86b79c4] disconnected
2022-09-19 14:40:57  WARN: Notification failed - Connection timed out (Connection timed out) - ConnectException (... < NotificatorFirebase:86 < NotificationManager:104 < ... < *:101 < ... < *:100 < *:124 < ConnectionManager:315 < ...)
2022-09-19 14:40:57  WARN: Failed to get notificator 
2022-09-19 14:40:57  WARN: You are using null notificatior, please check your configuration, notification not sent
2022-09-19 14:40:57  WARN: Failed to get notificator 
2022-09-19 14:40:57  WARN: You are using null notificatior, please check your configuration, notification not sent
2022-09-19 14:41:07  INFO: [Tc6849b4e] timed out
2022-09-19 14:41:07  INFO: [Tc6849b4e] disconnected
2022-09-19 14:41:24  INFO: [Tae5f08ea] timed out
2022-09-19 14:41:24  INFO: [Tae5f08ea] disconnected
2022-09-19 14:41:43  INFO: user: 306, report: events, from: 2022-08-20 14:41, to: 2022-09-19 14:41, devices: [-1], groups: []
2022-09-19 14:41:44  INFO: user: 306, report: events, from: 2022-08-20 14:41, to: 2022-09-19 14:41, devices: [952], groups: []
2022-09-19 14:43:08  WARN: Notification failed - Connection timed out (Connection timed out) - ConnectException (... < NotificatorFirebase:86 < NotificationManager:104 < ... < *:101 < ... < *:100 < *:124 < ConnectionManager:315 < ...)
2022-09-19 14:43:08  WARN: Failed to get notificator 
2022-09-19 14:43:08  WARN: You are using null notificatior, please check your configuration, notification not sent
2022-09-19 14:43:23  INFO: user: 306, action: login, from: 91.97.97.44

I tried to restart the traccar service, but the service stucked at restarting, as you see bellow...

2022-09-19 14:43:38  INFO: Stopping server...

Then I tried again to restart the traccar service with same linux command sudo systemctl restart traccar.service and then saw bellow logs with warning of DOS ALERT:

2022-09-19 14:45:08  INFO: Operating system name: Linux version: 5.4.0-125-generic architecture: amd64
2022-09-19 14:45:09  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: ojdkbuild version: 11.0.15+10-LTS
2022-09-19 14:45:09  INFO: Memory limit heap: 8014mb non-heap: 0mb
2022-09-19 14:45:09  INFO: Character encoding: UTF-8 charset: UTF-8
2022-09-19 14:45:09  INFO: Version: 5.3
2022-09-19 14:45:09  INFO: Starting server...
2022-09-19 14:45:09  INFO: HikariPool-1 - Starting...
2022-09-19 14:45:09  WARN: Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-09-19 14:45:09  INFO: HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@6ac4944a
2022-09-19 14:45:09  INFO: HikariPool-1 - Start completed.
2022-09-19 14:45:09  INFO: Clearing database change log checksums
2022-09-19 14:45:09  INFO: Successfully acquired change log lock
2022-09-19 14:45:10  INFO: Successfully released change log lock
2022-09-19 14:45:10  INFO: Successfully acquired change log lock
2022-09-19 14:45:10  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.json.JsonChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:45:10  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.yaml.YamlChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:45:10  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:45:11  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:45:11  INFO: Cannot load service: liquibase.hub.HubService: Provider liquibase.hub.core.StandardHubService could not be instantiated
2022-09-19 14:45:11  INFO: Successfully released change log lock
2022-09-19 14:45:11  INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2022-09-19 14:45:12  INFO: jetty-10.0.7; built: 2021-10-06T19:34:02.766Z; git: da8a4553af9dd84080931fa0f8c678cd2d60f3d9; jvm 11.0.15+10-LTS
2022-09-19 14:45:12  INFO: Session workerName=node0
2022-09-19 14:45:12  WARN: JAXBContext implementation could not be found. WADL feature is disabled.
2022-09-19 14:45:13  INFO: Started o.e.j.s.ServletContextHandler@40a7974{/,null,AVAILABLE}
2022-09-19 14:45:13  INFO: Started ServerConnector@150fc7a7{HTTP/1.1, (http/1.1)}{0.0.0.0:8082}
2022-09-19 14:45:13  INFO: Started Server@919086{STARTING}[10.0.7,sto=0] @4747ms
2022-09-19 14:45:13  INFO: Health check enabled with period 480000
2022-09-19 14:45:14  INFO: user: 186, action: login, from: 213.12.3.201
2022-09-19 14:45:15  INFO: Stopping server...
2022-09-19 14:45:16  INFO: Operating system name: Linux version: 5.4.0-125-generic architecture: amd64
2022-09-19 14:45:16  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: ojdkbuild version: 11.0.15+10-LTS
2022-09-19 14:45:16  INFO: Memory limit heap: 8014mb non-heap: 0mb
2022-09-19 14:45:16  INFO: Character encoding: UTF-8 charset: UTF-8
2022-09-19 14:45:16  INFO: Version: 5.3
2022-09-19 14:45:16  INFO: Starting server...
2022-09-19 14:45:16  INFO: HikariPool-1 - Starting...
2022-09-19 14:45:16  WARN: Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-09-19 14:45:16  INFO: HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@6ac4944a
2022-09-19 14:45:16  INFO: HikariPool-1 - Start completed.
2022-09-19 14:45:16  INFO: Clearing database change log checksums
2022-09-19 14:45:17  INFO: Successfully acquired change log lock
2022-09-19 14:45:17  INFO: Successfully released change log lock
2022-09-19 14:45:17  INFO: Successfully acquired change log lock
2022-09-19 14:45:17  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.json.JsonChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:45:17  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.yaml.YamlChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:45:18  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:45:18  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:45:18  INFO: Cannot load service: liquibase.hub.HubService: Provider liquibase.hub.core.StandardHubService could not be instantiated
2022-09-19 14:45:18  INFO: Successfully released change log lock
2022-09-19 14:45:19  INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2022-09-19 14:45:19  INFO: jetty-10.0.7; built: 2021-10-06T19:34:02.766Z; git: da8a4553af9dd84080931fa0f8c678cd2d60f3d9; jvm 11.0.15+10-LTS
2022-09-19 14:45:19  INFO: Session workerName=node0
2022-09-19 14:45:20  WARN: JAXBContext implementation could not be found. WADL feature is disabled.
2022-09-19 14:45:20  INFO: Started o.e.j.s.ServletContextHandler@65a4b9d6{/,null,AVAILABLE}
2022-09-19 14:45:20  INFO: Started ServerConnector@798cf6d2{HTTP/1.1, (http/1.1)}{0.0.0.0:8082}
2022-09-19 14:45:20  INFO: Started Server@7ead1d80{STARTING}[10.0.7,sto=0] @4919ms
2022-09-19 14:45:20  INFO: Health check enabled with period 480000
2022-09-19 14:45:23  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2bb89898[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.955S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:23  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@549bd163[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.942S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:23  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@483d893d[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.967S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:23  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@3dfc8d53[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.983S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:25  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@67069727[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.989S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:25  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@3212b144[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.988S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:25  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2f55381b[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.998S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:25  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@26e0e1e[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.994S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@6d55527[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.976S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@47c86214[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.994S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@527904e9[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.946S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@145ba0c8[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.975S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@43a17c99[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.976S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:27  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1edd83dc[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.989S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:27  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@52ca1896[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.978S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:28  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@6c4d178b[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.97S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:28  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@7836dc22[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.967S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:29  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@37068951[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.99S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:29  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1c871510[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.984S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:29  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@6c176aaf[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.985S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:30  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@5fd827f7[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.998S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:30  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@410c122a[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.969S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:30  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@17f7f88f[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.972S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:30  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@31e984fb[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.98S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:31  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@14422212[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.982S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:31  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1181e089[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.96S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:31  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1520af20[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.992S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:31  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@384daa72[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.958S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:32  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@6c74949d[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.969S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:32  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@4dc32c26[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.918S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:32  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2b94533b[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.957S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:33  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2e99a759[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.994S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:34  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@3456d4ae[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.991S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:34  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@3a9466d6[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.967S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:36  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1a471a93[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.925S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:36  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@7f55dc8e[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.938S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:36  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@18a49249[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.894S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:36  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@73b42df3[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.999S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:38  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@676c3bdf[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.979S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:38  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2131dee5[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.955S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:39  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@665092ed[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.993S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:39  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@778e75f9[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.988S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:39  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@4b6024ec[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.998S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:40  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@17efba96[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.98S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:46  INFO: user: 186, action: login, from: 213.16.32.32

Then I tried the third time to restart the traccar service, and after restarting it again, it is running normal up to now!

2022-09-19 14:46:07  INFO: Operating system name: Linux version: 5.4.0-125-generic architecture: amd64
2022-09-19 14:46:07  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: ojdkbuild version: 11.0.15+10-LTS
2022-09-19 14:46:07  INFO: Memory limit heap: 8014mb non-heap: 0mb
2022-09-19 14:46:07  INFO: Character encoding: UTF-8 charset: UTF-8
2022-09-19 14:46:07  INFO: Version: 5.3
2022-09-19 14:46:07  INFO: Starting server...
2022-09-19 14:46:07  INFO: HikariPool-1 - Starting...
2022-09-19 14:46:08  WARN: Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-09-19 14:46:08  INFO: HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@6ac4944a
2022-09-19 14:46:08  INFO: HikariPool-1 - Start completed.
2022-09-19 14:46:08  INFO: Clearing database change log checksums
2022-09-19 14:46:08  INFO: Successfully acquired change log lock
2022-09-19 14:46:08  INFO: Successfully released change log lock
2022-09-19 14:46:08  INFO: Successfully acquired change log lock
2022-09-19 14:46:08  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.json.JsonChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:46:08  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.yaml.YamlChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:46:09  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:46:10  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:46:10  INFO: Cannot load service: liquibase.hub.HubService: Provider liquibase.hub.core.StandardHubService could not be instantiated
2022-09-19 14:46:10  INFO: Successfully released change log lock
2022-09-19 14:46:10  INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2022-09-19 14:46:11  INFO: jetty-10.0.7; built: 2021-10-06T19:34:02.766Z; git: da8a4553af9dd84080931fa0f8c678cd2d60f3d9; jvm 11.0.15+10-LTS
2022-09-19 14:46:11  INFO: Session workerName=node0
2022-09-19 14:46:11  WARN: JAXBContext implementation could not be found. WADL feature is disabled.
2022-09-19 14:46:11  INFO: Started o.e.j.s.ServletContextHandler@3dea1ecc{/,null,AVAILABLE}
2022-09-19 14:46:11  INFO: Started ServerConnector@150fc7a7{HTTP/1.1, (http/1.1)}{0.0.0.0:8082}
2022-09-19 14:46:11  INFO: Started Server@919086{STARTING}[10.0.7,sto=0] @4697ms
2022-09-19 14:46:11  INFO: Health check enabled with period 480000

I didn't get where this problem comes from, or how to solve this for the future, does anyone else faced the same problem?

Thanks in advance for your help!

Anton Tananaev 3 years ago

It seems like you mixed up several completely unrelated things into one. A single command is probably unrelated to all disconnected devices. DOS alert is probably unrelated to restarting the service. etc

code_af 3 years ago

Thank you @Anton for you reply.

Maybe they are unrelated to each other, but don't you think there is a reason or problem exists for them? I am concerned for two things:

DOS Alert, was it really a DDOS attack or something? Is there a way to find more about this on our server side, maybe where do you think will be helpful to look into it?
Disconnecting devices are concerned but I don't think so if it was because of single command, but how it was possible that all devices were timed-out and disconnected from the server in a single minute, I am sure that these all devices didn't turn-off at the same time? Even I can see the firebase connection was also timed-out, therefore I am concerned that how it was possible?

Anton Tananaev 3 years ago

My point is that I don't think it's a good idea to discuss all the issues at the same time.

code_af 3 years ago

Okay. Then I will open a separate discussion for each issue.

Fabiano Gastaldi 3 years ago

Hi, i had the issue, see:
https://www.traccar.org/forums/topic/real-case-of-dos-alert-high-memory-usage-traccar-server-linux-56/

FVC RASTREAMENTO 8 months ago

Tem como alguém me ajudar a resolver esse problema

WARN 47 WARN: DOS ALERT: Request delayed-100ms, ip = 127, 0 * 0.1 overlimit-OverLimit@7d4cca2a[id-127.0.0.1, duration=PT0.2163995945, count=25], user=null

47 WARN . DOS ALERT: Request delayed=100ms, ip = 127, 0 * 0.1 overlimit=OverLimit@205efd1c[id-127.0.0.1, duration-PT0.2164926175, count-25], user-null

47 WARN: DOS ALERT: Request delayed-100ms, 1p = 127, 0 * 0.1 overlimit-Overlimit@3303da4[id=127.0.0.1, duration=PT0.2169382975, count-25], user-null 47 WARN user-null

: DOS ALERT: Request delayed-100ms, ip = 127 * 0 * 0.1 overlimit-OverLimit@ence45c[id-127.0.0.1, duration-PT0.2087009265, count-25],

47 WARN: DOS ALERT: Request delayed-100ms, 1p = 127.8 * 0.8 * 0.1 overlimit-OverLimit@1f1c7486[id-127.0.0.1, duration=PT0.2235617665, count-25], user-null

47 WARN: DOS ALERT: Request delayed 100ms, ip = 127, 0 * 0.1 overlimit-OverLimit@7ac60e2e[id-127.0.0.1, duration-PTB.2591690695, count=25], user-null

Anton Tananaev 8 months ago

Just increase the limit.

FVC RASTREAMENTO 8 months ago

Opa, você consegue me informar o caminho, para mim aumentar esse limite, não entendo muito, você pode deixar os comandos passo a passo por gentileza

Anton Tananaev 8 months ago

https://www.traccar.org/configuration-file/

FVC RASTREAMENTO 8 months ago

Você não consegue me passar o caminho correto passo a passo

FVC RASTREAMENTO 8 months ago

Aonde que eu posso fazer essa alteração o local correto