Device location not updating after 1-2 days server is been up

Shubham Aggarwal7 years ago

Hi,

We started using traccar few days back. We have been observing an issue with device position update feature.

Sample api we are using to update location of device:

https://traccar-xyz.com/?id=1234567890000&lat=13.19689295&lon=77.70693205&altitude=819.0&speed=0&hdop=3&timestamp=1530514091000

This seems to work for sometime (1-2 days) and after that positions are not being updated neither in the database nor the cache layer (tried hitting the GET positions api for that particular device) as if the api stops working completely altogether.

Tried checking logs for any errors. One difference i noticed that when the update stops working there are logs of below type:

2018-07-02 10:39:00 DEBUG: [0E99EB33: 5055 > 127.0.0.1] HEX: 485454502f312e3120323030204f4b0d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 10:39:00 DEBUG: [0E99EB33: 5055 < 127.0.0.1] HEX: 504f5354202f3f69643d383634343036303230343631363939266c61743d31322e3838383832313738266c6f6e3d37372e353733353532343426616c7469747564653d3833372e302673706565643d302668646f703d382674696d657374616d703d3135333035323634303938313720485454502f312e310d0a4d61782d466f7277617264733a2031300d0a582d4f726967696e616c2d55524c3a202f3f69643d383634343036303230343631363939266c61743d31322e3838383832313738266c6f6e3d37372e353733353532343426616c7469747564653d3833372e302673706565643d302668646f703d382674696d657374616d703d313533303532363430393831370d0a557365722d4167656e743a204170616368652d48747470436c69656e742f342e352e3220284a6176612f312e382e305f313132290d0a582d466f727761726465642d50726f746f3a2068747470730d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a582d466f727761726465642d466f723a2035322e37342e3135332e35333a35373830320d0a582d4f524947494e414c2d484f53543a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d464f525741524445442d504f52543a203434330d0a582d4152522d53534c3a20323034387c3132387c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74647c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74640d0a582d4152522d4c4f472d49443a2033336165336138362d396333312d346566302d386634632d6134663061653138393664350d0a5669613a20687474702f312e3120747261636361722d312d766d0d0a582d466f727761726465642d466f723a203137322e32302e302e340d0a582d466f727761726465642d50726f746f3a20687474700d0a582d466f727761726465642d486f73743a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d466f727761726465642d5365727665723a203137322e32302e312e340d0a486f73743a206c6f63616c686f73743a353035350d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 10:39:00 DEBUG: [0E99EB33: 5055 > 127.0.0.1] HEX: 485454502f312e3120323030204f4b0d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 10:39:20 DEBUG: [0E99EB33: 5055 < 127.0.0.1] HEX: 504f5354202f3f69643d383634343036303230303936353435266c61743d31392e3134303735303138266c6f6e3d37322e383237313339323126616c7469747564653d2d35372e302673706565643d312668646f703d362674696d657374616d703d3135333035323739333337323620485454502f312e310d0a4d61782d466f7277617264733a2031300d0a582d4f726967696e616c2d55524c3a202f3f69643d383634343036303230303936353435266c61743d31392e3134303735303138266c6f6e3d37322e383237313339323126616c7469747564653d2d35372e302673706565643d312668646f703d362674696d657374616d703d313533303532373933333732360d0a557365722d4167656e743a204170616368652d48747470436c69656e742f342e352e3220284a6176612f312e382e305f313132290d0a582d466f727761726465642d50726f746f3a2068747470730d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a582d466f727761726465642d466f723a2035322e37342e3135332e35333a35383234380d0a582d4f524947494e414c2d484f53543a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d464f525741524445442d504f52543a203434330d0a582d4152522d53534c3a20323034387c3132387c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74647c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74640d0a582d4152522d4c4f472d49443a2036333336666161352d626164362d343130372d623566642d6462386235303032356166360d0a5669613a20687474702f312e3120747261636361722d312d766d0d0a582d466f727761726465642d466f723a203137322e32302e302e340d0a582d466f727761726465642d50726f746f3a20687474700d0a582d466f727761726465642d486f73743a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d466f727761726465642d5365727665723a203137322e32302e312e340d0a486f73743a206c6f63616c686f73743a353035350d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 10:39:20 DEBUG: [0E99EB33: 5055 > 127.0.0.1] HEX: 485454502f312e3120323030204f4b0d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 10:39:20 DEBUG: [0E99EB33: 5055 < 127.0.0.1] HEX: 504f5354202f3f69643d383634343036303230353133393239266c61743d31382e39343136323134266c6f6e3d37322e383332303533373226616c7469747564653d2d36312e302673706565643d31382668646f703d332674696d657374616d703d3135333035323739343233343520485454502f312e310d0a4d61782d466f7277617264733a2031300d0a582d4f726967696e616c2d55524c3a202f3f69643d383634343036303230353133393239266c61743d31382e39343136323134266c6f6e3d37322e383332303533373226616c7469747564653d2d36312e302673706565643d31382668646f703d332674696d657374616d703d313533303532373934323334350d0a557365722d4167656e743a204170616368652d48747470436c69656e742f342e352e3220284a6176612f312e382e305f313132290d0a582d466f727761726465642d50726f746f3a2068747470730d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a582d466f727761726465642d466f723a2035322e37342e3135332e35333a35383235300d0a582d4f524947494e414c2d484f53543a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d464f525741524445442d504f52543a203434330d0a582d4152522d53534c3a20323034387c3132387c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74647c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74640d0a582d4152522d4c4f472d49443a2030626537326339612d373366332d343030332d383734632d3266326165336437666431650d0a5669613a20687474702f312e3120747261636361722d312d766d0d0a582d466f727761726465642d466f723a203137322e32302e302e340d0a582d466f727761726465642d50726f746f3a20687474700d0a582d466f727761726465642d486f73743a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d466f727761726465642d5365727665723a203137322e32302e312e340d0a486f73743a206c6f63616c686f73743a353035350d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 10:39:20 DEBUG: [0E99EB33: 5055 > 127.0.0.1] HEX: 485454502f312e3120323030204f4b0d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 10:39:30  INFO: [BB861AAE] disconnected
2018-07-02 10:39:40 DEBUG: [0E99EB33: 5055 < 127.0.0.1] HEX: 504f5354202f3f69643d383634343036303230313631343839266c61743d31322e3631363833333535266c6f6e3d37362e303331313339323926616c7469747564653d3734372e302673706565643d35342668646f703d332674696d657374616d703d3135333035323739353534303020485454502f312e310d0a4d61782d466f7277617264733a2031300d0a582d4f726967696e616c2d55524c3a202f3f69643d383634343036303230313631343839266c61743d31322e3631363833333535266c6f6e3d37362e303331313339323926616c7469747564653d3734372e302673706565643d35342668646f703d332674696d657374616d703d313533303532373935353430300d0a557365722d4167656e743a204170616368652d48747470436c69656e742f342e352e3220284a6176612f312e382e305f313132290d0a582d466f727761726465642d50726f746f3a2068747470730d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a582d466f727761726465642d466f723a2035322e37342e3135332e35333a35343232380d0a582d4f524947494e414c2d484f53543a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d464f525741524445442d504f52543a203434330d0a582d4152522d53534c3a20323034387c3132387c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74647c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74640d0a582d4152522d4c4f472d49443a2034353962373339342d343530632d343963632d383836352d3535383236623261613939310d0a5669613a20687474702f312e3120747261636361722d312d766d0d0a582d466f727761726465642d466f723a203137322e32302e302e340d0a582d466f727761726465642d50726f746f3a20687474700d0a582d466f727761726465642d486f73743a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d466f727761726465642d5365727665723a203137322e32302e312e340d0a486f73743a206c6f63616c686f73743a353035350d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 10:39:40 DEBUG: [0E99EB33: 5055 > 127.0.0.1] HEX: 485454502f312e3120323030204f4b0d0a436f6e74656e742d4c656e6774683a20300d0a0d0a

While the same api works again after restarting the server, the logs are like below

2018-07-02 23:59:40 DEBUG: [79BCB2F9: 5055 < 127.0.0.1] HEX: 504f5354202f3f69643d383634343036303230383131353437266c61743d32382e3433353734393932266c6f6e3d37372e313735373332323726616c7469747564653d3232372e302673706565643d362668646f703d392674696d657374616d703d3135333035373536353735353020485454502f312e310d0a4d61782d466f7277617264733a2031300d0a582d4f726967696e616c2d55524c3a202f3f69643d383634343036303230383131353437266c61743d32382e3433353734393932266c6f6e3d37372e313735373332323726616c7469747564653d3232372e302673706565643d362668646f703d392674696d657374616d703d313533303537353635373535300d0a557365722d4167656e743a204170616368652d48747470436c69656e742f342e352e3220284a6176612f312e382e305f313132290d0a582d466f727761726465642d50726f746f3a2068747470730d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a582d466f727761726465642d466f723a2035322e37342e3135332e35333a35343631360d0a582d4f524947494e414c2d484f53543a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d464f525741524445442d504f52543a203434330d0a582d4152522d53534c3a20323034387c3132387c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74647c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74640d0a582d4152522d4c4f472d49443a2033303335333738642d636336372d343261642d626234302d3939353039643465656662310d0a5669613a20687474702f312e3120747261636361722d312d766d0d0a582d466f727761726465642d466f723a203137322e32302e302e340d0a582d466f727761726465642d50726f746f3a20687474700d0a582d466f727761726465642d486f73743a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d466f727761726465642d5365727665723a203137322e32302e312e340d0a486f73743a206c6f63616c686f73743a353035350d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 23:59:40 DEBUG: [79BCB2F9: 5055 > 127.0.0.1] HEX: 485454502f312e3120323030204f4b0d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 23:59:40 DEBUG: [79BCB2F9: 5055 < 127.0.0.1] HEX: 504f5354202f3f69643d383634343036303230383131353437266c61743d32382e3433353734393932266c6f6e3d37372e313735373332323726616c7469747564653d3232372e302673706565643d362668646f703d392674696d657374616d703d3135333035373530353735333520485454502f312e310d0a4d61782d466f7277617264733a2031300d0a582d4f726967696e616c2d55524c3a202f3f69643d383634343036303230383131353437266c61743d32382e3433353734393932266c6f6e3d37372e313735373332323726616c7469747564653d3232372e302673706565643d362668646f703d392674696d657374616d703d313533303537353035373533350d0a557365722d4167656e743a204170616368652d48747470436c69656e742f342e352e3220284a6176612f312e382e305f313132290d0a582d466f727761726465642d50726f746f3a2068747470730d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a582d466f727761726465642d466f723a2035322e37342e3135332e35333a33333239340d0a582d4f524947494e414c2d484f53543a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d464f525741524445442d504f52543a203434330d0a582d4152522d53534c3a20323034387c3132387c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74647c433d494e2c20533d4b61726e6174616b612c204f3d496e7465726e6574205769646769747320507479204c74640d0a582d4152522d4c4f472d49443a2030356135353439612d306536312d343134312d613833612d6237616634646264383937300d0a5669613a20687474702f312e3120747261636361722d312d766d0d0a582d466f727761726465642d466f723a203137322e32302e302e350d0a582d466f727761726465642d50726f746f3a20687474700d0a582d466f727761726465642d486f73743a20747261636361722d6f6c612e736f757468696e6469612e636c6f75646170702e617a7572652e636f6d0d0a582d466f727761726465642d5365727665723a203137322e32302e312e340d0a486f73743a206c6f63616c686f73743a353035350d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 23:59:40 DEBUG: [79BCB2F9: 5055 > 127.0.0.1] HEX: 485454502f312e3120323030204f4b0d0a436f6e74656e742d4c656e6774683a20300d0a0d0a
2018-07-02 23:59:40  INFO: [79BCB2F9] id: 864406020798314, time: 2018-07-02 23:59:06, lat: 18.48490, lon: 73.93633, speed: 13.0, course: 0.0
2018-07-02 23:59:41  WARN: Geocoding failed - Empty address - GeocoderException (JsonGeocoder:69 < *:34 < *:96 < ...)
2018-07-02 23:59:41  INFO: [3C6B26CB] id: 864406020506030, time: 2018-07-02 23:54:15, lat: 12.71304, lon: 76.70259, speed: 22.0, course: 0.0
2018-07-02 23:59:41  WARN: Empty address
2018-07-02 23:59:41  INFO: [79BCB2F9] id: 864406020811547, time: 2018-07-02 23:54:17, lat: 28.43575, lon: 77.17573, speed: 6.0, course: 0.0
2018-07-02 23:59:41  INFO: [79BCB2F9] id: 864406020811547, time: 2018-07-02 23:44:17, lat: 28.43575, lon: 77.17573, speed: 6.0, course: 0.0

Get decoded payload in logs when api works. When it doesn't there are only hex values.

Checked system resources consumption also. CPU utilization is around 1% and memory utilization is around 10-12%.

Has this issue been noticed before? Please suggest what should be the further steps to resolve this.

Thanks

Anton Tananaev7 years ago

Try disabling geocoding and see if it helps.

Shubham Aggarwal7 years ago

Hi Anton!

I tried below things:

  1. Disable geocoding (as suggested)
  2. Definite Connection timeout limit to 12 minutes.

Things seems to working fine now. But as a feature request reverse geocoding is essential to product.

What could be the reason behind this? or I we want to use reverse geocoding then what could be the possible solution? Would a paid subscription of reverse geocoding be able to help in this issue?

Thanks

Anton Tananaev7 years ago

You can try to use reverse geocoding by request only. Also try different provider.