Upgraded a production server from 5.10 to 6.14.5 (installer, Liquibase migration clean, PostgreSQL). From the very first hour, position storage collapsed while devices stayed connected:
- Fleet: ~540 registered devices, ~230 active (gt06, startek, osmand). PostgreSQL 8GB droplet.
- Healthy on 5.10: ~45–50k positions/hour, ~165 distinct devices/hour.
- On 6.14.5: hourly rate decayed 22k → 15k → 10k → 5k over 10 hours; distinct devices with a fresh fix fell to ~19 and stayed there. The surviving ~19 were a fixed subset (mixed protocols, mixed user counts — no pattern we could find).
- Devices remained CONNECTED the whole time: 165 devices with lastupdate < 10 min, heartbeats flowing, /api/devices reporting them online — but their positions were not stored. No decoder errors, zero "filtered" log lines (filter config unchanged from 5.10), geocoder healthy (0.12s responses).
- gt06 rows that did get stored had a median servertime - fixtime lag of ~16 hours — i.e., only buffered replays were being accepted; live gt06 traffic effectively dead. startek rows that got stored had lag 0 but the set of startek devices storing anything shrank hour by hour.
- systemctl restart traccar did NOT recover it — connections re-established (148), but the same ~19-device subset kept storing and everyone else stayed frozen.
- Thread dump (kill -3) during the freeze: 52 threads, none BLOCKED, nothing stuck in geocoding or storage — looks like a logic path silently skipping the store, not a deadlock.
- Reverting the binary to 5.10 on the SAME migrated database recovered ingestion instantly (17 → 110+ fresh devices in minutes).
Separate defect found on 6.14.5 with the migrated schema: every audit insert fails —
INSERT INTO tc_actions(actionType, actionTime, address, attributes, userId, objectId, objectType) ... ERROR: integer out of range (PostgreSQL), logged as "Failed to store action create".
This resembles the old "device freezing" threads (6.3/6.4 era) but with restart NOT clearing it. Happy to provide the full thread dump, per-protocol hourly counts, and config (filter.enable=true, filter.invalid=true, geocoder nominatim with ignorePositions=false). What logging should we enable to catch the drop point if we attempt 6.x again?
Possible root cause found. Our config geocodes every position (geocoder.ignorePositions false) against a self-hosted Nominatim. If 6.14.5 processes positions in per-device queues that advance only on the geocoder callback, and the shared HTTP client has no timeouts (added in commit 17292f98 on 30 July, after 6.14.5), then a single hung geocoder request would permanently pin that device's queue — silently, matching everything we saw: devices die one by one, stay online via heartbeats, no errors, no blocked threads, restart re-freezes within minutes. Can you confirm whether that commit addresses this, and whether the fix will be in the next release? Would geocoder.ignorePositions=true be a safe interim workaround on 6.14.5?
It should not freeze permanently. I would definitely recommend disabling automatic geocoding for all positions.
Upgraded a production server from 5.10 to 6.14.5 (installer, Liquibase migration clean, PostgreSQL). From the very first hour, position storage collapsed while devices stayed connected:
Separate defect found on 6.14.5 with the migrated schema: every audit insert fails —
INSERT INTO tc_actions(actionType, actionTime, address, attributes, userId, objectId, objectType) ... ERROR: integer out of range (PostgreSQL), logged as "Failed to store action create".
This resembles the old "device freezing" threads (6.3/6.4 era) but with restart NOT clearing it. Happy to provide the full thread dump, per-protocol hourly counts, and config (filter.enable=true, filter.invalid=true, geocoder nominatim with ignorePositions=false). What logging should we enable to catch the drop point if we attempt 6.x again?