How to filter illogical and incorrect positions

Nils Leideck 2 months ago

Dear @Track-trace,

I know about the additional data and attributes from various devices. Also, not every vendor ships the same, even if the claim to use the same protocol. We harmonize data before we run the validation. The minimum validation is this:

/**
 * Validates an incoming Traccar position payload for plausibility.
 *
 * Two failure modes are distinguished:
 *   hardReject – the `valid` flag from Traccar itself is false, meaning
 *                the device signalled that it has no reliable GPS fix.
 *                No position data should be stored, but telemetry
 *                (battery, timestamp) is still valid and should be kept.
 *   softReject – the fix is flagged valid by the device but our own
 *                quality checks (accuracy, satellites, altitude, …) fail.
 *                Same behaviour: position is discarded, telemetry kept.
 *
 * @param {object} data - Raw position object from the Traccar webhook.
 * @returns {{ valid: boolean, hardReject: boolean, reason?: string, errors?: string[], failedChecks?: string[] }}
 */
function validatePosition(data) {
  if (data.valid !== true) {
    return { valid: false, hardReject: true, reason: 'invalid_flag' };
  }

  const errors = [];
  const failedChecks = [];

  if (!data?.deviceId) errors.push('deviceId missing');
  if (typeof data.latitude  !== 'number') errors.push('latitude invalid');
  if (typeof data.longitude !== 'number') errors.push('longitude invalid');
  if (Math.abs(data.latitude)  > 90)  errors.push('latitude out of range');
  if (Math.abs(data.longitude) > 180) errors.push('longitude out of range');

  if (typeof data.satellites === 'number' && data.satellites < MIN_SATELLITES) {
    failedChecks.push(`satellites=${data.satellites}`);
  }
  if (typeof data.rssi === 'number' && (data.rssi < 1 || data.rssi > MAX_RSSI)) {
    failedChecks.push(`rssi=${data.rssi}`);
  }
  if (typeof data.accuracy === 'number' && data.accuracy > MAX_ACCURACY) {
    failedChecks.push(`accuracy=${data.accuracy}`);
  }
  if (typeof data.altitude === 'number' && (data.altitude < -500 || data.altitude > MAX_ALTITUDE)) {
    failedChecks.push(`altitude=${data.altitude}`);
  }
  if (data.motion === false && typeof data.speed === 'number' && data.speed > 5) {
    failedChecks.push('motion_speed_mismatch');
  }

  return {
    valid: errors.length === 0 && failedChecks.length === 0,
    hardReject: false,
    errors,
    failedChecks
  };
}

After that, if this validation has passed, we validate the movement plausibility:

    // --- Phase 4: movement plausibility checks ---
    const prevLat  = pet.last_position_latitude;
    const prevLon  = pet.last_position_longitude;
    const prevTime = pet.last_position_timestamp;

    const species = pet.species || 'other';
    const config  = SPECIES_CONFIG[species] || SPECIES_CONFIG.other;

    const lat = positionData.latitude;
    const lon = positionData.longitude;

    if (prevLat && prevLon && prevTime) {
      const dist     = distanceMeters(prevLat, prevLon, lat, lon);
      const timeDiff = (new Date(positionData.fixTime) - new Date(prevTime)) / 1000;
      const calcSpeed = timeDiff > 0 ? (dist / timeDiff) * 3.6 : null;

      if (timeDiff > 0) {
        if (calcSpeed > config.maxJumpSpeed) {
          log('rejected_jump', { calcSpeed, species });
          return Response.json({ skipped: true, reason: 'gps_jump', calcSpeed });
        }
        if (dist > config.maxDistance) {
          log('rejected_distance', { dist });
          return Response.json({ skipped: true, reason: 'distance_too_large', dist });
        }
        if (dist < MIN_DISTANCE) {
          log('rejected_jitter', { dist });
          return Response.json({ skipped: true, reason: 'jitter' });
        }
        if (timeDiff > 300 && dist > 1000) {
          log('rejected_sleep_jump', { dist, timeDiff });
          return Response.json({ skipped: true, reason: 'sleep_jump' });
        }
      }
    }

After that, we smooth the position:

/**
 * Exponential moving average used to smooth consecutive GPS fixes.
 * A low alpha (0.25) keeps the filtered position close to history,
 * reducing the visual impact of single-fix outliers on the map.
 */
function smooth(prev, current, alpha = 0.25) {
  if (prev === null || prev === undefined) return current;
  return prev + alpha * (current - prev);
}

We calculate distance in meters with Haversine formula:

/**
 * Haversine formula — returns the great-circle distance between two
 * WGS-84 coordinates in metres. Used for jump and jitter detection.
 */
function distanceMeters(lat1, lon1, lat2, lon2) {
  const R = 6371000;
  const toRad = x => x * Math.PI / 180;
  const dLat = toRad(lat2 - lat1);
  const dLon = toRad(lon2 - lon1);
  const a =
    Math.sin(dLat / 2) ** 2 +
    Math.cos(toRad(lat1)) *
    Math.cos(toRad(lat2)) *
    Math.sin(dLon / 2) ** 2;
  return 2 * R * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
}

Can you give any advice on these validations or can you recommend any additional validation method?

-- Cheers, Nils

Track-trace 2 months ago

I actually think that you are better off focussing on using good quality tracking devices.

Nils Leideck 2 months ago

Well, as I said before, I'm afraid I have no control over that. But thanks anyway for your recommendation. We attach tracking devices to dogs, cats, horses, sheep, and other animals to monitor their movements. Do you have any recommendations for a high-quality device?

Track-trace 2 months ago

You use tracker with very long battery life. Small gps trackers with gsm simcard only last about 2 - 3 days.