Calculation of Distance from GPS Coordinates in DIAdem - National Instruments

HomeSupportCalculation of Distance from GPS Coordinates in DIAdem

Calculation of Distance from GPS Coordinates in DIAdem

Updated Nov 27, 2018

Reported In

Reported In shows products that are verified to work for the solution described in this article. This solution might also apply to other similar products or applications.

Software

DIAdem 2017

Issue Details

I have an application, in which my hardware measures GPS-coordinates of two points in less than 10 centimeter distance, namely their corresponding latitude and longitude values. My problem is, that when I am using the Spherical Law of Cosines to calculate the actual distance, I get always exactly zero as a result, although the points are several centimeters apart.

I use the Double datatype to store the raw latitude and longitude values, and also to store the resulting distance value.

Solution

The problem comes from the finite resolution of the Double datatype. According to the IEEE 754 standard the so called significand precision is 53 bits, so the maximum relative rounding error is 2^-53. It means, that considering decimal numbers, there will be maximum 53*log(2)~16 significant digits, i.e. nonzero digits stored by the fraction part of the variable. Let us consider the Spherical Law of Cosines in DIAdem now. If the physical distance is less than 6.371 cm-s, Zeta will be in reality less than 10^-8 radians, so in cos(Zeta)~1-Zeta^2/2 only the 17th significant digit is different from 9, it will be 5. After rounding we get for the argument of the acos-function exactly 1, so Zeta will be calculated to be exactly zero radian, and the calculated distance will be 0 cm.

To avoid this rounding error for distances under 1 km use the so called Equirectangular approximation: