Tag Archives: quantization error

Angle pt 4 – Quantization Errors

Fixed point types have quantization errors.  Quantization errors can be as high as the weight of the least significant bit. For an n-bit angle primitive, the weight of the least significant bit is 22-n. For values near zero the angle and its arctangent are approximately the same. Thus, the best case quantization error is the weight of the least signficant bit.
The tangent function is non-linear. Quantization error increases as the square of the secant. For values near +/-1 this means the quantization error is nearly twice the weight of the least significant bit.

To translate quantization errors into distances along the surface of the earth we will use pi radians equals 10800 nautical miles or 20,000 kilometers. (A nautical mile is about 6076 feet). As seen in the following table, it takes very few bytes to get high accuracy. For example, a four-byte angle primitive has a quantization error of between 0.366 inches (best case) and 0.733 inches (worst case = 2x best case). Using a four-byte angle type in lieu of an eight-byte double precision value in radians yields a 50% savings in memory and bandwidth.
prev continue

bytes radians distance units meters
1 0.024543693 84.375 nm 156250
2 9.58738E-05 0.329589844 nm 610.3515625
3 3.74507E-07 7.822608948 ft 2.384185791
4 1.46292E-09 0.366684794 in 0.009313226
5 5.71452E-12 0.001432362 in 3.63798E-05