LabVIEW To Fixed-Point Number Conversions Give Wrong Values

Updated Feb 11, 2020

Reported In

Software

  • LabVIEW

Issue Details

When I set integer word length and start changing word length values, sometimes I get wrong results when converting from single precision to fixed-point values, even though they are within minimum and maximum ranges, as shown below:


 

Solution

The above issue occurs when the combination of word and integer word lengths results in one of the following delta values:
  • 1.00974E-28,
  • 5.04871E-29,
  • 2.52435E-29.
To avoid this, increase word length to further reduce the delta value while maintaining the same minimum and maximum values, as shown below: