Unable to Obtain Precision of 7 Digits or More with SGL Datatypes in LabVIEW

Issue Details

• When I write the value 123456789 in a numeric control of SGL datatype on the LabVIEW Front Panel, it automatically gets coerced to 123456792. Why am I unable to obtain precision higher than 7 digits? I am using this in an FPGA VI. What is a possible alternative? 
• When I try to add two floating-point numbers with 7 or more digits, using numeric controls of SGL datatype, I get an incorrect output. I believe the output value is losing its precision beyond the 7th digit. Is this expected?
• I want to represent a very large number within the limits of 3.4e+38 using SGL datatype. Why am I unable to obtain uncompromised precision till the 9th or 10th digit even though it is a 32-bit datatype?

Solution

The SGL datatype is of (external link) IEEE 754 binary32 format which implies that even though it is a 32-bit datatype, only 23 bits are used to represent the (external link) significand in memory as shown below. The total precision of SGL datatype is (external link) 6 1/2 decimal digits. Thus, if a decimal string with more than 6 significant digits is converted to SGL representation, it might result in a round-off error produced by quantization of the real number in binary. 



The following alternatives might be useful while trying to obtain precision greater than 6 decimal digits.

• Use DBL, FXP, I32, I64, U32, or U64 datatypes based on your application requirements. Refer to Numeric Data Types Table for more details before using the same. 
• In an FPGA VI, use I32, I64, U32, U64 to represent integers and FXP to represent floating-point numbers. Refer to Deciding Which Data Type to Use in FPGA Designs (FPGA Module) for more details.