Unable to Obtain Precision of 7 Digits or More with SGL Datatypes in LabVIEW

Updated Sep 19, 2023

Reported In

Software

LabVIEW
LabVIEW FPGA Module

Issue Details

When I write the value 123456789 in a numeric control of SGL datatype on the LabVIEW Front Panel, it automatically gets coerced to 123456792. Why am I unable to obtain precision higher than 7 digits? I am using this in an FPGA VI. What is a possible alternative?
When I try to add two floating-point numbers with 7 or more digits, using numeric controls of SGL datatype, I get an incorrect output. I believe the output value is losing its precision beyond the 7th digit. Is this expected?
I want to represent a very large number within the limits of 3.4e+38 using SGL datatype. Why am I unable to obtain uncompromised precision till the 9th or 10th digit even though it is a 32-bit datatype?

Solution

The SGL datatype is of (external link) IEEE 754 binary32 format which implies that even though it is a 32-bit datatype, only 23 bits are used to represent the (external link) significand in memory as shown below. The total precision of SGL datatype is (external link) 6 1/2 decimal digits. Thus, if a decimal string with more than 6 significant digits is converted to SGL representation, it might result in a round-off error produced by quantization of the real number in binary.

The following alternatives might be useful while trying to obtain precision greater than 6 decimal digits.

Use DBL, FXP, I32, I64, U32, or U64 datatypes based on your application requirements. Refer to Numeric Data Types Table for more details before using the same.
In an FPGA VI, use I32, I64, U32, U64 to represent integers and FXP to represent floating-point numbers. Refer to Deciding Which Data Type to Use in FPGA Designs (FPGA Module) for more details.

Other Support Options

Ask the NI Community

Collaborate with other users in our discussion forums

Search the NI Community for a solution

Request Support from an Engineer

A valid service agreement or active software subscription may be required, and support options vary by country.

Open a service request

Learn about software subscriptions and services

Learn about hardware service programs