How Can I Read a Very Large CSV File in LabVIEW?

Issue Details

When I try to read data from a large CSV file (several hundreds of MB) by using Read Delimited Spreadsheet VI, I receive Memory is full error.

How can I modify my program in order to resolve this issue?

Solution

Read Delimited Spreadsheet VI by default will read the whole CSV file at once. It will read the file as a string and then converts it to a numeric array. As a result, it will make several copies of data which can result in the memory full error.

However, if you use Read From Text File VI, the number of rows to be read at a time can be specified, therefore, data can be read partially in order to prevent memory full error. The position of reading will be stored and new set of rows will be read in each loop iteration. 

Since the data is read as a string, you can use queue to transfer the data into another loop to covert them into a numeric array. It is recommended to use Parallel For Loops to improve the conversion performance. 

 You can also refer to the following VI Snippet on how to read a very large CSV file in LabVIEW.

Additional Information

Note: This image is a LabVIEW snippet, which includes LabVIEW code that you can reuse in your project. To use a snippet, right-click the image, save it to your computer, and drag the file onto your LabVIEW diagram.