If you are new to programming on an FPGA with LabVIEW, it will be worthwhile reviewing NI LabVIEW High-Performance FPGA Developer’s Guide
. This error can occur due to not following best practices when using FPGAs. For example, using lots of front panel items such as indicators an graphs.
There is a known issue in LabVIEW where simulating more than 2 GB of DRAM can result in a memory error. It is not related to LabVIEW's memory limit, so it will occur even when LabVIEW does not run out of memory.
Additionally, you can use the Performance Profiler
to track which programs are causing the memory growth to occur. Consider reducing data produced in this VI, changing the architecture to a more efficient one such as Producer/Consumer