Troubleshooting Memory Growth Issues in TestStand Systems

This article explains the causes of out-of-memory errors in TestStand systems and outlines a debugging process that can be followed to identify and resolve memory growth issues.

When creating a TestStand sequence that runs for extended periods of time, a key concern is creating sequences and code modules that do not cause significant memory growth over time. During development, it is important to carefully manage memory in test sequences and code modules to avoid out-of-memory errors.

Effects of Memory Growth on Test Systems

It is important to recognize that memory growth does not always indicate a problem with the test system. For example, if a code module is acquiring measurement data from hardware and storing the data in an array, the code module’s memory usage will increase as the data is stored. This type of memory growth is typically expected to occur.

However, if memory allocated by code modules or other test system components is not released when it is no longer needed, memory usage will continue to increase as the test system executes. This can eventually cause an out-of-memory error or crash once the test system uses all available memory. This uncontrolled memory growth over multiple test sequence iterations is often referred to as a memory leak.

 

Causes of Out-of-Memory Errors

Out-of-memory errors may occur for multiple reasons:

• Excessive memory growth during a single test loop.
  • This may occur if the test sequence for a single UUT attempts to allocate more memory than what is available to the test system. This often results in an out-of-memory error similar to the ones depicted in Memory Is Full Error in LabVIEW.
  • An example of this is test code that attempts to allocate a contiguous block of memory to store an array, string or other large data set.  
• Unbounded memory growth over multiple test loops.
  • This may occur if there is a memory leak in the test sequence or in a code module called by the test sequence.
  • Examples of memory leaks include: failing to release memory that is no longer needed, or forgetting to close instrument handles at the end of the sequence.

Memory Growth Troubleshooting Process

It can be difficult to track down memory growth issues in large test systems due to the amount of code that would require investigation. Additionally, memory growth may not cause an error until the test system has been executing for a long period of time, and may cause a crash or other instability in the test system instead.

For these reasons, troubleshooting a memory growth issue is a methodical process in which memory usage data must be analysed to understand the type of growth that is occurring. To begin, narrow down the test code to a simplified version which reproduces the memory growth, and examine the simplified code to resolve the memory growth issues. This process is described below in a step-by-step guide.

The other documents in this series are linked at relevant points in the troubleshooting process. They will provide details on the troubleshooting steps that should be taken, as well as guidance on how to interpret the results gathered at each stage in the process.

 

Step-By-Step Troubleshooting Guide

1. Review test system memory constraints by referring to the Memory Limits of TestStand Systems  document.
2. Using the linked document, examine test system settings and configuration to understand the amount of memory available to the test system. Things to consider are:
   • The types of code modules and whether they are 32- or 64-bit.
   • Code modules executing in-process or out-of-process.
   • Result collection settings.
   • Model plugin settings.
3. Use memory profiling tools to gather data about test system memory usage.
   • See the Use Task Manager to Monitor TestStand Memory Usage document for information on using memory profiling tools and interpreting results from the tools.
   1. Run simplified code to determine the type of memory that is growing and how much it is growing per test case loop. The linked article explains how to monitor:
      1. Virtual memory that is currently loaded into physical memory, other wise known as Private Working Set.
      2. Object handles, which can be references to instruments, LabVIEW VIs and other operations.
      3. Threads.
      4. User objects, which includes windows, menus and cursors.
      5. Graphics Device Interface (GDI) objects.
4. Simplify the TestStand sequence to focus on a smaller set of steps contributing to the problem. The Characterizing Memory Issues in TestStand Systems document provides guidance on how to do this. This troubleshooting process should follow the below structure:
   1. Use memory profiling tools from step 3 to gather memory growth data as you narrow down a test case.
   2. Determine whether result collection is contributing to memory growth, as described in the linked document. This will consist of:
      1. Disabling model plugins.
      2. Disabling result collection for specific test sequences or the entire test station.
      3. Execute with different TestStand process models and note whether a difference in memory growth can be observed. Follow the steps in Specify a Process Model for TestStand Sequence File  for guided instructions.
      4. Skip steps or sequences within the test case to narrow down the set of problematic steps.
      5. Narrow down the test sequence to specific code modules if possible.
5. Identify and resolve the memory growth issue by examining the test sequence and code modules.
   1. If the issue only occurs when result collection or model plugins are enabled, refer to the Prevent Memory Issues with TestStand Report Generation document to configure result processing settings to conserve memory.
   2. Narrow down the source of the problem to a specific sequence or code module. Use Identifying and Resolving Memory Growth Issues in TestStand Systems to determine which category the memory growth falls into.
   3. If the issue is occurring in a LabVIEW code module, refer to the LabVIEW Code Modules section of the article in step 5b.
   4. If the issue is occurring in a LabWindows/CVI code module, refer to the LabWindows/CVI Code Modules section of the article in step 5b.
   5. If the issue is occurring in a .NET code module, refer to the .NET Code Modules section of the article in step 5b.
6. After following the articles linked throughout this document, return to step 3 to determine whether the memory growth issue has been resolved.