Troubleshooting Memory Growth Issues in TestStand Systems

Updated Aug 20, 2021



  • TestStand

When creating a TestStand sequence that must run for a long duration, a key concern is creating sequences and code modules which do not cause significant memory growth over time. During development, this means that you must carefully manage memory in test sequences and code modules to avoid out-of-memory errors when the test system is executed for an extended period of time. The purpose of this document is to explain the causes of out-of-memory errors in TestStand systems and outline a debugging process that you can follow to identify and resolve memory growth issues in your test system.

Effects of Memory Growth on Test Systems

It is important to recognize that memory growth does not always indicate a problem with the test system. For example, if a code module is acquiring measurement data from hardware and storing the data in an array, the code module’s memory usage will increase as the data is stored. This type of memory growth is typically expected to occur.

However, if memory allocated by code modules or other test system components is not released when it is no longer needed, memory usage will continue to increase as the test system executes. This can eventually cause an out-of-memory error or crash once the test system uses all available memory. This uncontrolled memory growth over multiple test sequence iterations is often referred to as a memory leak.

Causes of Out-of-Memory Errors

Out-of-memory errors may occur for multiple reasons:

  • Excessive memory growth during a single test loop, which may occur if the test sequence for a single UUT attempts to allocate more memory than what is available to the test system. This often results in an out-of-memory error if the test code attempts to allocate a contiguous block of memory to store an array, string, or another large data set.
  • Unbounded memory growth over multiple test loops, which may occur if there is a memory leak in the test sequence or in a code module called by the test sequence.

Memory Growth Troubleshooting Process

It can be difficult to track down memory growth issues in large test systems due to the amount of code executed during a test. Additionally, memory growth may not cause an error until the test system has been executing for a long period of time, and may cause a crash or other instability in the test system rather than generating an out-of-memory error.

For these reasons, troubleshooting a memory growth issue is a methodical process in which you analyze memory usage data to understand what type of growth is occurring, narrow down the test code to a minimal test case which reproduces the memory growth, and examine the minimal test case to identify and resolve the memory growth issues. This process is described below in a step-by-step guide that you can follow as you investigate memory growth issues in your test system. The other documents in this series are linked at relevant points in the troubleshooting process. They will provide details on the troubleshooting steps you should take, as well as guidance on how you should interpret the results you gather at each stage in the process.

Step-By-Step Troubleshooting Guide

  1. Review test system memory constraints by referring to the Memory Limits of TestStand Systems document. This document explains memory limits imposed by the operating system, test system process, and other factors.
  2. Examine test system settings and configuration to understand the amount of memory available to the test system.
    1. Types of code modules and whether they are 32- or 64-bit
    2. Code modules executing in-process or out-of-process
    3. Result collection settings
    4. Model plugin settings
  3. Use memory profiling tools to gather data about test system memory usage.
    1. See the Using Memory Profiling Tools to Characterize Memory Growth Issues in TestStand Systems document for information on using memory profiling tools and interpreting results from the tools.
    2. Determine which tool(s) to use to monitor memory growth
      1. Task Manager
      2. Performance Monitor
    3. Run a test case to determine the type of memory that is growing and how much it is growing per test case loop.
      1. Private Working Set/Private Bytes
      2. .NET Heap
      3. Handles
      4. Threads
      5. GDI and User objects
  4. Narrow down a test case to focus on a smaller set of code contributing to the problem. The Characterizing Memory Issues in TestStand Systems document provides guidance on narrowing down a test case to a minimal set of code that reproduces the memory growth.
    1. Use memory profiling tools from step 3 to gather memory growth data as you narrow down a test case
    2. Determine whether result collection is contributing to memory growth
      1. Disable model plugins
      2. Disable result collection for specific test sequences or the entire test station
  5. Simplify the test sequence to determine which steps are contributing to memory growth
    1. Execute without a TestStand process model
    2. Skip steps or sequences within the test case to narrow down the set of problematic steps
    3. Narrow down the test sequence to specific code modules if possible
  6. Identify and resolve the memory growth issue by examining the test sequence and code modules.
    1. If the issue only occurs when result collection or model plugins are enabled, refer to the Addressing Memory Issues with Report Generation in TestStand document to configure result processing settings to conserve memory
    2. Once you have narrowed down the test case down to a specific test sequence or code module, examine the test sequence or code module to identify the cause of memory growth
    3. Refer to the Identifying and Resolving Memory Growth Issues in TestStand Systems document in this series for guidance on investigating memory growth issues in TestStand sequences and code modules
    4. If the issue is occurring in a LabVIEW code module, refer to the Investigating Memory Growth Issues in LabVIEW Code Modules Called from TestStand document
    5. If the issue is occurring in a LabWindows/CVI code module, refer to the LabWindows/CVI section of the Identifying and Resolving Memory Growth Issues in TestStand Systems document
    6. If the issue is occurring in a .NET code module, refer to the .NET section of the Identifying and Resolving Memory Growth Issues in TestStand Systems document.
  7. After resolving the memory growth issues you identified, return to step 3 to determine whether memory growth is still occurring elsewhere in the test system.