SystemLink Store and Forward Troubleshooting

Updated May 5, 2025

Reported In

Software

  • SystemLink

Issue Details

I want to use the SystemLink Store and Forward feature to upload data from my client system, but I need to troubleshoot one or more of the following issues:

  • Missing steps or results on the server.
  • Slow or delayed updates to steps or results on the server.
  • Results with Timed Out status.
  • Missing result file attachments.
  • Store and Forward Salt Beacon tags report large backlog or quarantine.

Solution

There are various reasons Store and Forward problems occur, including but not limited to:

  • Known issues in certain versions of the software
  • An insufficient forwarding interval
  • Folder access permission issues
  • Performance issues on the server

    First, refer to Resolving SystemLink Client Failures to Forward Data When Store and Forward is Enabled. If your clients are running an approved version, the following sections contain information on different troubleshooting steps you can take when dealing with the above reported issues.
     

Verify SystemLink packages are correctly updated.

Use NI Package Manager to verify that all the “SystemLink” packages are updated. Open NIPM and navigate to the Installed tab, uncheck the Products only checkbox and enable the Show full version numbers and hidden packages option after clicking on the gear icon at the top right corner. Then search for “SystemLink” in the package list. Some packages, such as the Python client packages, may be a minor version behind. The key packages that all need to match in version are:

  • NI SystemLink Forwarding Service
  • NI SystemLink Test Monitor Store
  • NI SystemLink File Store
  • NI SystemLink Asset Utilization Support 

Here is an example of an installation with incompatible versions. This system had errors running the forwarding service.
 
verifyComponents.png

Verify NI SystemLink Forwarding Service is running successfully.

  1. Launch Services from your Windows search bar.
  2. Search for the NI SystemLink Forwarding Service (the service should exist if the corresponding package is installed).
  3. Verify that it shows "Running" under the status column. If the service fails to start you might see the status changing when you refresh the Services user interface. The service is configured to retry the start if it runs into any errors so seeing the behavior change might be a clue that something else impedes the service from starting successfully. If this is the case there is a chance that Network or Local permissions are not enough for the local SYSTEM account which is what the services use by default.


Install Store and Forward Beacons

If you choose to use the using Store and Forward feature, you are strongly recommended to install the SystemLink Store and Forward Beacons on all the client systems. Otherwise, you could need many months to realize that the Store and Forward service is malfunctioning, because the symptoms can be subtle. 

These Store and Forward beacons will monitor folders used by the Store and Forward service to determine how many requests are waiting to be forwarded, and how many requests are quarantined due to errors reported from requests to the server. These metrics will be sent to the server using SystemLink Tags. Alarms can be set up to alert the user if a system has a large volume of pending requests or has any quarantined requests.
 

Manually Inspect the Store Directory

If it is not possible to install the Store and Forward beacons or if issues are detected with the client, then client systems can be manually inspected. Refer to Manually Inspecting SystemLink Store and Forward Stored Transactions for more information on how to do this.
 

Capture Windows Event Log Errors

You can refer to: Capture SystemLink Forwarding Service Windows Event Logs for more information on how to capture relevant event information using the Windows Event Viewer.
 

Capture Forwarding Service Logs

You can run the Forwarding Service manually from DOS with verbose logging enabled to get more specific details. Refer to Capture SystemLink Forwarding Service Logs  for more information on how to do this.
 

Configure Forwarding Interval

On clients where transactions are produced at a high rate, the default forwarding interval may be too long. Refer to Configure the SystemLink Forwarding Interval for instructions on how to edit this interval.
 

Store Access Permission

Permission issues in Windows can result in the Store and Forward to malfunction, refer to SystemLink Store and Forward Not Storing Transactions for more information on what the symptoms for this are and what steps you can take for this.

Forwarding Large Backlogs

If a client has not been connected to the server in a long time, a large backlog may form in the store. Once the client is reconnected to the server, it will begin to forward the stored transactions to the server at a high rate. Refer to Forwarding Large SystemLink Store and Forward Backlogs for recommendations on how to deal with this situation.

Additional Information

The SystemLink Client software has a feature called Store and Forward that enables data that is intended to be sent to the SystemLink server to be stored locally when disconnected from the server and then forwarded to the server when a connection is re-established. For information on different configuration modes for this feature refer to SystemLink Store and Forward Feature Configuration.