Manually Inspecting SystemLink Store and Forward Stored Transactions

When troubleshooting SystemLink Store and Forward problems it might be necessary to manually inspect the store directory, where data is cached locally before it is forwarded on to SystemLink. This article offers instructions how to perform such a manual inspection.

The store directory contains 3 subdirectories that each contain a different type of cached data:

• assetmgmt: updates on asset states, such as calibration events and new asset locations, etc.
• file: Files cached locally until forwarded to SystemLink
• testmon: Test results cached locally until forwarded to SystemLink

Each of these folders contain the following items:

•  __CACHE__: File that keeps track of information about the transactions that have been sent to the server. The “timestamp” field will indicate the last time the forwarding service attempted to forward a transaction.
• <guid>.jsonl: Files with random-looking GUID names. These contain the stored transactions, usually spread across multiple files. The format is specific to the forwarding service, but all the data is stored in plain text with each line representing a stored transaction.
• quarantine: Folder with jsonl files that log any transactions that received an error response from the server.  These transactions will NOT be re-tried by the Store and Forward service.

1. Browse to the folder C:\ProgramData\National Instruments\Skyline\Data\Store. This is where the stored transactions are saved.

2. Open each folder and look for the number of .jsonl transaction files and the timestamp from the __CACHE__ file. Consider the following:

• Files in the quarantine directory indicate a problem occurred that must be manually resolved. If there are many transactions, that can indicate a systemic issue that could already be addressed in an available patch.
• Files outside the quarantine directory are not necessarily an indication of a problem. This is where the transactions are held before they are forwarded to the server. However, a large volume of transactions can indicate a problem. The forwarding service may be crashing, or it may not be able to keep up with the rate of new data. The __CACHE__ timestamp can help determine which of these is the case.
• The timestamp contained in the __CACHE__ file indicates the last time the forwarding service forwarded a transaction. 
  • If the timestamp is suspiciously old, this can occur if the forwarding service is crashing while trying to forward transactions. In this case, you will need to get events or logs from the forwarding service (see Capture SystemLink Forwarding Service Windows Event Logs and Capture SystemLink Forwarding Service Logs). You can also compare the timestamp between the three subdirectories (asstmgmt, file, testmon). If they are inconsistent, this can be a clue to where a crash is occurring, because the folders are forwarded sequentially.
  • If the timestamp is current, this can indicate one of two problems:
    • The rate of new incoming data is higher than the rate of outgoing forwarded data. In this case, you can tweak the forwarding interval to improve performance, see Configure the SystemLink Forwarding Interval. 
    • You can also inspect the timestamp of individual rows in each of the jsonl files in the non-quarantine store. If the last row is a CloseFile transaction and has a timestamp older than the __CACHE__, that can be a symptom of a known bug fixed in the patch. These are .jsonl files that were not cleaned up. They can be deleted to improve performance.

3. If no .jsonl files are created, that could indicate a Windows access permission issue writing to the store. Check for the following symptoms and refer to SystemLink Store and Forward Not Storing Transactions:

• There are no .jsonl files in the Test Monitor Client Data Store folder (C:\ProgramData\National Instruments\Skyline\Data\Store\testmon).
• The __CACHE__ file indicates no transactions ("forwardedTransactionCounts" array is empty) after running a test, it also indicates that files were not stored.