Solution
The number of
Not Yet Processed files can increase over time if the Data Preprocessor Instance is not given enough time to balance scanning new files and processing existing files.
To calculate how long the Data Preprocessor Instance requires to process files, refer to the last step in
SystemLink Data Preprocessor Instance Not Outputting Data.
To reduce the number of
Not Yet Processed files:
- Decrease the frequency at which new files are added to the Data Preprocessor Instance.
- The Instance requires time to catch up on the backlog of files waiting to be processed. Try reducing how many files are added to the Raw Data folder per day.
- If the backlog of files is particularly large, it may be necessary to stop adding new files completely until the backlog has been dealt with.
- Increase the number of Parallel Compute Nodes.
- Open the NI SystemLink TDM Configuration application on the server PC.
- Select the Parallelization tab.
- Increase the value of Number of Compute Node Instances.
- The maximum possible value is 64. It represents the number of parallel processes that can be shared across the server's CPU cores.
- Note: increasing this value will also increase the maximum CPU load that the server experiences. It's important to find a balanced value that does not stress the server so much that it negatively impacts other processes.
- Split the Data Preprocessor Instance into 2 or more separate instances.
- Consider the types of sources that add new files to the Raw Data folder. For example, multiple test stations.
- Alternatively, consider the types of files that are included in the Raw Data folder and how many DataPlugins are currently used.
- It may be necessary to create separate Search Areas for different test stations or file types, to minimize how many new files must be scanned by a single Data Preprocessor Instance.