SystemLink Data Preprocessor Instance Not Outputting Data

Issue Details

• I have created a Data Preprocessor Instance in SystemLink. When the Instance is started, it does not generate any output files in my Processed Data folder.
• I created a Data Preparation Procedure (*.DPP) file to use with my SystemLink Data Preprocessor Instance. After placing new files in my Raw Data path, no new data is generated in my Processed Data path.
• My Data Preprocessor Instance previously worked, but has suddenly stopped outputting files in my Processed Data folder. Why is this?

Solution

• Ensure that the correct DataPlugin has been selected for the Data Preprocessor:
  1. From the Data Preparation panel, select the cog icon next to your Data Preprocessor Instance and click Manage.
  2. Select the DataPlugins tab.
  3. Ensure that the DataPlugin that supports your data file has been selected.
     • For example, if your Raw Data is in .TDMS format, select the TDMS DataPlugin as shown.

• Check the Processing Status of your Data Preprocessor for errors.
  1. From the Data Preparation panel, click the cog icon next to your Data Preprocessor Instance and click Manage.
  2. Navigate to the Processing Status tab.
  3. Expand your Raw Data folder.
  4. If the Status reports "Failed", click on it to see failure details.

• If your file format stores metadata in a separate file, ensure that this is also being moved to your Raw Data folder.
  • For example, *.TDM files store metadata in a separate *.TDX file. The *.TDX file must be moved with the *.TDM file.
• Ensure that your Data Preprocessor Instance has been properly started.
  • Navigate to the Data Preparation panel in SystemLink.
  • Click the cog icon next to your Data Preprocessor Instance and click Start.

• Verify that your Raw Data and Processed Data paths are valid.
  • Raw Data and Processed Data paths must be shared network paths or UNC paths.
  • UNC paths are prefaced by a \\ symbol. Refer to [External] File path formats on Windows systems for more information.
  • Refer to [External] File sharing over a network in Windows for instructions on how to set up a shared network folder.
• Ensure that there is not a backlog of files waiting to be processed.
  • A Data Preprocessor Instance must be given enough time to scan the Raw Data folder for new files, and process files that have already been scanned. If not given enough time, the number of files waiting to be scanned will build up. If the backlog grows too large, it's possible for the Data Processor Instance to stop processing because the server is strangled for resources.
  • To check whether a backlog is present:
    1. Select the Data Administration >> Data Preparation tab from the SystemLink web UI.
    2. Double-click the Data Preprocessor Instance.
    3. From the right-hand side, click the Pie Chart symbol and select Advanced Statistics.
    4. In the pop-up window, select the Processing Status tab.
    5. Compare the value of Not Yet Processed with Processed Successfully.
       • Not Yet Processed should not exceed Processed Successfully by a significant amount. There is no fixed value for the ratio of these statistics because the processing performance varies depending on the server's specifications. Instead, monitor the Not Yet Processed value over time to see if it grows.
    6. Close the pop-up window.
    7. Select the Processing Status tab.
    8. Expand the Raw Data folder and double-click on a file that was successfully processed.
    9. In the pop-up window, select the DataPlugins tab.
    10. Note the value of Processing Duration. 
        • Note: it may be necessary to repeat steps 8-10 for several files to determine an average processing time.
    11. Ensure that new files are added less frequently than the Processing Duration to prevent a backlog.
        • For example, if 1000 new files are added to the Instance per working day (8 hours = 480 minutes), this averages to around 2 files per minute (1000/480). If the average Processing Duration is longer than 1 minute, a backlog will build up.
        • To reduce the number of files in the backlog, refer to SystemLink Data Preprocessor Instance Has Backlog of Not Yet Processed Files.