Clean up Meta Information Clutter Using Systemlink’s Data Preparation Procedures

The name, unit, data type, etc. of measurement data may differ depending on the equipment used for the measurement and the organization performed during measurement. For example, in the case of speed, property names such as "speed", "Speed", and "speed (m / s)" may exist in each measurement data, and all names may indicate the same property information. In addition, speed units (km / s, m / s, etc.) and data types (character strings, numbers, etc.) may be mixed. When measurement data are stored in this kind of "disturbed state", it becomes difficult to accurately search, compared or analysse them in a database.
SystemLink TDM Data Preparation Add-on can be used to clean up meta-information disruptions and streamline data utilization prior to DataFinder indexing. This document explains how data preparation procedure is configured and used to process raw data for better utilization in a database (The example used here, will show how to change sample data properties such as names, units, and data types).

Create a data preparation procedure and check its contents

1. From DIAdem SCRIPT, select Settings>>SystemLink TDM>> Data Preparation Procedure.
2. Click New Data Preparation Procedure(dpp), set the Name, Description, Author and click OK.
3. You can save dpp file at this point or save it later after configuration is complete. To save the file,  click Save Data Preparatiion Procedure As button,and select a file location. 

The data preparation procedure consists of 6 steps: Replace Identifiers, Replace Values, Convert Units, Statistics, V & V, and Convert File. For more information about each step, refer to this article.  There are 6 different icons, each corresponding to a step. Except for the Convert File step, all steps are grayed out indicatiing that no steps have been configured yet. 
In the Convert File tab, select the type of data plugin you want to save the dpp file with. This data storage file format is necessary for optimized file loading and storage in SystemLink, when you create a service that uses a data preparation procedure. By default, the dpp file is saved with *.TDM format.  

Test data preparation procedures with DIAdem

Save the data preparation procedure and check how it operates on raw data by running it in DIAdem.

1. Click Test Data Preparation Procedure button.

   

2. Specify the data plug-in to use when selecting the data file, this will display the corresponding files. Use ExcelTDM to read * .xlsx files and National Instruments TDM Streaming to read * .tdms files.

• Since no processing content is set in this data preparation procedure, the data will be loaded into the Data Portal as shown below. Select file name (TR_M21_QT_7-7, TR_M17_QT_42-1 in this case) to check its properties.

        
   

• Select channel group (such as QT_7-7_Lower in the figure below) or channel (such as Temp_A) to check the channel group properties and channel properties.

Standardize data with data preparation procedures

Process_Example_Data_TDMS.dpp and Process_Example_Data_XLSX.dpp are used here to confirm data are processed accordingly to what is set in these dpp files. (Thesedpp files are located in C:\Users\Public\Documents\National Instruments\Data Preprocessor\Configuration folder, and also attached in the SampleDataFiles.zip attachement folder)
 

1. Click Open Data Preparation Procedure and select the dpp file(Process_Example_Data_TDMS.dpp or Process_Example_Data_XLSX.dpp) to load it into DIAdem.

2. From the Configure Data Preparation window, click Test Data Preparation Procedure, and select a data file to process. 

If you process the * .xlsx file with Process_Example_Data_XLSX.dpp, the processed data will look like below:
The left pictures represent raw data, and the right pictures represent the processed data (with Process_Example_Data_XLSX.dpp).
   

•  Test_Module and Test_Name, originally individual custom properties, are managed in a tree structure in the processed data. This conversion was set in the Replace Identifiers step of Process_Example_Data_XLSX.dpp  file. 

• Channel names, were changed from a Temp_A pattern to Temperature_A pattern. This change is set in the Replace Values step where channel names were mapped with the sepecic Rule  (to change all Temp_A-like pattern to Temperature_A-like format).

• Convert unit step set how channel data units are processed. In this example, the temperature default units and the conversion method between each unit was defined.

• Statistical values such as maximum, minimum, and average etc, can be calculated from the raw data and registered as custom properties in statistics. 

• Process_Example_Data_XLSX.dpp does not use this feature but it is possible to add VBScript processing in V & V step. 

• Process_Example_Data_XLSX.dpp is save with *.TDM data plugin as set in Convert File step. 

 

Use data preparation procedures with SystemLink

This section will explain how to set up and use the * .dpp file in SystemLink Web Application.

1. Select the Data preparation tab in SystemLink
2. In the Data Preparation >> Dashboard>> Instances, click New>> Data Preprocessor Instance to create and a new Data Preprocessor instance.
3. Input a Name for your data preprocessor instance and click Next.
4. Specify the folder to be monitored by this instance and the folder to output the processed data with a UNC path.
5. Select the DPP file (Process_Example_Data_XLSX.dpp) and click Finish.

 

6. Click the setting icon ⚙ next to the data preparation instance and select Manage.
7. Go to the DataPlugin tab and make sure ExcelTDM is Enabled. If not, select it and click Enable. (this corresponds to the data plugin sets in Convert file step)

8. Go back to Instances tab, select ⚙>>Start to start the data preprocessor instance.

When you move * .xlsx files into the folder to be monitored, the data preprocessor instance will detect it, process the data accordingly to what is set inside the ddp file and output the processed data in the specified folder.