Archived:Set Up a Preprocessing Routine for DataFinder Server

This tutorial shows you how to configure Data Preprocessor to standardize metadata and prepare data for automated analysis. This is an important part of the Technical Data Management process chain to ensure consistent and automated analysis for every data point that is collected. 

Note: This tutorial applies to NI DataFinder Server 2018 SP1 and earlier. In SystemLink 19.5, DataFinder Server and Analysis Server joined the SystemLink family of products and new Modules. Refer to the SystemLink Manual for updated information.  

The Data Preprocessor is designed to convert raw data from different sources to homogenous “smart data” by standardizing property identifiers and values, applying unit conversion, adding additional statistical calculations from the individual channel data, verifying the resulting data, and converting the data into a standard file format such as TDM or TDMS so that the modifications made during the preprocessing routine do not affect the original data file. This tutorial shows you how to change property identifiers and content to standardized names and values and add some additional signal characteristics using the Data Preprocessor. This tutorial uses files stored in your National Instruments public example folder. 
 

 

Use DIAdem to Define a Data Preprocessor Configuration

Use the Data Preprocessor Configuration dialogs to define the Data Preprocessor processing steps which will configure a Data Preprocessor in the TDM Server Manager. For convenience, it is easier to load one example file into the DIAdem Data Portal and use drag and drop for some configuration steps:

1. Select the DIAdem NAVIGATOR panel.
2. Click the Delete Internal Data symbol.

 

3. Select File>>Open. Change the file type to National Instruments TDM Streaming (*tdms). 

 

5. Navigate to the TDM Server Demo Data folder C:\Users\Public\Documents\National Instruments\DataFinder\Data and select a file.
6. Click Load to load a TDMS example file into the DIAdem Data Portal.

Create New DataFinder Preprocessor

1. Select the DIAdem SCRIPT panel.
2. Select Settings>>TDM Server>>Data Preprocessor.

 

3. Click the Data Preprocessor Configuration symbol.

 

4. Enter the Name of the new Data Preprocessor and an optional Description and Author.

 

5. Click OK.

Configure the Property Identifier Mapping

1. Select the Replace identifiers tab.
2. Enable Replace property identifiers.

   

3. Click Edit.
4. Click the Add DataPlugin Mapping symbol and select TDMS as DataPlugin name.

     

5. Click OK.
6. Select the TDMS column by clicking into the column header.
7. Click Edit DataPlugin Mapping to open the dialog box.

   

8. Select the root element in the Data Portal (the root of the tree structure).
9. Select all Custom Properties and drag and drop them into the still open Edit TDMS DataPlugin Mapping dialog.

   

10. Specify the <vacant> Standardized Identifier as defined in the table above. Where applicable, replace the separators like “_” with “~” using the Find and Replace dialog box along with the Transfer Property Names tool to copy the Original Identifier text over. 

   

11. Repeat this procedure for ChannelGroup and Channel on the respective tabs.

     

12. Click OK.
13. Click Close.

Configure the Property Value Mapping

1. Switch to the Replace Values tab.
2. Enable Replace property values.

   

3. Click Edit.
4. Expand Channel and select Name in the tree.

   

5. Select three channels (Temp_A, Temp_B, and Temp_C) in the Data Portal
6. Drag and drop the selected channels into the Mapping rules area of the Replace Property Values dialog box.

 

• For Temp_A, specify Temperature_A as Output Value
• For Temp_B, specify Temperature_B as Output Value
• For Temp_C, specify Temperature_C as Output Value

       

7. Click Yes to use this pattern and map all Temp_? channels to Temperature_?
8. Confirm the Edit rule dialog box by clicking OK.

   

9. Close the Replace Property Values dialog box.

Define Additional Channel Statistics

1. Switch to the Statistics tab.
2. Enable Calculate statistics.

 

3. Click Edit.
4. In the Calculate statistics dialog box, configure the following settings:

• Enable Minimum. Use Statistics~Minimum as Custom Property Name.
• Enable Maximum. Use Statistics~Maximum as Custom Property Name.
• Enable Arithmetic mean. Use Statistics~ArithmeticMean as Custom Property Name.
• Enable Standard deviation. Use Statistics~StandardDeviation as Custom Property Name.

   

5. Click Close.

Define the Standard File Format

1. Switch to the Convert File tab.
2. Define Save with DataPlugin as TDM.

   

3. Click Save Data Preprocessor Configuration As and save the DPP file to a folder you specify.

   

Verify the Data Preprocessor Routine

1. Select Data Preprocessor Chain Test with Debugger.

   

2. Browse to C:\Users\Public\Documents\National Instruments\DataFinder\Data.
3. Select National Instruments TDM Streaming (*tdms) from the File name drop-down menu.

   

4. Select a TDMS file and click Load. A pop-up dialog will appear letting you know the data has been processed and loaded into the Data Portal.

   

5. Explore the Data Portal and verify the channel names changed from Temp_X to Temperature_X and the Custom Properties show the new organization set in previous steps.

   

Deploy Data Preprocessor Package via Web Server

Now we can deploy the Data Preprocessor to the server via the TDM Server Manager. The Data Preprocessor uses a configuration file (*.DPP), which you created in DIAdem, to convert raw file data from one or more raw data folders to homogenous data, which you save in the respective processed data folder.

To set up the Data Preprocessor, complete the following steps in the wizard:

1. Open TDM Server Manager in your web browser and login using your credentials.
2. Select New>>Data Preprocessor to start the configuration wizard.

 

3. Enter ConvertExampleData as the Name of the new Data Preprocessor server. Click Next.

 

4. Enter the Name of the raw data area.
5. Specify the Raw data folder. Click the Choose a Folder symbol and select the TDM Server Demo Data.

   

6. Specify the Processed data folder. Click the Choose a Folder symbol and navigate to C:\Users\Public\Documents\National Instruments\TDM Server Demo\Processed_Data.
7. Click Next to continue the configuration wizard
8. Select Use custom configuration and specify your Data Preprocessor that you saved previously.

   

9. Click Next to continue the configuration wizard.
10. Select the ADVANCED license.

 

11. Click Finish to complete the initial configuration of the Data Preprocessor.
12. Select the newly created Data Preprocessor in the list and click Start to initiate processing.

 

Next Steps

After configuring the preprocessor routine, you may want to test the functionality. To do this, open up three folders and arrange on your desktop:

1. C:\Users\Public\Documents\National Instruments\DataFinder\Data (Contains Data)
2. C:\Users\Public\Documents\National Instruments\TDM Server Demo\Raw_Data (Empty)
3. C:\Users\Public\Documents\National Instruments\TDM Server Demo\Processed_Data (Empty)

Copy the data files into the Raw_Data folder and see how the Processed_Data folder updates.