Optimizing the Size of a DPP Output File in DIAdem

DIAdem Data Preparation Procedures (DPPs) process raw data files into meet a golden standard. This can include: changing property values, adding metadata, converting units and other things. Often, the output "golden" file is larger than the input raw file. This article explains how to optimize the DPP to minimize the size of the output file.

Note: the DPP will always add new properties, so the output file will almost always be bigger than the input. This article focuses on optimizing the output.

Why Are DIAdem DPP Processed Files Larger Than Raw Files? explains the different factors that contribute to the DPP output file size. Consider the following options for optimization:

• Only use harmonization where necessary.
• Only use unit conversion where necessary.
• Only add statistics that are needed.
• If using a Validation and Verification (V & V) script, ensure that the script is as efficient as possible.
• If the input and output file types are different formats, execute an Optimize Data Type calculation in DIAdem.

 

Only Use Harmonization Where Necessary

• When configuring the DPP in DIAdem, uncheck harmonization if it is not being used.
  • Harmonize Property Identifiers in the Replace Identifiers tab: This feature converts the names of file, group and channel properties so that they conform to a golden standard.
  • Harmonize Property Values in the Replace Values tab: This feature converts file, group and channel names or descriptions  to conform to a golden standard.
• If harmonization is required, remove anything that is not absolutely necessary for the final files.
  • When considering whether harmonization is necessary, consider the following:
    • Will the property be searched on my SystemLink server?
    • Is this property actually named differently across different files?
    • Will operators or end users need to view these properties?
    • Will the properties be included in a final report?

 

Only Use Unit Conversion Where Necessary

• When configuring the DPP in DIAdem, uncheck Unit Conversion in the Convert Units tab if it is not being used.
  • This feature converts channel units into a golden standard. For example, kV to V.
• If unit conversion is required, remove any conversion that is not absolutely necessary for the final files.
  • When considering whether conversion is necessary, consider the following:
    • Will the channel units be displayed in any reports or graphs?
    • Are operators logging a variety of different units for the same test type?
    • Is it possible to change the operator test sequence so that they must select a unit from a pre-defined list instead?

 

Only Add Statistics That Are Needed

• When configuring the DPP in DIAdem, uncheck the Calculate Statistical Characteristics Values in the Statistics tab if it is not being used.
• If statistics are required, remove any statistical properties that are not absolutely necessary for the final files.
  • When considering whether statistics are necessary, consider the following:
    • Will the statistical properties be searched for on my SystemLink Server?
    • Will the statistical values be displayed on a report or graph?

Optimize Data Types

DIAdem includes an Optimize Data Types calculation that can be used to reduce the DPP output file size manually and programmatically.

Optimization for TDM/TDX Output File Formats

If the DPP saves the output file as a .TDM or .TDX file, DIAdem's settings can be changed to automatically save optimize the data type.

1. From any DIAdem Panel, select Settings >> DIAdem Settings.
2. Select the NAVIGATOR tab.
3. Place a checkmark in the Save >> Optimize Data Type option.

Manual Optimization

For all output file formats, the DIAdem Optimize Data Type calculation can be used.

1. Select the DIAdem ANALYSIS Panel.
2. Select Channel Functions >> Optimize Data Type...
3. Drag and drop the Root item from the Data Portal into the Channels input.
   • This ensures that the calculation executes on all numeric data.
4. Click OK.
   • Note: This calculation may take several minutes to complete, depending on the size of your data file.
5. Save the contents of the Data Portal.

 

Programmatic Optimization

The calculation shown in the previous section can be implemented in a script so that it is included with the DPP.

1. When configuring the DPP in DIAdem, select the V & V tab.
2. Enable the Validation and Verification option and click Edit...
3. A freeform .VBSP script will open in the SCRIPT Panel.
4. Copy the following code into the .VBSP file.

   • Sub On_ValidationAndVerification(oContext)
     
     Dim i, j
     
     '--------------- Clear the log file ---------------
     Call LogFileDel()
     
     '--------------- Iterate through each ChannelGroup and optimize each channel ---------------
     For i = 1 to Data.Root.ChannelGroups.Count
       For j = 1 to Data.Root.ChannelGroups.Item(i).Channels.Count
         
         '--------------- For each "Time" channel, run the following analysis ---------------
         If Data.Root.ChannelGroups.Item(i).Channels.Item(j).Name = "Time" Then
           Set ChnResult = ChnOptimizeDataType("[" & Str(i) & "]/Time", "Standard", "Automatic", 6, True)
         
         '--------------- For all other channels, run the following analysis ---------------
         Else
           Call ChnOptimizeDataType("[" & Str(i) & "]/" & Data.Root.ChannelGroups.Item(i).Channels.Item(j).Name, "Standard", "Automatic", 6, True)
         End If
       Next
     Next
      
      
     End Sub 'On_ValidationAndVerification ------------------
     '-------------------------------------------------------
     

5. Save the DPP file and test it to verify that it works.
6. Once verified upload it to your SystemLink Data Preprocessor Instance.
   • If you already have an existing Data Preprocessor Instance, you can upload the new DPP file by going to <Data Processor Instance> >> Procedure >> Update Procedure.