Analysis Automation Takes Longer to Execute in SystemLink Compared to DIAdem

Issue Details

• I have created an Analysis Automation Procedure (.ANP file). I noticed that when executing the ANP from DIAdem, it completes much faster than it does from my SystemLink Analysis Server. Why is this?
• What causes an ANP to take longer to execute from SystemLink Server than it does when debugging directly in DIAdem?
• How can I reduce the execution time of my Analysis Automation in SystemLink?

Solution

It is expected that an Analysis Automation Procedure (.ANP file) will have a longer execution time in SystemLink Server than it does in DIAdem. The following factors contribute to the additional execution time.
 

Performing the Search Query

If the ANP has been configured with a Search Query, SystemLink must perform this search and identify the data files to execute on before the ANP analysis script can start. When testing the ANP directly in DIAdem, the data file is manually selected from a pop-up window. 

Therefore, when monitoring the execution time from DIAdem, this only takes into account the duration of the analysis script and not the Search Query. Once an ANP is triggered from SystemLink, additional time is required to perform the Search Query for each execution.

Launching DIAdem Workers

DIAdem Workers are background processes launched by SystemLink to execute the ANP. Their behaviour varies depending on whether the ANP uses Visual Basic Script (VBScript) or Python.

When executing an ANP directly in DIAdem, the following considerations must be made:

• If programming using VBScript, DIAdem launches a new instance of the VBScript Engine each time a script executes.
• If programming using Python, DIAdem launches one instance of the Python Engine which persists between Python scripts and is only released once DIAdem is closed.

When running an ANP from SystemLink:

• If the ANP uses VBScript, the DIAdem Worker persists in memory between tasks.
• If the ANP uses Python, the DIAdem Worker is automatically terminated between tasks. Therefore, SystemLink requires extra time to launch a new DIAdem Worker for each task.

Additional Information

To minimize the difference between the ANP execution time in DIAdem compared to SystemLink, consider the following points:

• Try increasing the number of DIAdem Workers that are launched to increase the number of parallel tasks that can be executed.
  • Refer to step 2 in Distributing Compute Nodes to configure the Number of Parallel Requests to the Compute Nodes option, which directly corresponds to the number of parallel DIAdem Workers.
• Reduce the server's CPU load and RAM consumption as much as possible.
  • The speed at which the DIAdem Workers are launched, and the speed at which they perform tasks, is dependent on the number of CPU cores and available RAM. If possible, increase the specifications of your server.
  • If the server's hardware cannot be changed, consider alleviating strain on the server. The ANP will execute more slowly if the server has a high CPU load and is prioritising other tasks.
• If using Python scripts in the ANP, the DIAdem Workers can be configured to persist between tasks. This will save time by preventing the server from launching a new DIAdem Worker for each task.
  • In the ANP analysis script, use dd.DIAdemWorkerNeedsRestart=FALSE. This allows the DIAdem Workers to persist between tasks, which can slightly reduce the execution time on the server. 
  • Note: ANPs using Python are designed to automatically terminate between tasks to avoid potential crosstalk between modules. Setting dd.DIAdemWorkerNeedsRestart may cause unexpected issues.