Analyzing DAQ-Related Interrupt Activity in CompactRIO Controllers

This article provides a case study to demonstrate how to apply and use the diagnostic workflow presented in Diagnosing Timing Jitter, Scheduling Delays, and Hang Conditions in NI Linux Real-Time Systems, when analyzing and studying NI-DAQmx Hardware-Timed Single Point (HWTSP) acquisition on NI Linux Real-Time Systems.

HWTSP is a sample mode intended for deterministic and real-time applications, particularly closed-loop control systems. In this mode, the hardware sample clock determines when each sample is acquired, while the application processes one sample at a time with no buffering. This allows the software loop to react to each hardware-timed sample with low latency.

Scenario

A cRIO-9045 controller runs an application that is continuously acquiring data through the NI-DAQmx API. The type of acquisition that is executing corresponds to a Hardware-Timed Single Point operation. The exact VI that is running on the controller is presented in the image below:

With the configuration above, the controller appears to behave as required. However, further analysis is expected to be carried out on the system to fully confirm that the NI-DAQmx task is not being affected by other processes running on the cRIO.

Setting Up the Code to Capture Traces

Before capturing traces, it is required to prepare the code appropriately, so it is easier to interpret them and to ensure that the event under investigation is captured. These are the modifications that were implemented on the original code:

As observed, code for starting and stopping capturing traces has been added around the while loop responsible for reading data coming from Module 2 on the cRIO. This way it is possible to ensure that the tracing operation ends as soon as the while loop is stopped or an error occurs.

Trace Analysis

After opening the trace.dat file in KernelShark, the information can be filtered to display the task and related events that correspond to the while loop responsible for the data acquisition. Please refer to the steps below for more details.

1. Navigate to Filter >>Show Events

2. From the Events Dialog Box, select sys_enter_ioctl, which is the tracepoint related device/driver control request

3. The Event List will be filtered and display the tasks that executed the sys_enter_ioctl event.

As observed, there are a few tasks that are associated with the sys_enter_ioctl. Nonetheless, LV_ESys2_Thr1 corresponds to the actual NI-DAQmx operation, as LV_ESys2_Thr* represents worker threads that are utilized to run parts of LabVIEW diagrams. Please note that the LabVIEW thread responsible for executing the NI-DAQmx task may vary between traces. Therefore, further analysis can be executed for the LV_ESys2_Thr1 task.

After filtering the trace to display only the information related to this task, the following pattern can be detected in the event list:

The events highlighted in red on the image above are explained below in more detail:

1. irq/136-atomicc: this is an interruption that wakes LV_ESys2_Thr1. When the irq/136-atomicc runs, the event sched/sched_switch is also executed, meaning that CPU 0 starts running the code associated with the NI-DAQmx task. Please note that the irq/136-atomicc interruption, is exclusive to compactRIO (cRIO) controllers. Its name will be different on PXI-based controllers. 
2. kmem/kfree: the kernel is freeing memory that was previously allocated. The call_site=nipalk... entries indicate the deallocation is occurring inside NI's Kernel Abstraction Layer (NIKAL), which is the element used by NI-DAQmx, VISA and other NI drivers.
3. syscalls/sys_exit_ioctl: The current ioctl() system call has finished executing in the kernel and control is being returned to the calling user-space thread. ioctl() is a Linux system call used by user-space applications to send device-specific commands or requests to kernel-space drivers.
4. syscalls/sys_enter_ioctl: indicates that the LabVIEW thread is issuing a new request to the kernel driver.
5. kmem/kmalloc: these events show that NIKAL allocates temporary kernel memory required for the driver operation.
6. timer_init and timer_start show that NIKAL configures a kernel timer associated with the request. The timer uses nNIKAL1_timerListTimeoutCallback as its callback, suggesting that the driver is establishing timeout handling for the ongoing operation.

According to the VI presented above, the sample rate has been configured to be 1 kHz. This means that a new sample will be acquired every 1 ms. To confirm in KernelShark that a new sample is in fact being read by the system every 1 ms, it is possible to use markers between two consecutive irq/136-atomicc events to calculate the delay between two acquired samples:

As shown in the image above, the delta between the two markers is approximately 1 ms (0.000 998 784 s), this means that the system is capturing the data at the expected rate. 

When analyzing the behavior above with the Diagnostic Workflow presented in Diagnosing Timing Jitter, Scheduling Delays, and Hang Conditions in NI Linux Real-Time Systems, it can be observed that on every loop iteration, a wake-up event occurs (i.e, irq/136-atomicc triggers the execution), and the process runs and receives CPU time on CPU 0. When it starts running, the kernel frees memory, executes the events associated with the driver (ioctl()), new memory is allocated on the kernel, and a timer is created. Therefore, all the steps of the workflow are executed:

This means that the task runs without any issues.