Diagnosing Timing Jitter, Scheduling Delays, and Hang Conditions in NI Linux Real-Time Systems

LabVIEW Real-Time (RT) applications may experience symptoms such as timing jitter, missed deadlines, or apparent system hangs. These symptoms can arise from different underlying mechanisms including CPU contention, interrupt pressure, or resource blocking.

This article presents a trace-based diagnostic workflow that helps identify where task execution stops progressing. The goal of trace analysis is to determine where execution stops progressing in the task lifecycle.

Task Execution Model

LabVIEW Real-Time tasks follow a sequence of events that allow them to execute periodically, as observed in the diagram below:

Each of the steps of the sequence above generates a specific event that can be identified on the traces as follows:

Note: Not all wake-up sources have a visible “configuration” event in trace data. Timer-based wakeups (e.g., hrtimer_start) are typically observable, while I/O and synchronization-based wakeups may only be visible when triggered.

After the task starts running, it will perform its work. Since tasks are run periodically, the stages above will be repeated. When troubleshooting and interpreting traces, the key is to identify which transition in this sequence fails.

Diagnostic Workflow

When investigating unexpected behavior on NI Linux RT systems, the goal is to determine where task execution stops progressing. If one of these transitions fails, the trace can provide clues about the underlying cause.

The following workflow provides a process for analyzing trace data, identifying potential root causes and actions that you can take to investigate further.

Below you can find a step-by-step explanation of the presented workflow.

Step 1: Identify The Affected Task

Start by identifying the real-Time task associated with the observed behavior. This may correspond to a Timed Loop, thread, or critical process. Please consider the following key points:

• Determine the task name or identifier. Some common tasks are:
  • Timed Loops. Can be identified by their name, which is configured in LabVIEW through the Input Node Structure Name field. 
  • LV_ESys2_Thr*. Worker threads used to run parts of LabVIEW diagrams. 
  • MainAppThread. It handles Main User Interface operation and communication. 
  • LV_Occurrence. A thread that handles LV Occurances (e.g. software interrupts).
  • LV_Socket_Mon. A thread that handles socket (TCP and UDP) traffic. 
• Locate the task in the trace timeline
• Focus the analysis on the events associated with that task

Step 2: Verify That a Wake-Up Event Occurs

Confirm that a wake-up event occurs. Different scenarios may apply depending on the type of task being analyzed:

• Time-based execution. Verify that the periodic timer responsible for waking the task up is functioning correctly and expiring as expected. Here, it is important to identify the following relevant trace events: timer/hrtimer_start and timer/hrtimer_expire_entry.
• Synchronization-based execution (lock). The wake-up is triggered when a lock is released or a synchronization condition is satisfied. Look for the following trace events: futex_wake, futex_unlock_pi, futex_cmp_requeue_pi. Additionally, identify the task that released the lock and the task that was waiting for it.
• I/O or hardware-driven execution (e.g., DAQ interrupts). The wake-up is triggered by an interrupt or driver event. In compactRIO (cRIO) controllers, this interrupt is irq/136-atomicc. Verify that the event is occurring as expected and that it is being triggered.

Step 3: Verify Task Wake-Up

Once the wake-up event occurs, the task should become runnable. You should identify the sched_wakeup event corresponding to the task in the captured traces. 

Step 4: Verify CPU Scheduling

After waking up, the scheduler must assign CPU time to the task. You should look for the sched_switch event.

Step 5: Verify Task Progress

Finally, confirm that the task is executed and makes progress once it receives CPU time. For this purpose, please consider the following suggestions:

• Consider using trace markers that you can visualize in the trace.dat file. For LabVIEW 2026 Q1, you can use the Log User String VI which is part of the RT Tracing VIs. For previous LabVIEW versions, please refer to the Tracing on NI Linux Real-Time documentation.
• If you are analyzing a Timed Loop, look for the sys_enter_clock_nanosleep and sys_exit_clock_nanosleep events, as they indicate when the loop enters and exits its sleep period, respectively. These events can be used to identify the end of one iteration and the beginning of the next.

Interpreting Failures

If the workflow is interrupted at some point, you may observe failures like scheduling latency (CPU contention), interrupt pressure, lock contention, blocking or deadlock conditions, and priority inversion in your system.

The following table summarizes how different trace observations may indicate specific types of issues.

The observations listed above represent common points where task execution may stop progressing. The following sections describe these scenarios in more detail

Scheduling Latency (CPU Contention)

If a task wakes up but does not immediately receive CPU time, the system may be experiencing scheduling latency. You may observe this as a delay between sched_wakeup and sched_switch. This delay indicates that another task or kernel activity is occupying the CPU. Possible causes include:

• Multiple real-Time tasks competing for the same CPU.
• Incorrect CPU affinity configuration.
• High system load.

In this scenario, it is recommended that you identify which task is running during the delay, and ensure that your CPU affinity settings and task priorities have been correctly set.

Interrupt Pressure

Heavy interruption activity can delay task execution even when the task is ready to run. The trace may display the following events: irq_handler_entry and irq_handler_exit. When interrupts dominate the CPU timeline, real-time tasks may experience increased latency. For these situations, it is recommended to look for interrupting sources, review hardware drivers generating interruptions and analyze interruption frequency. 

It is important to note that most interrupts on NI Linux Real-Time systems are handled by CPU 0. Therefore, if your controller has multiple CPU cores, you may consider assigning some of the timed structures in your application to other CPU cores to help reduce latency and improve overall system responsiveness.  

Lock Contention

When the task is executed but fails to make progress due to waiting on shared resources, the system may encounter lock contention. In the trace, this is observed as multiple futex_wait, futex_lock, futex_unlock and futex_cmp patterns. Possible causes include incorrect primitives specifically related to sharing information between loops (e.g., from a Timed Loops to a While Loop) and using controls and indicators inside a Timed Loop (this creates a lock between the loop and the thread in charge of the User Interface updates). In this scenario, it is recommended to identify the lock owner in the traces. 

Blocking or Deadlock Conditions

If the wake-up event does not occur again, the task may have failed to complete its previous iteration. In the trace, this behavior may be observed as missing hrtimer_start event, futex_wake, or interrupt activity. The recommended checks include:

• Identify which resource the task may be waiting for
• Review synchronization mechanisms used by the application
• Check whether another task is holding a lock for an extended period
• Verify that blocking operations use appropriate timeouts.
• When working with Timed Loops, ensure that the code within the loop can complete execution within the configured period through benchmarking. Otherwise, subsequent iterations may be delayed.

Below you can find a flow diagram that incorporates the information presented above as a summary. You can use it when diagnosing timing issues on your real-time applications.

Next Steps

The following case studies demonstrate how the diagnostic workflow described in this article can be applied to real trace captures:

• Diagnosing Scheduling Jitter in NI Linux Real-Time Using Kernel Traces
• Identifying Lock Contention in NI Linux Real-Time Trace Data
• Analyzing DAQ-Related Interrupt Activity in CompactRIO Controllers