When to Enable System Tracing on NI Linux Real‑Time Controllers

What problems does tracing help diagnose?

Tracing is a helpful tool that allows you to diagnose issues that cannot be easily observed when analyzing code developed in LabVIEW Real-Time. Through tracing, it is possible to determine which tasks were being executed by the kernel and the Operating System before an issue occurred.  

Capturing traces can help you investigate the following scenarios specifically:

• Jitter. If you have identified a critical task in your application that is not meeting the expected determinism even though it is running inside a Timed Loop, capturing traces can help you uncover which other tasks are negatively impacting the timing of your task. 
• System Hangs or Freezes. When your system is experiencing hangs or freezes, tracing can help you identify what the kernel was doing before the hang and determine how much activity was being executed on the different CPUs that are part of your controller.
• Lock Contention. Lock contention occurs when one or more tasks are waiting to use a shared resource. When running your application, this behavior can be observed as increased latency (for example, timed loops missing their deadlines and tasks that normally take microseconds suddenly require milliseconds to complete execution), reduced parallelism, the system freezes but eventually recovers. Through tracing, it is possible to observe how the different tasks are being handled by the kernel and which one is monopolizing a shared resource.

When not to use tracing

Capturing traces in NI Linux Real-Time is beneficial to help understand how timing and scheduling is being handled by the kernel when running your application, however, tracing won’t be helpful in the following situations:

• Issues related to the logic implemented in your code. As mentioned above, the traces will display helpful information about how the tasks are being managed by the kernel and the Operating System, but they will not show any values or magnitudes used in your code. For these cases, use traditional debugging. 
• Crashes. When a crash occurs, the system will suddenly restart, and any active tracing operation will be interrupted. The information contained in the trace will be incomplete, and you won’t be able to identify what happened before the crash. To monitor and diagnose crashes, consider using another strategy such as system logs, error logs and core dumps.
• Performance Profiling. While a trace can provide you with information about how tasks are executed on the system and you can validate how long it takes to complete an iteration, it is not possible to obtain data on CPU and memory usage, which is key for improving performance. Consider using the Profile Performance and Memory Window, included in LabVIEW, for this purpose.

Considerations before starting to capture traces

Before capturing traces, it is essential to properly configure the controller and system to ensure that the tracing session collects the relevant information required to analyze the unexpected behavior. Below you will find some general considerations as well as specific recommendations depending on the programming language being used for developing your application.

• General Considerations
  • Stop capturing traces after the event occurs. While being recorded, traces are stored in a limited-size circular buffer. If the buffer is full, the oldest traces are overwritten by new ones. As a result, the behavior you are trying to capture may be lost if newer trace data replaces it.
  • Give your timed loops meaningful names. When analyzing the behavior of a specific timed loop in your application, assigning a name to it can help you identify it in the traces. To modify the name of your timed loop, double-click on the Input Node and modify the Structure Name parameter accordingly.

• • Ensure trace-cmd is installed on the controller. This package allows you to extract the trace from the system and generate a file that can be analyzed with KernelShark. Starting with NI Linux Real-Time 2025 Q2, trace-cmd is installed automatically on the system. For NI Linux Real-Time 2025 Q1 and earlier, follow the instructions below to install the software:
    1. Access the shell on your Real-Time (RT) controller.
    2. Run the command opkg update
    3. Run the command opkg install trace-cmd

• LabVIEW 2025 Q3 (and earlier) and Other Programming Languages

Configure and enable tracing on your RT controller as explained in Tracing on NI Linux Real-Time under the ftrace section, and consider the following:

• • Enable tracing for all events. By capturing all the events, it will be possible to analyze what the system was doing before and during the behavior under investigation.
  • Increase the size of the buffer. If more information is required, you can adjust the size of the CPU buffers to be able to record more traces.
  • Adjust the /sys/kernel/debug/tracing directory permissions accordingly (only for LabVIEW-based applications).  Tracing is a privileged operation. If you are recording directly from your VI in LabVIEW (2025 Q1 and earlier), adjust file ownership and/or permissions by running the command through SSH (note: this change will not persist across a reboot):

chown -R lvuser:ni /sys/kernel/debug/tracing

• LabVIEW 2026 Q1 and Later

Configure tracing using the RT Tracing VIs that are automatically installed with the LabVIEW Real-Time Module 2026 Q1 and later and can be found by navigating to Controls Palette >> Real-Time >>RT Tracing. These VIs already use ftrace to capture traces and can export them into a .dat file that can be opened in KernelShark.

Options for increasing the size of the buffer and adding custom markers to the traces are also provided through these VIs. For more information, please refer to the following documentation: RT Tracing.