High Counts of CPU Interrupts on NI cRIO 904x Rev C

Issue Details

• I have NI cRIO 904x Revision C hardware and I observe that the CPU usage is unusually high. Particularly on i801_SMB interrupts.
• When I ran cat /proc/interrupts command on console out, I found out CPU interrupts on i801_SMB(IRQ20) is very high. However, I just reformatted my NI cRIO-904x and re-installed NI software. What is the solution to this issue?
• I tried upgrading to BIOS version 2.0.3f0, but I still observe high CPU count on i801_SMB. Is there another solution for this issue?
• The cRIO freezes and I observe RCU related error messages in kernel ring buffer (dmesg).

  • task:rcu_preempt  state:I stack:  0 pid: 13 ppid:  2 flags:0x00004000
    Call Trace:
    ? __schedule+0x1de/0x5b0
    ? schedule+0x7e/0x120
    ? schedule_timeout+0x182/0x2f0
    ? del_timer_sync+0xa0/0xa0
    ? rcu_gp_kthread+0x6ac/0xbd0
    ? rcu_cpu_kthread+0x280/0x280
    ? kthread+0x17c/0x1a0
    ? kthread_worker_fn+0x110/0x110
    ? ret_from_fork+0x1f/0x30
    Sending NMI from CPU 1 to CPUs 0:

Solution

This issue was caused by a SMBALERT_DIS bit of SLVCMD register that enables polling on i801_smbus interrupt, which increases the CPU usage. This was determined to be a bug. There are several solutions to this issue, depending on NI Linux Real-Time versions installed on NI cRIO 904x.

Before implementing the solution in this article, please ensure that you are using NI cRIO 904x Revision C normal or TPM controller AND observe high CPU Interrupts on i801_SMB. Please refer to Find the Revision of NI Hardware link on the method on how to check the revision version of your NI cRIO 904x. To check whether high CPU interrupts is caused by i801_SMB, please run cat /proc/interrupts command in console out. Please refer to Accessing the Shell on NI Linux Real-Time Devices link for the method on how to connect to your controller’s console via SSH and the picture below.
 
NI Linux Real Time version can be checked by going into your controller’s System Settings in NI MAX. Please refer to picture below.
 

If NI Linux Real-Time version is later than x64 4.9.47-rt37-6.0.0f1 version (NI CompactRIO 18.5):

• Update the BIOS version on your NI cRIO 904x to BIOS version 2.0.3f0 and later. The latest BIOS version can be downloaded from cRIO-904x Firmware and Additional Drivers link.
• Please refer to the readme file for a guide to update BIOS, which will be included with the BIOS installation file. You can also refer to CompactRIO 904x Firmware and Additional Drivers Readme link.
• Please take note that NI does not support downgrading BIOS version and is not responsible for any damage that is caused by downgrading BIOS.

If NI Linux Real-Time version is same or older than x64 4.9.47-rt37-6.0.0f1 version (NI CompactRIO 18.5), please implement either one of the following methods.

 

METHOD 1: Creating a RTOS file to set SMBALERT_DIS bit of SLVCMD register:

1. Please first update BIOS version on NI cRIO 9040x to BIOS version 2.0.3f0 and later. Please refer to “If NI Linux Real-Time version is later than x64 4.9.47-rt37-6.0.0f1 version (NI CompactRIO 18.5)” section for additional information.
2. Reboot your controller and initiate an SSH connection. Please refer to Accessing the Shell on NI Linux Real-Time Devices link for guide.
3. For this step, please create a RTOS script file. To create the RTOS file, input cat > /etc/rc5.d/S01smbfix line and press Enter. Input #! /bin/bash line and press Enter. Next, input devmem 0x91612011 8 0x5 line and press Enter. Finally, press <Ctrl-D>. This command line will create and save a RTOS file in the controller’s local location. Once the script file has been saved, the console will prompt to set permission for the script file.
4. Input chmod +x /etc/rc5.d/S01smbfix line and press Enter. This will set permission for the script file. This completes the process of creating the RTOS script file. Please refer to the picture below.

5. Next, exit from console out and reboot your controller by pressing the Restart button in NI MAX. Please refer to picture below

6. Reconnect to your controller’s console out and run devmem 0x91612011 8 command and check if the output is 0x05. Run cat /proc/interrupts command and check whether high CPU usage on i801_smbus interrupt is resolved. Please refer to the picture below.

METHOD 2: Instruct modprobe to ignore i801_smbus interrupt.

1. Initiate an SSH connection. Please refer to Accessing the Shell on NI Linux Real-Time Devices link for guide.
2. Input echo "options i2c-i801 disable_features=0x10" >> /etc/modprobe.d/i2c-i801.conf && reboot line and press Enter.
3. Run cat /proc/interrupts command and check whether high CPU usage on i801_smbus interrupt is resolved.

After implementing the solution above, the CPU usage caused by i801_smbus interrupt should be resolved.

Additional Information

If after trying out the solution above and still same issue occurs, please check if by running devmem 0x91612011 8 0x5 command was able reduce CPU share on i801_smbus interrupts. However, note that this might not be a permanent solution. To permanently implement this solution, install the attached RTOS script to NI cRIO 904x.

If you do not have NI cRIO 904x Rev C but observe high CPU usage in idle state, please reach out to NI Support for further troubleshooting.