Opening a Trace
To open a trace file, follow the steps outlined below:
- Launch KernelShark.
- Select File >> Open Trace File.
- Browse to the trace.dat file.
- Wait for the trace database to finish loading.
Once loaded, KernelShark will display two main areas: Graph and Event.
Understanding the User Interface
As mentioned before, the user interface is composed of the following elements:
Displays the information of the tasks executed per CPU or it can also show the data of a given task. Additionally, it also has a control area that can be used to navigate across the different plots.
The control area offers the following options:
It also contains marker buttons that can be used to place up to two markers on the graph (or the event list). This feature can be used for calculating the time difference between two events. The result is displayed in the AB Delta section in seconds.
It contains every recorded trace event in chronological order.
Each entry includes:
-
- CPU: the CPU that the event occurred on.
- Timestamp: the timestamp of the event in seconds.
- Task name: the name of the process that was running when the event occurred. Note: Linux may refer to tasks as threads or processes.
- Process ID (PID): identifier for the process that was running.
- Latency: shows the kernel execution state when each trace event occurred. It indicates whether interrupts were disabled, a reschedule was pending, the CPU was handling an interrupt, or preemption was disabled. For more details refer to the KernelShark documentation
- Event: the name of the event that occurred when the process ran.
- Event-specific information (Info): the data output of a particular event.
Additionally, you can use the Search section to look for a specific process or event.
Filtering
Considering the amount of data that can be captured in a trace file, KernelShark contains filters for both tasks and events, that can be used to display the most significant information for your investigation. To access the filters, navigate to the Filter menu on the toolbar.
You can select a filter from the following options:
- Show events. This option will display a dialog box with a list of events that are contained in the trace file. Here, you can select which events you want to visualize in KernelShark.
- Show tasks. This option will open the Tasks filter dialog box, from which one or multiple tasks can be selected.
- Show CPUs. Use this option to open the CPU filter dialog box. If your system contains more than one CPU core, use this filter to display only the events and tasks that were captured for the specific CPU you selected.
It is important to mention that you can apply multiple filters to the trace file and create a custom advanced filter.
Common Tracepoints
When analyzing trace files in KernelShark, you may find the following common events or tracepoints:
| Tracepoint | Purpose |
| sched_switch | CPU switches from one task to another |
| sched_wakeup | Task becomes runnable |
| sched_waking | Wake-up process initiated |
| timer/hrtimer_start | High-resolution timer armed |
| timer/hrtimer_expire_entry | High-resolution timer expires |
| irq_handler_entry | Hardware interrupt handler begins |
| irq_handler_exit | Hardware interrupt handler ends |
| softirq_entry | Software interrupt processing begins |
| softirq_exit | Software interrupt processing ends |
| futex_wait | Thread waits on synchronization object |
| futex_wake | Waiting thread is awakened |
| sys_enter_ioctl | Device/driver control request |
| sys_enter_recvmsg | Network receive request |
| sys_enter_poll | Wait for I/O or file-descriptor events |
| sys_exit_clock_nanosleep | Process has exited a sleep state. This event is common in Timed Loops, when they begin running an iteration |
| sys_enter_clock_nanosleep | Process has entered a sleep state. This event is common in Timed Loops, when they finish executing the logic that has been placed inside of them as part of the current iteration. |