DOKK / manpages / debian 12 / bpfcc-tools / criticalstat-bpfcc.8.en
criticalstat(8) System Manager's Manual criticalstat(8)

criticalstat - A tracer to find and report long atomic critical sections in kernel

criticalstat [-h] [-p] [-i] [-d DURATION]

criticalstat traces and reports occurrences of atomic critical sections in the kernel with useful stacktraces showing the origin of them. Such critical sections frequently occur due to use of spinlocks, or if interrupts or preemption were explicitly disabled by a driver. IRQ routines in Linux are also executed with interrupts disabled. There are many reasons. Such critical sections are a source of long latency/responsive issues for real-time systems.

This works by probing the preempt/irq and cpuidle tracepoints in the kernel. Since this uses BPF, only the root user can use this tool. Further, the kernel has to be built with certain CONFIG options enabled. See below.

Enable CONFIG_PREEMPT_TRACER, CONFIG_PREEMPTIRQ_EVENTS (CONFIG_PREEMPTIRQ_TRACEPOINTS in kernel 4.19 and later) and CONFIG_DEBUG_PREEMPT. Additionally, the following options should be DISABLED on older kernels: CONFIG_PROVE_LOCKING, CONFIG_LOCKDEP.

Print usage message.
Find long sections where preemption was disabled on local CPU.
Find long sections where interrupt was disabled on local CPU.
Only identify sections that are longer than DURATION in microseconds.

# criticalstat
# criticalstat -p
# criticalstat -d 500
# criticalstat -p -d 500

This tool can cause overhead if the application is spending a lot of time in kernel mode. The overhead is variable but can be 2-4% of performance degradation. If overhead is seen to be too much, please pass a higher DURATION to the -d option to filter more aggressively.

This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

Linux

Unstable - in development.

Joel Fernandes

Linux kernel's preemptoff and irqoff tracers.

2018-06-07 USER COMMANDS