DOKK / manpages / debian 12 / bpfcc-tools / funcslower-bpfcc.8.en
funcslower(8) System Manager's Manual funcslower(8)

funcslower - Trace slow kernel or user function calls.

funcslower [-hf] [-p PID] [-U | -K] [-m MIN_MS] [-u MIN_US] [-a ARGUMENTS] [-T] [-t] [-v] function [function ...]

This script traces a kernel or user function's entry and return points, and prints a message when the function's latency exceeded the specified threshold. Multiple functions are supported, and you can mix kernel functions with user functions in different libraries.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 1 is used. Recursive functions are not supported: only the inner-most recursive invocation will be traced.

Since this uses BPF, only the root user can use this tool.

CONFIG_BPF and bcc.

-p PID Trace this PID only.

Minimum duration to trace, in milliseconds. Default is 1 ms.
Minimum duration to trace, in microseconds.
Print the function's arguments, up to 6.
Print a HH:MM:SS timestamp with each entry.
Print a seconds timestamp with each entry, at microsecond resolution.
Print output in folded stack format.
Show stacks from user space only (no kernel space stacks).
Show stacks from kernel space only (no user space stacks).
Print the resulting BPF program, for debugging purposes.
The function to trace -- multiple functions are supported. If a plain function name is provided, the function is assumed to be a kernel function. For user functions, provide the library name and the function name, e.g. bash:readline or c:malloc.

# funcslower vfs_write
# funcslower -u 10 c:open
# funcslower -p 135 -u 10 c:malloc c:free
# funcslower -a 4 SyS_write
# funcslower -UK c:open

Time of the event as a human-readable HH:MM:SS format, or a timestamp in seconds at microsecond-accuracy from the first event seen.
Process name.
Process ID.
Latency of the operation in either microseconds (us) or milliseconds (ms).
The return value from the function. Often useful for diagnosing a relationship between slow and failed function calls.
The function name, followed by its arguments if requested.

Depending on the function(s) being traced, overhead can become severe. For example, tracing a common function like malloc() can slow down a C/C++ program by a factor of 2 or more. On the other hand, tracing a low-frequency event like the SyS_setreuid() function will probably not be as prohibitive, and in fact negligible for functions that are called up to 100-1000 times per second.

You should first use the funclatency and argdist tools for investigation, because they summarize data in-kernel and have a much lower overhead than this tool. To get a general idea of the number of times a particular function is called (and estimate the overhead), use the funccount tool, e.g.:

# funccount c:open

This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

Linux

Unstable - in development.

Sasha Goldshtein

funccount(8), funclatency(8), argdist(8), trace(8)

2017-03-30 USER COMMANDS