argdist - Trace a function and display a histogram or frequency
count of its parameter values. Uses Linux eBPF/bcc.
argdist [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-d
DURATION] [-n COUNT] [-v] [-T TOP] [-H specifier] [-C specifier] [-I header]
[-t TID]
argdist attaches to function entry and exit points, collects
specified parameter values, and stores them in a histogram or a frequency
collection that counts the number of times a parameter value occurred. It
can also filter parameter values and instrument multiple entry points at
once.
Since this uses BPF, only the root user can use this tool.
- -h
- Print usage message.
- -p PID
- Trace only functions in the process PID.
- -t TID
- Trace only functions in the thread TID.
- -z STRING_SIZE
- When collecting string arguments (of type char*), collect up to
STRING_SIZE characters. Longer strings will be truncated.
- -i INTERVAL
- Print the collected data every INTERVAL seconds. The default is 1
second.
- -d DURATION
- Total duration of trace in seconds.
- -n NUMBER
- Print the collected data COUNT times and then exit.
- -v
- Display the generated BPF program, for debugging purposes.
- -T TOP
- When collecting frequency counts, display only the top TOP entries.
- -H specifiers, -C
specifiers
- One or more probe specifications that instruct argdist which functions to
probe, which parameters to collect, how to aggregate them, and whether to
perform any filtering. See SPECIFIER SYNTAX below.
- -I header
- One or more header files that should be included in the BPF program. This
enables the use of structure definitions, enumerations, and constants that
are available in these headers. You should provide the same path you would
include in the BPF program, e.g. 'linux/blkdev.h' or 'linux/time.h'. Note:
in many cases, argdist will deduce the necessary header files
automatically.
The general specifier syntax is as follows:
{p,r,t,u}:{[library],category}:function(signature):type[,type...]:expr[,expr...][:filter]][#label]
- {p,r,t,u}
- Probe type - "p" for function entry, "r" for function
return, "t" for kernel tracepoint, "u" for USDT probe;
-H for histogram collection, -C for frequency count. Indicates where to
place the probe and whether the probe should collect frequency count
information, or aggregate the collected values into a histogram. Counting
probes will collect the number of times every parameter value was
observed, whereas histogram probes will collect the parameter values into
a histogram. Only integral types can be used with histogram probes; there
is no such limitation for counting probes.
- [library]
- Library containing the probe. Specify the full path to the .so or
executable file where the function to probe resides. Alternatively, you
can specify just the lib name: for example, "c" refers to libc.
If no library name is specified, the kernel is assumed.
- category
- The category of the kernel tracepoint. For example: net, sched,
block.
- function(signature)
- The function to probe, and its signature. The function name must match
exactly for the probe to be placed. The signature, on the other hand, is
only required if you plan to collect parameter values based on that
signature. For example, if you only want to collect the first parameter,
you don't have to specify the rest of the parameters in the signature.
When capturing kernel tracepoints, this should be the name of the event,
e.g. net_dev_start_xmit. The signature for kernel tracepoints should be
empty. When capturing USDT probes, this should be the name of the probe,
e.g. reloc_complete. The signature for USDT probes should be empty.
- [type[,type...]]
- The type(s) of the expression(s) to capture. This is the type of the keys
in the histogram or raw event collection that are collected by the
probes.
- [expr[,expr...]]
- The expression(s) to capture. These are the values that are assigned to
the histogram or raw event collection. You may use the parameters
directly, or valid C expressions that involve the parameters, such as
"size % 10". Tracepoints may access a special structure called
"args" that is formatted according to the tracepoint format
(which you can obtain using tplist). For example, the
block:block_rq_complete tracepoint can access args->nr_sector. USDT
probes may access the arguments defined by the tracing program in the
special arg1, arg2, ... variables. To obtain their types, use the tplist
tool. Return probes can use the argument values received by the function
when it was entered, through the $entry(paramname) special variable.
Return probes can also access the function's return value in $retval, and
the function's execution time in nanoseconds in $latency. Note that adding
the $latency or $entry(paramname) variables to the expression will
introduce an additional probe at the function's entry to collect this
data, and therefore introduce additional overhead.
- [filter]
- The filter applied to the captured data. Only parameter values that pass
the filter will be collected. This is any valid C expression that refers
to the parameter values, such as "fd == 1 && length >
16". The $entry, $retval, and $latency variables can be used here as
well, in return probes. The filter expression may also use the STRCMP
pseudo-function to compare a predefined string to a string argument. For
example: STRCMP("test.txt", file). The order of arguments is
important: the first argument MUST be a quoted literal string, and the
second argument can be a runtime string.
- [label]
- The label that will be displayed when printing the probed values. By
default, this is the probe specifier.
- Print a histogram of
allocation sizes passed to kmalloc:
- # argdist -H 'p::__kmalloc(u64 size):u64:size'
- Print a count of how
many times process 1005 called malloc with an allocation size of 16
bytes:
- # argdist -p 1005 -C 'p:c:malloc(size_t
size):size_t:size:size==16'
- Snoop on all strings
returned by gets():
- # argdist -C 'r:c:gets():char*:$retval'
- Print a histogram of
read sizes that were longer than 1ms:
- # argdist -H 'r::__vfs_read(void *file, void *buf, size_t
count):size_t:$entry(count):$latency > 1000000'
- Print frequency counts
of how many times writes were issued to a particular file descriptor number,
in process 1005:
- # argdist -p 1005 -C 'p:c:write(int fd):int:fd'
- Print a histogram of
error codes returned by read() in process 1005:
- # argdist -p 1005 -H 'r:c:read()'
- Print a histogram of
buffer sizes passed to write() across all processes, where the file
descriptor was 1 (STDOUT):
- # argdist -H 'p:c:write(int fd, const void *buf, size_t
count):size_t:count:fd==1'
- Count fork() calls in libc
across all processes, grouped by pid:
- # argdist -C 'p:c:fork():int:$PID;fork per process'
- Print histogram of
number of sectors in completing block I/O requests:
- # argdist -H 't:block:block_rq_complete():u32:nr_sector'
- Aggregate interrupts
by interrupt request (IRQ):
- # argdist -C 't:irq:irq_handler_entry():int:irq'
- Print the functions used
as thread entry points and how common they are:
- # argdist -C 'u:pthread:pthread_start():u64:arg2' -p 1337
- Print histograms of
sleep() and nanosleep() parameter values:
- # argdist -H 'p:c:sleep(u32 seconds):u32:seconds' -H
'p:c:nanosleep(struct timespec *req):long:req->tv_nsec'
- Spy on writes to STDOUT
performed by process 2780, up to a string size of 120 characters:
- # argdist -p 2780 -z 120 -C 'p:c:write(int fd, char* buf, size_t
len):char*:buf:fd==1'
- Group files being read from
and the read sizes from __vfs_read:
- # argdist -C 'p::__vfs_read(struct file *file, void *buf, size_t
count):char*,size_t:file->f_path.dentry->d_iname,count:file->f_path.dentry->d_iname[0]!=0'
This is from bcc.
- https://github.com/iovisor/bcc
Also look in the bcc distribution for a companion _examples.txt
file containing example usage, output, and commentary for this tool.
Unstable - in development.