NAME

uprobe - trace a given uprobe definition. User-level dynamic tracing. Uses Linux ftrace. EXPERIMENTAL.

SYNOPSIS

uprobe [-FhHsv] [-d secs] [-p PID] [-L TID] {-l target | uprobe_definition [filter]}

DESCRIPTION

This will create, trace, then destroy a given uprobe definition. See Documentation/trace/uprobetracer.txt in the Linux kernel source for the syntax of a uprobe definition, and "uprobe -h" for examples. With this tool, the probe alias is optional (it will default to something meaningful).

WARNING: This uses dynamic tracing of user-level functions, using some relatively new kernel code. I have seen this cause target processes to fail, either entering endless spin loops or crashing on illegal instructions. I believe newer kernels (post 4.0) are relatively safer, but use caution. Test in a lab environment, and know what you are doing, before use. Also consider other (more developed) user-level tracers (perf_events, LTTng, etc.).

Use extreme caution with the raw address mode: eg, "p:libc:0xbf130". uprobe does not check for instruction alignment, so tracing the wrong address (eg, mid-way through a multi-byte instruction) will corrupt the target's memory. Other tracers (eg, perf_events with debuginfo) check alignment.

Also beware of widespread tracing that interferes with the operation of the system, eg, tracing libc:malloc, which by-default will trace _all_ processes.

I wrote this because I kept testing different custom uprobes at the command line, and wanted a way to automate the steps. For generic user-level tracing, use perf_events directly.

Since this uses ftrace, only the root user can use this tool.

REQUIREMENTS

REQUIREMENTS: FTRACE and UPROBE CONFIG, which you may already have on recent kernel versions, file(1), ldconfig(8), objdump(1), and some version of awk. Also, currently only executes on Linux 4.0+ (see WARNING) unless -F is used.

OPTIONS

-F

Force. Trace despite kernel version warnings. Use on older kernels may expose you to (since fixed) bugs, which can lock up or crash target processes, which could also lock up the entire system. Test in a lab environment before use, and consider other more developed user-level tracers (perf_events, LTTng, etc.).

-d seconds

Set the duration of tracing, in seconds. Trace output will be buffered and printed at the end. This also reduces overheads by buffering in-kernel, instead of printing events as they occur.

The ftrace buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing, try increasing that size.

-h

Print usage message.

-H

Print column headers.

-s

Print user-level stack traces after each event. These are currently printed in hex, and need post-processing to see user-level symbols (eg, addr2line; I should automate that).

-v

Show the uprobe format file only (do not trace), identifying possible variables for use in a custom filter.

-p PID

Only trace user-level functions when this process ID is on-CPU.

-L TID

Only trace user-level functions when this thread ID is on-CPU.

uprobe_definition

A full uprobe definition, as documented by Documentation/trace/uprobetracer.txt in the Linux kernel source. Note that the probe alias name is optional with uprobe(8), and if not specified, it will default to something meaningful. See the EXAMPLES section.

filter

An ftrace filter definition.

EXAMPLES

These examples may need modification to match your target software function names and platform's register usage. If using platform specific registers becomes too painful in practice, consider a debuginfo-based tracer, which can trace variables names instead (eg, perf_events).

trace readline() calls in all running "bash" executables:

# uprobe p:bash:readline

trace readline() with explicit executable path:

# uprobe p:/bin/bash:readline

trace the return of readline() with return value as a string:

# uprobe 'r:bash:readline +0($retval):string'

trace sleep() calls in all running libc shared libraries:

# uprobe p:libc:sleep

trace sleep() with register %di (x86):

# uprobe 'p:libc:sleep %di'

trace this address (use caution: must be instruction aligned):

# uprobe p:libc:0xbf130

trace gettimeofday() for PID 1182 only:

# uprobe -p 1182 p:libc:gettimeofday

trace the return of fopen() only when it returns NULL:

# uprobe 'r:libc:fopen file=$retval' 'file == 0'

FIELDS

The output format depends on the kernel version, and headings can be printed using -H. The format is the same as the ftrace function trace format, described in the kernel source under Documentation/trace/ftrace.txt.

Typical fields are:

TASK-PID

The process name (which could include dashes), a dash, and the process ID.

CPU#

The CPU ID, in brackets.

||||

Kernel state flags. For example, on Linux 3.16 these are for irqs-off, need-resched, hardirq/softirq, and preempt-depth.

TIMESTAMP

Time of event, in seconds.

FUNCTION

User-level function name.

OVERHEAD

This can generate a lot of trace data quickly, depending on the frequency of the traced events. Such data will cause performance overheads. This also works without buffering by default, printing function events as they happen (uses trace_pipe), context switching and consuming CPU to do so. If needed, you can try the "-d secs" option, which buffers events instead, reducing overhead. If you think the buffer option is losing events, try increasing the buffer size (buffer_size_kb).

If you find a use for uprobe(8) where the overhead is prohibitive, consider the same enabling using perf_events where overhead should be reduced.

SOURCE

This is from the perf-tools collection:

https://github.com/brendangregg/perf-tools

Also look under the examples directory for a text file containing example usage, output, and commentary for this tool.

OS

Linux

STABILITY

Unstable - in development.

AUTHOR

Brendan Gregg

SEE ALSO

kprobe(8)