Output format

-B MODE, Set the buffer mode for stdout. Valid values are

-f FORMAT, Set the output format. Valid values are

-o FILENAME

--no-warnings

Tracing

-e PROGRAM

-I DIR

--include FILENAME

-l [SEARCH]

--unsafe

-k

stdin:48-57: WARNING: Failed to probe_read_user_str: Bad address (-14)
u:lib.so:"fn(char const*)" { printf("arg0:%s\n", str(arg0));}


                                                 ~~~~~~~~~

-kk

Process management

-p PID

-c COMMAND

--usdt-file-activation

Miscellaneous

--info

-h, --help

-V, --version

-v

-d

-dd

BPFTRACE_STRLEN

Default: 64

Number of bytes allocated on the BPF stack for the string returned by str().

Make this larger if you wish to read bigger strings with str().

Beware that the BPF stack is small (512 bytes).

Support for even larger strings is [being discussed]( <https://github.com/iovisor/bpftrace/issues/305>).

BPFTRACE_NO_CPP_DEMANGLE

Default: 0

C++ symbol demangling in user space stack traces is enabled by default.

This feature can be turned off by setting the value of this environment variable to 1.

BPFTRACE_MAP_KEYS_MAX

Default: 4096

This is the maximum number of keys that can be stored in a map. Increasing the value will consume more memory and increase startup times. There are some cases where you will want to: for example, sampling stack traces, recording timestamps for each page, etc.

BPFTRACE_MAX_PROBES

Default: 512

This is the maximum number of probes that bpftrace can attach to. Increasing the value will consume more memory, increase startup times and can incur high performance overhead or even freeze or crash the system.

BPFTRACE_CACHE_USER_SYMBOLS

Default: 0 if ASLR is enabled on system and -c option is not given; otherwise 1

By default, bpftrace caches the results of symbols resolutions only when ASLR (Address Space Layout Randomization) is disabled. This is because the symbol addresses change with each execution with ASLR. However, disabling caching may incur some performance penalty. Set this env variable to 1 to force bpftrace to cache.

BPFTRACE_VMLINUX

Default: None

This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset. If this value is not given, bpftrace searches vmlinux from pre defined locations. See src/attached_probe.cpp:find_vmlinux() for details.

BPFTRACE_BTF

Default: None

The path to a BTF file. By default, bpftrace searches several locations to find a BTF file. See src/btf.cpp for the details.

BPFTRACE_PERF_RB_PAGES

Default: 64

Number of pages to allocate per CPU for perf ring buffer. The value must be a power of 2.

If you’re getting a lot of dropped events bpftrace may not be processing events in the ring buffer fast enough. It may be useful to bump the value higher so more events can be queued up. The tradeoff is that bpftrace will use more memory.

BPFTRACE_MAX_BPF_PROGS

Default: 512

This is the maximum number of BPF programs (functions) that bpftrace can generate. The main purpose of this limit is to prevent bpftrace from hanging since generating a lot of probes takes a lot of resources (and it should not happen often).

Overview

The bpftrace (bt) language is inspired by the D language used by dtrace and uses the same program structure. Each script consists of an preamble and one or more action blocks.

preamble
actionblock1
actionblock2

Preprocessor and type definitions take place in the preamble:

#include <linux/socket.h>
#define RED "\033[31m"
struct S {


  int x;
}

Each action block consists of three parts:

probe[,probe]
/predicate/ {


  action
}

Probes

Predicate

Action

A basic script that traces the open(2) and openat(2) system calls can be written as follows:

BEGIN
{


        printf("Tracing open syscalls... Hit Ctrl-C to end.\n");
}
tracepoint:syscalls:sys_enter_open,
tracepoint:syscalls:sys_enter_openat
{


        printf("%-6d %-16s %s\n", pid, comm, str(args->filename));
}

This script has two action blocks and a total of 3 probes. The first action block uses the special BEGIN probe, which fires once during bpftrace startup. This probe is used to print a header, indicating that the tracing has started.

The second action block uses two probes, one for open and one for openat, and defines an action that prints the file being open ed as well as the pid and comm of the process that execute the syscall. See the PROBES section for details on the available probe types.

Identifiers

Identifiers must match the following regular expression: [_a-zA-Z][_a-zA-Z0-9]*

Comments

Both single line and multi line comments are supported.

// A single line comment
i:s:1 { // can also be used to comment inline
/*


 a multi line comment
*/


  print(/* inline comment block */ 1);
}

Data Types

The following fundamental integer types are provided by the language.

Floating-point

Floating-point numbers are not supported by BPF and therefore not by bpftrace.

Constants

Integers constants can be defined in the following formats:

Octal constants have to be prefixed with a 0, e.g. 0123. Hexadecimal constants start with either 0x or 0X, e.g. 0x10. Scientific are written in the <m>e<n> format which is a shorthand for m*10^n, e.g. $i = 2e3;. Note that scientific literals are integer only due to the lack of floating point support, 1e-3 is not valid.

To improve the readability of big literals a underscore _ can be used as field separator, e.g. 1_000_123_000.

Integer suffixes as found in the C language are parsed by bpftrace to ensure compatibility with C headers/definitions but they’re not used as size specifiers. 123UL, 123U and 123LL all result in the same integer type with a value of 123.

Character constants can be defined by enclosing the character in single quotes, e.g. $c = 'c';.

String constants can be defined by enclosing the character string in double quotes, e.g. $str = "Hello world";.

Characters and strings support the following escape sequences:

Type conversion

Integer and pointer types can be converted using explicit type conversion with an expression like:

$y = (uint32) $z;
$py = (int16 *) $pz;

Integer casts to a higher rank are sign extended. Conversion to a lower rank is done by zeroing leading bits.

Operators and Expressions

Arithmetic Operators

The following operators are available for integer arithmetic:

Logical Operators

Bitwise Operators

Relational Operators

The following relational operators are defined for integers and pointers.

The following relation operators are available for comparing strings and integer arrays.

Assignment Operators

The following assignment operators can be used on both map and scratch variables:

All these operators are syntactic sugar for combining assignment with the specified operator. @ -= 5 is equal to @ = @ - 5.

Increment and Decrement Operators

The increment (++) and decrement (--) operators can be used on integer and pointer variables to increment their value by one. They can only be used on variables and can either be applied as prefix or suffix. The difference is that the expression x++ returns the original value of x, before it got incremented while ++x returns the value of x post increment. E.g.

$x = 10;
$y = $x--; // y = 10; x = 9
$a = 10;
$b = --$a; // a = 9; b = 9

Note that maps will be implicitly declared and initialized to 0 if not already declared or defined. Scratch variables must be initialized before using these operators.

Variables and Maps

bpftrace knows two types of variables, scratch and map.

'scratch' variables are kept on the BPF stack and only exists during the execution of the action block and cannot be accessed outside of the program. Scratch variable names always start with a $, e.g. $myvar.

'map' variables use BPF 'maps'. These exist for the lifetime of bpftrace itself and can be accessed from all action blocks and user-space. Map names always start with a @, e.g. @mymap.

All valid identifiers can be used as name.

The data type of a variable is automatically determined during first assignment and cannot be changed afterwards.

Associative Arrays

Associative arrays are a collection of elements indexed by a key, similar to the hash tables found in languages like C++ (std::map) and Python (dict). They’re a variant of 'map' variables.

@name[key] = expression
@name[key1,key2] = expression

Just like with any variable the type is determined on first use and cannot be modified afterwards. This applies to both the key(s) and the value type.

The following snippet creates a map with key signature [int64, string[16]] and a value type of int64:

@[pid, comm]++

Variable scoping

Pointers

Pointers in bpftrace are similar to those found in C.

Tuples

bpftrace has support for immutable N-tuples (n > 1). A tuple is a sequence type (like an array) where, unlike an array, every element can have a different type.

Tuples are a comma separated list of expressions, enclosed in brackets, (1,2) Individual fields can be accessed with the . operator. Tuples are zero indexed like arrays are.

i:s:1 {


  $a = (1,2);


  $b = (3,4, $a);


  print($a);


  print($b);


  print($b.0);
}

Prints:

(1, 2)
(3, 4, (1, 2))
3

Arrays

bpftrace supports accessing one-dimensional arrays like those found in C.

Constructing arrays from scratch, like int a[] = {1,2,3} in C, is not supported. They can only be read into a variable from a pointer.

The [] operator is used to access elements.

struct MyStruct {


  int y[4];
}
kprobe:dummy {


  $s = (struct MyStruct *) arg0;


  print($s->y[0]);
}

Structs

C like structs are supported by bpftrace. Fields are accessed with the . operator. Fields of a pointer to a struct can be accessed with the -> operator.

Custom struct can be defined in the preamble

Constructing structs from scratch, like struct X var = {.f1 = 1} in C, is not supported. They can only be read into a variable from a pointer.

struct MyStruct {


  int a;
}
kprobe:dummy {


  $ptr = (struct MyStruct *) arg0;


  $st = *$ptr;


  print($st.a);


  print($ptr->a);
}

Conditionals

Conditional expressions are supported in the form of if/else statements and the ternary operator.

The ternary operator consists of three operands: a condition followed by a ?, the expression to execute when the condition is true followed by a : and the expression to execute if the condition is false.

condition ? ifTrue : ifFalse

Both the ifTrue and ifFalse expressions must be of the same type, mixing types is not allowed.

The ternary operator can be used as part of an assignment.

$a == 1 ? print("true") : print("false");
$b = $a > 0 ? $a : -1;

If/else statements, like the one in C, are supported.

if (condition) {


  ifblock
} else if (condition) {


  if2block
} else {


  elseblock
}

Loops

Since kernel 5.3 BPF supports loops as long as the verifier can prove they’re bounded and fit within the instruction limit.

In bpftrace loops are available through the while statement.

while (condition) {


  block;
}

Within a while-loop the following control flow statements can be used:

i:s:1 {


  $i = 0;


  while ($i <= 100) {


    printf("%d ", $i);


    if ($i > 5) {


      break;


    }


    $i++


  }


  printf("\n");
}

Loop unrolling is also supported with the unroll statement.

unroll(n) {


  block;
}

The compiler will evaluate the block n times and generate the BPF code for the block n times. As this happens at compile time n must be a constant greater than 0 (n > 0).

The following two probes compile into the same code:

i:s:1 {


  unroll(3) {


    print("Unrolled")


  }
}
i:s:1 {


  print("Unrolled")


  print("Unrolled")


  print("Unrolled")
}

avg

variants

Calculate the running average of n between consecutive calls.

i:s:1 {


  @x++;


  @y = avg(@x);


  print(@x);


  print(@y);
}

Internally this keeps two values in the map: value count and running total. The average is computed in user-space when printing by dividing the total by the count.

clear

variants

async

Clear all keys/values from map m.

i:ms:100 {


  @[rand % 10] = count();
}
i:s:10 {


  print(@);


  clear(@);
}

count

variants

Count how often this function is called.

Using @=count() is conceptually similar to @++. The difference is that the count() function uses a map type optimized for this (PER_CPU), increasing performance. Due to this the map cannot be accessed as a regular integer.

i:ms:100 {


  @ = count();
}
i:s:10 {


  print(@);


  clear(@);
}

delete

variants

Delete a single key from a map. For a single value map this deletes the only element. For an associative-array the key to delete has to be specified.

k:dummy {


  @scalar = 1;


  @associative[1,2] = 1;


  delete(@scalar);


  delete(@associative[1,2]);


  delete(@associative); // error
}

hist

variants

Create a log2 histogram of n.

kretprobe:vfs_read {


  @bytes = hist(retval);
}

Results in:

@:
[1M, 2M)               3 |                                                    |
[2M, 4M)               2 |                                                    |
[4M, 8M)               2 |                                                    |
[8M, 16M)              6 |                                                    |
[16M, 32M)            16 |                                                    |
[32M, 64M)            27 |                                                    |
[64M, 128M)           48 |@                                                   |
[128M, 256M)          98 |@@@                                                 |
[256M, 512M)         191 |@@@@@@                                              |
[512M, 1G)           394 |@@@@@@@@@@@@@                                       |
[1G, 2G)             820 |@@@@@@@@@@@@@@@@@@@@@@@@@@@                         |

lhist

variants

Create a linear histogram of n. lhist creates M ((max - min) / step) buckets in the range [min,max) where each bucket is step in size. Values in the range (-inf, min) and (max, inf) get their get their own bucket too, bringing the total amount of buckets created to M+2.

i:ms:1 {


  @ = lhist(rand %10, 0, 10, 1);
}
i:s:5 {


  exit();
}

Prints:

@:
[0, 1)               306 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         |
[1, 2)               284 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@            |
[2, 3)               294 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@          |
[3, 4)               318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@       |
[4, 5)               311 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@        |
[5, 6)               362 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[6, 7)               336 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    |
[7, 8)               326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      |
[8, 9)               328 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@     |
[9, 10)              318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@       |

max

variants

Update the map with n if n is bigger than the current value held.

min

variants

Update the map with n if n is smaller than the current value held.

stats

variants

stats combines the count, avg and sum calls into one.

kprobe:vfs_read {


  @bytes[comm] = stats(arg2);
}

@bytes[bash]: count 7, average 1, total 7
@bytes[sleep]: count 5, average 832, total 4160
@bytes[ls]: count 7, average 886, total 6208
@

sum

variants

Calculate the sum of all n passed.

zero

variants

async

Set all values for all keys to zero.

bswap

variants

bswap reverses the order of the bytes in integer n. In case of 8 bit integers, n is returned without being modified. The return type is an unsigned integer of the same width as n.

buf

variants

buf reads length amount of bytes from address data. The maximum value of length is limited to the BPFTRACE_STRLEN variable. For arrays the length is optional, it is automatically inferred from the signature.

buf is address space aware and will call the correct helper based on the address space associated with data.

The buf_t object returned by buf can safely be printed as a hex encoded string with the %r format specifier.

Bytes with values >=32 and <=126 are printed using their ASCII character, other bytes are printed in hex form (e.g. \x00). The %rx format specifier can be used to print everything in hex form, including ASCII characters. The similar %rh format specifier prints everything in hex form without \x and with spaces between bytes (e.g. 0a fe).

i:s:1 {


  printf("%r\n", buf(kaddr("avenrun"), 8));
}

\x00\x03\x00\x00\x00\x00\x00\x00
\xc2\x02\x00\x00\x00\x00\x00\x00

cat

variants

async

Dump the contents of the named file to stdout. cat supports the same format string and arguments that printf does. If the file cannot be opened or read an error is printed to stderr.

t:syscalls:sys_enter_execve {


  cat("/proc/%d/maps", pid);
}

55f683ebd000-55f683ec1000 r--p 00000000 08:01 1843399                    /usr/bin/ls
55f683ec1000-55f683ed6000 r-xp 00004000 08:01 1843399                    /usr/bin/ls
55f683ed6000-55f683edf000 r--p 00019000 08:01 1843399                    /usr/bin/ls
55f683edf000-55f683ee2000 rw-p 00021000 08:01 1843399                    /usr/bin/ls
55f683ee2000-55f683ee3000 rw-p 00000000 00:00 0

cgroup_path

variants

Convert cgroup id to cgroup path. This is done asynchronously in userspace when the cgroup_path value is printed, therefore it can resolve to a different value if the cgroup id gets reassigned. This also means that the returned value can only be used for printing.

A string literal may be passed as an optional second argument to filter cgroup hierarchies in which the cgroup id is looked up by a wildcard expression (cgroup2 is always represented by "unified", regardless of where it is mounted).

The currently mounted hierarchy at /sys/fs/cgroup is used to do the lookup. If the cgroup with the given id isn’t present here (e.g. when running in a Docker container), the cgroup path won’t be found (unlike when looking up the cgroup path of a process via /proc/.../cgroup).

BEGIN {


  $cgroup_path = cgroup_path(3436);


  print($cgroup_path);


  print($cgroup_path); /* This may print a different path */


  printf("%s %s", $cgroup_path, $cgroup_path); /* This may print two different paths */
}

cgroupid

variants

compile time

cgroupid retrieves the cgroupv2 ID of the cgroup available at path.

BEGIN {


  print(cgroupid("/sys/fs/cgroup/system.slice"));
}

exit

variants

async

Terminate bpftrace, as if a SIGTERM was received. The END probe will still trigger (if specified) and maps will be printed.

join

variants

async

join joins all the string array arr with sep as separator into one string. This string will be printed to stdout directly, it cannot be used as string value.

The concatenation of the array members is done in BPF and the printing happens in userspace.

tracepoint:syscalls:sys_enter_execve {


  join(args->argv);
}

kaddr

variants

compile time

Get the address of the kernel symbol name.

The following script:

kptr

variants

Marks ptr as a kernel address space pointer. See the address-spaces section for more information on address-spaces. The pointer type is left unchanged.

ksym

variants

async

Retrieve the name of the function that contains address addr. The address to name mapping happens in user-space.

The ksym_t type can be printed with the %s format specifier.

kprobe:do_nanosleep
{


  printf("%s\n", ksym(reg("ip")));
}

Prints:

do_nanosleep

macaddr

variants

Create a buffer that holds a macaddress as read from mac This buffer can be printed in the canonical string format using the %s format specifier.

kprobe:arp_create {


  printf("SRC %s, DST %s\n", macaddr(sarg0), macaddr(sarg1));
}

Prints:

SRC 18:C0:4D:08:2E:BB, DST 74:83:C2:7F:8C:FF

ntop

variants

ntop returns the string representation of an IPv4 or IPv6 address. ntop will infer the address type (IPv4 or IPv6) based on the addr type and size. If an integer or char[4] is given, ntop assumes IPv4, if a char[16] is given, ntop assumes IPv6. You can also pass the address type (e.g. AF_INET) explicitly as the first parameter.

pton

variants

compile time

pton converts a text representation of an IPv4 or IPv6 address to byte array. pton infers the address family based on . or : in the given argument. pton comes in handy when we need to select packets with certain IP addresses.

override

variants

unsafe

Kernel 4.16

Helper bpf_override

Supported probes

When using override the probed function will not be executed and instead rc will be returned.

k:__x64_sys_getuid
/comm == "id"/ {


  override(2<<21);
}

uid=4194304 gid=0(root) euid=0(root) groups=0(root)

This feature only works on kernels compiled with CONFIG_BPF_KPROBE_OVERRIDE and only works on functions tagged ALLOW_ERROR_INJECTION.

bpftrace does not test whether error injection is allowed for the probed function, instead if will fail to load the program into the kernel:

ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
Error attaching probe: 'kprobe:vfs_read'

reg

variants

Supported probes

Get the contents of the register identified by name. Valid names depend on the CPU architecture.

signal

variants

unsafe

Kernel 5.3

Helper bpf_send_signal

Probe types: k(ret)probe, u(ret)probe, USDT, profile

Send a signal to the process being traced. The signal can either be identified by name, e.g. SIGSTOP or by ID, e.g. 19 as found in kill -l.

kprobe:__x64_sys_execve
/comm == "bash"/ {


  signal(5);
}

$ ls
Trace/breakpoint trap (core dumped)

sizeof

variants

compile time

Returns size of the argument in bytes. Similar to C/C++ sizeof operator. Note that the expression does not get evaluated.

str

variants

Helper probe_read_str, probe_read_{kernel,user}_str

str reads a NULL terminated (\0) string from data. The maximum string length is limited by the BPFTRACE_STR_LEN env variable, unless length is specified and shorter than the maximum. In case the string is longer than the specified length only length - 1 bytes are copied and a NULL byte is appended at the end.

When available (starting from kernel 5.5, see the --info flag) bpftrace will automatically use the kernel or user variant of probe_read_{kernel,user}_str based on the address space of data, see ADDRESS-SPACES for more information.

strerror

variants

Convert errno code to string. This is done asynchronously in userspace when the strerror value is printed, hence the returned value can only be used for printing.

#include <errno.h>
BEGIN {


  print(strerror(EPERM));
}

strftime

variants

async

Format the nanoseconds since boot timestamp timestamp_ns according to the format specified by fmt. The time conversion and formatting happens in user space, therefore the timestr_t value returned can only be used for printing using the %s format specifier.

bpftrace uses the strftime(3) function for formatting time and supports the same format specifiers.

i:s:1 {


  printf("%s\n", strftime("%H:%M:%S", nsecs));
}

bpftrace also supports the following format string extensions:

strncmp

variants

strncmp compares up to n characters string s1 and string s2. If they’re equal 0 is returned, else a non-zero value is returned.

bpftrace doesn’t read past the length of the shortest string.

The use of the == and != operators is recommended over calling strncmp directly.

strcontains

variants

strcontains compares whether the string haystack contains the string needle. If needle is contained 1 is returned, else zero is returned.

bpftrace doesn’t read past the length of the shortest string.

system

variants

unsafe async

system lets bpftrace run the specified command (fork and exec) until it completes and print its stdout. The command is run with the same privileges as bpftrace and it blocks execution of the processing threads which can lead to missed events and delays processing of async events.

i:s:1 {


  time("%H:%M:%S: ");


  printf("%d\n", @++);
}
i:s:10 {


  system("/bin/sleep 10");
}
i:s:30 {


  exit();
}

Note how the async time and printf first print every second until the i:s:10 probe hits, then they print every 10 seconds due to bpftrace blocking on sleep.

Attaching 3 probes...
08:50:37: 0
08:50:38: 1
08:50:39: 2
08:50:40: 3
08:50:41: 4
08:50:42: 5
08:50:43: 6
08:50:44: 7
08:50:45: 8
08:50:46: 9
08:50:56: 10
08:50:56: 11
08:50:56: 12
08:50:56: 13
08:50:56: 14
08:50:56: 15
08:50:56: 16
08:50:56: 17
08:50:56: 18
08:50:56: 19

system supports the same format string and arguments that printf does.

t:syscalls:sys_enter_execve {


  system("/bin/grep %s /proc/%d/status", "vmswap", pid);
}

time

variants

async

Format the current wall time according to the format specifier fmt and print it to stdout. Unlike strftime() time() doesn’t send a timestamp from the probe, instead it is the time at which user-space processes the event.

bpftrace uses the strftime(3) function for formatting time and supports the same format specifiers.

uaddr

variants

Supported probes

Does not work with ASLR, see issue #75 <https://github.com/iovisor/bpftrace/issues/75>

The uaddr function returns the address of the specified symbol. This lookup happens during program compilation and cannot be used dynamically.

The default return type is uint64*. If the ELF object size matches a known integer size (1, 2, 4 or 8 bytes) the return type is modified to match the width (uint8*, uint16*, uint32* or uint64* resp.). As ELF does not contain type info the type is always assumed to be unsigned.

uprobe:/bin/bash:readline {


  printf("PS1: %s\n", str(*uaddr("ps1_prompt")));
}

uptr

variants

Marks ptr as a user address space pointer. See the address-spaces section for more information on address-spaces. The pointer type is left unchanged.

usym

variants

async

Supported probes

Equal to ksym but resolves user space symbols

uprobe:/bin/bash:readline
{


  printf("%s\n", usym(reg("ip")));
}

Prints:

readline

path

variants

Kernel 5.10

Helper bpf_d_path

Return full path referenced by struct path pointer in argument.

This function can only be used by functions that are allowed to, these functions are contained in the btf_allowlist_d_path set in the kernel.

unwatch

variants

async

Removes a watchpoint

skboutput

variants

Kernel 5.5

Helper bpf_skb_output

Write sk_buff skb 's data section to a PCAP file in the path, starting from offset to offset + length.

The PCAP file is encapsulated in RAW IP, so no ethernet header is included. The data section in the struct skb may contain ethernet header in some kernel contexts, you may set offset to 14 bytes to exclude ethernet header.

Each packet’s timestamp is determined by adding nsecs and boot time, the accuracy varies on different kernels, see nsecs.

This function returns 0 on success, or a negative error in case of failure.

Environment variable BPFTRACE_PERF_RB_PAGES should be increased in order to capture large packets, or else these packets will be dropped.

Usage

# cat dump.bt
kfunc:napi_gro_receive {


  $ret = skboutput("receive.pcap", args->skb, args->skb->len, 0);
}
kfunc:dev_queue_xmit {


  // setting offset to 14, to exclude ethernet header


  $ret = skboutput("output.pcap", args->skb, args->skb->len, 14);


  printf("skboutput returns %d\n", $ret);
}
# export BPFTRACE_PERF_RB_PAGES=1024
# bpftrace dump.bt
...
# tcpdump -n -r ./receive.pcap  | head -3
reading from file ./receive.pcap, link-type RAW (Raw IP)
dropped privs to tcpdump
10:23:44.674087 IP 22.128.74.231.63175 > 192.168.0.23.22: Flags [.], ack 3513221061, win 14009, options [nop,nop,TS val 721277750 ecr 3115333619], length 0
10:23:45.823194 IP 100.101.2.146.53 > 192.168.0.23.46619: 17273 0/1/0 (130)
10:23:45.823229 IP 100.101.2.146.53 > 192.168.0.23.46158: 45799 1/0/0 A 100.100.45.106 (60)

print

variants

async

variants

print prints a the value, which can be a map or a scalar value, with the default formatting for the type.

i:ms:10 { @=hist(rand); }
i:s:1 {


  print(@);


  print(123);


  print("abc");


  exit();
}

Prints:

@:
[16M, 32M)             3 |@@@                                                 |
[32M, 64M)             2 |@@                                                  |
[64M, 128M)            1 |@                                                   |
[128M, 256M)           4 |@@@@                                                |
[256M, 512M)           3 |@@@                                                 |
[512M, 1G)            14 |@@@@@@@@@@@@@@                                      |
[1G, 2G)              22 |@@@@@@@@@@@@@@@@@@@@@@                              |
[2G, 4G)              51 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
123
abc

Note that maps are printed by reference while scalar values are copied. This means that updating and printing maps in a fast loop will likely result in bogus map values as the map will be updated before userspace gets the time to dump and print it.

The printing of maps supports the optional top and div arguments. top limits the printing to the top N entries with the highest integer values

BEGIN {


  $i = 11;


  while($i) {


    @[$i] = --$i;


  }


  print(@, 2);


  clear(@);


  exit()
}

@[9]: 9
@[10]: 10

The div argument scales the values prior to printing them. Scaling values before storing them can result in rounding errors. Consider the following program:

k:f {


  @[func] += arg0/10;
}

With the following sequence as numbers for arg0: 134, 377, 111, 99. The total is 721 which rounds to 72 when scaled by 10 but the program would print 70 due to the rounding of individual values.

Changing the print call to print(@, 5, 2) will take the top 5 values and scale them by 2:

@[6]: 3
@[7]: 3
@[8]: 4
@[9]: 4
@[10]: 5

printf

variants

async

printf() formats and prints data. It behaves similar to printf() found in C and many other languages.

The format string has to be a constant, it cannot be modified at runtime. The formatting of the string happens in user space. Values are copied and passed by value.

bpftrace supports all the typical format specifiers like %llx and %hhu. The non-standard ones can be found in the table below:

Supported escape sequences

Colors are supported too, using standard terminal escape sequences:

print("\033[31mRed\t\033[33mYellow\033[0m\n")

BEGIN and END

These are special built-in events provided by the bpftrace runtime. BEGIN is triggered before all other probes are attached. END is triggered after all other probes are detached.

Note that specifying an END probe doesn’t override the printing of 'non-empty' maps at exit. To prevent the printing all used maps need be cleared, which can be done in the END probe:

END {


    clear(@map1);


    clear(@map2);
}

hardware

variants

shortname

The hardware probe attaches to pre-defined hardware events provided by the kernel.

They are implemented using performance monitoring counters (PMCs): hardware resources on the processor. There are about ten of these, and they are documented in the perf_event_open(2) man page. The event names are:

The count option specifies how many events must happen before the probe fires. If count is left unspecified a default value is used.

hardware:cache-misses:1e6 { @[pid] = count(); }

interval

variants

shortnames

The interval probe fires at a fixed interval as specified by its time spec. Interval fire on one CPU at the time, unlike [profile] probes.

iterator

variants

shortnames

These are eBPF iterator probes, that allow iteration over kernel objects.

Iterator probe can’t be mixed with any other probe, not even other iterator.

Each iterator probe provides set of fields that could be accessed with ctx pointer. User can display set of available fields for iterator via -lv options as described below.

Examples:

# bpftrace -e 'iter:task { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }'
Attaching 1 probe...
systemd:1
kthreadd:2
rcu_gp:3
rcu_par_gp:4
kworker/0:0H:6
mm_percpu_wq:8
...
# bpftrace -e 'iter:task_file { printf("%s:%d %d:%s\n", ctx->task->comm, ctx->task->pid, ctx->fd, path(ctx->file->f_path)); }'
Attaching 1 probe...
systemd:1 1:/dev/null
systemd:1 2:/dev/null
systemd:1 3:/dev/kmsg
...
su:1622 1:/dev/pts/1
su:1622 2:/dev/pts/1
su:1622 3:/var/lib/sss/mc/passwd
...
bpftrace:1892 1:pipe:[35124]
bpftrace:1892 2:/dev/pts/1
bpftrace:1892 3:anon_inode:bpf-map
bpftrace:1892 4:anon_inode:bpf-map
bpftrace:1892 5:anon_inode:bpf_link
bpftrace:1892 6:anon_inode:bpf-prog
bpftrace:1892 7:anon_inode:bpf_iter

It’s possible to pin iterator with specifying optional probe ':pin' part, that defines the pin file. It can be specified as absolute path or relative to /sys/fs/bpf.

relative pin

# bpftrace -e 'iter:task:list { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }'
Program pinned to /sys/fs/bpf/list

# cat /sys/fs/bpf/list
systemd:1
kthreadd:2
rcu_gp:3
rcu_par_gp:4
kworker/0:0H:6
mm_percpu_wq:8
rcu_tasks_kthre:9
...

Examples with absolute pin file:

absolute pin

# bpftrace -e '
iter:task_file:/sys/fs/bpf/files {


  printf("%s:%d %s\n", ctx->task->comm, ctx->task->pid, path(ctx->file->f_path));
}'
Program pinned to /sys/fs/bpf/files

# cat /sys/fs/bpf/files
systemd:1 anon_inode:inotify
systemd:1 anon_inode:[timerfd]
...
systemd-journal:849 /dev/kmsg
systemd-journal:849 anon_inode:[eventpoll]
...
sssd:1146 /var/log/sssd/sssd.log
sssd:1146 anon_inode:[eventpoll]
...
NetworkManager:1155 anon_inode:[eventfd]
NetworkManager:1155 /var/lib/sss/mc/passwd (deleted)

kfunc and kretfunc

variants

shortnames

requires (--info)

kfuncs attach to kernel function similar to kprobe and kretprobe. They make use of eBPF trampolines which allows kernel code to call into BPF programs with near zero overhead.

kfunc s make use of BTF type information to derive the type of function arguments at compile time. This removes the need for manual type casting and makes the code more resilient against small signature changes in the kernel. The function arguments are available in the args struct which can be inspected by doing verbose listing (see LISTING PROBES). These arguments are also available in the return probe (kretfunc).

# bpftrace -lv 'kfunc:tcp_reset'
kfunc:tcp_reset


    struct sock * sk


    struct sk_buff * skb

kfunc:x86_pmu_stop {


  printf("pmu %s stop\n", str(args->event->pmu->name));
}

kretfunc:fget {


  printf("fd %d name %s\n", args->fd, str(retval->f_path.dentry->d_name.name));
}

fd 3 name ld.so.cache
fd 3 name libselinux.so.1
fd 3 name libselinux.so.1
...

kfunc:kvm:x86_emulate_insn { @ = count(); }

@ = 347603

kprobe and kretprobe

variants

shortnames

kprobe s allow for dynamic instrumentation of kernel functions. Each time the specified kernel function is executed the attached BPF programs are ran.

kprobe:tcp_reset {


  @tcp_resets = count()
}

Function arguments are available through the argX and sargX builtins, for register args and stack args respectively. Whether arguments passed on stack or in a register depends on the architecture and the number or arguments in used, e.g. on x86_64 the first non-floating point 6 arguments are passed in registers, all following arguments are passed on the stack. Note that floating point arguments are typically passed in special registers which don’t count as argX arguments which can cause confusion. Consider a function with the following signature:

void func(int a, double d, int x)

Due to d being a floating point x is accessed through arg1 where one might expect arg2.

bpftrace does not detect the function signature so it is not aware of the argument count or their type. It is up to the user to perform Type conversion when needed, e.g.

kprobe:tcp_connect
{


  $sk = ((struct sock *) arg0);


  ...
}

kprobe s are not limited to function entry, they can be attached to any instruction in a function by specifying an offset from the start of the function.

kretprobe s trigger on the return from a kernel function. Return probes do not have access to the function (input) arguments, only to the return value (through retval). A common pattern to work around this is by storing the arguments in a map on function entry and retrieving in the return probe:

kprobe:d_lookup
{


        $name = (struct qstr *)arg1;


        @fname[tid] = $name->name;
}
kretprobe:d_lookup
/@fname[tid]/
{


        printf("%-8d %-6d %-16s M %s\n", elapsed / 1e6, pid, comm,


            str(@fname[tid]));
}

profile

variants

shortnames

Profile probes fire on each CPU on the specified interval.

software

variants

shortnames

The software probe attaches to pre-defined software events provided by the kernel. Event details can be found in the perf_event_open(2) man page.

The event names are:

tracepoint

variants

shortnames

Tracepoints are hooks into events in the kernel. Tracepoints are defined in the kernel source and compiled into the kernel binary which makes them a form of static tracing. Which means that unlike kprobe s new tracepoints cannot be added without modifying the kernel.

The advantage of tracepoints is that they generally provide a more stable interface than kprobe s do, they do not depend on the existence of a kernel function.

Tracepoint arguments are available in the args struct which can be inspected with verbose listing, see the LISTING PROBES section for more details.

tracepoint:syscalls:sys_enter_openat {


  printf("%s %s\n", comm, str(args->filename));
}

irqbalance /proc/interrupts
irqbalance /proc/stat
snmpd /proc/diskstats
snmpd /proc/stat
snmpd /proc/vmstat
snmpd /proc/net/dev
[...]

Additional information

uprobe, uretprobe

variants

shortnames

uprobe s or user-space probes are the user-space equivalent of kprobe s. The same limitations that apply kprobe and kretprobe also apply to uprobe s and uretprobe s.

When tracing libraries, it is sufficient to specify the library name instead of a full path. The path will be then automatically resolved using /etc/ld.so.cache:

# bpftrace -e 'uprobe:libc:malloc { printf("Allocated %d bytes\n", arg0); }'
Allocated 4 bytes
...

If the traced binary has DWARF included, function arguments are available in the args struct which can be inspected with verbose listing, see the LISTING PROBES section for more details.

It is important to note that for uretprobe s to work the kernel runs a special helper on user-space function entry which overrides the return address on the stack. This can cause issues with languages that have their own runtime like Golang:

example.go

func myprint(s string) {


  fmt.Printf("Input: %s\n", s)
}
func main() {


  ss := []string{"a", "b", "c"}


  for _, s := range ss {


    go myprint(s)


  }


  time.Sleep(1*time.Second)
}

bpftrace

# bpftrace -e 'uretprobe:./test:main.myprint { @=count(); }' -c ./test
runtime: unexpected return pc for main.myprint called from 0x7fffffffe000
stack: frame={sp:0xc00008cf60, fp:0xc00008cfd0} stack=[0xc00008c000,0xc00008d000)
fatal error: unknown caller pc

usdt

variants

shortnames

watchpoint and asyncwatchpoint

variants

shortnames

These are memory watchpoints provided by the kernel. Whenever a memory address is written to (w), read from (r), or executed (x), the kernel can generate an event.

In the first form, an absolute address is monitored. If a pid (-p) or a command (-c) is provided, bpftrace takes the address as a userspace address and monitors the appropriate process. If not, bpftrace takes the address as a kernel space address.

In the second form, the address present in argN when function is entered is monitored. A pid or command must be provided for this form. If synchronous (watchpoint), a SIGSTOP is sent to the tracee upon function entry. The tracee will be SIGCONTed after the watchpoint is attached. This is to ensure events are not missed. If you want to avoid the SIGSTOP + SIGCONT use asyncwatchpoint.

Note that on most architectures you may not monitor for execution while monitoring read or write.

Examples

Print hit when a read from or write to 0x10000000 happens:

# bpftrace -e 'watchpoint:0x10000000:8:rw { printf("hit!\n"); exit(); }' -c ./testprogs/watchpoint

Print the call stack every time the jiffies variable is updated:

# bpftrace -e "watchpoint:0x$(awk '$3 == "jiffies" {print $1}' /proc/kallsyms):8:w {


  @[kstack] = count();
}
i:s:1 { exit(); }"
......
@[


    do_timer+12


    tick_do_update_jiffies64.part.22+89


    tick_sched_do_timer+103


    tick_sched_timer+39


    __hrtimer_run_queues+256


    hrtimer_interrupt+256


    smp_apic_timer_interrupt+106


    apic_timer_interrupt+15


    cpuidle_enter_state+188


    cpuidle_enter+41


    do_idle+536


    cpu_startup_entry+25


    start_secondary+355


    secondary_startup_64+164
]: 319

"hit" and exit when the memory pointed to by arg1 of increment is written to.

# cat wpfunc.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
__attribute__((noinline))
void increment(__attribute__((unused)) int _, int *i)
{


  (*i)++;
}
int main()
{


  int *i = malloc(sizeof(int));


  while (1)


  {


    increment(0, i);


    (*i)++;


    usleep(1000);


  }
}
# bpftrace -e 'watchpoint:increment+arg1:4:w { printf("hit!\n"); exit() }' -c ./wpfunc