Tutorial: Tracing02 - monitor xdp tracepoints

In this lesson we will show how to attach to and monitor all xdp related tracepoints and some related info to user space stat application.

Table of Contents

• Tracepoints
• Assignments
  • Assignment 1: Monitor all xdp tracepoints
• Alternative solutions
  • bpftrace
  • perf record

Tracepoints

Tracepoints are useful for debugging XDP, especially for XDP_REDIRECT.

To gain performance XDP_REDIRECT does RX-bulking towards destinations, which unfortunately means that XDP-prog doesn’t get errors directly returned through the BPF-helper call bpf_redirect() (or bpf_redirect_map). Instead these errors can be debugged via using the XDP tracepoint available in the kernel.

The bpf library expects the tracepoint eBPF program to be stored in a section with following name:

tracepoint/<sys>/<tracepoint>

where <sys> is the tracepoint subsystem and <tracepoint> is the tracepoint name, which can be done with following construct:

SEC("tracepoint/xdp/xdp_exception")
int trace_xdp_exception(struct xdp_exception_ctx *ctx)

Via the libbpf library open and load the bpf_object the usual way. E.g.

	obj = bpf_object__open_file(cfg->filename, NULL)
	bpf_object__load(obj);

You can then iterate through all the programs and attach every program to the tracepoint:

bpf_object__for_each_program(prog, obj) {
	...
	tp_link = bpf_program__attach_tracepoint(prog, "xdp", tp);
	err = libbpf_get_error(tp_link);
	...
}

for more details please check load_bpf_and_trace_attach function in trace_load_and_stats.c object.

Assignments

Assignment 1: Monitor all xdp tracepoints

$ sudo ./trace_load_and_stats
XDP-event       CPU:to  pps          drop-pps     extra-info
XDP_REDIRECT    total   0            0            Success
XDP_REDIRECT    total   0            0            Error
Exception       0       0            11           XDP_UNKNOWN
Exception       1       0            2            XDP_UNKNOWN
Exception       2       0            36           XDP_UNKNOWN
Exception       3       0            29           XDP_UNKNOWN
Exception       4       0            3            XDP_UNKNOWN
Exception       5       0            8            XDP_UNKNOWN
Exception       total   0            91           XDP_UNKNOWN
cpumap-kthread  total   0            0            0
devmap-xmit     total   0            0            0.00

Alternative solutions

bpftrace

The bpftrace tool is easy to construct an oneliner that can capture and e.g. count the events of a given tracepoint. E.g. attaching to all XDP tracepoints and counting them:

sudo bpftrace -e 'tracepoint:xdp:* { @cnt[probe] = count(); }'
Attaching 12 probes...
^C

@cnt[tracepoint:xdp:mem_connect]: 18
@cnt[tracepoint:xdp:mem_disconnect]: 18
@cnt[tracepoint:xdp:xdp_exception]: 19605
@cnt[tracepoint:xdp:xdp_devmap_xmit]: 1393604
@cnt[tracepoint:xdp:xdp_redirect]: 22292200

To extract the “ERRNO” being return as part of the err parameter, this bpftrace oneliner can be useful:

sudo bpftrace -e \
 'tracepoint:xdp:xdp_redirect*_err {@redir_errno[-args->err] = count();}
  tracepoint:xdp:xdp_devmap_xmit {@devmap_errno[-args->err] = count();}'

perf record

The perf tool also supports recording tracepoints of the box:

perf record -a -e xdp:xdp_redirect_err \
     -e xdp:xdp_redirect_map_err \
     -e xdp:xdp_exception \
     -e xdp:xdp_devmap_xmit