C types mapping in bpf2go: why are things done a certain way?

[mejedi]

I am a newcomer to cilium/ebpf. I have experience with coding against libbpf in C. Recently, I did a Golang project with small parts in C. Golang app was building on top of a custom C library exposing app-specific interfaces on top of libbpf. It works but developer experience was subpar as one had to be familiar both with cgo and ebpf (2 times as hairy as it needs to be).

Super excited about cilium/ebpf. Coding against ebpf apis in Golang and having C types mapped in Go is so nice and clean.

Playing with bpf2go I had a few wtf moments. Maybe my expectations about C types mapping were formed by cgo. Maybe there are good reasons for things to be done in a certain way. I'm sharing in hopes that this input might be useful to improve the developer experience of cilium/ebpf.

Structure definitions

Unless explicitly requested via -type flag OR being used as a K/V type in a map, structures are emitted inline.

struct foo {
    int gunk;
};

struct bar {
    struct foo a;
    struct foo b;
};

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __type(key, __u32);
    __type(value, struct bar);
} lit SEC(".maps");

maps to

type DpBar struct {
    A struct{ Gunk int32 }
    B struct{ Gunk int32 }
}

It breaks the natural assumption that entities of the same type in C maps to entities of the same type in Go.

Further, is not it natural to assume that a subset of C types as exposed via BTF should all have corresponding Golang types generated?

Sure thing, one can add -type foo to bpf2go invocation and then it appears as expected:

type DpBar struct {
    A DpFoo
    B DpFoo
}

type DpFoo struct{ Gunk int32 }

Having to update -type flags as the code evolves feels somewhat redundant and error-prone.

Enum definitions

Unless we pass -type foo,

enum foo {
    HELLO, WORLD
};

struct bar {
    enum foo a;
};

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __type(key, __u32);
    __type(value, struct bar);
} lit SEC(".maps");

translates into

type DpBar struct{ A uint32 }

while the natural expectation is to get

type DpBar struct{ A DpFoo }

type DpFoo uint32

const (
    DpFooHELLO DpFoo = 0
    DpFooWORLD DpFoo = 1
)

I would also argue that prepending the type name DpFoo to form constant names (e.g. DpFooHELLO) is a questionable design choice as constant names on C side typically already have a unique type name-derived prefix or suffix (as all enum constants share the same namespace).

Unions

Only the first "branch" is emitted. Nested type declarations are not generated, unless the type is used in a map or mentioned in -type flag. It is consistent with type handling elsewhere.

So

struct foo {
    int eee;
};

struct dummy {
    __u64 gunk;
};

union bar {
    struct foo a;
    struct dummy d;
};

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __type(key, __u32);
    __type(value, union bar);
} lit SEC(".maps");

with -type bar results in

type DpBar struct {
    A DpFoo
    _ [4]byte
}

type DpFoo struct{ Eee int32 }

I don't see a way to access the second branch d (besides unsafe typecast).

For reference, cgo maps union U to [sizeof(U)]byte.

Which would've enabled to produce and consume data via serialisation.

[lmb]

Thanks for your ideas :) Let me try to break things down:

Heuristics for selecting which types to generate

It breaks the natural assumption that entities of the same type in C maps to entities of the same type in Go.

This is not a good assumption when writing kprobes for example. Here you'll often pull in vmlinux.h to inspect some data in the kernel, but it's not clear to me that generating the kernel types is appropriate at all. Writing out "all named structs and typedefs" is too simplistic a heuristic I think.

That doesn't mean that the current heuristic is great either: I'd be happy if we could improve it so that less -type was required.

Auto-generate types for enums

This is a special case of the problem you describe above, no? Maybe this is a good candidate to improve the heuristic which types to emit. Maybe something like enums referred to from a generated type are also generated? Key question: should this apply recursively or just the direct "children"?

Enum names

I would also argue that prepending the type name DpFoo to form constant names (e.g. DpFooHELLO) is a questionable design choice as constant names on C side typically already have a unique type name-derived prefix or suffix (as all enum constants share the same namespace).

If I understand you correctly, you'd prefer DpHELLO instead of DpFooHELLO? The problem is that skipping the Foo can lead to name clashes on the Go side because struct and enum have distinct namespaces in C.

struct HELLO {
  // bla
}

enum {
    HELLO = 1,
};

This is valid C that would fail to generate valid Go. The simplest fix to this is using distinct prefixes.

Unions

I don't see a way to access the second branch d (besides unsafe typecast).

Correct. FWIW you can serialise and de-serialise from these types as well. I realise now that there might be no way to access data in the padding though, since we never read or write padding.

For reference, cgo maps union U to [sizeof(U)]byte.

Yeah that might have been the better choice. Unfortunately I wasn't aware of what CGO did!

[mejedi]

Hi @lmb, thank you for looking into this and sharing your perspectives. I'll follow the same structure you've established.

Heuristics for selecting which types to generate

Here you'll often pull in vmlinux.h to inspect some data in the kernel, but it's not clear to me that generating the kernel types is appropriate at all [when doing kprobes].

I doubt that there will be much of the kernel types in BTF. Definitely not the entire vmlinux.h crowd.

Never used gcc for BPF, but clang does a very good job pruning BTF. Types that remain are:

1. global variables, including map definitions,
2. non-eliminated functions,
3. arguments of __builtin_btf_type_id and __builtin_preserve_access_index, and
4. any type reachable¹ by following the type graph from 1, 2, 3.

In the end, we might capture just a few of the kernel types when CO-RE is employed. Is it of a concern if we would've generated Golang types for them by default? We could narrow that down by doing a pass over BTF graph (e.g. only include types needed for VARs).

I'd like to propose the following:

1. Unless -no-global-types is requested, slurp VAR types from BTF. Recognise map definitions so that we don't include the outer structure, but drill down to .key and .value.
2. Add types from -type options.
3. Follow the graph recursively and add types as we visit them.

Enum names

If I understand you correctly, you'd prefer DpHELLO instead of DpFooHELLO? The problem is that skipping the Foo can lead to name clashes on the Go side...

Now that I understand it, I probably like it more than what cgo does. They enforce prefixes on type names, eg. struct Foo becomes struct_Foo.

Unions

For reference, cgo maps union U to [sizeof(U)]byte.

Yeah that might have been the better choice. Unfortunately I wasn't aware of what CGO did!

Well, on the other hand bpf2go supports unnamed unions embedded in structs which cgo doesn't. I think I finally came to appreciate bpf2go way of doing things.

In my case, the union is defined by me, so I can replace

union payload { ... };
struct value {
  int tag;
  union payload val;
}

with

union payload { ... };
struct value {
  int tag;
  union {
    __u8 val_blob[sizeof(union payload)];
    union payload val;
  };
}

to get the desired effect.

---

Footnotes

1. rule 4 above has a quirk: a structure type referenced by a pointer in a designated section is only included if the section name starts with .maps:

$ cat whackamole.c
#include <linux/types.h>
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct foo {
    int frob;
};

struct bar { struct foo *p; };

struct bar bar SEC(".whackamole");
$ clang -g -c -target bpf whackamole.c
$ bpftool btf dump file whackamole.o
[1] STRUCT 'bar' size=8 vlen=1
	'p' type_id=2 bits_offset=0
[2] PTR '(anon)' type_id=5
[3] VAR 'bar' type_id=1, linkage=global
[4] DATASEC '.whackamole' size=0 vlen=1
	type_id=3 offset=0 size=8 (VAR 'bar')
[5] FWD 'foo' fwd_kind=struct

[lmb]

How about we split this into stages?

1. Add VAR types to the set of to-be-emitted types. That seems really sensible to me and shouldn't cause problems.
2. Experiment with traversing the graph from to-be-emitted types. For example, we could add all named types to the to-be-emitted set, or only typedefs, or something completely different. This would at least involve testing the change on some open source b2g users to see what the impact would be.

[jvgutierrez]

I got a third-party defined struct that looks like:

struct hc_real_definition {                                                              
  union {                                                                                
    __be32 daddr;                                                                        
    __be32 v6daddr[4];                                                                   
  };                                                                                     
  __u8 flags;                                                                            
};

and bpf2go yields:

type ipipHcRealDefinition struct {
    Daddr uint32
    _     [12]byte
    Flags uint8
    _     [3]byte
}

as mentioned above, the cgo approach would be definitely useful for cases like the described above

[lmb]

Can you define your own C type and cast to that instead? I realised that is a bit cumbersome, but might be acceptable with a couple of __static_assert thrown in?