How to sort perf events

[lazybetrayer]

I want to sort perf events, like iovisor/gobpf does: https://github.com/iovisor/gobpf/blob/16120a1bf4d4abc1f9cf37fecfb86009a1631b9f/elf/perf.go#L318

There's no way to read event from a specific cpu ring buffer or check if it's empty.
So I have to read events until all cpu have send an event with timestamp > starttime.
But if a cpu didn't send any event for some period, all events will be delayed.
Then I add a timer to force send events every 50ms.
With this timer there are some out-of-order events occasionally.
If I remove this timer there will be no out-of-order-events.
I think this may be related to how perf.Reader process cpu ring buffer: it doesn't process next cpu ring buffer until the current one is emptied.

Please give me some advice how to sort perf events correctly. Thanks.

evch := make(chan Event, 1024)

// another goroutine read events from perf.Reader and send to evch

d := time.Millisecond * 50
ticker := time.NewTicker(d)
var events []Event

bm := bitmap.New(nCPU)

for {
    start := NowNanoseconds()
    ticker.Reset(d)
    bm.Clear()

INNER_LOOP:
    for {
        select {
        case ev, ok := <-evch:
            if !ok {
                return
            }
            events = append(events, ev)
            if ev.ts > start {
                bm.Set(ev.cpu)
            }
            if bm.IsLowestNBitsSet(nCPU) {
                break INNER_LOOP
            }
        case <-ticker.C:
            break INNER_LOOP
        case <-ctx.Done():
            return
        }
    }

    k := len(events)
    if k == 0 {
        continue
    }

    // sort events by timestamp

    for i, ev := range events {
        if ev.ts > start {
            k = i
            break
        }

        // send event to another channel
    }
    events = events[k:]
}

[lmb]

There's no way to read event from a specific cpu ring buffer or check if it's empty.

You can check whether all rings are empty by setting a deadline that is in the past, for example:

reader.SetDeadline(time.Now())

This will make Read return immediately if all rings are empty.

I think this may be related to how perf.Reader process cpu ring buffer: it doesn't process next cpu ring buffer until the current one is emptied.

I'm not sure this is true. Even if Reader did round robin between rings, it might choose a CPU first that has an event with a later "time" than another CPU, no?

My suggestion would be to swap your Ticker for a Timer, and reset that timer whenever you receive a sample. Then the 50ms becomes the max delay you're willing to wait for a batch to complete. That doesn't prevent out of order events either (I'm not sure that's possible), but makes it exceedingly unlikely.

[ti-mo]

The tl;dr is: you can't avoid out-of-order events. Include the value of ktime_get_ns() as one of the values in the struct you send to user space and design the downstream logic in a way that allows ingesting them out-of-order. For example, when ingesting byte/packet counters, reject any events for a given 5-tuple that have timestamps older than the latest update for the flow.

[lazybetrayer]

Hi @lmb @ti-mo, did you see the code in iovisor/gobpf? Can you tell me what's wrong in there code? Thanks.

[lmb]

Sorry, I don't have the time to go read and understand that code. As far as I understand you want to somehow sort your events and I think that is possible with the current API. You just have to write some code which does the sorting on top of Reader and include a timestamp in your event, as Timo has explained.