bug: go unwinding stops at systemstack

[felixge]

It seems like the Go unwinder stops when it hits the systemstack frame. This frame usually marks the boundary between the g0 stack associated with the M and the real goroutine stack that called into it.

Steps to reproduce

Run the program below. I used go1.26, but I suspect the problem impacts all recent Go versions.

package main

import (
	"os"
	"runtime"
	"runtime/pprof"
	"time"
)

func main() {
	f, err := os.Create("cpu.pprof")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	pprof.StartCPUProfile(f)
	defer pprof.StopCPUProfile()

	pcs := make([]uintptr, 32)
	deadline := time.After(30 * time.Second)
	for {
		select {
		case <-deadline:
			return
		default:
			runtime.Callers(0, pcs)
		}
	}
}

While running the program, I ran the profiler at version fbc3686a (2026-03-20) and the devfiler at version 35fe854b (2026-03-18).

Expected Behavior

The program is spending most of its time in runtime.Callers, so I would expect the flame graph to show main calling into runtime.Callers. Go's built-in CPU profiler works as expected:

Actual Behavior

The eBPF profiler seems to stop unwinding at the systemstack, so devfiler produces the following flame graph:

Additional Notes

I suspect the problem is caused by the need to walk a stack transition. In theory gopclntab should have all the required metadata for this, so hopefully this is fixable.

If time allows, I'll try to take a shot at this myself. But I figured I should document the problem to begin with.

[florianl]

Stopping stack unwinding at this point is expected. When extracting stack deltas, certain Go internal functions are identified that stop stack unwinding.

opentelemetry-ebpf-profiler/nativeunwind/elfunwindinfo/elfgopclntab.go

Lines 29 to 42 in fbc3686

	// Go runtime functions for which we should not attempt to unwind further
	var goFunctionsStopDelta = map[string]*sdtypes.UnwindInfo{
	"runtime.mstart": &sdtypes.UnwindInfoStop, // topmost for the go runtime main stacks
	"runtime.goexit": &sdtypes.UnwindInfoStop, // return address in all goroutine stacks
	// stack switch functions that would need special handling for further unwinding.
	// See PF-1101.
	"runtime.mcall": &sdtypes.UnwindInfoStop,
	"runtime.systemstack": &sdtypes.UnwindInfoStop,
	// signal return frame
	"runtime.sigreturn": &sdtypes.UnwindInfoSignal,
	"runtime.sigreturn__sigaction": &sdtypes.UnwindInfoSignal,
	}

We usually stop stack unwinding at this point, as these functions make use of assembly and would cause incorrect stack unwinding.

[fabled]

IIRC the missing piece was to get access to current G and/or M (needs TLS access and some go introspection data). The golabels plugin now does that, so sharing that code or generalizing it a bit more should be sufficient to allow implementing this.