Skip to content

signal SIGBUS: bus error code=0x2 at elfgopclntab.go:520 #1316

@lgwozdecki

Description

@lgwozdecki

I'm experiencing that crash from time to time with the crash log as below.
I'm also adding the crash analysis done by AI.

version: v0.0.202610
system: Ubuntu 22

profiler output:

fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x799d9298c5e0 pc=0x78222d]
goroutine 43 gp=0xc00028b340 m=21 mp=0xc000ad7008 [running]:
runtime.throw({0xe586ab?, 0x4201b1?})
#011/usr/local/go/src/runtime/panic.go:1094 +0x48 fp=0xc000c37898 sp=0xc000c37868 pc=0x480208
runtime.sigpanic()
#011/usr/local/go/src/runtime/signal_unix.go:922 +0x10a fp=0xc000c378f8 sp=0xc000c37898 pc=0x481f6a
go.opentelemetry.io/ebpf-profiler/nativeunwind/elfunwindinfo.(*Gopclntab).getFuncMapEntry(...)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/nativeunwind/elfunwindinfo/elfgopclntab.go:520
go.opentelemetry.io/ebpf-profiler/nativeunwind/elfunwindinfo.(*Gopclntab).Symbolize.func1(...)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/nativeunwind/elfunwindinfo/elfgopclntab.go:569
sort.Search(...)
#011/usr/local/go/src/sort/search.go:65
go.opentelemetry.io/ebpf-profiler/nativeunwind/elfunwindinfo.(*Gopclntab).Symbolize(0xc000324808?, 0x799df4a21840?)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/nativeunwind/elfunwindinfo/elfgopclntab.go:568 +0x4d fp=0xc000c37958 sp=0xc000c378f8 pc=0x78222d
go.opentelemetry.io/ebpf-profiler/interpreter/go.(*goInstance).Symbolize(0xc000c37ac8?, {0xc0005e55b0, 0xc000c37ae8?, 0x40cee5?}, 0xc002e4ede0, {{0xc00066e120?}})
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/interpreter/go/go.go:113 +0xea fp=0xc000c37ab8 sp=0xc000c37958 pc=0xb29d4a
go.opentelemetry.io/ebpf-profiler/interpreter.(*MultiInstance).Symbolize(0xd34e80?, {0xc0005e55b0, 0x2, 0xc00}, 0xc002e4ede0, {{0x1?}})
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/interpreter/multi.go:121 +0x86 fp=0xc000c37b20 sp=0xc000c37ab8 pc=0xb05e66
go.opentelemetry.io/ebpf-profiler/processmanager.(*ProcessManager).symbolizeFrame(0xc001d6c0e0, 0x2c6b8, {0xc0005e55b0, 0x2, 0xc00}, 0xc002e4ede0, {{0xc000c37cb8?}})
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/manager.go:199 +0x1c2 fp=0xc000c37c28 sp=0xc000c37b20 pc=0xb67762
go.opentelemetry.io/ebpf-profiler/processmanager.(*ProcessManager).convertFrame(0xc001d6c0e0, 0x2c6b8, {0xc0005e55b0, 0x2, 0xc00}, 0xc002e4ede0)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/manager.go:230 +0xe5 fp=0xc000c37d20 sp=0xc000c37c28 pc=0xb67a85
go.opentelemetry.io/ebpf-profiler/processmanager.(*ProcessManager).HandleTrace(0xc001d6c0e0, 0xc0005e5508)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/manager.go:359 +0x605 fp=0xc000c37f28 sp=0xc000c37d20 pc=0xb68865
go.opentelemetry.io/ebpf-profiler/tracer.(*Tracer).HandleTrace(0xc001d26680, 0xc0005e5508)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/tracer/tracer.go:1341 +0x25 fp=0xc000c37f50 sp=0xc000c37f28 pc=0xb8ed65
go.opentelemetry.io/ebpf-profiler/internal/controller.startTraceHandling.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/internal/controller/controller.go:185 +0xb2 fp=0xc000c37fe0 sp=0xc000c37f50 pc=0xb906b2
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000c37fe8 sp=0xc000c37fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/internal/controller.startTraceHandling in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/internal/controller/controller.go:179 +0xf9
goroutine 1 gp=0xc000002380 m=nil [chan receive, 95 minutes]:
runtime.gopark(0x3ff?, 0x14?, 0x0?, 0x0?, 0x10101?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc001dd4c60 sp=0xc001dd4c40 pc=0x48032e
runtime.chanrecv(0xc00008c230, 0x0, 0x1)
#011/usr/local/go/src/runtime/chan.go:667 +0x44e fp=0xc001dd4cd8 sp=0xc001dd4c60 pc=0x414fee
runtime.chanrecv1(0xc001dbfee8?, 0xfafd58?)
#011/usr/local/go/src/runtime/chan.go:509 +0x12 fp=0xc001dd4d00 sp=0xc001dd4cd8 pc=0x414b72
main.mainWithExitCode()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/main.go:140 +0x94a fp=0xc001dd4f38 sp=0xc001dd4d00 pc=0xbc7caa
main.main()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/main.go:59 +0x13 fp=0xc001dd4f50 sp=0xc001dd4f38 pc=0xbc7333
runtime.main()
#011/usr/local/go/src/runtime/proc.go:285 +0x29d fp=0xc001dd4fe0 sp=0xc001dd4f50 pc=0x44a03d
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001dd4fe8 sp=0xc001dd4fe0 pc=0x488181
goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle), 3 minutes]:
runtime.gopark(0x6d491bb47ee?, 0x0?, 0x0?, 0x0?, 0x0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00005afa8 sp=0xc00005af88 pc=0x48032e
runtime.goparkunlock(...)
#011/usr/local/go/src/runtime/proc.go:466
runtime.forcegchelper()
#011/usr/local/go/src/runtime/proc.go:373 +0xb3 fp=0xc00005afe0 sp=0xc00005afa8 pc=0x44a373
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00005afe8 sp=0xc00005afe0 pc=0x488181
created by runtime.init.7 in goroutine 1
#011/usr/local/go/src/runtime/proc.go:361 +0x1a
goroutine 3 gp=0xc000002e00 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00005b780 sp=0xc00005b760 pc=0x48032e
runtime.goparkunlock(...)
#011/usr/local/go/src/runtime/proc.go:466
runtime.bgsweep(0xc00007a000)
#011/usr/local/go/src/runtime/mgcsweep.go:323 +0xdf fp=0xc00005b7c8 sp=0xc00005b780 pc=0x431b1f
runtime.gcenable.gowrap1()
#011/usr/local/go/src/runtime/mgc.go:212 +0x25 fp=0xc00005b7e0 sp=0xc00005b7c8 pc=0x425b25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00005b7e8 sp=0xc00005b7e0 pc=0x488181
created by runtime.gcenable in goroutine 1
#011/usr/local/go/src/runtime/mgc.go:212 +0x66
goroutine 4 gp=0xc000002fc0 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x164a04?, 0x0?, 0x0?, 0x0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00005bf78 sp=0xc00005bf58 pc=0x48032e
runtime.goparkunlock(...)
#011/usr/local/go/src/runtime/proc.go:466
runtime.(*scavengerState).park(0x170b460)
#011/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00005bfa8 sp=0xc00005bf78 pc=0x42f5a9
runtime.bgscavenge(0xc00007a000)
#011/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00005bfc8 sp=0xc00005bfa8 pc=0x42fb59
runtime.gcenable.gowrap2()
#011/usr/local/go/src/runtime/mgc.go:213 +0x25 fp=0xc00005bfe0 sp=0xc00005bfc8 pc=0x425ac5
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00005bfe8 sp=0xc00005bfe0 pc=0x488181
created by runtime.gcenable in goroutine 1
#011/usr/local/go/src/runtime/mgc.go:213 +0xa5
goroutine 5 gp=0xc000003500 m=nil [GOMAXPROCS updater (idle), 95 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00005c788 sp=0xc00005c768 pc=0x48032e
runtime.goparkunlock(...)
#011/usr/local/go/src/runtime/proc.go:466
runtime.updateMaxProcsGoroutine()
#011/usr/local/go/src/runtime/proc.go:6706 +0xe7 fp=0xc00005c7e0 sp=0xc00005c788 pc=0x458347
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x488181
created by runtime.defaultGOMAXPROCSUpdateEnable in goroutine 1
#011/usr/local/go/src/runtime/proc.go:6694 +0x37
goroutine 6 gp=0xc0000036c0 m=nil [finalizer wait, 95 minutes]:
runtime.gopark(0x459315?, 0x42ea9c?, 0xe0?, 0xe1?, 0x490013?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00005a620 sp=0xc00005a600 pc=0x48032e
runtime.runFinalizers()
#011/usr/local/go/src/runtime/mfinal.go:210 +0x107 fp=0xc00005a7e0 sp=0xc00005a620 pc=0x4249a7
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x488181
created by runtime.createfing in goroutine 1
#011/usr/local/go/src/runtime/mfinal.go:172 +0x3d
goroutine 7 gp=0xc000003880 m=nil [cleanup wait]:
runtime.gopark(0x799da7280800?, 0xc00005cf90?, 0xb5?, 0xa2?, 0x799da7280800?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00005cf68 sp=0xc00005cf48 pc=0x48032e
runtime.goparkunlock(...)
#011/usr/local/go/src/runtime/proc.go:466
runtime.(*cleanupQueue).dequeue(0x170b740)
#011/usr/local/go/src/runtime/mcleanup.go:439 +0xc5 fp=0xc00005cfa0 sp=0xc00005cf68 pc=0x421bc5
runtime.runCleanups()
#011/usr/local/go/src/runtime/mcleanup.go:635 +0x45 fp=0xc00005cfe0 sp=0xc00005cfa0 pc=0x422285
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x488181
created by runtime.(*cleanupQueue).createGs in goroutine 1
#011/usr/local/go/src/runtime/mcleanup.go:589 +0xa5
goroutine 8 gp=0xc000003c00 m=nil [select, 95 minutes, locked to thread]:
runtime.gopark(0xc00005d7a8?, 0x2?, 0x58?, 0x0?, 0xc00005d794?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00005d618 sp=0xc00005d5f8 pc=0x48032e
runtime.selectgo(0xc00005d7a8, 0xc00005d790, 0x0?, 0x0, 0x0?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc00005d758 sp=0xc00005d618 pc=0x45cea5
runtime.ensureSigM.func1()
#011/usr/local/go/src/runtime/signal_unix.go:1085 +0x194 fp=0xc00005d7e0 sp=0xc00005d758 pc=0x47a9b4
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00005d7e8 sp=0xc00005d7e0 pc=0x488181
created by runtime.ensureSigM in goroutine 1
#011/usr/local/go/src/runtime/signal_unix.go:1068 +0xc5
goroutine 9 gp=0xc0002781c0 m=3 mp=0xc000061008 [syscall, 95 minutes]:
runtime.notetsleepg(0x172d620, 0xffffffffffffffff)
#011/usr/local/go/src/runtime/lock_futex.go:123 +0x29 fp=0xc00005dfa0 sp=0xc00005df78 pc=0x41a789
os/signal.signal_recv()
#011/usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc00005dfc0 sp=0xc00005dfa0 pc=0x482269
os/signal.loop()
#011/usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00005dfe0 sp=0xc00005dfc0 pc=0xbc4eb3
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x488181
created by os/signal.Notify.func1.1 in goroutine 1
#011/usr/local/go/src/os/signal/signal.go:152 +0x1f
goroutine 17 gp=0xc00028a000 m=nil [select, 95 minutes]:
runtime.gopark(0xc0000567b0?, 0x2?, 0x0?, 0x0?, 0xc00005679c?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000056628 sp=0xc000056608 pc=0x48032e
runtime.selectgo(0xc0000567b0, 0xc000056798, 0x0?, 0x0, 0x0?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000056768 sp=0xc000056628 pc=0x45cea5
os/signal.NotifyContext.func1()
#011/usr/local/go/src/os/signal/signal.go:289 +0x66 fp=0xc0000567e0 sp=0xc000056768 pc=0xbc49e6
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000567e8 sp=0xc0000567e0 pc=0x488181
created by os/signal.NotifyContext in goroutine 1
#011/usr/local/go/src/os/signal/signal.go:288 +0x158
goroutine 18 gp=0xc00028a1c0 m=nil [select, 3 minutes]:
runtime.gopark(0xc000056f90?, 0x2?, 0x60?, 0xb2?, 0xc000056f6c?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000056df0 sp=0xc000056dd0 pc=0x48032e
runtime.selectgo(0xc000056f90, 0xc000056f68, 0x0?, 0x0, 0x0?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000056f30 sp=0xc000056df0 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/periodiccaller.Start.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:21 +0xe5 fp=0xc000056fe0 sp=0xc000056f30 pc=0x723245
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000056fe8 sp=0xc000056fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/periodiccaller.Start in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:17 +0x99
goroutine 19 gp=0xc00028a380 m=nil [chan receive, 95 minutes]:
runtime.gopark(0x47db45?, 0x10?, 0x80?, 0x49?, 0x799df4ba65c0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc0000576a8 sp=0xc000057688 pc=0x48032e
runtime.chanrecv(0xc00030e3f0, 0xc000057790, 0x1)
#011/usr/local/go/src/runtime/chan.go:667 +0x44e fp=0xc000057720 sp=0xc0000576a8 pc=0x414fee
runtime.chanrecv2(0xfafb50?, 0xc0003180a0?)
#011/usr/local/go/src/runtime/chan.go:514 +0x12 fp=0xc000057748 sp=0xc000057720 pc=0x414b92
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000289060, {0xfafb50, 0xc0003180a0})
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:88 +0xe5 fp=0xc0000577b8 sp=0xc000057748 pc=0x91a1a5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc0000577e0 sp=0xc0000577b8 pc=0x919fa8
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000577e8 sp=0xc0000577e0 pc=0x488181
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:52 +0x11a
goroutine 20 gp=0xc00028a8c0 m=nil [chan receive, 95 minutes]:
runtime.gopark(0xc0002b8180?, 0xb?, 0x0?, 0x0?, 0x0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000094ea8 sp=0xc000094e88 pc=0x48032e
runtime.chanrecv(0xc00030e460, 0xc000333f90, 0x1)
#011/usr/local/go/src/runtime/chan.go:667 +0x44e fp=0xc000094f20 sp=0xc000094ea8 pc=0x414fee
runtime.chanrecv2(0xfafb50?, 0xc0003180f0?)
#011/usr/local/go/src/runtime/chan.go:514 +0x12 fp=0xc000094f48 sp=0xc000094f20 pc=0x414b92
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000289090, {0xfafb50, 0xc0003180f0})
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:88 +0xe5 fp=0xc000094fb8 sp=0xc000094f48 pc=0x91a1a5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc000094fe0 sp=0xc000094fb8 pc=0x919fa8
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000094fe8 sp=0xc000094fe0 pc=0x488181
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:52 +0x11a
goroutine 21 gp=0xc00028aa80 m=nil [chan receive, 95 minutes]:
runtime.gopark(0x918f66?, 0xc0002903c0?, 0x98?, 0x2f?, 0xc0003182d0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00008fea8 sp=0xc00008fe88 pc=0x48032e
runtime.chanrecv(0xc00030e4d0, 0xc00010df90, 0x1)
#011/usr/local/go/src/runtime/chan.go:667 +0x44e fp=0xc00008ff20 sp=0xc00008fea8 pc=0x414fee
runtime.chanrecv2(0xfafb50?, 0xc000318140?)
#011/usr/local/go/src/runtime/chan.go:514 +0x12 fp=0xc00008ff48 sp=0xc00008ff20 pc=0x414b92
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002890c0, {0xfafb50, 0xc000318140})
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:88 +0xe5 fp=0xc00008ffb8 sp=0xc00008ff48 pc=0x91a1a5
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc00008ffe0 sp=0xc00008ffb8 pc=0x919fa8
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x488181
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/grpcsync/callback_serializer.go:52 +0x11a
goroutine 16 gp=0xc00028ac40 m=nil [select]:
runtime.gopark(0xc000c33f60?, 0x4?, 0xa0?, 0xd6?, 0xc000c33f10?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000c33d90 sp=0xc000c33d70 pc=0x48032e
runtime.selectgo(0xc000c33f60, 0xc000c33f08, 0x0?, 0x0, 0x4?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000c33ed0 sp=0xc000c33d90 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/reporter.(*runLoop).Start.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/reporter/runloop.go:28 +0x178 fp=0xc000c33fe0 sp=0xc000c33ed0 pc=0xb03858
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000c33fe8 sp=0xc000c33fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/reporter.(*runLoop).Start in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/reporter/runloop.go:21 +0xca
goroutine 33 gp=0xc000278700 m=nil [chan receive, 95 minutes]:
runtime.gopark(0xc0002e2c38?, 0xc000143680?, 0x48?, 0xc8?, 0x414aa0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc0000586e8 sp=0xc0000586c8 pc=0x48032e
runtime.chanrecv(0xc0002e2150, 0x0, 0x1)
#011/usr/local/go/src/runtime/chan.go:667 +0x44e fp=0xc000058760 sp=0xc0000586e8 pc=0x414fee
runtime.chanrecv1(0xc00008c770?, 0xc000163b30?)
#011/usr/local/go/src/runtime/chan.go:509 +0x12 fp=0xc000058788 sp=0xc000058760 pc=0x414b72
go.opentelemetry.io/ebpf-profiler/reporter.(*OTLPReporter).Start.func3()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/reporter/otlp_reporter.go:103 +0x39 fp=0xc0000587e0 sp=0xc000058788 pc=0xb02519
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000587e8 sp=0xc0000587e0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/reporter.(*OTLPReporter).Start in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/reporter/otlp_reporter.go:102 +0x26b
goroutine 25 gp=0xc00028ae00 m=nil [select]:
runtime.gopark(0xc000057f68?, 0x2?, 0xe0?, 0xe1?, 0xc000057eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000057d68 sp=0xc000057d48 pc=0x48032e
runtime.selectgo(0xc000057f68, 0xc000057ee8, 0xc000293eb0?, 0x0, 0xc001ce0000?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000057ea8 sp=0xc000057d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c1b0)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000057fc8 sp=0xc000057ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000057fe0 sp=0xc000057fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000057fe8 sp=0xc000057fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 14 gp=0xc0002788c0 m=nil [IO wait]:
runtime.gopark(0xc000ac82a0?, 0x0?, 0xe0?, 0x88?, 0xb?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc00008eb20 sp=0xc00008eb00 pc=0x48032e
runtime.netpollblock(0x4d9658?, 0x412fa6?, 0x0?)
#011/usr/local/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc00008eb58 sp=0xc00008eb20 pc=0x442917
internal/poll.runtime_pollWait(0x799df4a1aa00, 0x72)
#011/usr/local/go/src/runtime/netpoll.go:351 +0x85 fp=0xc00008eb78 sp=0xc00008eb58 pc=0x47f505
internal/poll.(*pollDesc).wait(0xc000107480?, 0xc000388000?, 0x0)
#011/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00008eba0 sp=0xc00008eb78 pc=0x4f9387
internal/poll.(*pollDesc).waitRead(...)
#011/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000107480, {0xc000388000, 0x8000, 0x8000})
#011/usr/local/go/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc00008ec38 sp=0xc00008eba0 pc=0x4fa679
net.(*netFD).Read(0xc000107480, {0xc000388000?, 0x799df4ba65c0?, 0x10?})
#011/usr/local/go/src/net/fd_posix.go:68 +0x25 fp=0xc00008ec80 sp=0xc00008ec38 pc=0x6a6705
net.(*conn).Read(0xc00005e4b8, {0xc000388000?, 0x47db65?, 0x18?})
#011/usr/local/go/src/net/net.go:196 +0x45 fp=0xc00008ecc8 sp=0xc00008ec80 pc=0x6b5dc5
bufio.(*Reader).Read(0xc0001437a0, {0xc000000124, 0x9, 0xdc5220?})
#011/usr/local/go/src/bufio/bufio.go:245 +0x197 fp=0xc00008ed00 sp=0xc00008ecc8 pc=0x530617
io.ReadAtLeast({0xfa6600, 0xc0001437a0}, {0xc000000124, 0x9, 0x9}, 0x9)
#011/usr/local/go/src/io/io.go:335 +0x8e fp=0xc00008ed48 sp=0xc00008ed00 pc=0x4cf3ae
io.ReadFull(...)
#011/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000000124, 0x9, 0xc000000000?}, {0xfa6600?, 0xc0001437a0?})
#011/agent/go/pkg/mod/golang.org/x/net@v0.51.0/http2/frame.go:250 +0x65 fp=0xc00008ed98 sp=0xc00008ed48 pc=0xa3d445
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0000000e0)
#011/agent/go/pkg/mod/golang.org/x/net@v0.51.0/http2/frame.go:513 +0x6b fp=0xc00008ee20 sp=0xc00008ed98 pc=0xa3db0b
google.golang.org/grpc/internal/transport.(*framer).readFrame(0xc000107500)
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/transport/http_util.go:480 +0x45 fp=0xc00008ee60 sp=0xc00008ee20 pc=0xa6ff25
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0000fa908, 0xc000266d90)
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/transport/http2_client.go:1658 +0x1c5 fp=0xc00008efc0 sp=0xc00008ee60 pc=0xa62945
google.golang.org/grpc/internal/transport.NewHTTP2Client.gowrap4()
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/transport/http2_client.go:411 +0x25 fp=0xc00008efe0 sp=0xc00008efc0 pc=0xa58aa5
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x488181
created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 11
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/transport/http2_client.go:411 +0x1e53
goroutine 15 gp=0xc000278a80 m=nil [select]:
runtime.gopark(0xc001dd6ed8?, 0x2?, 0x80?, 0xb5?, 0xc001dd6eac?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc001dd6d38 sp=0xc001dd6d18 pc=0x48032e
runtime.selectgo(0xc001dd6ed8, 0xc001dd6ea8, 0xa6f4d6?, 0x0, 0xc000390000?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc001dd6e78 sp=0xc001dd6d38 pc=0x45cea5
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00008bf00, 0x1)
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/transport/controlbuf.go:420 +0x108 fp=0xc001dd6f08 sp=0xc001dd6e78 pc=0xa4e748
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0002ba5a0)
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/transport/controlbuf.go:597 +0x78 fp=0xc001dd6f68 sp=0xc001dd6f08 pc=0xa4ef18
google.golang.org/grpc/internal/transport.NewHTTP2Client.func6()
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/transport/http2_client.go:469 +0xd2 fp=0xc001dd6fe0 sp=0xc001dd6f68 pc=0xa58992
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001dd6fe8 sp=0xc001dd6fe0 pc=0x488181
created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 11
#011/agent/go/pkg/mod/google.golang.org/grpc@v1.79.1/internal/transport/http2_client.go:467 +0x244b
goroutine 34 gp=0xc000278c40 m=nil [GC worker (idle)]:
runtime.gopark(0x6e5ad300ba0?, 0x3?, 0xaf?, 0xb8?, 0x0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000093f38 sp=0xc000093f18 pc=0x48032e
runtime.gcBgMarkWorker(0xc00008ccb0)
#011/usr/local/go/src/runtime/mgc.go:1463 +0xeb fp=0xc000093fc8 sp=0xc000093f38 pc=0x42820b
runtime.gcBgMarkStartWorkers.gowrap1()
#011/usr/local/go/src/runtime/mgc.go:1373 +0x25 fp=0xc000093fe0 sp=0xc000093fc8 pc=0x4280e5
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x488181
created by runtime.gcBgMarkStartWorkers in goroutine 1
#011/usr/local/go/src/runtime/mgc.go:1373 +0x105
goroutine 35 gp=0xc000278e00 m=nil [GC worker (idle)]:
runtime.gopark(0x6e5ad24ad31?, 0x3?, 0xc4?, 0x43?, 0x0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc001d58738 sp=0xc001d58718 pc=0x48032e
runtime.gcBgMarkWorker(0xc00008ccb0)
#011/usr/local/go/src/runtime/mgc.go:1463 +0xeb fp=0xc001d587c8 sp=0xc001d58738 pc=0x42820b
runtime.gcBgMarkStartWorkers.gowrap1()
#011/usr/local/go/src/runtime/mgc.go:1373 +0x25 fp=0xc001d587e0 sp=0xc001d587c8 pc=0x4280e5
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001d587e8 sp=0xc001d587e0 pc=0x488181
created by runtime.gcBgMarkStartWorkers in goroutine 1
#011/usr/local/go/src/runtime/mgc.go:1373 +0x105
goroutine 26 gp=0xc001d50000 m=nil [select, 1 minutes]:
runtime.gopark(0xc000383f68?, 0x2?, 0x20?, 0x0?, 0xc000383eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000383d68 sp=0xc000383d48 pc=0x48032e
runtime.selectgo(0xc000383f68, 0xc000383ee8, 0xc002f422d8?, 0x0, 0xc001d3171c?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000383ea8 sp=0xc000383d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c1c8)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000383fc8 sp=0xc000383ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000383fe0 sp=0xc000383fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000383fe8 sp=0xc000383fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 27 gp=0xc001d501c0 m=nil [select, 1 minutes]:
runtime.gopark(0xc000387768?, 0x2?, 0x20?, 0x0?, 0xc0003876ec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000387568 sp=0xc000387548 pc=0x48032e
runtime.selectgo(0xc000387768, 0xc0003876e8, 0xc000576f38?, 0x0, 0xc00027fb50?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc0003876a8 sp=0xc000387568 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c1e0)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc0003877c8 sp=0xc0003876a8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc0003877e0 sp=0xc0003877c8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0003877e8 sp=0xc0003877e0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 28 gp=0xc001d50380 m=nil [select, 1 minutes]:
runtime.gopark(0xc000387f68?, 0x2?, 0x20?, 0x0?, 0xc000387eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000387d68 sp=0xc000387d48 pc=0x48032e
runtime.selectgo(0xc000387f68, 0xc000387ee8, 0xc0005760b0?, 0x0, 0xc0002ddf90?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000387ea8 sp=0xc000387d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c1f8)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000387fc8 sp=0xc000387ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000387fe0 sp=0xc000387fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000387fe8 sp=0xc000387fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 29 gp=0xc001d50540 m=nil [select, 1 minutes]:
runtime.gopark(0xc000384768?, 0x2?, 0x20?, 0x0?, 0xc0003846ec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000384568 sp=0xc000384548 pc=0x48032e
runtime.selectgo(0xc000384768, 0xc0003846e8, 0xc000a4e020?, 0x0, 0xc001ca4810?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc0003846a8 sp=0xc000384568 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c210)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc0003847c8 sp=0xc0003846a8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc0003847e0 sp=0xc0003847c8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0003847e8 sp=0xc0003847e0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 30 gp=0xc001d50700 m=nil [select, 1 minutes]:
runtime.gopark(0xc000386768?, 0x2?, 0x20?, 0x0?, 0xc0003866ec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000386568 sp=0xc000386548 pc=0x48032e
runtime.selectgo(0xc000386768, 0xc0003866e8, 0xc002f423c8?, 0x0, 0xc0002ddf8c?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc0003866a8 sp=0xc000386568 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c228)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc0003867c8 sp=0xc0003866a8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc0003867e0 sp=0xc0003867c8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0003867e8 sp=0xc0003867e0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 31 gp=0xc001d508c0 m=nil [select, 1 minutes]:
runtime.gopark(0xc000058f68?, 0x2?, 0x20?, 0x0?, 0xc000058eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000058d68 sp=0xc000058d48 pc=0x48032e
runtime.selectgo(0xc000058f68, 0xc000058ee8, 0xc002f42380?, 0x0, 0xc00027fb5c?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000058ea8 sp=0xc000058d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c240)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000058fc8 sp=0xc000058ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000058fe0 sp=0xc000058fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000058fe8 sp=0xc000058fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 32 gp=0xc001d50a80 m=nil [select, 1 minutes]:
runtime.gopark(0xc000059768?, 0x2?, 0x20?, 0x0?, 0xc0000596ec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000059568 sp=0xc000059548 pc=0x48032e
runtime.selectgo(0xc000059768, 0xc0000596e8, 0xc0005761a0?, 0x0, 0xc00117c004?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc0000596a8 sp=0xc000059568 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c258)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc0000597c8 sp=0xc0000596a8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc0000597e0 sp=0xc0000597c8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000597e8 sp=0xc0000597e0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 49 gp=0xc001d50c40 m=nil [select]:
runtime.gopark(0xc000059f68?, 0x2?, 0xc9?, 0xc1?, 0xc000059eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000059d68 sp=0xc000059d48 pc=0x48032e
runtime.selectgo(0xc000059f68, 0xc000059ee8, 0xc000293dd8?, 0x0, 0xc001ce0004?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000059ea8 sp=0xc000059d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c270)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000059fc8 sp=0xc000059ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000059fe0 sp=0xc000059fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000059fe8 sp=0xc000059fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 50 gp=0xc001d50e00 m=nil [select]:
runtime.gopark(0xc001d59f68?, 0x2?, 0xc9?, 0xc1?, 0xc001d59eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc001d59d68 sp=0xc001d59d48 pc=0x48032e
runtime.selectgo(0xc001d59f68, 0xc001d59ee8, 0xc000293ec8?, 0x0, 0xc00129a4b8?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc001d59ea8 sp=0xc001d59d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c288)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc001d59fc8 sp=0xc001d59ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc001d59fe0 sp=0xc001d59fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001d59fe8 sp=0xc001d59fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 51 gp=0xc001d50fc0 m=nil [select, 5 minutes]:
runtime.gopark(0xc000092f68?, 0x2?, 0xc9?, 0xc1?, 0xc000092eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000092d68 sp=0xc000092d48 pc=0x48032e
runtime.selectgo(0xc000092f68, 0xc000092ee8, 0xc000293a90?, 0x0, 0xc00027fb54?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000092ea8 sp=0xc000092d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c2a0)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000092fc8 sp=0xc000092ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000092fe0 sp=0xc000092fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000092fe8 sp=0xc000092fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 52 gp=0xc001d51180 m=nil [select, 1 minutes]:
runtime.gopark(0xc000380f68?, 0x2?, 0xc9?, 0xc1?, 0xc000380eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000380d68 sp=0xc000380d48 pc=0x48032e
runtime.selectgo(0xc000380f68, 0xc000380ee8, 0xc002f423b0?, 0x0, 0xc00129aab4?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000380ea8 sp=0xc000380d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c2b8)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000380fc8 sp=0xc000380ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000380fe0 sp=0xc000380fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000380fe8 sp=0xc000380fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 53 gp=0xc001d51340 m=nil [select, 5 minutes]:
runtime.gopark(0xc000381768?, 0x2?, 0xc9?, 0xc1?, 0xc0003816ec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000381568 sp=0xc000381548 pc=0x48032e
runtime.selectgo(0xc000381768, 0xc0003816e8, 0xc000576f80?, 0x0, 0xc001ca4808?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc0003816a8 sp=0xc000381568 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c2d0)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc0003817c8 sp=0xc0003816a8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc0003817e0 sp=0xc0003817c8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0003817e8 sp=0xc0003817e0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 54 gp=0xc001d51500 m=nil [select, 1 minutes]:
runtime.gopark(0xc000381f68?, 0x2?, 0x20?, 0x0?, 0xc000381eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000381d68 sp=0xc000381d48 pc=0x48032e
runtime.selectgo(0xc000381f68, 0xc000381ee8, 0xc002f42308?, 0x0, 0xc001c576c0?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000381ea8 sp=0xc000381d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c2e8)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000381fc8 sp=0xc000381ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000381fe0 sp=0xc000381fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000381fe8 sp=0xc000381fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 55 gp=0xc001d516c0 m=nil [select]:
runtime.gopark(0xc000382768?, 0x2?, 0xc9?, 0xc1?, 0xc0003826ec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000382568 sp=0xc000382548 pc=0x48032e
runtime.selectgo(0xc000382768, 0xc0003826e8, 0xc000a4e0b0?, 0x0, 0xc00129a4bc?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc0003826a8 sp=0xc000382568 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c300)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc0003827c8 sp=0xc0003826a8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc0003827e0 sp=0xc0003827c8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0003827e8 sp=0xc0003827e0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 56 gp=0xc001d51880 m=nil [select, 1 minutes]:
runtime.gopark(0xc000382f68?, 0x2?, 0x20?, 0x0?, 0xc000382eec?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000382d68 sp=0xc000382d48 pc=0x48032e
runtime.selectgo(0xc000382f68, 0xc000382ee8, 0xc000292aa0?, 0x0, 0xc00117c000?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000382ea8 sp=0xc000382d68 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.(*asyncUpdateWorker).serve(0xc00028c318)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:102 +0xc5 fp=0xc000382fc8 sp=0xc000382ea8 pc=0xb7e0c5
go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x25 fp=0xc000382fe0 sp=0xc000382fc8 pc=0xb7de25
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000382fe8 sp=0xc000382fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/processmanager/ebpf.newAsyncMapUpdaterPool in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/processmanager/ebpf/asyncupdate.go:56 +0x105
goroutine 57 gp=0xc001d51a40 m=nil [select, 1 minutes]:
runtime.gopark(0xc000095f90?, 0x2?, 0x0?, 0x0?, 0xc000095f6c?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000095df0 sp=0xc000095dd0 pc=0x48032e
runtime.selectgo(0xc000095f90, 0xc000095f68, 0x0?, 0x0, 0x0?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000095f30 sp=0xc000095df0 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/periodiccaller.Start.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:21 +0xe5 fp=0xc000095fe0 sp=0xc000095f30 pc=0x723245
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/periodiccaller.Start in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:17 +0x99
goroutine 58 gp=0xc001d51c00 m=nil [select]:
runtime.gopark(0xc001dd7f90?, 0x2?, 0xe4?, 0x6f?, 0xc001dd7f6c?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc001dd7df0 sp=0xc001dd7dd0 pc=0x48032e
runtime.selectgo(0xc001dd7f90, 0xc001dd7f68, 0x100000009dc?, 0x0, 0x1011f00000140?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc001dd7f30 sp=0xc001dd7df0 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/periodiccaller.Start.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:21 +0xe5 fp=0xc001dd7fe0 sp=0xc001dd7f30 pc=0x723245
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001dd7fe8 sp=0xc001dd7fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/periodiccaller.Start in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:17 +0x99
goroutine 59 gp=0xc001d51dc0 m=nil [select]:
runtime.gopark(0xc000badf60?, 0x3?, 0x20?, 0xce?, 0xc000badf22?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000badda8 sp=0xc000badd88 pc=0x48032e
runtime.selectgo(0xc000badf60, 0xc000badf1c, 0xc000b60900?, 0x0, 0x4000000a0c?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000badee8 sp=0xc000badda8 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/tracer.(*Tracer).processPIDEvents(0xc001d26680, {0xfafb50, 0xc000318000})
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/tracer/events.go:49 +0x126 fp=0xc000badfb8 sp=0xc000badee8 pc=0xb83866
go.opentelemetry.io/ebpf-profiler/tracer.(*Tracer).StartPIDEventProcessor.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/tracer/events.go:41 +0x28 fp=0xc000badfe0 sp=0xc000badfb8 pc=0xb83708
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000badfe8 sp=0xc000badfe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/tracer.(*Tracer).StartPIDEventProcessor in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/tracer/events.go:41 +0x74
goroutine 36 gp=0xc0002796c0 m=nil [select, 1 minutes]:
runtime.gopark(0xc001d55790?, 0x2?, 0xdb?, 0xd4?, 0xc001d5576c?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc001d555f0 sp=0xc001d555d0 pc=0x48032e
runtime.selectgo(0xc001d55790, 0xc001d55768, 0x0?, 0x0, 0x2?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc001d55730 sp=0xc001d555f0 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/periodiccaller.Start.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:21 +0xe5 fp=0xc001d557e0 sp=0xc001d55730 pc=0x723245
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001d557e8 sp=0xc001d557e0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/periodiccaller.Start in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:17 +0x99
goroutine 37 gp=0xc000279880 m=nil [select, 93 minutes]:
runtime.gopark(0xc00037bf40?, 0x5?, 0x40?, 0xee?, 0xc00037bf0e?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc001dd5d90 sp=0xc001dd5d70 pc=0x48032e
runtime.selectgo(0xc001dd5f40, 0xc00037bf04, 0x0?, 0x0, 0x0?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc001dd5ed0 sp=0xc001dd5d90 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/kallsyms.(*Symbolizer).reloadWorker(0xc00007dc00, {0xfafb50, 0xc000318000}, 0xc001dcc1b0)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/kallsyms/kallsyms.go:560 +0x10a fp=0xc001dd5fb0 sp=0xc001dd5ed0 pc=0x73ba4a
go.opentelemetry.io/ebpf-profiler/kallsyms.(*Symbolizer).StartMonitor.gowrap1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/kallsyms/kallsyms.go:624 +0x2c fp=0xc001dd5fe0 sp=0xc001dd5fb0 pc=0x73bfac
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001dd5fe8 sp=0xc001dd5fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/kallsyms.(*Symbolizer).StartMonitor in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/kallsyms/kallsyms.go:624 +0xd3
goroutine 38 gp=0xc000279a40 m=nil [IO wait]:
runtime.gopark(0x15a62e8?, 0x2000000000017?, 0xa8?, 0xa0?, 0x0?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000b4fd68 sp=0xc000b4fd48 pc=0x48032e
runtime.netpollblock(0x1000?, 0x1000?, 0x0?)
#011/usr/local/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc000b4fda0 sp=0xc000b4fd68 pc=0x442917
internal/poll.runtime_pollWait(0x799df4a1a800, 0x72)
#011/usr/local/go/src/runtime/netpoll.go:351 +0x85 fp=0xc000b4fdc0 sp=0xc000b4fda0 pc=0x47f505
internal/poll.(*pollDesc).wait(0x4b?, 0x40?, 0x1)
#011/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000b4fde8 sp=0xc000b4fdc0 pc=0x4f9387
internal/poll.(*pollDesc).waitRead(...)
#011/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).RawRead(0xc001d34ae0, 0xc001205440)
#011/usr/local/go/src/internal/poll/fd_unix.go:710 +0x125 fp=0xc000b4fe48 sp=0xc000b4fde8 pc=0x4ff425
os.(*rawConn).Read(0xc00005e1f8, 0x0?)
#011/usr/local/go/src/os/rawconn.go:31 +0x4a fp=0xc000b4fe70 sp=0xc000b4fe48 pc=0x50806a
github.com/mdlayher/kobject.(*sysConn).TryRead(0xc00005e258, {0xc001cf0000, 0x1000, 0x1000})
#011/agent/go/pkg/mod/github.com/mdlayher/kobject@v0.0.0-20200520190114-19ca17470d7d/client_linux.go:64 +0x184 fp=0xc000b4fed0 sp=0xc000b4fe70 pc=0x737ac4
github.com/mdlayher/kobject.(*Client).Receive(0xc001dcc1b0)
#011/agent/go/pkg/mod/github.com/mdlayher/kobject@v0.0.0-20200520190114-19ca17470d7d/client.go:70 +0xc2 fp=0xc000b4ff78 sp=0xc000b4fed0 pc=0x737642
go.opentelemetry.io/ebpf-profiler/kallsyms.(*Symbolizer).pollKobjectClient(0xc00007dc00, {0x200000000000000?, 0xa3b?}, 0xc001dcc1b0)
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/kallsyms/kallsyms.go:603 +0x27 fp=0xc000b4ffb0 sp=0xc000b4ff78 pc=0x73bd07
go.opentelemetry.io/ebpf-profiler/kallsyms.(*Symbolizer).StartMonitor.gowrap2()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/kallsyms/kallsyms.go:625 +0x2c fp=0xc000b4ffe0 sp=0xc000b4ffb0 pc=0x73bf4c
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000b4ffe8 sp=0xc000b4ffe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/kallsyms.(*Symbolizer).StartMonitor in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/kallsyms/kallsyms.go:625 +0x13c
goroutine 39 gp=0xc000279c00 m=17 mp=0xc000ad6008 [syscall]:
syscall.Syscall6(0xe8, 0x4c, 0xc0002c3440, 0x8, 0xffffffffffffffff, 0x0, 0x0)
#011/usr/local/go/src/syscall/syscall_linux.go:96 +0x39 fp=0xc001d57ca8 sp=0xc001d57c48 pc=0x4db179
syscall.Syscall6(0xe8, 0x4c, 0xc0002c3440, 0x8, 0xffffffffffffffff, 0x0, 0x0)
#011<autogenerated>:1 +0x3d fp=0xc001d57cf0 sp=0xc001d57ca8 pc=0x4dbcdd
golang.org/x/sys/unix.EpollWait(0xfa60a0?, {0xc0002c3440?, 0x8?, 0x8?}, 0xffffffffffffffff?)
#011/agent/go/pkg/mod/golang.org/x/sys@v0.41.0/unix/zsyscall_linux_amd64.go:55 +0x4f fp=0xc001d57d60 sp=0xc001d57cf0 pc=0x56422f
github.com/cilium/ebpf/internal/unix.EpollWait(...)
#011/agent/go/pkg/mod/github.com/cilium/ebpf@v0.20.0/internal/unix/types_linux.go:114
github.com/cilium/ebpf/internal/epoll.(*Poller).Wait(0xc00135a240, {0xc0002c3440, 0x8, 0x8}, {0xc001d2a268?, 0x0?, 0x0?})
#011/agent/go/pkg/mod/github.com/cilium/ebpf@v0.20.0/internal/epoll/poller.go:181 +0x245 fp=0xc001d57e60 sp=0xc001d57d60 pc=0x661205
github.com/cilium/ebpf/perf.(*Reader).ReadInto(0xc000000000, 0xc001d57f80)
#011/agent/go/pkg/mod/github.com/cilium/ebpf@v0.20.0/perf/reader.go:364 +0x2b7 fp=0xc001d57f50 sp=0xc001d57e60 pc=0x6638d7
go.opentelemetry.io/ebpf-profiler/tracer.startPerfEventMonitor.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/tracer/events.go:110 +0x9c fp=0xc001d57fe0 sp=0xc001d57f50 pc=0xb83dbc
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001d57fe8 sp=0xc001d57fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/tracer.startPerfEventMonitor in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/tracer/events.go:103 +0x178
goroutine 40 gp=0xc000279dc0 m=nil [chan send]:
runtime.gopark(0xc000367e40?, 0x49041c?, 0x8?, 0x0?, 0xfa6120?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000367de8 sp=0xc000367dc8 pc=0x48032e
runtime.chansend(0xc00008d0a0, 0xfa0778, 0x1, 0xc000367e01?)
#011/usr/local/go/src/runtime/chan.go:283 +0x3ab fp=0xc000367e58 sp=0xc000367de8 pc=0x41404b
runtime.chansend1(0xfa6120?, 0xc00007c860?)
#011/usr/local/go/src/runtime/chan.go:161 +0x17 fp=0xc000367e88 sp=0xc000367e58 pc=0x413c97
go.opentelemetry.io/ebpf-profiler/tracer.(*Tracer).startTraceEventMonitor.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/tracer/events.go:268 +0x211 fp=0xc000367fe0 sp=0xc000367e88 pc=0xb843f1
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000367fe8 sp=0xc000367fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/tracer.(*Tracer).startTraceEventMonitor in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/tracer/events.go:157 +0x245
goroutine 41 gp=0xc00028afc0 m=nil [select]:
runtime.gopark(0xc000090f80?, 0x3?, 0x0?, 0x0?, 0xc000090f52?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000090dd8 sp=0xc000090db8 pc=0x48032e
runtime.selectgo(0xc000090f80, 0xc000090f4c, 0xa8000000065?, 0x0, 0x1a7200000080?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc000090f18 sp=0xc000090dd8 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/periodiccaller.StartWithManualTrigger.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:49 +0xf6 fp=0xc000090fe0 sp=0xc000090f18 pc=0x723516
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/periodiccaller.StartWithManualTrigger in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:45 +0xae
goroutine 42 gp=0xc00028b180 m=nil [select]:
runtime.gopark(0xc001dd8f90?, 0x2?, 0xf7?, 0xc4?, 0xc001dd8f6c?)
#011/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc001dd8df0 sp=0xc001dd8dd0 pc=0x48032e
runtime.selectgo(0xc001dd8f90, 0xc001dd8f68, 0x800000000000015?, 0x0, 0x125f018000008?, 0x1)
#011/usr/local/go/src/runtime/select.go:351 +0x845 fp=0xc001dd8f30 sp=0xc001dd8df0 pc=0x45cea5
go.opentelemetry.io/ebpf-profiler/periodiccaller.Start.func1()
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:21 +0xe5 fp=0xc001dd8fe0 sp=0xc001dd8f30 pc=0x723245
runtime.goexit({})
#011/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001dd8fe8 sp=0xc001dd8fe0 pc=0x488181
created by go.opentelemetry.io/ebpf-profiler/periodiccaller.Start in goroutine 1
#011/builds/psr/perfopt/profile-processor/opentelemetry-ebpf-profiler/periodiccaller/periodiccaller.go:17 +0x99

AI analysis (may contain hallucinations):

SIGBUS / BUS_ADRERR (bus error code 0x2) at elfgopclntab.go:520 inside getFuncMapEntry, called from Symbolize via sort.Search.

Code 0x2 = BUS_ADRERR = "nonexistent physical address". The virtual address was valid (mapped), but the kernel had reclaimed the underlying physical page.

Why Physical Pages Get Reclaimed

Gopclntab.data is backed by a mmap(MAP_SHARED) mapping of the ELF file on disk. After symbolization, ReleaseResources() → SetDontNeed() → syscall.Madvise(r.data, MADV_DONTNEED) is called on the same buffer. MADV_DONTNEED tells the kernel it may discard the pages. When a subsequent access to a discarded page occurs, it normally page-faults and re-reads from the file. However, on some systems or under memory pressure the kernel can cause a BUS_ADRERR if the underlying file is no longer accessible (truncated, deleted, unmounted, etc.) or if the mapping is to a file that has a page that cannot be re-fetched.

The Actual Race

The crash is not fundamentally a use-after-free but rather a concurrent MADV_DONTNEED on memory that is actively being read — but there's something more subtle here.

Looking at Symbolize in elfgopclntab.go:568-569:

  index := sort.Search(g.numFuncs+1, func(i int) bool {
      funcPc, _ := g.getFuncMapEntry(i)  // i = 0..numFuncs inclusive
      return funcPc > pc
  }) - 1

getFuncMapEntry(g.numFuncs) at line 519:

  fmap := (*pclntabFuncMap118)(unsafe.Pointer(&g.functab[index*int(g.funcMapSize)]))

g.functab is a subslice of mmap'd data. After MADV_DONTNEED is called on the backing region by another goroutine (ReleaseResources() is called for every PID's interpreter instances after each trace is handled, manager.go:373-377), the kernel may reclaim those physical pages. The next access to those pages causes a page fault. Under normal circumstances that re-reads from file, but if:

  • The file is opened from a container overlay filesystem that was deleted under the process
  • The file page cache entry has been invalidated or the file is from a network filesystem

...it results in BUS_ADRERR.

Call Chain Summary

  goroutine 43:
    HandleTrace → convertFrame → symbolizeFrame → MultiInstance.Symbolize
      → goInstance.Symbolize (go.go:113)
        → Gopclntab.Symbolize (elfgopclntab.go:568)
          → sort.Search (calls getFuncMapEntry for i=0..numFuncs)
            → getFuncMapEntry (elfgopclntab.go:519-520)
              → accesses g.functab[i * funcMapSize]  ← SIGBUS here

  Concurrent: ReleaseResources() (manager.go:374) on the same PID's goInstance
    → goInstance.ReleaseResources (go.go:134)
      → Gopclntab.SetDontNeed
        → madvise(MADV_DONTNEED) on the same mmap backing g.functab

Note that ReleaseResources is called on the same trace's PID, after all frames have been symbolized — so the crash likely occurs when there are multiple concurrent goroutines processing different traces from the same PID: goroutine A calls ReleaseResources after finishing its trace while goroutine B is still in Symbolize for another trace from the same PID.

Fix Direction

The fix needs to ensure MADV_DONTNEED is not called on pages that are actively being read. Options:

  1. Remove MADV_DONTNEED from ReleaseResources for Go instances — the mmap reference counting (g.dataRef) already handles memory lifecycle correctly; MADV_DONTNEED is a pure optimization hint that is causing crashes.
  2. Guard Symbolize and SetDontNeed with a RWMutex on goData — Symbolize holds a read lock, SetDontNeed holds a write lock.
  3. Only call SetDontNeed after all instances of a goData are done — since goData is shared across all goInstance objects for a given binary (via ref-counting), MADV_DONTNEED should only happen when the last instance is released, not after every individual trace.

Option 1 is the most conservative and safest. Option 3 is the most semantically correct (it defers the hint to when the data is truly no longer needed).

The crash is reproducible because:

  • The profiler processes many traces in parallel (goroutine 43 was one of many handlers)
  • Multiple Go processes from the same binary share a goData and its Gopclntab
  • The MADV_DONTNEED racing with Symbolize on the mmapped pages causes BUS_ADRERR under the right filesystem/memory conditions

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions