Mappings are repeatedly parsed

[bobrik]

We have a V8 based piece of software (edgeworker, approximately workerd) that has a ton of mappings:

ivan@107m94:~$ sudo wc -l /proc/*/maps | sort -n | tail -n5
6922 /proc/557384/maps
22106 /proc/3927685/maps
52283 /proc/2264585/maps
128198 /proc/2264341/maps
351702 total

All top3 instances are of that software.

A big chunk of time is spent parsing mappings:

I can see in strace that the most map-heavy process gets re-parsed a lot:

ivan@107m94:~$ sudo strace -t -f -p $(pidof ebpf-profiler) 2>&1 | grep --line-buffered /proc/2264341
[pid 910473] 05:45:25 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910237] 05:45:27 readlinkat(AT_FDCWD, "/proc/2264341/exe", "/usr/local/bin/edgeworker", 128) = 25
[pid 910465] 05:45:41 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910243] 05:45:44 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910465] 05:45:46 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910472] 05:45:48 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910472] 05:45:49 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910465] 05:45:51 readlinkat(AT_FDCWD, "/proc/2264341/exe", "/usr/local/bin/edgeworker", 128) = 25
[pid 909883] 05:45:55 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910472] 05:45:59 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910243] 05:46:05 openat(AT_FDCWD, "/proc/2264341/cgroup", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910472] 05:46:05 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910472] 05:46:08 readlinkat(AT_FDCWD, "/proc/2264341/exe", "/usr/local/bin/edgeworker", 128) = 25
[pid 910472] 05:46:08 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910473] 05:46:11 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910473] 05:46:11 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910473] 05:46:13 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910243] 05:46:25 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910237] 05:46:27 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910237] 05:46:27 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910465] 05:46:30 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 909925] 05:46:30 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910465] 05:46:33 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910465] 05:46:33 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910465] 05:46:35 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910237] 05:46:37 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 909885] 05:46:39 readlinkat(AT_FDCWD, "/proc/2264341/exe", "/usr/local/bin/edgeworker", 128) = 25
[pid 910243] 05:46:40 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910473] 05:46:43 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910473] 05:46:43 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910473] 05:46:45 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910465] 05:46:50 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910243] 05:46:53 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910243] 05:47:05 openat(AT_FDCWD, "/proc/2264341/cgroup", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910243] 05:47:06 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 909925] 05:47:10 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910472] 05:47:10 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 910472] 05:47:14 readlinkat(AT_FDCWD, "/proc/2264341/exe",  <unfinished ...>
[pid 910473] 05:47:16 openat(AT_FDCWD, "/proc/2264341/maps", O_RDONLY|O_CLOEXEC <unfinished ...>

Sprinkling some debug printk in bpf code reveals that this is the most called codepath:

opentelemetry-ebpf-profiler/support/ebpf/interpreter_dispatcher.ebpf.c

Lines 271 to 275 in 159c896

	case metricID_UnwindNativeErrWrongTextSection:;
	u64 pid_tgid = (u64)trace->pid << 32 | trace->tid;
	if (report_pid(ctx, pid_tgid, record->ratelimitAction)) {
	increment_metric(metricID_NumUnknownPC);
	}

There are indeed many unknown frames. We have v8 interpreter disabled (and it has never worked for us here).

Looking at top offenders in terms of time grouped by exe path, I can also see nginx there, which has a bunch of lua and many worker processes, which I assume is responsible here too.

The actual top of GetMappings timing:

<even smaller nginx with lua>: 1025.23 ms
<smaller nginx with lua>: 1562.27 ms
<big nginx with lua>: 3600.90 ms
<v8 based thingy>: 14815.62 ms

[fabled]

Yes, continuous reparsing mappings can get time consuming. It also triggers the synchronization code that consumes memory. Etc. Especially if program has tons of anonymous non-executable mappings (e.g. musl allocator, or some other allocators doing it) or just the program itself does it. Things get hairy.

But in practice the reparsing is triggered if the ebpf code sees code executing from location that it does not know about. This typically happens on programs with VM that is continously JITting code. Especially if we don't support the VM things get hairy - in this case nginx/lua. We should probably synthesize for the unhandled executable pages an "unsupported mapping" stack delta so that the bpf code will not trigger reparsing the mappings for those.

V8 is supported, but if its continuously seeing new executable mappings this can happen. Due to the nature of the VMs there's probably no good way to limit this. Dotnet is similar VM.

Since minimum kernel version has been bumped some time ago, and #1178 is discussing bumping it further. I wonder if we could take advantage of:

• perf_event_open with PERF_RECORD_MMAP (since kernel 3.7?) and/or PERF_RECORD_MMAP2 (since kernel 3.12?) events
• bpf_find_vma helper (though this needs kernel 5.17+)

Particularly the ebpf helper would be useful for optimizing certain things in the bpf code. But perhaps its still early to bump the kernel requirement so high.

[bobrik]

V8 is supported, but if its continuously seeing new executable mappings this can happen.

While it is supported, it does not work us (for whatever reason) and we explicitly disable it to keep cardinality in check. Can the problem be that we keep parsing mappings, but that just has no effect on the bpf side success rate?

perf_event_open with PERF_RECORD_MMAP (since kernel 3.7?) and/or PERF_RECORD_MMAP2 (since kernel 3.12?) events

PERF_RECORD_MMAP2 is pretty chatty on my system:

2026/03/23 22:54:32 stopped: 474352 events in 119.9s (3956.6 events/s)
2026/03/23 22:54:32 top 5 files by event count:
2026/03/23 22:54:32   1. //anon (443938 events)
2026/03/23 22:54:32   2. [vdso] (3662 events)
2026/03/23 22:54:32   3. /usr/lib/x86_64-linux-gnu/libc.so.6 (3582 events)
2026/03/23 22:54:32   4. /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (3573 events)
2026/03/23 22:54:32   5. /usr/lib/x86_64-linux-gnu/libm.so.6 (1115 events)
2026/03/23 22:54:32 top 5 comms by event count:
2026/03/23 22:54:32   1. nginx-ssl (263652 events)
2026/03/23 22:54:32   2. nginx-fl (179770 events)
2026/03/23 22:54:32   3. <unknown> (30219 events)
2026/03/23 22:54:32   4. nginx-cache (474 events)
2026/03/23 22:54:32   5. clang-19 (19 events)

Things also tend to come un batches (note only two distinct addresses below):

2026/03/23 23:00:36 #276665 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276666 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276667 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276668 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276671 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276672 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276675 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276676 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276677 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276678 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276683 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276684 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276689 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276690 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276691 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276692 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276693 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276694 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276695 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276696 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276697 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276698 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276699 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276700 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276701 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276702 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276703 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276704 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276705 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276706 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276708 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276709 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276710 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276711 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276717 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276718 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276719 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276720 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276721 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276722 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276723 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276724 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276729 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276730 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276731 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276732 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276733 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon
2026/03/23 23:00:36 #276734 pid = 2541221 (nginx-fl), addr = 0x55de86e00000, len = 0x11a0000, file = //anon
2026/03/23 23:00:36 #276735 pid = 2541221 (nginx-fl), addr = 0x55de86de0000, len = 0x11c0000, file = //anon

Most of the events are for nginx + luajit, which we're moving away from. If we switched to perf events today, we'd probably spend more time handling events for those short lived mappings than we do on /proc/<pid>/maps parsing.

Note that the most mapping heavy process does not show up anywhere near the top.

Particularly the ebpf helper would be useful for optimizing certain things in the bpf code.

Could you elaborate how the helper would be useful?

[bobrik]

I think it's a combination of low backoff ceiling (~25s) and the fact that backoff is getting reset. My high level understanding is that we no bad PCs to appear in a 5s window, and that triggers the backoff reset:

opentelemetry-ebpf-profiler/support/ebpf/tracemgmt.h

Lines 137 to 145 in 4ea0940

	if (diff_ts < limit_window_ts + (5000 * 1000000ULL)) {
	// PID event within 5 seconds, increase limit window size if possible
	if (attempt < (fast_timer ? fast_max_attempts : default_max_attempts)) {
	attempt++;
	}
	} else {
	// Silence for at least 5 seconds. Reset back to zero.
	attempt = 0;
	}

I cranked up the ceiling to 204s and disabled backoff.

• Before:

• After:

It's pretty clear that less time is being spent parsing mappings. Here's a diff flamegraph:

This can probably be driven further down by increasing backoff higher and by not unloading exited binaries as aggressively.

Surprisingly, little has changed in terms of CPU usage overall (green has the changes, yellow does not):

The above is cadvisor for the whole cgroup. Below is ebpf profiler itself based on samples, which is more pronounced:

I'm not really sure what drives this difference.