[CRASH] f_malloc: defragmentation leaves a stale `pf` back-pointer; SIGSEGV in `fm_remove_free()`

[Lt-Flash]

OpenSIPS version you are running

version: opensips 4.1.0-dev (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, F_PARALLEL_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 11:11:52 Sep  2 2026 with cc 9

Built from master at 32d67aed92.

The bug

f_malloc fragments carry pf, a pointer to the physically preceding
fragment, so a merge must re-point the following fragment at the survivor.
fm_free() does this; fm_malloc()'s defragmentation loop does not:

/* fm_free() - correct */
f->size += neigh->size + FRAG_OVERHEAD;
FRAG_NEXT(neigh)->pf = f;

/* fm_malloc() defragmentation - the fixup is missing */
fm_remove_free(fm, n);
frag->size += n->size + FRAG_OVERHEAD;

After the merge, the fragment following the pair still points at n, which is
now inside frag's payload. When it is freed, fm_free() reads that stale
pf as FRAG_PREV, frag_is_free() (which is just ((_f)->prev)) tests
payload bytes, and fm_remove_free() executes *pf = n->u.nxt_free; through
whatever it found:

#0  fm_remove_free (n=0x7fcc9a112548, fm=...) at mem/f_malloc.c:176
#1  fm_free (...) at mem/f_malloc_dyn.h:264
#2  _shm_free_bulk (file="h_table.c", line=204) at ../../mem/shm_mem.h:598
#3  free_cell (dead_cell=0x7fcc9a112708) at h_table.c:204
#4  delete_cell (...) at timer.c:243
#5  wait_handler (...) at timer.c:484

A general protection fault, not a NULL dereference — consistent with prev
read out of payload. (I have the core dump and can provide it.)

Fix

--- a/mem/f_malloc_dyn.h
+++ b/mem/f_malloc_dyn.h
@@ -144,6 +144,7 @@ void *fm_malloc(struct fm_block *fm, unsigned long size,
                        {
                                fm_remove_free(fm, n);
                                frag->size += n->size + FRAG_OVERHEAD;
+                               FRAG_NEXT(frag)->pf = frag;

fm_malloc_init() sets up fm->last_frag as a real header, so no bounds
check is needed — as in fm_free().

To Reproduce

A transaction per request that is suspended, against a pool small enough to
defragment constantly. Suspension is what exposes it: a synchronous route frees
its cell inline, before the layout can go wrong, while async() leaves it for
tm's timer — hence wait_handler in the trace. sleep is used only because it
is a stock async function.

log_level=2
stderror_enabled=yes
syslog_enabled=yes
udp_workers=4

socket=udp:YOUR_IP:5060

mpath="/path/to/modules/"

loadmodule "proto_udp.so"
loadmodule "signaling.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "maxfwd.so"
loadmodule "sipmsgops.so"
loadmodule "cfgutils.so"

modparam("tm", "fr_timeout", 5)
modparam("tm", "fr_inv_timeout", 30)

route {
	if (!mf_process_maxfwd_header(10)) {
		send_reply(483, "Too Many Hops");
		exit;
	}
	if (is_method("INVITE")) {
		if (!t_newtran()) {
			send_reply(500, "no transaction");
			exit;
		}
		$avp(one) = "a moderately long profile value so the fragment is not tiny 01234567890123456789";
		$avp(two) = $ci;
		$avp(three) = "short";
		async(sleep(1), resume_route);
		exit;
	}
	if (is_method("ACK|BYE|CANCEL")) {
		send_reply(200, "OK");
		exit;
	}
	send_reply(405, "Method Not Allowed");
	exit;
}

route[resume_route] {
	t_reply(200, "OK");
	exit;
}

1. opensips -f repro.cfg -a F_MALLOC -m 32 -M 16
2. INVITEs at ~2000 cps for ~2 min (several SIPp processes — one saturates
   near 1000-1250 cps). The log fills with not enough contiguous free shm memory ... attempting defragmentation.
3. A worker faults and the process exits, after ~12,000 defragmentation passes.

Relevant System Logs

WARNING:core:fm_malloc: not enough contiguous free shm memory (949616 bytes
        left, need 5704), attempting defragmentation...
ERROR:tm:sip_msg_cloner: no more share memory
ERROR:tm:new_t: out of mem
CRITICAL:core:sig_usr: segfault in process pid: 136109, id: 3
kernel: traps: opensips[136109] general protection fault

Verification

build	trials	crashed
master	4	4
master + the patch above	4	0 (~35,000 defrag passes each)

Only f_malloc is affected: it is the only allocator holding a pointer to the
physically preceding fragment. q_malloc has no defragmentation pass,
hp_malloc coalesces forward only, and f_parallel_malloc has no pf.

OS/environment information

• Operating System: Ubuntu 20.04.6 LTS (x86_64), kernel 5.4.0-216-generic
• OpenSIPS installation: git (master, 32d67aed92), built from source, gcc 9.4.0
• Other: the build carries -DDBG_MALLOC, so the allocator must be named
  explicitly (-a F_MALLOC) or the runtime default is Q_MALLOC_DBG.

[gmaruzz]

Independent reproduction of this on a release build (4.0.1), with two cores analysed under matching opensips-dbg. The signature matches yours exactly. But it reproduced without any defragmentation happening, which may matter for the completeness of the proposed fix — details at the end.

Environment

version: opensips 4.0.1 (x86_64/linux)
git revision: 1747e6ed20
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC,
       HP_MALLOC, F_PARALLEL_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT

Debian package 4.0.1-1 from apt.opensips.org, opensips-dbg:amd64 4.0.1-1 matching. Started -m 4096 -M 128. The faulting block is pkg, in the MI/HTTP worker (id: 2). Two independent crashes on two separate machines, same OpenSIPS config.

Trigger

An rtp_relay:update MI call (over mi_http) that matches a live call, re-anchoring it to a different rtpengine node:

{"engine":"route","set":1,"node":"udp:<address of a disabled engine>"}

2 of 2 attempts that matched a real in-progress call crashed; calls where the filter matched nothing did not. This is only a trigger, not the defect — as your analysis predicts, it is simply the first code path to free a fragment adjacent to an already-corrupted one.

Backtrace

#0 fm_remove_free (fm=<pkg block>, n=<the SDP buffer>)   mem/f_malloc.c:176
#1 fm_free        (p=<the SDP buffer>)                   mem/f_malloc_dyn.h:264
#2 rtp_relay_reinvite (tmp, leg=0, body, release_body=1) modules/rtp_relay/rtp_relay_ctx.c:2397

rtp_relay_ctx.c:2397 is pkg_free(body->s), the last line of rtp_relay_reinvite. f_malloc_dyn.h:264 is the second join branch, neigh = FRAG_PREV(f).

The values, both cores

value	host A	host B
body->s (pointer being freed)	0x7f6813160340	0x7f8bc7c87340
body->len	401	401
f = FM_FRAG(p) (= p-56)	0x7f6813160308	0x7f8bc7c87308
f->size	0x1000000010000	0x2000000010000
f->pf	== p (i.e. f+56)	== p (i.e. f+56)
f->prev	0x0	0x0
pf = n->prev (dereferenced at f_malloc.c:176)	0x3230393837312048	0x3230393837312048
hash bucket	2097	2097

f->pf points at f+56 — its own payload. pf = 0x3230393837312048 is little-endian ASCII "H 178902": bytes 16–23 of the SDP body itself (o=FreeSWITCH 1789026742 ...). Line 176 is *pf = n->u.nxt_free;, so the write goes through a pointer read out of SDP text. Identical on both machines because both carry the same FreeSWITCH o= line.

The sequence:

1. First join branch computes FRAG_NEXT(f) from the bogus size → unmapped, but the < fm->last_frag guard fails, so it is never dereferenced.
2. Second branch takes neigh = FRAG_PREV(f) = f->pf = p, pointing into the payload.
3. frag_is_free(neigh) reads neigh->prev — SDP text, non-zero → "free".
4. fm_remove_free(fm, p) executes *pf = ... through 0x3230393837312048. SIGSEGV, si_code = 128 (SI_KERNEL, general protection fault).

Ruled out

• Not a double-free. DBG_MALLOC is compiled in, so check_double_free() runs on every fm_free; it never fired. f->prev == NULL means frag_is_free(f) is false — the fragment was genuinely allocated.
• Not an overrun of the freed buffer. The 401-byte body is intact end to end: the SDP tail (...a=rtcp:38503\r\n) sits exactly at p+400, and the bytes past it are unrelated live heap, not overrun payload.
• Correct ownership. The buffer comes from bencode_dictionary_get_string_dup() (bencode.h:463, BENCODE_MALLOC = pkg_malloc), so rtp_relay is the rightful owner and the free is legitimate.

What is corrupt is the neighbouring fragment's header, exactly as this issue describes.

The part that may matter for the fix

Both crashes happened with no defragmentation in evidence and no memory pressure at all:

• real_used = 16,362,232 of 134,217,728 bytes in the pkg block at crash time — 12.2 % used, read directly out of both cores ($25/$26/$27 = block name "pkg", size, real_used, used_frags). 20,585 fragments.
• Zero occurrences of defragment / not enough contiguous in the journal — searched both across the surviving crash window (12:40–13:10, ~5,500 log lines retained per host) and across the entire retained journal on both machines, all boots. Count is 0 on both.
• Zero E_CORE_PKG_THRESHOLD events fired on either host (the event is registered at startup, never raised).

Since this issue reproduces under heavy defragmentation load, and fm_split_frag and fm_realloc both write pf correctly in 4.0.1, I could not determine how the stale pf came to exist here. The mechanism match is unambiguous — a pf pointing into its own payload is not ambiguous about anything — but the route to it appears to be a different one than the defragmentation loop, at least on this workload.

That may mean the FRAG_NEXT(frag)->pf = frag fixup in the defrag loop is necessary but not sufficient. Worth checking whether another path can leave the same stale back-pointer without defragmenting.

Happy to run further gdb on the preserved cores if any specific value would help.

[gmaruzz]

^ it's Claude, obviously

[Lt-Flash]

@gmaruzz thank you for the cores and the careful write-up - the part you flagged as not fitting turned out to be the key. Your crash is a different defect, now filed separately as #4255.

With the rtp_relay route engine, rtp_relay_route_fill_body() assigns the script return value uncopied (*body = val.rs), and script_return_set() builds that value with an inline buffer (v->val.rs.s = v->buf, offset +32). rtp_relay_reinvite(..., release_body=1) then pkg_free()s that interior pointer at rtp_relay_ctx.c:2397 - your frame #2. I reproduced it 2/2 on stock 4.0.

That accounts for the three things that did not add up:

• f->pf pointing into its own payload is not a stale back-pointer. f = p-56 lands on the allocation itself, so f->pf reads val.rs.s, which is v->buf, the pointer being freed.
• check_double_free() cannot fire: frag_seems_valid() only tests first_frag <= f->pf <= last_frag, which holds by construction. It never validates size.
• No defragmentation is involved, which is why you found none at 12.2% pkg used.

Your {"engine":"route"} trigger is the tell too: the bencode_dictionary_get_string_dup() ownership argument applies to the rtpengine engine, not the route one.

So this issue is unaffected - the defragmentation fixup still rests on its own reproduction.