wasm2c: fail closed when table allocation runs out of memory

[sxvncry]

Fixes #2854

PoC (same script attached to the issue): https://github.com/user-attachments/files/32163301/poc.sh

Summary

The wasm2c runtime table allocators (wasm-rt-impl-tableops.inc, template
for funcref/externref/exnref) never check the calloc() result. On
allocation failure the table keeps the requested huge size while data
is NULL, so the bounds check i < table->size passes for every index and
generated code touches table->data[i] at NULL + i*sizeof(element) with
i up to ~2e9 — a guest read/write/control-flow-hijack primitive in host
memory (low 64 GiB of the address space).

The memory allocator in the same runtime already checks its calloc result
and aborts (wasm-rt-mem-impl-helper.inc, from #2786); the table path
never got the same treatment.

Root cause

wasm2c/wasm-rt-impl-tableops.inc line 54:

table->size = elements;
table->max_size = max_elements;
table->data = calloc(table->size, sizeof(WASM_RT_TABLE_ELEMENT_TYPE));
/* no NULL check */

Trigger: (table 2000000000 4000000000 funcref) → 2e9 * 32 = 64 GB
calloc fails under memory pressure / address-space limit → broken
invariant.

Impact (poc.sh, 6/6 pass, unpatched HEAD)

1. WRITE + control-flow hijack: elem segment writes a guest funcref to
   NULL + 8192*32 = 0x40000; the host maps a callback struct there, the
   guest function pointer lands in it and the host calls it.
2. READ: table.get distinguishes host memory contents at the same
   address (ref.is_null leaks null-ness of 32-byte words across 0–64 GB).
3. DoS: table.set crashes the host deterministically.
4. All three table types share the template and show data=NULL size=2e9.

Fix

Mirror the memory path: after calloc, if elements != 0 and data is
NULL, reset size/max_size to 0 and abort(). The size reset closes
the window even if the host installs a WASM_RT_TRAP handler that
intercepts the failure path. The grow path already handles failure
correctly — this patch makes allocate fail closed too.

[sbc100]

Presumably we should take a consistent approach here across all of wasm2c?

I believe we have calls to calloc / malloc and realloc within wasm-rt-impl. I'm not sure if abort is really the best behaviour, altghough perhaps as a default is not bad? Real embedder can always provide there own implementation of wasm-rt I guess?

Can you also add a test case for this? Presumably (table 2000000000 4000000000 funcref) would be enough?

@keithw @shravanrn

[sbc100]

Use table->size instead of elements here since that is what was passed to calloc.

[sbc100]

Is there any point in this line if the next line is abort ? Maybe just remove it?

[shravanrn]

@sbc100 Polite suggestion. This is clearly a security issue (albeit only in a scenario where the wasm file being compiled is adversarially produced). Let's not churn on minor stylistic points. Thanks for reporting and fixing this @sxvncry

[shravanrn]

@sbc100 I may follow up in a separate commit to rewrite this to a standard test here. I don't think we want to maintain a separate test tool .py for one test that will likely never get broken (i can also leave it in if you prefer)

[shravanrn]

looks like the rebase is causing build failures.

@sxvncry If you are upto it, can you see if you add the failing wat file as a normal test like test/wasm2c/address-overflow.txt with the default test runner instead of a new runner tool? If this doesn't work for some reason, could you look into fixing your custom runner to pass the tests?

Alternately, if you have trouble with the test addition fix, you can submit just the code patch that you did earlier, and I can take over adding the test. (If you want to do this, please let me know)

[sbc100]

Happy to defer to you on this one @shravanrn

[sxvncry]

Thanks @shravanrn. Please feel free to take over the test addition and rewrite it as a standard test. The code patch itself is ready from my side.

[shravanrn]

@sxvncry sounds good.