Bound native NIO buffer retention after large socket writes

server/docker-compose.yml

@@ -16,7 +16,8 @@ services:
     # `web` depends_on `vector` so that we don't lose logs on docker compose down
     depends_on:
       - vector
-    command: sh -c 'java $${JAVA_OPTS} --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -Djdk.attach.allowAttachSelf -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:StartFlightRecording=disk=true,maxsize=1G,name=instant,filename=instant_$(date +%s%N).jfr -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=localhost -server -jar target/instant-standalone.jar'
+    # Bound cached NIO copy buffers and return freed native pages; JAVA_OPTS can override these defaults.
+    command: sh -c 'java -Djdk.nio.maxCachedBufferSize=131072 -XX:TrimNativeHeapInterval=60000 -Xlog:trimnative=info $${JAVA_OPTS} --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -Djdk.attach.allowAttachSelf -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:StartFlightRecording=disk=true,maxsize=1G,name=instant,filename=instant_$(date +%s%N).jfr -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=localhost -server -jar target/instant-standalone.jar'
 
   vector:
     # Mirrored from timberio/vector:0.56.0-alpine into our private ECR so

server/infra/README.md

@@ -41,6 +41,53 @@ does not: it reports one impacted instance for much of the day without a
 request-level cause, and the group's ELB health check already replaces
 instances that fail.
 
+## Native memory
+
+The API sets `-Djdk.nio.maxCachedBufferSize=131072`. When NIO writes a heap
+buffer to a socket, it copies the data into a temporary native buffer. Corretto
+26 caches these buffers on long-lived platform/carrier threads without a size
+limit by default. Undertow's gathering writes can leave many large buffers in
+each IO thread's cache after the WebSocket messages finish. These buffers are
+outside the Java heap and are excluded from the direct-buffer MXBean and
+`MaxDirectMemorySize` accounting.
+
+The 128 KiB cap applies to each cached buffer. It preserves reuse of ordinary
+Java socket buffers while freeing larger temporary buffers after I/O. It does
+not limit message sizes or the memory needed by writes in progress. The cache
+can hold up to 1,024 entries per thread; this is not a process-wide native
+memory limit. The property is read at NIO initialization, so changes require a
+JVM restart.
+
+Two September 28 hosts reached about 120.7 GiB RSS on 123.1 GiB machines while
+their last reported heap pressure was below 50%. Profiling identified repeated
+34.6 MiB native allocations in `Util.getTemporaryDirectBuffer` during Undertow
+WebSocket writes. A read-only cache census found 4.97 GiB retained on the
+surviving host, including 4.96 GiB on its 32 IO threads. The newer host already
+held 1.67 GiB. The failed processes were unavailable for a cache census, so
+these measurements do not retrospectively assign every byte of their RSS.
+
+The container also sets `-XX:TrimNativeHeapInterval=60000` with
+`-Xlog:trimnative=info`. A dedicated JVM thread returns freed glibc pages to the
+OS every minute and logs reclamation and duration. A previous trim reclaimed
+4.6 GiB on the surviving process. This complements the cache cap: trimming
+cannot reclaim buffers that the NIO cache still owns. Heap sizing remains in
+`JAVA_OPTS`.
+
+After rollout, observe host RSS and `MemAvailable`, trim duration, request and
+reactivity latency, GC pressure, and large-message traffic through a full backup
+cycle. Backpressured large writes can allocate/free temporary buffers repeatedly;
+the cache cap therefore trades some allocation work for bounded retention.
+Local socket tests establish payload correctness and memory reclamation, not
+production latency bounds.
+
+`JAVA_OPTS` follows these defaults, allowing an explicit override. To restore
+the original NIO cache behavior, append
+`-Djdk.nio.maxCachedBufferSize=9223372036854775807`. To disable trimming, append
+`-XX:TrimNativeHeapInterval=0`. Roll the configuration and preserve the other JVM
+options, including heap settings.
+
+## Deployment
+
 The controller invokes the full down-policy ARN with cooldown; IAM restricts
 execution to the exact group ARN. The old low-CPU alarms have no direct scaling
 actions, so a missing controller keeps extra capacity running.