diff --git a/server/docker-compose.yml b/server/docker-compose.yml index 035f0700c6..0a15db270d 100644 --- a/server/docker-compose.yml +++ b/server/docker-compose.yml @@ -16,7 +16,8 @@ services: # `web` depends_on `vector` so that we don't lose logs on docker compose down depends_on: - vector - command: sh -c 'java $${JAVA_OPTS} --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -Djdk.attach.allowAttachSelf -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:StartFlightRecording=disk=true,maxsize=1G,name=instant,filename=instant_$(date +%s%N).jfr -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=localhost -server -jar target/instant-standalone.jar' + # Bound cached NIO copy buffers and return freed native pages; JAVA_OPTS can override these defaults. + command: sh -c 'java -Djdk.nio.maxCachedBufferSize=131072 -XX:TrimNativeHeapInterval=60000 -Xlog:trimnative=info $${JAVA_OPTS} --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -Djdk.attach.allowAttachSelf -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:StartFlightRecording=disk=true,maxsize=1G,name=instant,filename=instant_$(date +%s%N).jfr -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=localhost -server -jar target/instant-standalone.jar' vector: # Mirrored from timberio/vector:0.56.0-alpine into our private ECR so diff --git a/server/infra/README.md b/server/infra/README.md index 2b6de5628d..66604814b9 100644 --- a/server/infra/README.md +++ b/server/infra/README.md @@ -41,6 +41,53 @@ does not: it reports one impacted instance for much of the day without a request-level cause, and the group's ELB health check already replaces instances that fail. +## Native memory + +The API sets `-Djdk.nio.maxCachedBufferSize=131072`. When NIO writes a heap +buffer to a socket, it copies the data into a temporary native buffer. Corretto +26 caches these buffers on long-lived platform/carrier threads without a size +limit by default. Undertow's gathering writes can leave many large buffers in +each IO thread's cache after the WebSocket messages finish. These buffers are +outside the Java heap and are excluded from the direct-buffer MXBean and +`MaxDirectMemorySize` accounting. + +The 128 KiB cap applies to each cached buffer. It preserves reuse of ordinary +Java socket buffers while freeing larger temporary buffers after I/O. It does +not limit message sizes or the memory needed by writes in progress. The cache +can hold up to 1,024 entries per thread; this is not a process-wide native +memory limit. The property is read at NIO initialization, so changes require a +JVM restart. + +Two September 28 hosts reached about 120.7 GiB RSS on 123.1 GiB machines while +their last reported heap pressure was below 50%. Profiling identified repeated +34.6 MiB native allocations in `Util.getTemporaryDirectBuffer` during Undertow +WebSocket writes. A read-only cache census found 4.97 GiB retained on the +surviving host, including 4.96 GiB on its 32 IO threads. The newer host already +held 1.67 GiB. The failed processes were unavailable for a cache census, so +these measurements do not retrospectively assign every byte of their RSS. + +The container also sets `-XX:TrimNativeHeapInterval=60000` with +`-Xlog:trimnative=info`. A dedicated JVM thread returns freed glibc pages to the +OS every minute and logs reclamation and duration. A previous trim reclaimed +4.6 GiB on the surviving process. This complements the cache cap: trimming +cannot reclaim buffers that the NIO cache still owns. Heap sizing remains in +`JAVA_OPTS`. + +After rollout, observe host RSS and `MemAvailable`, trim duration, request and +reactivity latency, GC pressure, and large-message traffic through a full backup +cycle. Backpressured large writes can allocate/free temporary buffers repeatedly; +the cache cap therefore trades some allocation work for bounded retention. +Local socket tests establish payload correctness and memory reclamation, not +production latency bounds. + +`JAVA_OPTS` follows these defaults, allowing an explicit override. To restore +the original NIO cache behavior, append +`-Djdk.nio.maxCachedBufferSize=9223372036854775807`. To disable trimming, append +`-XX:TrimNativeHeapInterval=0`. Roll the configuration and preserve the other JVM +options, including heap settings. + +## Deployment + The controller invokes the full down-policy ARN with cooldown; IAM restricts execution to the exact group ARN. The old low-CPU alarms have no direct scaling actions, so a missing controller keeps extra capacity running.