Describe the bug
dd-task-scheduler runs SymDBEnablement.startSymbolExtraction() on a 4-hour periodic schedule. For each loaded JAR, SymbolAggregator.scanJar() streams over every .class ZIP entry and calls parseJarEntry(), which obtains an entry stream but never closes it:
private void parseJarEntry(...) {
...
InputStream inputStream = jarFile.getInputStream(jarEntry); // acquires an Inflater
int readBytes;
baos.reset();
while ((readBytes = inputStream.read(buffer)) != -1) {
baos.write(buffer, 0, readBytes);
}
// inputStream is never closed — no try-with-resources, no finally
...
}Closing a ZipFile entry stream is what returns its Inflater to the ZipFile's internal inflater cache for reuse. Because the stream is never closed, every compressed entry allocates a brand-new Inflater via ZipFile.getInputStream() → ZipFile.getInflater() → new Inflater(). Inflater.init(Native Method) allocates a native zlib context via malloc. These are off-heap native allocations counted by the Linux cgroup memory controller but not by the JVM heap, so they are invisible to JVM heap metrics and not bounded by -XX:MaxDirectMemorySize. Each one lives until GC finalizes the abandoned stream or the JarFile is closed at the end of that jar's scan.
In a large application with hundreds of dependency JARs (thousands of .class entries per jar), this produces a sudden native memory burst of ~230 MB in 2–3 seconds, which — added to baseline JVM footprint — exceeds the container's cgroup memory limit and triggers an OOMKill.
The unclosed stream is still present in the latest release, v1.64.2 (verified against SymbolAggregator.java at the v1.64.2 tag, method parseJarEntry).
To Reproduce
Originally observed in production:
- Deploy a Java service with many dependency JARs (fat JAR or exploded classpath).
- Set a tight container memory limit (e.g. 512 Mi).
- Enable dd-trace-java v1.63.1+ (via Remote Configuration or
DD_DYNAMIC_INSTRUMENTATION_ENABLED=true).
- Wait ~4 hours for
AgentTaskScheduler$PeriodicTask to fire SymDBEnablement.startSymbolExtraction().
- Container OOMKills. Repeat every 4 hours as each OOMKill restarts the JVM and resets the scheduler.
Minimal reproduction (synthetic, attached)
A self-contained Docker repro is attached (symdb-oom-repro). It generates synthetic dependency jars (8,001 tiny .class entries each — no proprietary code), runs a hello-world main under dd-java-agent v1.64.2 with DD_DYNAMIC_INSTRUMENTATION_ENABLED=true, DD_SYMBOL_DATABASE_UPLOAD_ENABLED=true, and DD_INTERNAL_FORCE_SYMBOL_DATABASE_UPLOAD=true (so extraction fires at startup instead of waiting for Remote Config; no Datadog backend needed), in a 512 MiB container.
Verified results:
- Full repro: with the agent attached but the generated jars untouched, VmRSS is flat at ~143-145 MB for a 10-second baseline phase; within ~1-2 seconds of loading one class per generated jar (which queues them for the SymDB scan), the container is OOMKilled. Exit code 137,
OOMKilled: true. One run caught RSS pinned at ~520 MB (just under the 524,288 kB limit) for ~10s before the kill — matching the production trajectory.
- Control: identical app, limit, and agent with SymDB disabled — RSS stable at ~161 MB, survives indefinitely, exit 0.
- Agent-free mechanism demo (
InflaterLeakDemo.java, ~40 lines replicating parseJarEntry's exact I/O pattern): VmRSS climbs ~30 MB → ~378 MB scanning one 8,001-entry jar (~44 KB native per unclosed Inflater). Adding the single missing close() (-Dclose=true) keeps VmRSS flat at ~65 MB over the same jars.
Expected behavior
SymDB JAR scanning should not produce a native memory burst large enough to OOMKill a container that was otherwise stable. Closing each entry stream would let ZipFile reuse a single cached Inflater per jar, bounding native usage at roughly one zlib context instead of one per class file.
Environment
| Field |
Value |
| dd-trace-java |
v1.64.2 |
| JVM |
OpenJDK 25.492-b09 (Eclipse Temurin 8u492-b09) |
| Platform |
Linux (EKS), cgroups v2 |
| Container memory limit |
512 Mi |
| MaxDirectMemorySize |
64 MB (does NOT bound Inflater native allocs) |
| SymDB activation |
via Datadog Remote Configuration (no local env var set) |
Evidence — full allocating stack trace (captured via SIGQUIT at moment of burst)
"dd-task-scheduler" #5 daemon prio=5 os_prio=0 nid=0x48 runnable
java.lang.Thread.State: RUNNABLE
at java.util.zip.Inflater.init(Native Method) ← native malloc
at java.util.zip.Inflater.<init>(Inflater.java:103)
at java.util.zip.ZipFile.getInflater(ZipFile.java:477) ← cache empty: streams never closed
at java.util.zip.ZipFile.getInputStream(ZipFile.java:396)
- locked <0x00000000f5cea710> (a java.util.jar.JarFile)
at java.util.jar.JarFile.getInputStream(JarFile.java:471)
- locked <0x00000000f5cea710> (a java.util.jar.JarFile)
at com.datadog.debugger.symbol.SymbolAggregator.parseJarEntry(SymbolAggregator.java:279)
at com.datadog.debugger.symbol.SymbolAggregator.lambda$scanJar$2(SymbolAggregator.java:217)
at com.datadog.debugger.symbol.SymbolAggregator$$Lambda$498/199842660.accept(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at com.datadog.debugger.symbol.SymbolAggregator.scanJar(SymbolAggregator.java:216)
at com.datadog.debugger.symbol.SymDBEnablement.extractSymbolForLoadedClasses(SymDBEnablement.java:157)
at com.datadog.debugger.symbol.SymDBEnablement.startSymbolExtraction(SymDBEnablement.java:115)
at com.datadog.debugger.symbol.SymDBEnablement$$Lambda$492/1469801459.run(Unknown Source)
at datadog.trace.util.AgentTaskScheduler$RunnableTask.run(AgentTaskScheduler.java:63)
at datadog.trace.util.AgentTaskScheduler$RunnableTask.run(AgentTaskScheduler.java:58)
at datadog.trace.util.AgentTaskScheduler$PeriodicTask.run(AgentTaskScheduler.java:342)
at datadog.trace.util.AgentTaskScheduler$Worker.run(AgentTaskScheduler.java:294)
at java.lang.Thread.run(Thread.java:750)
Memory trajectory at OOMKill (from cgroup memory.current, 1s resolution)
Three kills at exact 4-hour intervals after JVM start. Each kill shows the same pattern:
Kill 1 (04:50:38Z): 243 MB → +60 → +69 → +78 → +73 → +88 MB → OOMKill @ ~471 MB
Kill 2 (08:51:12Z): 233 MB → +51 → +50 → +80 → +80 → +91 MB → OOMKill @ 456 MB
Kill 3 (12:50:57Z): 252 MB → +73 → +74 → +64 → +64 → +105 MB → OOMKill @ 497 MB
Net burst: ~228–245 MB native, in 2–3 seconds, on a service with only ~60 MB JVM heap in use.
Workaround
Set DD_DYNAMIC_INSTRUMENTATION_ENABLED=false (or DD_SYMBOL_DATABASE_UPLOAD_ENABLED=false) to disable SymDB. The service is stable without it.
Proposed fix
Close the entry stream in parseJarEntry() so ZipFile reuses its cached Inflater — one native zlib context per jar instead of one per class entry:
try (InputStream inputStream = jarFile.getInputStream(jarEntry)) {
int readBytes;
baos.reset();
while ((readBytes = inputStream.read(buffer)) != -1) {
baos.write(buffer, 0, readBytes);
}
}(parseFileEntry() for directory scans already uses try-with-resources; parseJarEntry() is the outlier.)
Describe the bug
dd-task-scheduler runs
SymDBEnablement.startSymbolExtraction()on a 4-hour periodic schedule. For each loaded JAR,SymbolAggregator.scanJar()streams over every.classZIP entry and callsparseJarEntry(), which obtains an entry stream but never closes it:Closing a
ZipFileentry stream is what returns itsInflaterto theZipFile's internal inflater cache for reuse. Because the stream is never closed, every compressed entry allocates a brand-newInflaterviaZipFile.getInputStream()→ZipFile.getInflater()→new Inflater().Inflater.init(Native Method)allocates a native zlib context via malloc. These are off-heap native allocations counted by the Linux cgroup memory controller but not by the JVM heap, so they are invisible to JVM heap metrics and not bounded by-XX:MaxDirectMemorySize. Each one lives until GC finalizes the abandoned stream or theJarFileis closed at the end of that jar's scan.In a large application with hundreds of dependency JARs (thousands of
.classentries per jar), this produces a sudden native memory burst of ~230 MB in 2–3 seconds, which — added to baseline JVM footprint — exceeds the container's cgroup memory limit and triggers an OOMKill.The unclosed stream is still present in the latest release, v1.64.2 (verified against
SymbolAggregator.javaat the v1.64.2 tag, methodparseJarEntry).To Reproduce
Originally observed in production:
DD_DYNAMIC_INSTRUMENTATION_ENABLED=true).AgentTaskScheduler$PeriodicTaskto fireSymDBEnablement.startSymbolExtraction().Minimal reproduction (synthetic, attached)
A self-contained Docker repro is attached (
symdb-oom-repro). It generates synthetic dependency jars (8,001 tiny.classentries each — no proprietary code), runs a hello-world main under dd-java-agent v1.64.2 withDD_DYNAMIC_INSTRUMENTATION_ENABLED=true,DD_SYMBOL_DATABASE_UPLOAD_ENABLED=true, andDD_INTERNAL_FORCE_SYMBOL_DATABASE_UPLOAD=true(so extraction fires at startup instead of waiting for Remote Config; no Datadog backend needed), in a 512 MiB container.Verified results:
OOMKilled: true. One run caught RSS pinned at ~520 MB (just under the 524,288 kB limit) for ~10s before the kill — matching the production trajectory.InflaterLeakDemo.java, ~40 lines replicatingparseJarEntry's exact I/O pattern): VmRSS climbs ~30 MB → ~378 MB scanning one 8,001-entry jar (~44 KB native per unclosedInflater). Adding the single missingclose()(-Dclose=true) keeps VmRSS flat at ~65 MB over the same jars.Expected behavior
SymDB JAR scanning should not produce a native memory burst large enough to OOMKill a container that was otherwise stable. Closing each entry stream would let
ZipFilereuse a single cachedInflaterper jar, bounding native usage at roughly one zlib context instead of one per class file.Environment
Evidence — full allocating stack trace (captured via SIGQUIT at moment of burst)
Memory trajectory at OOMKill (from cgroup memory.current, 1s resolution)
Three kills at exact 4-hour intervals after JVM start. Each kill shows the same pattern:
Net burst: ~228–245 MB native, in 2–3 seconds, on a service with only ~60 MB JVM heap in use.
Workaround
Set
DD_DYNAMIC_INSTRUMENTATION_ENABLED=false(orDD_SYMBOL_DATABASE_UPLOAD_ENABLED=false) to disable SymDB. The service is stable without it.Proposed fix
Close the entry stream in
parseJarEntry()soZipFilereuses its cachedInflater— one native zlib context per jar instead of one per class entry:(
parseFileEntry()for directory scans already uses try-with-resources;parseJarEntry()is the outlier.)