Fix: Spring 7 Webflux NoSuchElementError on HttpHeaders#entrySet()

[mtomik]

What does this PR do?

Since the original PR is stuck for some time,
I created another one with the same commits + a bit better solution for the core of this issue.

Also when I was running these newer tests I found one more issue:

Spring7ServerFunctionalInstrumentationTest#dispatchError test was failing due to TracedSubscriber not removing the reference of the subscription on cancel() from contextMap. so the solution might be the CancellationAwareSubscription that is just the wrapper calling the discardIf() after cancel.

Is it possible that the issue was there even with older versions of reactor (from older Spring version), but now in that Spring 7 ( reactor 3.8.6 ) it was always failing.

Checklist

• This is an enhancement of existing features, or a new feature in existing plugins
  • I have updated CHANGELOG.next-release.md
  • I have added tests that prove my fix is effective or that my feature works
  • Added an API method or config option? Document in which version this will be introduced
  • I have made corresponding changes to the documentation
• This is a bugfix
  • I have updated CHANGELOG.next-release.md
  • I have added tests that would fail without this fix
• This is a new plugin
  • I have updated CHANGELOG.next-release.md
  • My code follows the style guidelines of this project
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated supported-technologies.md
  • Added an API method or config option? Document in which version this will be introduced
  • Added an instrumentation plugin? Describe how you made sure that old, non-supported versions are not instrumented by accident.
• This is something else
  • I have updated CHANGELOG.next-release.md

[cla-checker-service]

💚 CLA has been signed

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mtomik]

thanks for the review 👍

the failed build was due to incompatibility of that sub cancel with java 7. I reworked that using Instrumentation -SubscriptionCancelInstrumentation that will call the discardIf on cancelled subscription

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[jackshirazi]

run docs-build

[jackshirazi]

@mtomik are you able to complete the remaining suggested feedback, I was waiting for this to do a release but if it's not going to be completed soon I'll push this to the next release

[mtomik]

I found there one more issue. cancel of the subscription was called before collecting all the headers could be finished and that could cause this exception:

java.lang.NullPointerException: null
at java.util.Objects.requireNonNull(Objects.java:220)
at org.springframework.http.HttpHeaders$CaseInsensitiveEntrySet$CaseInsensitiveEntry.getValue(HttpHeaders.java:2334)
at org.springframework.http.HttpHeaders$CaseInsensitiveEntrySet$CaseInsensitiveEntry.getValue(HttpHeaders.java:2319)
at org.springframework.http.HttpHeaders.lambda$forEach$0(HttpHeaders.java:2030)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.springframework.http.HttpHeaders.forEach(HttpHeaders.java:2030)
at co.elastic.apm.agent.springwebflux.WebfluxHelper.copyHeaders(WebfluxHelper.java:320)
at co.elastic.apm.agent.springwebflux.WebfluxHelper.fillResponse(WebfluxHelper.java:310)
at co.elastic.apm.agent.springwebflux.WebfluxHelper.endTransaction(WebfluxHelper.java:182)
at co.elastic.apm.agent.springwebflux.TransactionAwareSubscriber.cancelTransaction(TransactionAwareSubscriber.java:214)
at co.elastic.apm.agent.springwebflux.TransactionAwareSubscriber.cancel(TransactionAwareSubscriber.java:91)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:658)

so I fixed it with the correct order + added the same error handling that we got for status code there.

@jackshirazi that should be all from me. if the review of my last changes might block your release, no problem. lets move it to another one then.

[SylvainJuge]

I just checked the matching performance in the test standard output, and it's totally acceptable:

   Advice name                                                                                    Type ns            Method ns         
   ExecutorInstrumentation$ExecutorRunnableInstrumentation                                             91,301,814         13,214,102   
   JavaxFilterInstrumentation                                                                          46,153,889            616,531   
   InitServiceNameInstrumentation$JavaxInitServiceNameInstrumentation                                  32,963,871          1,479,452   
   SubscriptionCancelInstrumentation                                                                   25,883,477          8,483,966   
   JavaxServletInstrumentation                                                                         30,973,489            656,024   
   JavaxAsyncInstrumentation$JavaxStartAsyncInstrumentation                                            27,429,023            660,029   
   ExecutorInstrumentation$ExecutorInvokeAnyAllInstrumentation                                         14,844,983         11,137,511   
   RunnableCallableForkJoinTaskInstrumentation                                                         16,824,272          1,212,259   
   ExecutorInstrumentation$ExecutorCallableInstrumentation                                              9,833,095          3,811,010   
   ServletContainerInitializerServiceNameInstrumentation$JavaxInitServiceNameInstrumentation           12,067,723                  0   
   ExecutorInstrumentation$ForkJoinPoolInstrumentation                                                  9,219,421          2,371,271   
   ForkJoinTaskInstrumentation                                                                          8,114,142            558,088   
   JavaxFilterChainInstrumentation                                                                      6,956,189            147,314   
   JavaxRequestStreamRecordingInstrumentation                                                           6,463,642            466,558   
   DispatcherHandlerInstrumentation                                                                     6,393,830            468,880   
   HandlerAdapterInstrumentation                                                                        4,913,156          1,504,565   
   ReactorInstrumentation                                                                               5,121,863          1,232,776   
   JakartaServletVersionInstrumentation$JakartaInit                                                     6,340,990                  0   
   JavaxServletVersionInstrumentation$JavaxInit                                                         4,625,790            579,637   
   JavaxAsyncInstrumentation$JavaxAsyncContextInstrumentation                                           4,688,562            190,258   
   JakartaServletVersionInstrumentation$JakartaService                                                  4,754,681                  0   
   JavaxServletVersionInstrumentation$JavaxService                                                      4,353,296            351,680   
   JakartaServletInstrumentation                                                                        3,074,576                  0   
   WarmupInstrumentation                                                                                2,764,028                  0   
   BodyInsertersInstrumentation                                                                         1,995,783            722,432   
   JakartaFilterChainInstrumentation                                                                    2,679,905                  0   
   SystemSingleEnvVariablesInstrumentation                                                              2,247,976            323,788   
   JakartaAsyncInstrumentation$JakartaAsyncContextInstrumentation                                       2,485,043                  0   
   JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation                                         2,409,624                  0   
   InitServiceNameInstrumentation$JakartaInitServiceNameInstrumentation                                 2,360,988                  0   
   JakartaFilterInstrumentation                                                                         2,317,672                  0   
   ServletContainerInitializerServiceNameInstrumentation$JakartaInitServiceNameInstrumentation          2,218,153                  0   
   JakartaRequestStreamRecordingInstrumentation                                                         2,167,849                  0   
   SystemAllEnvVariablesInstrumentation                                                                 1,637,550            243,327   

This instrumentation uses 3 classes, and none of them is an outlier in the table above.

• HandlerAdapterInstrumentation
• DispatcherHandlerInstrumentation
• BodyInsertersInstrumentation

[jackshirazi]

I think there are a bunch in reactor that don't use those in the names. Maybe add that as a special case?

Suggested change

	return nameContains("Subscriber").or(nameContains("Subscription"));
	return nameStartsWith("reactor.")
	.or(nameContains("Subscriber"))
	.or(nameContains("Subscription"));

[SylvainJuge]

Asking an LLM to find known implementations of Subscription yields the following list, so reactor. is a good prefix, we might also add io.reactivex. and io.smallrye.mutiny. but I'm definitely not familiar with any of those reactor frameworks. Here it would not hurt to have a few false positives rather than lots of false positives creating lots of useless overhead, so this should be fine as a first step, and we can refine further if needed.

reactor.core.publisher.FluxInterval$IntervalRunnable
reactor.core.publisher.FluxSwitchMap$SwitchMapInner
reactor.core.publisher.SinkManyUnicastNoBackpressure
io.reactivex.internal.operators.flowable.FlowableAmb$AmbCoordinator
io.reactivex.internal.operators.flowable.FlowablePublishMulticast$OutputCanceller
io.smallrye.mutiny.operators.multi.MultiFlatMapOp$FlatMapInner
io.smallrye.mutiny.operators.multi.MultiCombineLatestOp$CombineLatestCoordinator
io.smallrye.mutiny.operators.multi.MultiGroupByOp$State
io.smallrye.mutiny.operators.multi.MultiZipOp$ZipCoordinator
io.smallrye.mutiny.operators.multi.processors.UnicastProcessor
io.smallrye.mutiny.operators.multi.MultiOperatorProcessor
io.smallrye.mutiny.operators.multi.builders.IntervalMulti$IntervalRunnable
org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier
org.springframework.http.server.reactive.ChannelSendOperator$WriteCompletionBarrier

[mtomik]

one drawback of this additional name matching is increased instruction time spent. So I run the numbers again:

current filter

2026-09-04 00:03:18,209 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 257,422,186ns
| Advice name                                        | Type ns         | Method ns       |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      46,278,675 |       5,203,226 |
| SubscriptionCancelInstrumentation                  |      17,952,170 |      11,488,096 |
| JakartaFilterInstrumentation                       |      24,123,333 |               0 |

OR starts with "reactor."

2026-09-03 23:59:18,358 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 282,551,157ns
| Advice name                                        | Type ns         | Method ns       |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      48,814,752 |       6,779,428 |
| SubscriptionCancelInstrumentation                  |      42,561,353 |       8,489,875 |
| JakartaFilterInstrumentation                       |      23,363,543 |               0 |
| JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation |      20,613,139 |               0 |

OR all 3 starts with you mentioned

2026-09-04 00:02:17,662 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 289,889,601ns
| Advice name                                        | Type ns         | Method ns       |
| SubscriptionCancelInstrumentation                  |      47,053,567 |      14,306,418 |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      48,076,345 |       5,605,395 |
| JakartaFilterInstrumentation                       |      24,638,337 |               0 |
| JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation |      22,838,079 |               0 |

if we want to optimize the filter even further - with something like just nameEndsWith("Subscriber") we cover most of them with just this overhead.

2026-09-04 00:12:48,681 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 259,733,930ns
| Advice name                                        | Type ns         | Method ns       |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      62,550,712 |       5,650,035 |
| JakartaFilterInstrumentation                       |      24,774,004 |               0 |
| SubscriptionCancelInstrumentation                  |      14,498,788 |       9,413,708 |
| JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation |      20,837,849 |               0 |

the nameContains("Subscription") was there just to match it even when someone did his own implementation - bigger chance that we catch it

nameEndsWith("Subscriber") OR all 3 your package prefixes

2026-09-04 00:09:40,953 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 281,649,260ns
| Advice name                                        | Type ns         | Method ns       |
| SubscriptionCancelInstrumentation                  |      38,772,826 |      14,850,740 |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      48,154,398 |       5,293,119 |
| JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation |      24,675,265 |               0 |
| JakartaFilterInstrumentation                       |      23,984,791 |               0 |

the thing is that if we miss some of them, we just rely on GC to clean it up later. so it just about finding that balance I guess 😄 but at the end, every one of these will be better than it was before, where we missed all of them