MdcPropagationContext throwing ArrayIndexOutOfBoundsException under high TPS

[srinunagulapalli]

Expected Behavior

No response

Actual Behaviour

We are using Micronaut MdcPropagationContext for our propagation context. We are initializing it and propagating it in the first server filter.
We also update this by adding new key-value pairs and also update the existing ones from multiple places mentioned below.

1. In the HttpServerFilter
2. In the HttpClientFilter

Our application has multiple declarative API clients to make calls to integrations and we are updating only a single key in the HttpClientFilter. So, when multiple integration calls are made, the same key will be updated by multiple filters within a single server request to the API.
This works perfectly fine under low TPS. But, when the application is subjected to high TPS, we are seeing a strange behavior where in it is throwing java.lang.ArrayIndexOutOfBoundsException: Index 21 out of bounds for length 21.
But, When we moved updating that particular key's value in the MdcPropagationContext from the HttpClientFilter to the HttpServerFilter, we did not see any exception thrown.
We suspected this could be a race condition and also tried making the update of the key's value in the MdcPropagationContext a synchronized operation, but we still observe the same behavior.

We would definitely appreciate any help to find out what is causing the issue and how to fix it.

Steps To Reproduce

No response

Environment Information

No response

Example Application

No response

Version

4.7.5

[dstepanov]

At least post a full stacktrace.
The best would be to have a reproducer.

[srinunagulapalli]

Here is the stack trace of the two exceptions being thrown.

Exception in thread "DefaultDispatcher-worker-36" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[Dispatchers.Default, Continuation at com.example.demo.$executeAsync$1.invokeSuspend()@]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:130) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:107) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704) Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [io.micronaut.core.async.propagation.MicronautPropagatedContext@7662d2c4, DeferredCoroutine{Active}@5881018c, Dispatchers.Default] Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 19 out of bounds for length 19 at org.apache.logging.log4j.internal.map.UnmodifiableArrayBackedMap.add(UnmodifiableArrayBackedMap.java:196) at org.apache.logging.log4j.internal.map.UnmodifiableArrayBackedMap.copyAndPutAll(UnmodifiableArrayBackedMap.java:297) at org.apache.logging.log4j.spi.DefaultThreadContextMap.putAll(DefaultThreadContextMap.java:83) at org.apache.logging.log4j.ThreadContext.putAll(ThreadContext.java:277) at org.apache.logging.slf4j.Log4jMDCAdapter.setContextMap(Log4jMDCAdapter.java:56) at org.slf4j.MDC.setContextMap(MDC.java:246) at io.micronaut.context.propagation.slf4j.MdcPropagationContext.setCurrent(MdcPropagationContext.java:54) at io.micronaut.context.propagation.slf4j.MdcPropagationContext.updateThreadContext(MdcPropagationContext.java:41) at io.micronaut.context.propagation.slf4j.MdcPropagationContext.updateThreadContext(MdcPropagationContext.java:31) at io.micronaut.core.propagation.PropagatedContextImpl.updateThreadState(PropagatedContextImpl.java:227) at io.micronaut.core.propagation.PropagatedContextImpl.propagate(PropagatedContextImpl.java:206) at io.micronaut.core.async.propagation.MicronautPropagatedContext.updateThreadContext(MicronautPropagatedContext.kt:36) at io.micronaut.core.async.propagation.MicronautPropagatedContext.updateThreadContext(MicronautPropagatedContext.kt:30) at kotlinx.coroutines.internal.ThreadContextKt.updateThreadContext(ThreadContext.kt:74) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:223) ... 4 more

---

Exception in thread "DefaultDispatcher-worker-30" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[Dispatchers.Default, Continuation at com.example$executeAsync$1.invokeSuspend()@]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:130) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:107) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704) Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [io.micronaut.core.async.propagation.MicronautPropagatedContext@11cecb19, DeferredCoroutine{Active}@, Dispatchers.Default] Caused by: java.util.ConcurrentModificationException at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1579) at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1612) at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1610) at org.apache.logging.log4j.internal.map.UnmodifiableArrayBackedMap.copyAndPutAll(UnmodifiableArrayBackedMap.java:288) at org.apache.logging.log4j.spi.DefaultThreadContextMap.putAll(DefaultThreadContextMap.java:83) at org.apache.logging.log4j.ThreadContext.putAll(ThreadContext.java:277) at org.apache.logging.slf4j.Log4jMDCAdapter.setContextMap(Log4jMDCAdapter.java:56) at org.slf4j.MDC.setContextMap(MDC.java:246) at io.micronaut.context.propagation.slf4j.MdcPropagationContext.setCurrent(MdcPropagationContext.java:54) at io.micronaut.context.propagation.slf4j.MdcPropagationContext.updateThreadContext(MdcPropagationContext.java:41) at io.micronaut.context.propagation.slf4j.MdcPropagationContext.updateThreadContext(MdcPropagationContext.java:31) at io.micronaut.core.propagation.PropagatedContextImpl.updateThreadState(PropagatedContextImpl.java:227) at io.micronaut.core.propagation.PropagatedContextImpl.propagate(PropagatedContextImpl.java:206) at io.micronaut.core.async.propagation.MicronautPropagatedContext.updateThreadContext(MicronautPropagatedContext.kt:36) at io.micronaut.core.async.propagation.MicronautPropagatedContext.updateThreadContext(MicronautPropagatedContext.kt:30) at kotlinx.coroutines.internal.ThreadContextKt.updateThreadContext(ThreadContext.kt:74) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:223) ... 4 more

[dstepanov]

Looks like log4j bug. Can you try to use logback?