Problem: I'm encountering an IllegalReferenceCountException in production. The issue occurs when I attempt to log the original response body before deserialization fails.
- how can I debug this error?
- could you explain what's the error?
Env
- Spring WebFlux (Spring Boot 3.4.3)
- Netty + Reactor + Coroutine(Kotlin)
- Kotlin
Suspicious Code The following custom Decoder is intended to log the response body when deserialization fails.
@Bean
fun webClientCustomizer(): WebClientCustomizer {
return WebClientCustomizer { webClientBuilder: WebClient.Builder ->
val delegate = Jackson2JsonDecoder(JacksonMapperFactory.createObjectMapper())
val loggingJackson2JsonDecoder = LoggingJacksonDecoder(delegate)
webClientBuilder.clientConnector(ReactorClientHttpConnector(HttpClient.create()))
.exchangeStrategies(
ExchangeStrategies.builder().codecs {
it.defaultCodecs().jackson2JsonDecoder(loggingJackson2JsonDecoder)
}.build(),
)
}
}
private class LoggingJacksonDecoder(
private val delegate: Jackson2JsonDecoder,
) : Jackson2JsonDecoder(delegate.objectMapper) {
override fun decode(dataBuffer: DataBuffer, targetType: ResolvableType, mimeType: MimeType?, hints: MutableMap<String, Any>?): Any {
try {
DataBufferUtils.retain(dataBuffer)
return super.decode(dataBuffer, targetType, mimeType, hints)
} catch (e: DecodingException) {
dataBuffer.readPosition(0)
val sourceString = dataBuffer.toString(Charsets.UTF_8)
throw DecodingException("JacksonDecodingError. source=$sourceString", e)
} finally {
DataBufferUtils.release(dataBuffer)
}
}
}
Stack Trace
An exception has been observed post termination, use DEBUG level to see the full stack: ioty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
[e7001a4d-1, L:/***.**.**.**:43618 - R:secret/**.*.***.***:80] Error was received while reading the incoming data. The connection will be closed.
ioty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at ioty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
at ioty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:148)
at ioty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
at ioty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)
at ioty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
at reactorty.channel.FluxReceive.onInboundNext(FluxReceive.java:380)
at reactorty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:435)
at reactorty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:841)
at reactorty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:115)
at ioty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at ioty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at ioty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at ioty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
at ioty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at ioty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at ioty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at ioty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at ioty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at ioty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at ioty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at ioty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at ioty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at ioty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at ioty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at ioty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at ioty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at ioty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at ioty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
at ioty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
at ioty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
at ioty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at ioty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at ioty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
try bufferSize exceed problem -> already adjust and the error doesn't occurred.
expected RefCountError doesn't occurred.