I'm encountering intermittent failures with a Jenkins job running on a remote node. The job typically takes 2-3 hours to complete, but fails unpredictably with an "Unexpected termination of the channel" error.
Environment:
- Jenkins version: 2.462.3
- Remote node OS: SUSE Linux Enterprise Server 15 SP4
- Remote node JAVA version:
openjdk 17.0.14 2025-01-21
OpenJDK Runtime Environment (build 17.0.14+7-suse-150400.3.51.1-x8664)
OpenJDK 64-Bit Server VM (build 17.0.14+7-suse-150400.3.51.1-x8664, mixed mode, sharing)
Failure Pattern:
- Failures occur at random points during execution.
- Sometimes fails at job start, other times mid-execution.
Error Log (Build Log):
Started by timer
Running as SYSTEM
Building remotely on remote-node-xxx (worker-xxx) in workspace /data/jenkins/workspace/job-name
FATAL: java.io.IOException: Unexpected termination of the channel
java.io.EOFException
at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
at java.base/java.io.ObjectInputStream.readStreamHeader(Unknown Source)
at java.base/java.io.ObjectInputStream.<init>(Unknown Source)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to CCloud-de-i588324
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1826)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:1041)
at hudson.FilePath.act(FilePath.java:1229)
at hudson.FilePath.act(FilePath.java:1218)
at hudson.FilePath.mkdirs(FilePath.java:1409)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1241)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1894)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:446)
Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:346)
at hudson.remoting.Channel.terminate(Channel.java:1122)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Finished: FAILURE
What I've tried:
- Upgrade JAVA from 11 to 17
- Restart remote node
- Check
sshd
log but not helped (no error found)
Questions:
- What could cause these sporadic channel termination errors?
- How can I effectively debug the connection issues between Jenkins master and the remote node?
- Are there specific logs I should be monitoring on either the master or remote node to identify the root cause?
Any guidance on troubleshooting approaches or potential solutions would be greatly appreciated.