I have this declarative pipeline to test Jenkins behaviour:

pipeline {
    agent {
        docker {
            label 'standard'
            image 'debian:stable'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
        }
    }

    stages {
        stage('On root agent') {
            steps {
                sh 'bash --version'
            }
        }

        stage('On other agent') { // this never runs
            agent any

            steps {
              sh '''
                bash --version 
              '''
            }
        }
    }
}

Or this scripted pipeline version

node {
  withDockerContainer(image: 'debian:stable', args: '-v /var/run/docker.sock:/var/run/docker.sock') {
    stage("first") {
      sh 'bash --version'
    }

    node {
      stage("second") {
        sh "bash --version"
      }
    }
  }
}

In this pipeline, we want the second stage to be run on a different agent than the root, docker, one. But this second-stage step never runs, at least with the Jenkins setup my company is running with.

Some logs:

Pull request #17677 updated
...
[Pipeline] Start of Pipeline
[Pipeline] milestone
Trying to pass milestone 51
[Pipeline] node
Running on jenkins-agent-n2-custom-8-32768-hswu4b
 in /tmp/....
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: git
using credential github-blablacarbot-ssh
Fetching changes from the remote Git repository
Fetching without tags
...
[GitHub Checks] GitHub check (name: Jenkins, status: in_progress) has been published.
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . debian:stable
.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
jenkins-agent-n2-custom-8-32768-hswu4b does not seem to be running inside a container
$ docker run -t -d -u 0:0 -v /var/run/docker.sock:/var/run/docker.sock -w ... debian:stable cat
$ docker top c67eb42eef77ae16fce698c21eabbcb14d8e6591130429ce2413bd57dc7e02f8 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (On root agent)
[Pipeline] sh
+ bash --version
GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (On other agent)
[Pipeline] node
Running on jenkins-agent-n2-custom-8-32768-8cbh5v
 in /tmp...
[Pipeline] {
[Pipeline] checkout
The recommended git tool is: git
...
[GitHub Checks] GitHub check (name: Jenkins, status: in_progress) has been published.
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
process apparently never started in /tmp/...
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1 c67eb42eef77ae16fce698c21eabbcb14d8e6591130429ce2413bd57dc7e02f8
$ docker rm -f --volumes c67eb42eef77ae16fce698c21eabbcb14d8e6591130429ce2413bd57dc7e02f8
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
[GitHub Checks] GitHub check (name: Jenkins, status: completed) has been published.

GitHub has been notified of this commit’s build result

Finished: FAILURE

Why can't the process in the 2nd agent start?

In a real-life scenario, I'm exploring this option for performance purposes: the majority of my stages can run on the same agent (so let's use the root one so as not to have to pop a new one for each stage), but some heavy stage can benefit from running isolated in another agent.

I have this declarative pipeline to test Jenkins behaviour:

pipeline {
    agent {
        docker {
            label 'standard'
            image 'debian:stable'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
        }
    }

    stages {
        stage('On root agent') {
            steps {
                sh 'bash --version'
            }
        }

        stage('On other agent') { // this never runs
            agent any

            steps {
              sh '''
                bash --version 
              '''
            }
        }
    }
}

Or this scripted pipeline version

node {
  withDockerContainer(image: 'debian:stable', args: '-v /var/run/docker.sock:/var/run/docker.sock') {
    stage("first") {
      sh 'bash --version'
    }

    node {
      stage("second") {
        sh "bash --version"
      }
    }
  }
}

In this pipeline, we want the second stage to be run on a different agent than the root, docker, one. But this second-stage step never runs, at least with the Jenkins setup my company is running with.

Some logs:

Pull request #17677 updated
...
[Pipeline] Start of Pipeline
[Pipeline] milestone
Trying to pass milestone 51
[Pipeline] node
Running on jenkins-agent-n2-custom-8-32768-hswu4b
 in /tmp/....
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: git
using credential github-blablacarbot-ssh
Fetching changes from the remote Git repository
Fetching without tags
...
[GitHub Checks] GitHub check (name: Jenkins, status: in_progress) has been published.
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . debian:stable
.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
jenkins-agent-n2-custom-8-32768-hswu4b does not seem to be running inside a container
$ docker run -t -d -u 0:0 -v /var/run/docker.sock:/var/run/docker.sock -w ... debian:stable cat
$ docker top c67eb42eef77ae16fce698c21eabbcb14d8e6591130429ce2413bd57dc7e02f8 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (On root agent)
[Pipeline] sh
+ bash --version
GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (On other agent)
[Pipeline] node
Running on jenkins-agent-n2-custom-8-32768-8cbh5v
 in /tmp...
[Pipeline] {
[Pipeline] checkout
The recommended git tool is: git
...
[GitHub Checks] GitHub check (name: Jenkins, status: in_progress) has been published.
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
process apparently never started in /tmp/...
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1 c67eb42eef77ae16fce698c21eabbcb14d8e6591130429ce2413bd57dc7e02f8
$ docker rm -f --volumes c67eb42eef77ae16fce698c21eabbcb14d8e6591130429ce2413bd57dc7e02f8
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
[GitHub Checks] GitHub check (name: Jenkins, status: completed) has been published.

GitHub has been notified of this commit’s build result

Finished: FAILURE

Why can't the process in the 2nd agent start?

In a real-life scenario, I'm exploring this option for performance purposes: the majority of my stages can run on the same agent (so let's use the root one so as not to have to pop a new one for each stage), but some heavy stage can benefit from running isolated in another agent.

• jenkins
• jenkins-pipeline

1 Answer 1

I remember running into the same problem and even thinking that it was working some time before. And it turned out that (at least according to the documentation) for running a pipeline on different agents Jenkins only offers two options:

• using agent none on the top level:

  Defining agent none at the top-level of the Pipeline ensures that an Executor will not be assigned unnecessarily. Using agent none also forces each stage section to contain its own agent section.

• using some agent, but not with a docker { ... } block on the top level. This could be combined with reuseNode on stages' agent blocks to optimize the resource allocation and performance:

  reuseNode

  A boolean, false by default. If true, run the container on the node specified at the top-level of the Pipeline, in the same workspace, rather than on a new node entirely.

  This option is valid for docker and dockerfile, and only has an effect when used on an agent for an individual stage.

Also, I would recommend against using agent any as it makes the build environment unknown and can eventually lead to errors and not reproducible builds.

So, your pipeline could look like this:

pipeline {
    agent none

    stages {
      stage('On standard agent') {
        agent {     
          docker {
            label 'standard'
            image 'debian:stable'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
          }
        }
        steps {
          sh 'bash --version'
        }
      }

      stage('On other agent') {
        agent {     
          docker {
            label 'other'
            image 'some-other-image:1.0.0'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
          }
        }
        steps {
          sh '''
            bash --version 
          '''
        }
      }
    }
}

Or like this:

pipeline {
    agent {
      label `standard`
    }
    stages {
      stage('On standard agent') {
        agent {     
          docker {
            label 'standard'
            image 'debian:stable'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
            reuseNode true
          }
        }
        // It would make sense to define nested stages instead of steps here
        steps {
          sh 'bash --version'
        }
      }

      stage('On other agent') {
        agent {     
          docker {
            label 'other'
            image 'some-other-image:1.0.0'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
          }
        }
        steps {
          sh '''
            bash --version 
          '''
        }
      }
    }
}