I am using snakemake v 8.28 with the snakemake-executor-plugin-slurm to submit my workflow to a slurm scheduler. Each instance of a rule is submitted as it's own job to the scheduler. By default, the slurm job name is set to a large hash string that is uninformative. I would like to set the --job-name parameter for the underlying sbatch or srun calls to be the rule name. Or even better yet, a combo of the rule and sample name. I have been experimenting with the following workflow profile set in profiles/default/config.yaml:

#Cluster configuration
executor: slurm

default-resources:
    slurm_partition: "mypartition"
    slurm_account: "myaccount"
    cpus_per_task: 2
    mem_mb_per_cpu: 7500
    runtime: 30
    slurm_extra: "--job-name={rule}"

This returns an error:

Error:
  WorkflowError:
    Failed to evaluate resources value '--job-name={rulename}'.
        String arguments may need additional quoting. E.g.: --default-resources "tmpdir='/home/user/tmp'" or --set-resources "somerule:someresource='--nice=100'". This also holds for setting resources inside of a profile, where you might have to enclose them in single and double quotes, i.e. someresource: "'--nice=100'".

I have tried various combinations of quoting, but get the same error.

When using an older version of snakemake (~7.19) I was able to do this using the --cluster parameter with a PBS scheduler:

snakemake \
--cluster "qsub \
-q myqueue -l select=1:ncpus={threads}:mem=16GB \
-N {rule} \
-o {log} -j oe"

Is this possible with the newer version of snakemake and/or using a slurm scheduler?

It would be even more desirable to include my sample wildcard, something like this:

default-resources:
    slurm_partition: "mypartition"
    slurm_account: "myaccount"
    cpus_per_task: 2
    mem_mb_per_cpu: 7500
    runtime: 30
    slurm_extra: "--job-name={sample}.{rule}"

I am using snakemake v 8.28 with the snakemake-executor-plugin-slurm to submit my workflow to a slurm scheduler. Each instance of a rule is submitted as it's own job to the scheduler. By default, the slurm job name is set to a large hash string that is uninformative. I would like to set the --job-name parameter for the underlying sbatch or srun calls to be the rule name. Or even better yet, a combo of the rule and sample name. I have been experimenting with the following workflow profile set in profiles/default/config.yaml:

#Cluster configuration
executor: slurm

default-resources:
    slurm_partition: "mypartition"
    slurm_account: "myaccount"
    cpus_per_task: 2
    mem_mb_per_cpu: 7500
    runtime: 30
    slurm_extra: "--job-name={rule}"

This returns an error:

Error:
  WorkflowError:
    Failed to evaluate resources value '--job-name={rulename}'.
        String arguments may need additional quoting. E.g.: --default-resources "tmpdir='/home/user/tmp'" or --set-resources "somerule:someresource='--nice=100'". This also holds for setting resources inside of a profile, where you might have to enclose them in single and double quotes, i.e. someresource: "'--nice=100'".

I have tried various combinations of quoting, but get the same error.

When using an older version of snakemake (~7.19) I was able to do this using the --cluster parameter with a PBS scheduler:

snakemake \
--cluster "qsub \
-q myqueue -l select=1:ncpus={threads}:mem=16GB \
-N {rule} \
-o {log} -j oe"

Is this possible with the newer version of snakemake and/or using a slurm scheduler?

It would be even more desirable to include my sample wildcard, something like this:

default-resources:
    slurm_partition: "mypartition"
    slurm_account: "myaccount"
    cpus_per_task: 2
    mem_mb_per_cpu: 7500
    runtime: 30
    slurm_extra: "--job-name={sample}.{rule}"

• snakemake
• snakemake-slurm

1 Answer 1

The slurm executor plugin should respect the regular snakemake --jobname flag without the need to define a slurm_extra resource. If it does not, I'd say that is a bug in the slurm executor.

As a Slurm user I've found that the Snakemake DRMAA executor is generally better behaved than the slurm executor. And you can definitely set the job name. Here is my config:

default-resources:
- tmpdir='/lustre/tmp'
- time_h=24
- mem_mb=6000
- n_cpus=1
- extra_slurm_flags=''
drmaa: -p standard --qos=edgen --account=edg01
  --time={resources.time_h}:00:00 --mem={resources.mem_mb} --mincpus={resources.n_cpus}
  {resources.extra_slurm_flags} -e slurm_output/{rule}.%A.err -o slurm_output/{rule}.%A.out
jobname: '{rulename}.snakejob.{jobid}.sh'
jobscript: /lustre/software/snakemake/profile/snakemake_jobscript.sh
jobs: 200
local-cores: 4
latency-wait: 10
max-jobs-per-second: 1
set-resource-scopes:
- extra_slurm_flags=local

You will need the Python drmaa library installed, plus the SLURM DRMAA package from https://github/natefoo/slurm-drmaa.