The problem
I have a Snakemake script snakefile.py. In that script I have rules all, call_variants and 4 other rules.
When I ran:
snakemake --printshellcmds -s snakefile.py
One of samples failed with an error:
/usr/bin/bash: line 1: 357013 Segmentation fault (core dumped) <...>
RuleException:
CalledProcessError in file snakefile.py, line 93:
Command 'set -euo pipefail; <command>' returned non-zero exit status 139.
I do not know what may cause the problem, because only the small portion of 47 samples fail, and not even the same samples in different runs, as you will see below.
What I've tried
After the command:
snakemake --printshellcmds -s snakefile.py
which resulted in failing in calling variants on samples 1 and 2, I deleted all files computed with this command and sequentially ran the commands below:
1.
snakemake --keep-going --printshellcmds -s snakefile.py
The result: rule call_variants failed with the same error on samples 2, 3, 4, 5, 6, 7. The remaining of 47 samples completed successfully, including sample 1, failed to be called in the original command.
2.
snakemake --keep-going --printshellcmds -s snakefile.py
The result: samples 2, 3, 4, 5, 6 completed successfully, sample 7 failed with the same error.
3.
snakemake --keep-going --printshellcmds -s snakefile.py
The result: sample 7 completed successfully.
So it took me to run the same command with --keep-going option 3 times to get all my samples computed. The samples that failed to be computed in the previous runs computed successfully in subsequent runs. The log files of failed samples for call_variants rule shell command do not contain any error messages and are just truncated compared to the log files of succeeded samples.
It is worth noting, that the only rule that failed was call_variants, none of the other 4 rules caused any trouble.
The other interesting observation is that if I execute the shell command that caused the Snakemake rule to fail directly in the shell it runs successfully. The command is exactly the same as the one ran by Snakemake, since I copy it from Snakemake log.
The rule call_variants calls variants with xAtlas variant caller. I have the similar scripts for other 2 variant callers, they run without the error. The script for xAtlas also ran without the error before moving to another server and subsequent new installation of xAtlas.
Software set-up:
Snakemake v.8.30.0
Ubuntu 22.04.3 LTS
The problem
I have a Snakemake script snakefile.py. In that script I have rules all, call_variants and 4 other rules.
When I ran:
snakemake --printshellcmds -s snakefile.py
One of samples failed with an error:
/usr/bin/bash: line 1: 357013 Segmentation fault (core dumped) <...>
RuleException:
CalledProcessError in file snakefile.py, line 93:
Command 'set -euo pipefail; <command>' returned non-zero exit status 139.
I do not know what may cause the problem, because only the small portion of 47 samples fail, and not even the same samples in different runs, as you will see below.
What I've tried
After the command:
snakemake --printshellcmds -s snakefile.py
which resulted in failing in calling variants on samples 1 and 2, I deleted all files computed with this command and sequentially ran the commands below:
1.
snakemake --keep-going --printshellcmds -s snakefile.py
The result: rule call_variants failed with the same error on samples 2, 3, 4, 5, 6, 7. The remaining of 47 samples completed successfully, including sample 1, failed to be called in the original command.
2.
snakemake --keep-going --printshellcmds -s snakefile.py
The result: samples 2, 3, 4, 5, 6 completed successfully, sample 7 failed with the same error.
3.
snakemake --keep-going --printshellcmds -s snakefile.py
The result: sample 7 completed successfully.
So it took me to run the same command with --keep-going option 3 times to get all my samples computed. The samples that failed to be computed in the previous runs computed successfully in subsequent runs. The log files of failed samples for call_variants rule shell command do not contain any error messages and are just truncated compared to the log files of succeeded samples.
It is worth noting, that the only rule that failed was call_variants, none of the other 4 rules caused any trouble.
The other interesting observation is that if I execute the shell command that caused the Snakemake rule to fail directly in the shell it runs successfully. The command is exactly the same as the one ran by Snakemake, since I copy it from Snakemake log.
The rule call_variants calls variants with xAtlas variant caller. I have the similar scripts for other 2 variant callers, they run without the error. The script for xAtlas also ran without the error before moving to another server and subsequent new installation of xAtlas.
Software set-up:
Snakemake v.8.30.0
Ubuntu 22.04.3 LTS
1 Answer
Reset to default 0My first suspicion here would be a memory-related error. These will show up in the kernel log:
$ sudo dmesg -T
as OOM events. You could also use strace on the application to look for malloc() calls that fail, but you do have to make sure you run strace on the underlying binary, not any wrapper script that might be being invoked in the rule.
If the application has just enough memory available when run outside of Snakemake then it may be the overhead of Snakemake pushing it over the edge. Also, with Snakemake are you running one job at a time or allowing it to run multiple jobs in parallel? Are you using multiple threads within the rule?