I am using snakemake with rule need visit public read-only s3 storage
# test
$ aws s3 ls 1000genomes/phase1/phase1.exome.alignment.index.bas.gz --no-sign-request
2012-05-01 03:58:41 423691 phase1.exome.alignment.index.bas.gz
$ aws s3 cp s3://1000genomes/phase1/phase1.exome.alignment.index.bas.gz ./ --no-sign-request
download: s3://1000genomes/phase1/phase1.exome.alignment.index.bas.gz to ./phase1.exome.alignment.index.bas.gz
$ ls phase1.exome.alignment.index.bas.gz
phase1.exome.alignment.index.bas.gz
# public visit ok
# snakemake
storage:
provider="s3",
max_requests_per_second=10
rule download_data:
input:
vcf=lambda wildcards: storage.s3("s3://1000genomes/phase1/phase1.exome.alignment.index.bas.gz")
output:
'output/data/phase1.exome.alignment.index.bas.gz'
shell:
# just a example here, aws s3 cp can instead, but we have more complex process
"cp {input.vcf} {output}"
snakemake -s /path/to/snakemake.smk
The following required arguments are missing for plugin s3: --storage-s3-access-key (or environment variable SNAKEMAKE_STORAGE_S3_ACCESS_KEY), --storage-s3-secret-key (or environment variable SNAKEMAKE_STORAGE_S3_SECRET_KEY).
I treid to set SNAKEMAKE_STORAGE_S3_ACCESS_KEY
and SNAKEMAKE_STORAGE_S3_SECRET_KEY
environment to ''.
export SNAKEMAKE_STORAGE_S3_ACCESS_KEY=''
export SNAKEMAKE_STORAGE_S3_SECRET_KEY=''
but its seems not work.
snakemake -s /path/to/snakemake.smk
Failed to check existence of s3://1000genomes/phase1/phase1.exome.alignment.index.bas.gz
RuntimeError: 34 (AWS_ERROR_INVALID_ARGUMENT): An invalid argument was passed to a function.