Here is answer about how move files from one s3 bucket to another using sync mand from mand line (cli):
aws s3 sync s3://from_my_bucket s3://to_my_other_bucket
Can this be invoked from javascript sdk? With searching I did not find any sync named method. So is it supported in sdk?
Here is answer about how move files from one s3 bucket to another using sync mand from mand line (cli):
aws s3 sync s3://from_my_bucket s3://to_my_other_bucket
Can this be invoked from javascript sdk? With searching I did not find any sync named method. So is it supported in sdk?
4 Answers
Reset to default 1I found this really helpful article on cloning S3 bucket using Node.JS package "aws-sdk": [Node Clone S3 Bucket by Rajesh Babu][1]
I am quoting his approach here:
[1]: https://blog.bitsrc.io/a-practical-guide-to-building-a-node-js-service-on-an-aws-s3-bucket-aff19105ba83
- Fetch the list of Keys from the bucket and the target Prefix. (check AWS-SDK Javascript APIs)
- Separate the files and directories, because we clone the directories and download the files.
- Clone all the directories first, and then move on to download the files.
- Download the files through streams and log success and failure respectively.
You can see how this node package does it.
Basically, it makes use of several S3 instance methods to sync two buckets.
I needed to sync thousands of files, and decided to make use of AWS Data Pipeline to do it. I update the pipeline definition to point to the correct source and destination folders from my Lambda function, and then invoke it.
It uses 2 S3DataNodes and a CopyActivity.
This should be the defacto answer: No but there are official examples. Here is the README to their Go example: https://github./aws/aws-sdk-go/blob/main/example/service/s3/sync/README.md