Hope you are doing great.
We have a daily datasync job which is orchestrated using Lambdas and AWS API. The source locations are AWS S3 buckets and the target locations are GCP cloud storage buckets. However recently we started getting an error on datasync tasks (It worked fine before) with a lot of failed transfers due to the error "S3 PutObject Failed":
[ERROR] Deferred error: s3:c68 close("s3://target-bucket/some/path/to/file.jpg"): 40978 (S3 Put Object Failed)
We use manifest to backup files. I didn't change anything in IAM roles etc. I don't understand why It just stopped working. Some S3 PUT works but the majority fail
Did anyone run into the same issue ?
Hope you are doing great.
We have a daily datasync job which is orchestrated using Lambdas and AWS API. The source locations are AWS S3 buckets and the target locations are GCP cloud storage buckets. However recently we started getting an error on datasync tasks (It worked fine before) with a lot of failed transfers due to the error "S3 PutObject Failed":
[ERROR] Deferred error: s3:c68 close("s3://target-bucket/some/path/to/file.jpg"): 40978 (S3 Put Object Failed)
We use manifest to backup files. I didn't change anything in IAM roles etc. I don't understand why It just stopped working. Some S3 PUT works but the majority fail
Did anyone run into the same issue ?
Share Improve this question asked yesterday AlexisAlexis 754 bronze badges1 Answer
Reset to default 0I've encountered similar problems with S3 PutObject failures in cross-cloud transfer jobs. Here are some things to check:
- Quota limits - Check if you've hit any AWS or GCP quotas. This can happen if your data volume increased or if other processes are using the same resources.
- IAM token expiration - Even though you didn't change IAM roles, check if your tokens or credentials might be expiring during the job. This is common with long-running transfers.
- Network connectivity - Verify there haven't been any changes to VPC settings, security groups, or network policies between your environments.
- File size or format issues - Are the failing transfers related to specific file types or sizes? Sometimes large files need different transfer settings.
In general, look at the full CloudWatch logs for your Lambda functions to see if there are more detailed error messages before the "S3 Put Object Failed" error.