I am trying to configure AWS Firehose to write to a specific s3 directory. For this, in the prefix section of Firehose console, I gave "Sample/". Attaching the screenshot.
Even after this, in the S3, its creating the folder structure S3bucket/Sample/YYYY/MM/dd/file.json.
Is there a way to omit the expressions (eg.YYYY/MM/dd) and let the firehose use only the prefix, like this => S3bucket/Sample/file.json?
I am trying to configure AWS Firehose to write to a specific s3 directory. For this, in the prefix section of Firehose console, I gave "Sample/". Attaching the screenshot.
Even after this, in the S3, its creating the folder structure S3bucket/Sample/YYYY/MM/dd/file.json.
Is there a way to omit the expressions (eg.YYYY/MM/dd) and let the firehose use only the prefix, like this => S3bucket/Sample/file.json?
Share Improve this question asked Nov 20, 2024 at 15:31 hari krishnanhari krishnan 133 bronze badges2 Answers
Reset to default 0One solution is to add lambda function that triggers on S3 events and changes the final file name. In your case removing the date and other stuff and final name like, file.json
You can avoid the default time partitions using the dynamic partitioning. That will override the default partitions.
Only caveat is that the dynamic partitioning should be enabled while creating the firehose stream.
You can refer the documentation on how to enable it. https://docs.aws.amazon/firehose/latest/dev/dynamic-partitioning-partitioning-keys.html