I have a multiple files in a s3 bucket i have to create a table in snowflake through dbt based on the last file that was loaded in the s3 bucket. what would be the ideal approach?
I have a multiple files in a s3 bucket i have to create a table in snowflake through dbt based on the last file that was loaded in the s3 bucket. what would be the ideal approach?
Share Improve this question asked Mar 17 at 14:36 John PaulJohn Paul 113 bronze badges1 Answer
Reset to default 0You could use dbt External Tables package https://github/dbt-labs/dbt-external-tables and create an external table in Snowflake that looks at the S3 bucket.
Then in dbt you could query the stage and use METADATA$FILE_LAST_MODIFIED field that is attached to every file, in order to process only the last one.
Have a look at this as well: https://medium/slateco-blog/doing-more-with-less-usingdbt-to-load-data-from-aws-s3-to-snowflake-via-external-tables-a699d290b93f