最新消息:雨落星辰是一个专注网站SEO优化、网站SEO诊断、搜索引擎研究、网络营销推广、网站策划运营及站长类的自媒体原创博客

Fivetran: from AWS Postgres to GCP Snowflake - Slow - Stack Overflow

programmeradmin1浏览0评论

Both in the East region, our prod DB is running in AWS while our Snowflake datastore in GCP. We replicate with Fivetran every hour 293 tables. Some tables are small (10K records) while other tables are large (250+ MM records).

C suite last month came to me with a requirement to increase the replication frequency to 15 minutes. Just "turning the dial up" doesn't work: I get my inbox almost immediately spammed with Fivetran email notifications that my replication is running behind.

Our prod DB handles inserts and updates. No deletes. When I set up the replication, I created a read replica of prod in AWS and set XMIN as the update method. Now I wonder if switching to logical replication would increase the performance.

I spent an hour yesterday with Fivetran professional services. They were as stumped as me. Short of pruning the tables that we don't need to have in Snowflake (not even sure it's an option), I am not sure what to do to get to the 15 minutes requested.

Has anyone here ever experienced such behavior? What were the symptoms? What was your solution?

发布评论

评论列表(0)

  1. 暂无评论