Use Case
I am ingesting data using Fivetran, which syncs data from an Oracle database directly into my Databricks table. Fivetran manages the creation, updates, and inserts on these tables. As a result, my source is a static table in the Bronze layer.
Goal
I want to use Delta Live Tables (DLT) to stream data from the Bronze layer to the Silver and Gold layers.
Implementation
I have a SQL notebook with the following code:
CREATE OR REFRESH STREAMING TABLE cdc_test_silver;
APPLY CHANGES INTO live.cdc_test_silver
FROM lakehouse_poc.bronze.cdc_test
KEYS (ID)
SEQUENCE BY ModificationTime;
The objective is to create the Silver Delta Live Table using the Bronze Delta Table as the source.
Issue Encountered
I am receiving the following error:
Source data for the APPLY CHANGES target 'lakehouse_poc.bronze.cdc_test_silver' must be a streaming query.
Question
How can I handle this issue and successfully stream data from Bronze to Silver using Delta Live Tables?