I've encountered an issue with Delta Live Table in both my Development and Production Workspaces. The data is arriving correctly in my Azure Storage Account; however, the checkpoint is being stored in the path dbfs:/. I haven't modified the Storage Location, in fact, the data is being written to the tables correctly. The problem is that it's performing a full refresh since the checkpoint has started from scratch. Is there a bug in Databricks?
I've encountered an issue with Delta Live Table in both my Development and Production Workspaces. The data is arriving correctly in my Azure Storage Account; however, the checkpoint is being stored in the path dbfs:/. I haven't modified the Storage Location, in fact, the data is being written to the tables correctly. The problem is that it's performing a full refresh since the checkpoint has started from scratch. Is there a bug in Databricks?
Share Improve this question asked Mar 14 at 13:08 Antonio FernándezAntonio Fernández 1035 silver badges12 bronze badges 1- The checkpoint is used when doing an update run so only the new data is processed. If you perform a full refresh, the pipeline will disregard the checkpoint contents. It will fetch all data present in the source folder and reset all the tables. So what kind of run are you executing? – Anupam Chand Commented Mar 15 at 7:58
1 Answer
Reset to default 0***I've encountered an issue with Delta Live Table in both my Development and Production Workspaces. The data is arriving correctly in my Azure Storage Account; however, the checkpoint is being stored in the path dbfs:/. I haven't modified the Storage Location, in fact, the data is being written to the tables correctly. The problem is that it's performing a full refresh since the checkpoint has started from scratch. Is there a bug in Databricks?
***However, the checkpoint is being stored in the dbfs:/ path, and because of this, DLT is performing a full refresh since the checkpoint has started from scratch
***By default, Delta Live Tables store checkpoint information in dbfs:/delta/ (which is within the Databricks file system). If you're using an external storage account (e.g., Azure Blob Storage or ADLS), it's crucial to specify the checkpoint location explicitly in your DLT pipeline configuration. Otherwise, Databricks will default to dbfs:/ and may not be able to track the checkpoint properly across sessions or runs.
***If your pipeline is configured to write data to an external Azure Storage Account but checkpoints are being stored in dbfs:/, the full refresh behavior can occur because the system can't track the incremental changes, leading it to treat the dataset as if it's being processed from scratch every time.
***When creating a Delta Live Table pipeline, ensure that the checkpoint location is set correctly within the pipeline settings.To fix this, you should specify the checkpoint location for your Delta Live Table pipeline. This will ensure that the checkpoints are saved in the correct external storage location, and the pipeline can track incremental changes.
***If using the Databricks UI, Go to the Delta Live Tables pipeline in the Databricks workspace.
In the pipeline configuration settings, look for "Advanced settings".
In the "Checkpoint location" field, specify the external storage location (ADLS or Blob Storage).
***For best practices, it's recommended to store Delta Live Table checkpoints in an external storage account (e.g., ADLS or Blob Storage) instead of dbfs:/ for better scalability and reliability.