In Databricks, I ran vacuum on all the tables in loop after successful run, when i check the history for all the tables which contain operation "Vacuum start" and captured "sizeOfDataToDelete" which was closer to 1TB but when check the container metrics its still remains the size of before performing vacuum, it didn't free up the space below attached sample of the output, what could be the issue?

In Databricks, I ran vacuum on all the tables in loop after successful run, when i check the history for all the tables which contain operation "Vacuum start" and captured "sizeOfDataToDelete" which was closer to 1TB but when check the container metrics its still remains the size of before performing vacuum, it didn't free up the space below attached sample of the output, what could be the issue?

• apache-spark
• databricks
• azure-databricks

1 Answer 1

In Databricks, Vaccuum removes only the files which are not part of delta log . I am assuming you have used DELETE command to delete the data you no longer need first. If so by default deleted data will be marked as delete but not deleted till 7 days. That is the default time till which the files exists although not part of the delta log. https://docs.databricks/aws/en/sql/language-manual/delta-vacuum

VACUUM removes all files from the table directory that are not managed by Delta, as well as data files that are no longer in the latest state of the transaction log for the table and are older than a retention threshold. VACUUM will skip all directories that begin with an underscore (_), which includes the _delta_log. Partitioning your table on a column that begins with an underscore is an exception to this rule; VACUUM scans all valid partitions included in the target Delta table. Delta table data files are deleted according to the time they have been logically removed from Delta’s transaction log plus retention hours, not their modification timestamps on the storage system. The default threshold is 7 days.