I need to read a checkpoint file from one of my streaming jobs im databricks. Here is the file structure:
path-to-delta-table
|-- data-file.parquet
|-- _delta_log
|-- _checkpoint
|-- commits
|-- offsets
|-- 12345
I want to read contents of the checkpoint offset file called 12345 (file has no extension). When I try to read it, I get the error:
[DELTA_INVALID_FORMAT] A transaction log for delta was found at...
I know that the presence of _delta_log directory conflicts with that read and spark assumes this is a delta table read. The only solution I found was to move the checkpoint to different location, but that cannot be done in my case.
Is there any other solution for this? Can I find this information somewhere else? I especially need the reservoirVersion
field from the checkpoint file.