I am using Python diskcache as a queue. One or more producers pushes data to the cache while a singular consumer pops the entries, formats some json and pushes it to a MQTT topic. I am using this setup to cover me at times of network outage.

It works great, with one exception: If one of the producers restarts for some reason the cache is emptied. I am assuming that a new cache is created that wipes out the old data.

Is there a way around this?

I am using the Deque implementation and not setting any eviction/culling policies etc., everything is default.

I reversed the implementation so the consumer creates the cache, but the issue persists if the consumer restarts. Better, but still an issue.