Python pickles are great! They allow you to serialize a whole lot of stuff: primitives, lists, dicts, & even classes. If your’re writing python they are very handy but persisting them robustly requires some thought. Let’s say your application maintains some state. When your app fires up, you would like to read that state and resume your operation. One way of doing this is just to pickle your statefull objects to a known file: state.pickle, for example.
import pickle
with open("state.pickle", "wb") as f
pickle.dump(appStateObject, f)
Clean. Simple. There’s just one problem. Every time the file gets opened, it is truncated and the dump starts serializing the object to disk. If their is a problem during serialization you would end up with a foobared pickle. You could serialize the whole object to memory first, then write it out. If you application fails (or is terminated) while writing to disk… it will still have a foobared pickle. So what to do? Write it to another file.
import pickle
import tempfile
import os.path
f = None
filename = "state.pickle"
tfile = None
try:
with tempfile.NamedTemporaryFile(dir=os.path.dirname(filename), delete=False) as tfile:
pickle.dump(appStateObject, tfile)
os.rename(tfile.name, filename)
finally:
if tfile and os.path.exists(tfile.name):
os.remove(file.name)
The trick is to save the pickle in the same directory as the final file but under another filename. If things go south, any existing pickle won’t be foobared. If the file gets written out without errors, os.rename() will overwrite the existing pickle (if it exists) in a atomic manner on POSIX systems. The same holds true for any file you open in write mode and truncation occurs. Never assume that you will leave the file in a usable state if things go wrong.
You could resort to storing your object in a database that has robust transaction support. That would be a lot of baggage to store a simple pickle.
