Pickle

pickle is Python’s built-in object serialization module. It converts Python object structures into byte streams and reconstructs them later.¹

Why I avoid pickle for data storage#

Pickle solves a Python convenience problem, not a data-storage problem. It serializes a Python object. That makes it Python-specific, environment-dependent, and unsuitable as a general interchange format. Almost every kind of data I store has a better alternative.

It is not universal#

Someone using R, Julia, a database, or ordinary command-line tools cannot readily inspect a pickle. Even another Python environment may not be enough. Functions and classes are stored by qualified name, so the expected modules and definitions must remain importable when the file is loaded.¹

The interpreter and library environment become part of the hidden schema. NumPy’s documentation cites portability as a reason to disallow pickle-backed object arrays: required libraries may be absent, and pickled data may be incompatible across Python versions.²

The format is underspecified#

There are multiple pickle protocols and several library-specific ways to produce pickle-like files. A filename rarely tells me which method, Python version, library versions, or object definitions I need. The loading environment becomes undocumented metadata.

Pickle is also opaque to ordinary inspection. I cannot examine its columns, types, or identifiers with common command-line and database tools without invoking Python deserialization. Shared research data should expose their structure instead of hiding it inside a reconstruction procedure.

Prefer data formats to object snapshots#

I prefer formats whose structure is explicit and whose readers are not tied to one Python object graph.

• Tables: Parquet. It is a standardized, open-source columnar format, and Arrow provides readers across multiple languages.³
• Numeric arrays: .npy or .npz, without Python objects. NumPy itself cites security and portability as reasons to disallow pickle-backed object arrays.²
• Structured records, mappings, and metadata: JSON or JSON Lines. JSON is a text-based, language-independent interchange format.⁴ It is not limited to small files; JSON Lines supports large datasets that can be processed one record at a time.⁵
• Human-edited configuration: TOML or YAML.

Each dataset should also include a README, schema, provenance, and checksums. Plain text is often the best even when the main data are binary.

Loading data should not execute code#

Python’s documentation warns that a malicious pickle can execute arbitrary code when loaded.¹ This is another reason to avoid it, but the portability problem is sufficient on its own. A collaborator should not need the original Python environment or trust executable behavior just to inspect a dataset.

When pickle is acceptable#

The narrow exception is a trusted, temporary, disposable cache inside one controlled Python environment, when regenerating it is trivial. Even there, pickle is a convenience rather than a canonical data storage format. I would not use it as the only copy of shared data, an interchange format, or an archival artifact.