Method / tool

content hash

A cryptographic fingerprint (typically SHA-256) computed from a file's bytes, used to give each input and output a stable, path-independent identity — the primitive that makes artifact provenance verifiable even when files are copied, renamed, or moved across storage systems.

The problem — File paths are mutable. A raw TIFF named plate01_run3.tif can be overwritten, moved to a new directory, renamed to fix a typo, or silently re-exported from a vendor system with identical metadata but different bytes. A provenance record that references the file by path cannot distinguish the original from any of its successors. When a pipeline reruns three months later and picks up the "same" file by path, it may be computing on a different image — with no error, no warning, and no record of the substitution. This is how provenance collapses silently.

What it is / how it works — A content hash is a fixed-length fingerprint computed by a deterministic function (SHA-256 is the standard) over the exact bytes of a file. The same bytes always produce the same hash; a single changed bit produces a completely different hash. By recording the hash of each input alongside its path, a pipeline gives every artifact a stable, path-independent identity: the provenance record says "processed sha256:3a7f…d4c2," not "processed plate01_run3.tif." If the file at that path later changes, the hash no longer matches and the discrepancy is detectable — the artifact becomes traceable in the way the microscopy metadata perspective (Huisman et al., 2019) frames as a prerequisite for auditable provenance. In Fovea pipelines, content hashes are the primitive beneath run manifest lineage: each manifest entry records the source image hash, the model or parameter hash, and the output hash, so every artifact can be re-derived or its derivation challenged. The hash also enables deduplication across studies — the same raw plate deposited twice under different filenames resolves to one node in the provenance graph.

Where it breaks — Hashing is only useful if it is checked. A pipeline that records input hashes but never validates them at rerun is logging provenance without enforcing it. The second failure mode is hashing after transformation: if a pipeline reads a vendor file, silently resamples it, and then hashes the resampled version, the hash does not identify the original raw data — it identifies a derived copy. The discipline is to hash the canonical raw input (the vendor file as received) before any transformation, and to propagate that identity through every downstream artifact that depends on it.

A provenance record that stores a file path without a content hash is a note about where data was, not what it was. Hash inputs before processing — the path changes; the bytes do not.

References

Appears in these notes

  • Every Output Should Know Where It Came FromProvenance is a property of the artifact, not the pipeline — every mask, profile, and count matrix should carry the lineage that produced it. Where verification asks whether a run reproduces, provenance asks whether a single output, found alone, can still explain itself.

← Back to the constellation