Content-addressed storage

Content-addressed storage (CAS) identifies stored objects using a digest derived from their representation. A request names the content expected, while a storage service or retrieval protocol locates a copy. This separates an object's identity from a particular server or pathname. RFC 6920 standardizes one way to name information with hashes; individual storage systems use their own object formats and identifier schemes. ^[1]

CAS is useful for archives, version control, and shared build artifacts. Its central promise is precise identity and verification under the chosen hash scheme. Persistence, trustworthy authorship, and successful reconstruction require additional mechanisms. This article covers the software storage pattern, representative implementations, and its limits for digital preservation; it is not a survey of associative-memory hardware or a commercial product catalogue.

A simplified store accepts bytes, computes a digest, and records a mapping from that digest to the bytes. Repeating a write with identical input can reuse the existing object. Reading by digest allows the recipient to hash the retrieved representation and check it against the requested value. The Venti archival server is a concrete implementation of this block-level design: it uses a hash of a block's contents as the identifier for read and write operations, applies a write-once policy, and coalesces duplicate blocks. The paper further describes these writes as idempotent because repeated writes of the same data need no additional storage. This is payload-based idempotence, distinct from the caller-intent and retry semantics discussed in idempotency keys. These are properties of Venti's design, not guarantees supplied by every content-addressed store. ^[2]

The digest is not a compressed copy from which an arbitrary object can be recovered. A separate copy must exist. Nor is a content name necessarily a network address: a client may resolve it through a local index, an archive resolver, or a network lookup. RFC 6920 distinguishes identifying information from locating and transporting it. ^[1]

An illustrative workflow is:

producer -> serialize object -> hash representation -> retain bytes
                                      |
                              publish identifier
                                      |
reader   -> locate a copy -> retrieve bytes -> recompute and compare

The arrows describe separate responsibilities. A completed hash calculation establishes neither that retention succeeded nor that another reader can reach the stored copy. This is an architectural inference from separating naming and storage, not a claim about a particular service's durability contract.

An identifier is meaningful only alongside a specification for its input. A system may hash a raw file, a header followed by file data, a serialized directory, or the root of a graph of blocks. Git, for example, hashes an object header containing its type and length together with its content. A Git blob identifier therefore need not equal a raw-file checksum. ^[3]

IPFS content identifiers include information needed to interpret the hash and encoded object. For a chunked file, the digest generally identifies a root block rather than the concatenated file bytes. Import settings, such as chunking and codec choices, can change the CID even when the original file bytes are identical. ^[4]

Equivalent human meaning is also different from identical bytes. These two JSON documents express the same simple mapping but contain different byte sequences:

{"a":1,"b":2}

{ "b": 2, "a": 1 }

RFC 8785 defines a JSON Canonicalization Scheme for producing a deterministic representation within its specified input constraints. A generic instruction to “sort the keys” is not a complete implementation: serialization rules, number handling, strings, and permitted inputs also matter. Canonicalization should be an explicit part of an interoperable format, not an undocumented cleanup step. ^[5]

Large structures can contain references to smaller content-addressed objects. In Git, blobs hold file content; trees associate names and modes with blobs or other trees; commits refer to a tree and record history-related metadata. Changing a filename can therefore change a tree while leaving the file's blob unchanged. Two commits can also refer to the same tree while differing in commit metadata. ^[3]

This leads to Merkle DAGs, where a node's identity incorporates references to its descendants. Software Heritage uses such a structure to connect archived contents, directories, revisions, releases, and snapshots. The distinction between these object types matters: identifying a file is not the same as identifying the repository state in which it appeared. ^[6]

Content addressing does not make editing impossible. An edit produces a new immutable object, while a separate reference can designate the new current version. Unchanged objects may be shared across versions. The storage and retrieval costs depend on object boundaries, representation, and implementation; whole-file duplication and chunk-level reuse are different designs, not universal properties of CAS.

The 2002 Venti paper describes a network block store designed for archival data, with write-once blocks and duplicate-block coalescing. Its applications included logical backup, physical backup, and snapshot file systems. Its archival policy went further than merely assigning hashes: the authors deliberately designed an append-only repository. The paper calls its performance results preliminary and bases its feasibility discussion on data from more than a decade of use of two Plan 9 file systems; those are source-specific prototype and workload observations, not a general performance result for all content-addressed stores. ^[2]

Bazel separates an action cache, mapping action identities to result metadata, from a content-addressable store holding output files. This is a useful distinction between “which computation's result?” and “which resulting bytes?” Reading an existing cached object is reuse, not an independent execution of the computation. ^[7]

A Nix implementer's 2020 discussion similarly distinguishes names derived from build inputs from names derived from produced content. If a changed recipe produces identical output, content addressing can reveal that equality and potentially avoid unnecessary downstream work. The article explains a design motivation at that time; it should not be read as a statement about every current Nix store mode. ^[8]

Software Heritage's SWHID scheme separates a core intrinsic identifier from optional qualifiers. Qualifiers can record context, such as an origin or path, and select a fragment such as particular lines. This makes a useful distinction for citations: object identity and the context in which an object was encountered are related but different information. ^[6]

A fixed-size digest cannot assign a different value to every possible arbitrary-length input. Systems depend on an appropriate hash function making relevant collisions infeasible. Historical examples require historical qualification: Venti used SHA-1, but NIST's December 2022 guidance recommends migration from SHA-1 to SHA-2 or SHA-3 because of security weaknesses. A legacy identifier format is not a recommendation for a new security-sensitive design. ^[9]

Checking a digest also does not identify who supplied the expected digest or establish why that source should be trusted. If an attacker can replace both an artifact and the value a reader treats as authoritative, simple equality checking does not authenticate the release. RFC 6920 explicitly treats trust in the name and security of the hash as separate considerations. ^[1]

IPFS distinguishes cached data from pinned data: pinning protects specified objects from a node's garbage collection. Direct and recursive pins differ because retaining a root block alone may not retain its descendants. A retention plan must match the structure needed for reconstruction. ^[10]

An immutable name does not itself provide replication, backups, available network paths, or an operator committed to retaining data. Likewise, protecting objects from one garbage collector does not make a machine immune to disk loss. Those are practical consequences of the storage responsibilities, and preservation assessments should record them separately from successful digest verification.

Deduplication can reuse identical objects within the store's comparison boundary. It does not imply that separate operators share storage, that differently serialized equivalents deduplicate, or that all historical versions cost nothing. A changed chunk layout can reduce reuse even if much of a file is conceptually unchanged. Published capacity or speed claims need their workload, chunking scheme, and measurement method.

This original Python example compares raw bytes. It illustrates naming only; it does not implement a storage server, a Git object, or an IPFS CID.

from hashlib import sha256

first = b"cedar\n"
copy = bytes(first)
changed = b"cedar\r\n"

def identifier(data):
    return "sha256:" + sha256(data).hexdigest()

assert identifier(first) == identifier(copy)
assert identifier(first) != identifier(changed)
print("identical bytes match; changed line ending differs")

The assertions were run locally for this contribution on 2026-09-07. They establish the two stated comparisons for these inputs, not collision resistance or storage durability. A useful follow-on exercise is to archive the two byte sequences, remove one local copy, and distinguish an identifier still being known from its corresponding content still being retrievable. That exercise is proposed, not reported as performed.

Reproducible builds ask whether specified artifacts can be recreated from the documented source, instructions, and environment. Content addressing helps name and compare the inputs and outputs. It does not cause a nondeterministic build to become deterministic or recreate missing dependencies. ^[11]

For an archival handoff, a useful proposed record contains the identifier scheme, representation rules, retained locations, exact source revision, reconstruction instructions, and known verification limits. Such a record lets a later investigator distinguish byte identity, availability, rebuildability, and the truth of any claims made inside the artifact.