Building on the DISC · The Book of Swarm

This chapter builds on the distributed chunk store and introduces data structures and processes that enable higher-level functionality, offering a rich experience handling data. In particular, we show how chunks can be organised to represent files (4.1.1) and how files can be organised to represent collections (4.1.2). We also introduce key–value maps (4.1.4) and briefly discuss the potential of arbitrary functional data structures. We then shift our attention to presenting our solution for providing confidentiality and access control (4.2).

In 4.3, we introduce Swarm feeds, which are designed to represent a wide variety of sequential data, such as versioning updates of mutable resources or indexing messages for real-time data exchange: offering a system of persisted pull messaging. To implement push notifications of all kinds, 4.4 introduces the novel concept of Trojan chunks that allow messages to be disguised as chunks and directed to their intended recipient in the swarm. We explain how Trojan chunks and feeds can be combined to form a fully-fledged communication system with robust privacy features.

4.1 Data structures

In the first two chapters, we made the assumption that data is structured in the form of chunks, i.e. fixed-size data blobs. However, in this section, we will present the algorithms and structures that enable the representation of data of arbitrary length. We will introduce Swarm manifests, which form the basis of representing collections, indexes, and routing tables, allowing Swarm to host websites and offer URL-based addressing.

4.1.1 Files and the Swarm hash

In this section, we introduce the concept of the Swarm hash, which provides a mechanism for combining chunks to represent larger sets of structured data, such as files. The idea behind the Swarm hashing algorithm is that chunks can be arranged in a Merkle tree. In this structure, leaf nodes correspond to chunks from consecutive segments of input data, while intermediate nodes correspond to chunks that are composed of the chunk references of their children. These combined chunks are packaged together to form another chunk, as illustrated in Figure 4.1.

Figure 4.1: Swarm hash: data input is segmented to 4-kilobyte chunks (gray), that are BMT hashed. Their hashes are packaged into intermediate chunks starting on level

0

, all the way until a single chunk remains on level

n

Branching factor and reference size

The branching factor of the tree is calculated as the chunk size divided by the reference size. For unencrypted content, the chunk reference is simply the BMT hash of the chunk, which is 32 bytes, so the branching factor is just 4096/32 = 128. A group of chunks referenced under an intermediate node is referred to as a batch. If the content is encrypted, the chunk reference consists of the concatenation of the chunk hash and the decryption key. Both are 32 bytes long, so an encrypted chunk reference will be 64 bytes, and therefore the branching factor is 64.

**Figure 4.2:** Intermediate chunk. It encapsulates references to its children.

As a result, a single chunk can represent an intermediate node in the Swarm hash tree, in which case its content can be segmented to references, allowing retrieval of their children. These child nodes themselves may be intermediate chunks, as illustrated in Figure 4.2. By recursively unpacking these from the root chunk downwards, we can eventually obtain a sequence of data chunks.

Chunk span and integrity of depth

The length of data subsumed under an intermediate chunk is called the chunk span. In order to be able to tell if a chunk is a data chunk or not, the chunk span is prepended to the chunk data in a 64-bit little-endian binary representation. When calculating the BMT hash of a chunk, this span constitutes the metadata that needs to be added to the BMT root and hashed together, resulting in the chunk address. When assembling a file starting from a hash, it is possible to tell if a chunk is a data chunk or an intermediate chunk simply by looking at the span. If the span is larger than 4K, the chunk is an intermediate chunk and its content needs to be interpreted as a series of hashes of its children; otherwise it is a data chunk.

In theory, if the length of the file is already known, spans of intermediate chunks are unnecessary since we could calculate the number of intermediate levels required for the tree. However, using spans disallows reinstating the intermediate levels as data layers. In this way, we impose integrity of the depth.

Appending and resuming aborted uploads

The Swarm hash has the interesting property that any data span corresponding to an intermediate chunk is also a file and can therefore be referenced as if the intermediate chunk was its root hash. This property is significant because it enables appending to a file while retaining a historical reference to the earlier state, without duplicating chunks, except for the incomplete right edge of the Merkle tree. Appending data to a file is particularly useful for scenarios such as resuming uploads after a crash partway through uploading big files.

Random access

Note that all chunks in a file, except for the right edge, are completely filled. Given that chunks have a fixed size, it is possible to calculate the path to a specific chunk and the offset to search within that chunk for any arbitrary data offset ahead of time. Because of this, random access to files is supported right away (see Figure 4.3).

Compact inclusion proofs for files

Suppose we were to prove the inclusion of a substring in a file at a particular offset. We have observed that the offset, when applied to the data, follows a deterministic path traversing the Swarm hash. Since a substring inclusion proof simply reduces to a series of proofs of data segment paths, the chunk addresses are a result of a BMT hash, where the base segments are 32 bytes long. This means that in intermediate chunks, BMT base segments align with the addresses of their children. As a consequence, proving that a child of an intermediate chunk at a particular span offset is equivalent to giving a segment inclusion proof on the child hash. Therefore, substring inclusion in files can be proved with a sequence of BMT inclusion proofs where the length of the sequence corresponds to the depth of the Swarm hash tree (see Figure 4.4).

Note that such inclusion proofs are possible even in the case of encrypted data. This is because the decryption key for a segment position can be selectively disclosed without revealing any information that could compromise the encryption of other parts in the chunk.

In this section, we have presented the Swarm hash, a data structure over chunks that represents files, which supports the following functionalities:

random access: The file can be read from any arbitrary offset with no extra cost.
append: Supports appending without duplication.
length-preserving edits: Supports length-preserving edits without the duplication of unmodified parts.
compact inclusion proofs: Allow inclusion proofs with resolution of 32 bytes in space logarithmic in file size.

4.1.2 Collections and manifests

The Swarm manifest serves as a framework that defines a mapping between arbitrary paths and files to represent collections. It includes metadata associated with the collection and its objects (files). A manifest entry contains a reference to a file or, more precisely, a reference to the Swarm root chunk of the representation of file (see 4.1.1) and also specifies the media mime type of the file to ensure that browsers can appropriately handle it. A manifest can be thought of as (1) a routing table, (2) a directory tree, or (3) an index, which makes it possible for Swarm to implement (1) web sites, (2) file-system directories, or (3) key–value stores (see 4.1.4), respectively. The use of manifests is essential for URL-based addressing within Swarm (see 4.1.3).

Manifests are represented as a compacted trie¹ in which individual trie nodes are serialised as chunks. The paths are associated with a manifest entry that specifies at least the reference. The reference may point to an embedded manifest if the path is a common prefix of more than one path in the collection, thereby implementing branching in the trie, as depicted in Figure 4.5.

**Figure 4.5:** Manifest structure. Nodes represent a generic trie node: it contains the forks which describe continuations sharing a prefix. Forks are indexed by the next byte of the key, whose value contains the Swarm reference to the child node as well as the longest prefix (compaction).

**Figure 4.6:** Manifest entry is a data structure that contains the reference to a file including metadata about a file or directory pertaining to the assembler, access control, and HTTP headers.

A manifest entry is essentially a reference that provides information about a file or directory. The metadata pertains to the following areas of concern:

Parameters for the downloader component, responsible for assembling chunks into a byte stream. This includes access control information, erasure coding parameters, and the publisher needed for chunk recovery.
Information relevant to client-side rendering, handled by the browser. This may include content type headers, or generically HTTP headers, that are picked up by the local swarm client’s API and set in the response header when the file is retrieved.
File information mapped to the file system during downloading, such as file permissions.

The high-level API for manifests offers functionality for uploading and downloading files and directories. It also provides an interface for adding documents to a collection on a path and deleting a document from a collection. Note that deletion here only means that a new manifest is created in which the path in question is omitted. There is no other notion of deletion in Swarm, i.e. the referenced value in the deleted manifest entry still remains in Swarm. Swarm exposes the manifest API via the bzz URL scheme.

4.1.3 URL-based addressing and name resolution

Earlier, we introduced the low-level network component of Swarm as a distributed immutable store of chunks (DISC, see 2.2.1). In the previous two sections, we discussed how files (4.1.1) and collections (4.1.2) can be represented in Swarm and identified using chunk references. Manifests provide a way to index individual documents in a collection, enabling them to serve as representations of websites hosted in Swarm. The root manifests serve as the entry-points to virtually hosted sites on Swarm and are therefore analogous to hosting servers. In the current web, domain names resolve to the IP addresses of host servers, and URL paths (of static sites) are mapped to entries in the directory tree based on their path relative to the document root set for the host. Analogously, in Swarm, domain names resolve to references to the root manifests, and URL paths are mapped to manifest entries based on their path.

When the HTTP API serves a URL, the following steps are performed:

domain name resolution: Swarm resolves the host part to a reference to a root manifest,
manifest traversal: recursively traverse embedded manifests along the path matching the URL path to arrive at a manifest entry,
serving the file: the file referenced in the manifest entry is retrieved and rendered in the browser with headers (notably content type) obtained from the metadata of manifest entry.

Swarm supports domain name resolution using the Ethereum Name Service (ENS). ENS is the system that, analogously to the DNS of the old web, translates human-readable names into system-specific identifiers, i.e., references in the case of Swarm. In order to use ENS, a Swarm node needs to be connected to an EVM-based blockchain that supports the Ethereum API (ETH mainnet, Ropsten, ETC, etc). Users of ENS can register a domain name on the blockchain and set it to resolve to a reference, most commonly the content hash of a public (unencrypted) manifest root. In the case that this manifest represents a directory containing the assets of a website, the default path for the hash may be set to be the desired root HTML page. When an ENS name is navigated to using a Swarm-enabled browser or gateway, Swarm will simply render the root HTML page and serve the rest of the assets provided in the relative path. Swarm facilitates easy website hosting, while also providing an interface to older pre-existing browsers and offering a decentralised improvement over the traditional DNS system.

4.1.4 Maps and key–value stores

This section describes two methods for implementing a simple distributed key–value store in Swarm. Both rely solely on tools and APIs which have been already introduced.

The first technique involves using manifests: Paths represent keys and the reference in the manifest entry with the particular path point to the value. This approach benefits from a full API enabling insert, update and remove through the bzz manifest API. Since manifests are structured as a compacted trie, this key–value store is scalable. Index metadata requires storage logarithmic to the number of key–value pairs. Lookup requires logarithmic bandwidth. The data structure allows for iteration that respects key order.

Single owner chunks also provide a way to define a key–value store.

The second technique simply posits that the index of the single owner chunk be constructed as a concatenation of the hash of the database name and the key. This structure only provides insert, without update or remove. Both insert and lookup are constant space and bandwidth. However, lookup is not safe against false negatives, i.e., if the chunk that represents the key–value pair is not found, this does not mean it has never been created (e.g. it may have been garbage collected). Thus, the single owner chunk based key–value store is best used as (1) a bounded cache of recomputable values, (2) mapping between representations such as a translation between a Swarm hash and a Keccak256 hash as used in the Ethereum blockchain state trie nodes, or (3) conventional relational links, such as likes, upvotes, and comments on social media posts.

4.2 Access control

This section first addresses the confidentiality of content using encryption. Encryption becomes especially useful once users are granted the ability to manage others’ access to restricted content. This encompasses scenarios such as managing private shared content and granting authorisation for members to access specific areas of a web application. In this way, we provide a robust and simple API to manage access control, something that is traditionally handled through centralised gate-keeping which is subject to frequent and disastrous security breaches.

4.2.1 Encryption

This section focuses on achieving confidentiality in a distributed public data storage. We explore how to fulfill the natural requirement for many use cases to store private information while ensuring that it remains accessible only to specific authorised parties using Swarm.

It is clear that the pseudo-confidentiality provided by the server-based access control predominantly used in current web applications is inadequate. In Swarm, nodes are expected to share the chunks with each other, in fact, storers of chunks are incentivised to serve them to anyone who requests them. This decentralised architecture makes it infeasible for nodes to act as the gatekeepers trusted with controlling access to the data. Moreover, since any node in the network could potentially be a storer, the confidentiality solution must not reveal any information that could distinguish a private chunk from random data. As a consequence of this, the only way to prevent unauthorized parties from accessing private chunks is through encryption. In Swarm, if a requestor is authorized to access a chunk, they must possess a decryption key that allows them to decrypt the chunk. Unauthorized parties, on the other hand, must not have access to the decryption key. Incidentally, this mechanism also serves as the basis for plausible deniability.

Encryption at the chunk level is described in 2.2.4. It has the desirable property of being virtually independent of the chunk store layer, using the exact same underlying infrastructure for storing and retrieving chunks as unencrypted content. The only difference between accessing private and public data is the presence of a decryption/encryption key in the chunk references and the associated minor cryptographic computational overhead.

The storage API’s raw GET endpoint allows both encrypted and unencrypted chunk references. Decryption is triggered if the chunk reference is double size; consisting of the address of the encrypted chunk and a decryption key. Using the address, the encrypted chunk is retrieved, stored, and decrypted using the supplied decryption key. The API responds with the resulting plaintext.

The storage API’s POST endpoint expects users to indicate if they want to have encryption on the upload or not. In both cases, the chunk will be stored and push-synced to the network, but if encryption is desired, the encrypted chunk needs to be created first. If no further context is given, a random encryption key is generated which is used as a seed to generate random padding to fill the chunk up to a complete 4096 bytes if needed, and finally this plaintext is encrypted with the key. In the case of encryption, the API POST call returns the Swarm reference, which consists of the Swarm hash as the chunk address and the encryption key.

In order to guarantee the uniqueness of encryption keys as well as to ease the load on the OS’s entropy pool, it is recommended (but not required) to generate the key as the MAC of the plaintext using a (semi-) permanent random key stored in memory. This key can be permanent and generated using scrypt (Percival 2009) with a password provided upon startup. Instead of the plaintext, a namespace and path of the manifest entry can be used as context. Using a key derivation function in this way has the consequence that chunk encryption will be deterministic as long as the context is the same: If we exchange one byte of a file and encrypt it with the same context, all data chunks of the file except the one that was modified will end up being encrypted exactly as the original. Encryption is therefore deduplication-friendly.

4.2.2 Managing access

This section describes the process the client needs to follow in order to obtain the full reference to the encrypted content. This protocol relies on basic meta-information, which is simply encoded as plaintext metadata and explicitly included in the root manifest entry for a document. This level of access, known as root access, does not require special privileges.

In contrast, granted access is a type of selective access that necessitates both root access and access credentials, i.e. an authorised private key or passphrase. Granted access allows different levels of privileges for accessing the content by multiple parties sharing the same root access. This approach allows for updating the content without changing access credentials. Granted access is implemented using an additional layer of encryption on references.

The symmetric encryption of the reference is called the encrypted reference, and the symmetric key used in this layer is called the access key.

In the case of granted access, the root access meta-information contains both the encrypted reference and the additional information required for obtaining the access key using the access credentials. Once the access key is acquired, the reference to the content is obtained by decrypting the encrypted reference with the access key. The resulting full reference consists of the address root chunk and the decryption key for the root chunk. The requested data can then be retrieved and decrypted using the standard method.

The access key can be obtained from a variety of sources, three of which we will define.

**Figure 4.7:** Access key as session key for single party access.

First, a session key is derived from the provided credentials. In the case of granting access to a single party, the session key is used directly as the access key, as depicted in Figure 4.7. However, in the case of multiple parties, an additional mechanism is used to transform the session key into the access key.

Passphrase

The simplest credential for deriving a session key is a passphrase. The session key is obtained from a passphrase using scrypt with parameters that are specified within the root access meta-information. The output of scrypt is a 32-byte key that may be directly used for Swarm encryption and decryption algorithms.

In typical use cases, the passphrase is distributed by an off-band means with adequate security measures, or exchanged in person. Any user knowing the passphrase from which the key was derived will be able to access the content.

Asymmetric derivation

A more sophisticated credential is a private key, identical to those used throughout Ethereum for accessing accounts, i.e. an elliptic curve using secp256k1. In order to obtain the session key, an elliptic curve Diffie-Hellman (ECDH) key agreement must be performed between the content publisher and the grantee. The resulting shared secret is hashed together with a salt. The content publisher’s public key as well as the salt are included among metadata in the root access manifest. Based on the standard assumptions of ECDH, this session key can only be computed by the publisher and the grantee and no-one else. Once again, when access is granted to a single public key, the session key derived this way can be directly used as the access key for the decryption of the encrypted reference. Figure 4.8 summarises the use of credentials to derive the session key.

**Figure 4.8:** Credentials to derive session key.

4.2.3 Selective access to multiple parties

In order to manage access by multiple parties to the same content, an additional layer is introduced to obtain the access key from the session key. In this variant, grantees can be authenticated using either type of credentials, however, the session key derived as described above is not directly used as the access key for decrypting the reference. Instead, two keys are derived from the session key: a lookup key and an access key decryption key. These keys are obtained by hashing the session key with two different constants ( $0$ and $1$ , respectively).

When granting access, the publisher needs to generate a global access key to encrypt the full reference, and then encrypts it with the access key decryption keys for each grantee. Thereafter, a lookup table is created, mapping each grantee’s lookup key to their encrypted access key. Then, for each lookup key, the access key is encrypted with the corresponding access key decryption key.

This lookup table is implemented as an access control trie (ACT) in Swarm manifest format. The paths in the ACT correspond to the lookup keys and manifest entries containing the ciphertext of the encrypted access keys as metadata attribute values. The ACT manifest is an independent resource referenced by a URL, which is included among the root access metadata to indicate whether or not an ACT is to be used.

When accessing content, the user follows these steps: they retrieve the root access meta data, identify the ACT resource, and then calculate their session key using either their passphrase and the scrypt parameters or the publisher public key, their private key, and a salt. From the session key, they can derive the lookup key by hashing it with $0$ , and then retrieve the manifest entry from the ACT. For this, they will need to know the root of the ACT manifest and then use the lookup key as the URL path. If the entry exists, the user obtains the value of the access key attribute in the form of a ciphertext that is decrypted with a key derived by hashing the session key with the constant $1$ . The resulting access key can then be used to decrypt the encrypted reference included in the root access manifest, as seen in Figure 4.9. Once the manifest root is unlocked, all references contain the decryption key.

**Figure 4.9:** Access control for multiple grantees involves an additional layer to get from the session key to the access key. Each user must lookup the global access key specifically encrypted to them. Both the key to look up and the key for decrypting the access key are derived from the session key, which, in turn, requires their credentials.

This access control scheme offers several desirable properties:

Checking and looking up one’s own access is logarithmic in the size of the ACT.
The size of the ACT merely provides an upper bound on the number of grantees, without disclosing any information beyond this upper bound about the set of grantees to third parties. Even those included in the ACT can only learn that they are grantees, but obtain no information about other grantees beyond an upper bound on their number.
Granting access to an additional key requires extending the ACT by a single entry, which is logarithmic in the size of the ACT.
Revoking access requires changing the access key and therefore the rebuilding of the ACT. Note that this also requires that the publisher retain a record of the public keys of grantees after the initial creation of the ACT.

4.2.4 Access hierarchy

In the simplest case, the access key is a symmetric key. However, this is just a special case of the more flexible solution, where the access key consists of a symmetric key and a key derivation path by which it is derived from a root key. In this case, a derivation path may also be included in addition to the encrypted reference. Any party with an access key whose derivation is a prefix to the derivation path of the reference can decrypt the reference by deriving its key using their own key and the rest of the reference’s derivation path.

This allows for a tree-like hierarchy of roles, possibly reflecting an organisational structure. As long as role changes are "promotions", i.e. they result in increased privileges, modifying a single ACT entry for each role change is sufficient.

4.3 Feeds: mutability in an immutable store

Feeds are a unique feature of Swarm that constitute the primary use case for single owner chunks. They have a wide range of applications, including versioning revisions of a mutable resource, indexing sequential updates to a topic, publishing parts to streams, or posting consecutive messages in communication channels. Feeds implement persisted pull-messaging and can also be interpreted as a pub-sub system.

In Section 4.3.1, we introduce how feeds are composed of single owner chunks with an indexing scheme, the choice of which we discuss in 4.3.2. We then delve into the importance of feed integrity and methods for verifying and enforcing it in 4.3.3. Section 4.3.4 describes epoch-based feeds which provide a search mechanism for feeds that receive sporadic updates. Finally, in 4.3.5, we demonstrate how feeds can be used as an outbox for sending and receiving subsequent messages within a communication channel.

4.3.1 Feed chunks

A feed chunk is a single owner chunk with the associated constraint that the identifier is composed of the hash of a feed topic and a feed index. The topic is a 32-byte arbitrary byte array, typically the Keccak256 hash of one or more human-readable strings specifying the topic and optionally the subtopic of the feed (see Figure 4.10 for a visual representation).

Figure 4.10: Feed chunks are single owner chunks where the identifier is the hash of the topic and an index. Indexes are deterministic sequences calculated according to an indexing scheme. Subsequent indexes for the same topic represent identifiers for feed updates.

The index of a feed can take various forms, defining some of the potential types of feeds. The ones discussed in this section are: (1) simple feeds that use incremental integers as their index (4.3.2); (2) epoch-based feeds that use an epoch ID (4.3.4); and (3) private channel feeds that use nonces generated through a double ratchet key chain. (4.3.5). The common characteristic among all these feed types is that both the publisher (owner) and the consumer must be aware of the indexing scheme in order to interact with the feed effectively.

Publishers have exclusive ownership of the feed chunks and are the only ones authorised to post updates to their feed. Posting an update requires (1) constructing the identifier from the topic and the correct index, and (2) signing the identifier concatenated with the hash of the arbitrary content of the update. Since the identifier designates an address in the owner’s subspace of addresses, this signature effectively assigns the payload to this address (see 2.2.3). In this way, all items published on a particular feed can be verified to have been created solely by the owner of the corresponding private key.

On the consumer side, users can retrieve a feed by specifying the chunk address. Retrieving a specific update requires the consumer to construct the address from the owner’s public key and the identifier. To calculate the identifier, the user needs two pieces of information: the topic and the appropriate index, for which they need to know the indexing scheme.

Feeds enable Swarm users to represent a sequence of content updates. The content of each update serves as the payload, which the feed owner signs against the identifier. The payload can be a swarm reference, allowing users to retrieve the associated data.

4.3.2 Indexing schemes

Different types of feeds require different indexing schemes and different lookup strategies. In the following sections, we introduce a few largely independent dimensions in which feeds can be categorised and which appear relevant in making a choice.

The actual indexing scheme used, or even the presence of one (i.e. if the single-owner chunk is a feed chunk at all), is left unrevealed in the structure of a feed chunk, as this information is not needed for the validation of a chunk by forwarding nodes. Including the subtype explicitly in the structure would only result in unnecessary information leakage.

Update semantics

Updates of a feed can be categorised into three subtypes, each with distinct semantics.

Feeds that represent revisions of the same semantic entity are called mutable resource updates. These resources mutate because the underlying semantic entity undergoes changes, such as updates to your CV or the expansion of a resource description, like the Wikipedia entry about a Roman emperor. Users will typically be interested in the latest update of such resources, with past versions having only historical significance. As the term ’revision’ suggests, the interpretation of an update in the case of mutable resources is substitutive.

The second subtype of feeds, known as series updates, represents a series of content linked by a common thread, theme, or author. In a series, each update is considered as an alternative and independent instantiation or episode that unfolds in a chronological order, such as include social media status updates, a person’s blog posts, or blocks of a blockchain. Series updates present a cohesive narrative or progression of information, allowing users to engage with the content in a sequential and interconnected manner.

Finally, there are partitions expressed as feeds, where updates are meant to be accumulative: subsequent updates are sequentially added to earlier ones. A common example is a video stream that consists of multiple parts. Partition feeds mainly differ from series in that the individual feed updates are not meaningful or interpretable on their own. Instead, the temporal sequence of updates may represent a processing order that corresponds to some serialisation of the structure of the resource rather than reflecting temporal succession. When accessing a partition feed, it may be necessary to accumulate all the parts in order to ensure the integrity of the represented resource. This means that each update builds on the previous ones, resulting in a comprehensive representation of the entire resource.

If subsequent updates of a feed include a reference to a data structure that indexes the previous updates (e.g. a key–value store using the timestamp for the update or simply the root hash of the concatenation of update content), then the lookup strategy in for all three feed types reduces to retrieving the latest update.

Update frequency

Feeds that are updated over time may be categorised into several types. Some are sporadic feeds with irregular asynchronicities, i.e. updates that can have unpredictable gaps. Another type is periodic feeds, where updates are published at regularly recurring intervals.

Additionally, we will discuss real-time feeds, where the update frequencies may not follow a regular pattern but instead vary within the temporal span of real-time human interaction, i.e. they are punctuated by intervals in the second to minute range.

Subscriptions

Feeds can be interpreted as pub/sub systems that offer persistence, enabling asynchronous pulls. In what follows, we analyse how the choice of indexing scheme affects the implementation of subscriptions to feeds as pub/sub.

In order to cater for subscribers of a feed, the updates need to be tracked. When we have knowledge of the latest update, we can employ periodic polling to fetch subsequent updates. If the feed is periodic, one can start polling after a known period. Alternatively, if the feed updates are frequent enough (at most a 1-digit integer orders of magnitude rarer than the desired polling frequency), then polling is also feasible. However, if the feed is sporadic and updates occur unpredictably, polling may not be practical. In such scenarios, alternative methods like push notifications (see 4.4.1 and 4.4.4) become preferable for ensuring timely updates.

If we have missed out on polling for a period of time due to being offline, or just created the subscription, we can also rely on push notifications or use a lookup strategy to retrieve the necessary updates.

Looking up partitions poses no difficulty as each update needs to be fetched and accumulated. In this case, the strategy of just iterating over the successive indexes cannot be improved. For periodic feeds, we can just calculate the index for a given time, hence asynchronous access is efficient and trivial. However, looking up the latest version of a sporadically updated feed requires some search and hence benefits from epoch-based indexing.

Aggregate indexing

A set of sporadic feeds can be turned into a periodic one using feed aggregation. Imagine, for example, a multi-user forum like Reddit, where each registered participant would publish comments on a post using sporadic feeds. In this scenario, would be impractical for each user to monitor the comment feed of every other user and search through their sporadic feeds for updates in order to retrieve all the comments on the thread. A more efficient approach is to just do it once though for all users. Indexers do exactly that: they aggregate everyone’s comments into an index, a data structure whose root can then be published as a periodic feed, see Figure 4.11. The frequency of updates can be chosen to provide a real-time feed experience; even if the rate of change does not justify it, i.e. some updates may be redundant, the cost amortises over all users who use the aggregate feed, making it economically sustainable.

Figure 4.11: Feed aggregation serves to merge information from several source feeds in order to save consumers from duplicate work. Left: 6 nodes involved in group communication (discussing a post, having real time chat, or asynchronous email thread). Each node publishes their contributions as outbox feed updates (small coloured circles). Each participant polls the other’s epoch-based feeds, duplicating work with the lookup. Right: the 6 nodes now register as sources with an aggregator which polls the nodes’ feed and creates indices that aggregate the sources into one data structure which each participant can then pull.

A service of this kind can be offered with arbitrary levels of security, yet trustlessly, without relying on reputation. Using consensual data structures for the aggregation, incorrect indexes can be proven using affordable and concise inclusion proofs (see 4.1.1) and therefore any challenges related to correctness can be evaluated on chain. Providers face the risk of losing their deposit if a challenge remains unrefuted, which acts as a strong incentive for them to uphold a high standard of service quality.

4.3.3 Integrity

We consider a feed to have integrity when each of its updates is unambiguous. Formally, this means that for each index, the respective feed identifier is only ever assigned to a single payload. Incidentally, this also implies that the corresponding feed chunk has integrity. As discussed in 2.2.3, this is a prerequisite for consistent retrieval. If the payloads of the successive updates are imagined as blocks of a blockchain, then the criterion of integrity requires feed owners to avoid creating forks in their chain.

In fact, the integrity of a feed can only be guaranteed by the owner. However, it is important to consider whether the integrity can be effectively checked or enforced. Owners can commit to the integrity of their feeds by staking a deposit on the blockchain, which they stand to lose if they are found to double sign on an update. While this may provide a strong disincentive to fork a feed in the long run, it alone does not offer sufficient guarantees to consumers of the feed with respect to integrity. Because of this, we must design indexing schemes that actively enforce the desired integrity standards.

Authoritative version history

Mutable resource update feeds track versions pretty much the same way as the Ethereum Name Service does. When a version is consolidated, such as a website update, the owner wants to register the content address of the current version. In order to guarantee that there is no dispute over history, the payload needs to incorporate the hash of the previous payload. This requirement implies that the payload must be a composite structure. However, if the goal is to have a payload consisting of solely a manifest or manifest entry so that it can be rooted to a URL path or directly displayed, this is not possible. Additionally, if in cases where the feed content is not a payload hash, ENS registers a payload hash despite the absence of the corresponding chunk in Swarm, thus violating the semantics of ENS.

An indexing scheme that incorporates the previous payload hash into the subsequent index operates in a similar manner to a blockchain. It expresses the owner’s unambiguous commitment to a particular history and requires any consumer who reads and uses it to acknowledge and accept that history. Looking up such feeds is only possible by retrieving each update since the last known one. The address refers to the update chunk, so registering the update address both guarantees historic integrity and preserves ENS semantics so that the registered address is just a Swarm reference to a chunk. Such feeds establish an authoritative version history, i.e. they provide a secure audit trail of the revisions made to a mutable resource.

Real-time integrity check

A deterministically indexed feed provides the ability to perform a real-time integrity check. In the context of feeds that represent blockchains (ledgers/side-chains), integrity refers to having a non-forking and unique chain commitment. The ability to enforce this in real-time allows fast and secure definitions of transaction finality.

We illustrate this with an example of an off-chain p2p payment network where each node’s locked-up funds are allocated to a fixed set of creditors (see more detail in Trón et al. 2019a). The creditors of a node need to verify the accuracy of the reallocations, i.e. that the total increases are covered by countersigned decreases. If a debitor keeps publishing a deposit allocation table for an exhaustive list of creditors, by issuing two alternatives to targeted creditors, the debitors will be able to orchestrate a double spend. Conversely, if there is certainty in the uniqueness of this allocation table, the creditor can confidently conclude finality.

We claim that using Swarm feeds, this uniqueness constraint can be verified in real-time.

The key insight is that it is impossible to meaningfully control the responses to a single owner chunk request: Even if an attacker has control over the entire neighbourhood of the chunk address, there is no systematic way to respond with particular versions to particular requestors.² This is due to the inherent ambiguity of the originator of the request in the forwarding Kademlia protocol. Let us imagine that the attacker, using some sophisticated traffic analysis, has the chance of $1 / n$ (asymptotic ceiling) to identify the originator and give a differential response. By sending multiple requests from random addresses, however, one can test integrity and consider a consistent response a requirement for concluding finality. The probability that the attacker can give a consistent differential response to a creditor testing with $k$ independent requests is $1 / n^{k}$ . With linear bandwidth cost in $k$ , we can achieve exponential degrees of certainty about the uniqueness of an update. If a creditor observes consistency in the responses, it can conclude that there is no alternative allocation table.

By requiring the allocation tables to be disseminated as feed updates, we can leverage the advantages of permissionlessness, availability, and anonymity to enforce feed integrity. If the feed is a blockchain-like ledger, a real-time integrity check translates to fork finality.

4.3.4 Epoch-based indexing

In order to use single owner chunks to implement sporadic feeds with flexible update frequency, we introduce epoch-based feeds as an indexing scheme. In this scheme, the identifier of a single owner chunk incorporates anchors related to the time of publishing. In order to be able to find the latest update, we introduce an adaptive lookup algorithm.

Epoch grid

An epoch is a defined time period starting at a specific point in time, known as the epoch base time, and lasts for a specific duration. The lengths of these periods are expressed as powers of 2 in seconds, ranging from the shortest period of $2^{0} = 1$ second to the longest period of $2^{31}$ seconds.

An epoch grid is the arrangement of epochs, where each row represents an alternative partitioning of time into various disjoint epochs of the same length. Rows, also known as levels, are indexed by the logarithm of the epoch length putting level 0 with 1 second epoch length at the bottom by convention, see Figure 4.12.

**Figure 4.12:** Epoch grid showing the first few updates of an epoch-based feed. Epochs occupied are marked in yellow and are numbered to reflect the order of updates they represent.

When representing a epoch-based feed in an epoch grid, each update is assigned to a specific epoch within the grid based on its timestamp. In particular, an update is mapped to the longest free epoch that includes the timestamp. This structure gives the series of updates a contiguous structure, which allows for easy search. The contiguity requirement implies that by knowing the epoch of the previous update, the subsequent update can be mapped to a specific epoch without ambiguity.

To identify a specific epoch within the epoch grid, we need to know both the epoch base time and the level. This pair is called the epoch reference. To calculate the epoch base time for any given instant in time $t$ at a particular level $l$ , the $l$ least significant bits of $t$ are dropped. The level requires one byte, and the epoch base time (using Linux seconds) 4 bytes, so the epoch reference can be serialised in 5 bytes. It is worth noting that the epoch reference of the initial update of any epoch-based feed is always the same.

Mapping epochs to feed update chunks

The serialised epoch reference serves as the feed index for mapping feed updates to feed chunks. The topic of the feed hashed together with the index results in the feed identifier used in constructing the single owner chunk that expresses the feed chunk.

To determine the appropriate epoch for storing a subsequent update, the publisher needs to know the location of the previous update. If the publisher does not keep track of this information, they can use the lookup algorithm to find their most recent update.

Lookup algorithm

When consumers retrieve feeds, their objectives typically revolve around looking up the state of the feed at a particular time (historical lookup) or to retrieving the latest update.

If historical lookups based on a target time are required, the update can incorporate a data structure that maps timestamps to corresponding states. In such cases, finding any update later than the target can be used to deterministically look up the state at an earlier time.

If no such index is available, historical lookups need to find the shortest filled epoch whose timestamp is earlier than the target.

To select the best starting epoch from which to walk our grid, we have to assume the worst case scenario, which is that the resource has never been updated since the last time we saw it. If we don’t know when the resource was last updated, we assume it to be 0.

We can guess a start level as the position of the first non-zero bit of $lastUpdate ⊻ NOW$ counting from the left. The bigger the time difference between the last update time and the current time, the higher the level will be.

4.3.5 Real-time data exchange

Feeds can be used to represent a communication channel, i.e. the outgoing messages of a persona. This type of feed, called an outbox feed can be created to provide email-like communication or instant messaging, or even the two combined. For email-like asynchronicities, epoch-based indexing can be used, while deterministic sequence indexing is more suitable for instant messaging. In group chat or group email, confidentiality is managed through an access control trie over the data structure, which indexes each party’s contribution to the thread. Communication clients can retrieve the feeds of each group member relating to a particular thread and merge their timelines for rendering.

Even forums could be implemented with such an outbox mechanism described above. However, as the number of registered participants increases, aggregating all outboxes on the client side may become impractical. In such cases, index aggregators or other schemes may be necessary to crowdsource the combination of the data.

Two-way private channels

Private two-party communication can also be implemented using outbox feeds, see Figure 4.13. The parameters of such feeds are established during the initial key exchange or registration protocol (see 4.4.2), ensuring that the parties consent on the indexing scheme as well as the encryption used.

**Figure 4.13:** Swarm feeds as outboxes for private communication. Outbox feeds represent consecutive messages from a party in a conversation. The indexing scheme can follow a key management system with strong privacy which obfuscates the communication channel itself and renders interception attacks prohibitively expensive.

For real-time instant messaging using a series feed, it is important to have an indexing scheme that supports deterministic continuations for at least a few updates ahead. This enables sending retrieve requests for upcoming updates in advance, i.e. during or even prior to processing the previous messages. When these retrieve requests arrive at the nodes closest to the requested update address, it is expected that the chunk will not be available since the other party will not have sent them yet. However, even these storer nodes are incentivised to keep retrieve requests alive until they expire (as discussed in 2.3.1). This means that up until the end of their time-to-live setting (30 seconds), the requests will function as subscriptions: the arrival of the update chunk triggers the delivery response to the open request as if it was the notification sent to the subscriber. This reduces the expected message latency to less than twice the average time of one-way forwarding paths, see Figure 4.14.

Figure 4.14: Advance requests for future updates. The diagram shows the timeline of events during instant messaging between two parties A and B, using outbox feeds. The columns represent the neighbourhood locations of the feed update addresses. The circles show the time of protocol messages arriving: colors indicate the origin of data, empty circles are retrieve requests, and full circles are push-sync deliveries arriving at the respective neighbourhood. Note that the outbox addresses are deterministic 3 messages ahead, allowing retrieve requests to be sent before the corresponding updates arrive. Importantly, the latency between one party sending a message

m

and the other receiving it is shown as

δ (m)

. Messages

A_{3}

and

A_{4}

arrive before

A_{2}

which can be reported and repaired. If address predictability was only limited to 1 message ahead, both

B_{2}

and

B_{3}

would have much longer latencies. Also note that the latencies of

B_{2}

and

B_{3}

are helped by advance requests: the retrieve requests for

B_{4}

and

B_{5}

are sent upon receipt of

B_{1}

and

B_{2}

and arrive at their neighbourhood at the same time as the messages

B_{2}

and

B_{3}

arrive at theirs, respectively. If address predictability was limited to 1 message ahead, this would negatively impact the latencies of

B_{2}

and

B_{3}

Post-compromise security

A key management solution called double ratchet is the de-facto industry standard used for encryption in instant messaging. It is customary to use the extended triple Diffie–Hellmann key exchange (X3DH) to establish the initial parameters for the double-ratchet key chains (see 4.4.2).

The double-ratchet approach combines a ratchet based on a continuous key-agreement protocol with a ratchet based on a key-derivation function (Perrin and Marlinspike 2016). This scheme can be generalised (Alwen et al. 2019) and understood as a combination of well-understood primitives. It has been demonstrated to provide (1) forward secrecy, (2) backward secrecy,³ and (3) immediate decryption and message loss resilience.

**Figure 4.15:** Future secrecy for update addresses

In addition to the confidentiality due end-to-end encryption, Swarm offers further resistance against attacks. Due to the forwarding Kademlia protocol, the sender can remain ambiguous and deniable. Furthermore, the normal push-sync and pull-sync traffic helps to obfuscate messages. To make it really hard for an attacker, the sequence of indexes can also provide future secrecy if we add more key chains to the double-ratchet machinery. Beside root, sending, and receiving encryption key chains, two additional keys are introduced: outgoing and incoming outbox index key chains, see Figure 4.15. As a result of this measure, the underlying communication channel is obfuscated, i.e. intercepting an outbox update chunk and knowing its index reveals nothing about previous or subsequent outbox update indexes. This makes subsequent messages prohibitively difficult and costly to monitor or intercept.

In 4.3.3, we used factoring in the payload hash into the indexing scheme to achieve non-mergeability of chains (unambiguous history). Inspired by this, we propose to also factor in the payload hash into the subsequent feed update index. This introduces an additional property called recover security, which, intuitively, ensures that once an adversary manages to forge a message from A to B, then no future message from A to B will be accepted by B. This is guaranteed if the authenticity of A’s messages to B affects the subsequent feed index. If there is a mismatch (indicating a forged message), the messages will be looked up at the wrong address, leading to the abandonment of the communication channel and the initiation of a new one. By implementing this approach, the communication channel achieves complete confidentiality and becomes a zero-leak solution for real-time messaging.

4.4 Pss: direct push messaging with mailboxing

This section introduces pss, Swarm’s direct node-to-node push messaging solution. Functionalities of and motivation for its existence are playfully captured by alternative resolutions of the term:

postal service on Swarm: Delivering messages if recipient is online or depositing for download if not.
pss is bzz whispered: Beyond the association to Chinese whispers, it surely carries the spirit and aspiration of Ethereum Whisper.⁴ Pss piggybacks on Swarm’s distributed storage for chunks and hence inherits their full incentivisation for relaying and persistence. At the same time it borrows from Whisper’s crypto, envelope structure and API.
pss! instruction to hush/whisper: Evokes an effort to not disclose information to third parties, which is found exactly in the tagline for pss: truly zero-leak messaging where beside anonymity and confidentiality, the very act of messaging is also undetectable.
pub/sub system: The API allows publishing and subscription to a topic.

First, in 4.4.1, we introduce Trojan Chunks, which are messages sent to storers that masquerade as chunks whose content address happens to fall in the proximity of their intended recipient. Section 4.4.2 discusses the use of pss to send contact messages to establish real-time communication channels. In 4.4.3, we explore the mining of feed identifiers to target a specific neighbourhood with the address of a single owner chunk and present the construct of an addressed envelope. Finally, building on Trojan chunks and addressed envelopes, 4.4.4 introduces update notification requests.

4.4.1 Trojan chunks

Cutting-edge systems that promise private messaging often struggle to offer truly zero-leak communication (Kwon et al. 2016). While linking the sender and recipient is cryptographically proven to be impossible, resistance to traffic analysis is harder to achieve. Maintaining sufficiently large anonymity sets requires high volumes available at all times. In the absence of mass adoption, guaranteeing high message rate in dedicated messaging networks necessitates constant fake traffic. However, with Swarm, there is an opportunity to disguise messages as chunk traffic, effectively obfuscating the act of messaging itself.

We define a Trojan chunk as a content-addressed chunk with a fixed internal content structure (see Figure 4.17):

span

8-byte little-endian uint64 representation of the length of the message

nonce

32-byte arbitrary nonce

Trojan message

4064-byte asymmetrically encrypted message ciphertext with underlying plaintext composed of

: 2-byte little-endian encoding of the length of the message in bytes $0 \leq l \leq 4030$ ,
: $32$ byte obfuscated topic ID
: $m$ bytes of a message
: $4030 - m$ random bytes.

**Figure 4.16:** A pss message is a Trojan chunk that wraps an obfuscated topic identifier with a Trojan message, which in turn wraps the actual message payload to be interpreted by the application that handles it.

**Figure 4.17:** The Trojan chunk wraps an asymmetrically encrypted Trojan message.

Knowing the public key of the recipient, the sender follows a specific process to send a trojan message. First, the message is wrapped in a trojan message format by prefixing it with length information and padding it to 4030 bytes. Then, the sender encrypts it using the recipient’s public key to obtain the ciphertext payload of the trojan chunk by asymmetric encryption. The sender then generates a random nonce such that when it is prepended to the payload, the chunk hashes to an address that starts with a destination target prefix. The destination target is a bit sequence that represents a specific neighbourhood in the address space. If the target is a partial address derived as a prefix of the recipient’s overlay address, matching the target means that the chunk falls in the neighbourhood of the recipient. If only the public key is known, it is assumed that it is the bzz account of the recipient, i.e. their overlay address can be calculated from it⁵ (see 2.1.2). The sender then uploads the resulting chunk to Swarm with postage stamps of their choice which then ends up being synced to the recipient address’ neighbourhood. If the recipient node is online and the bit length of the matching target is greater than the recipient’s neighbourhood depth, they will receive the chunk. In practice, targets should be $n + c$ bits long, where $n$ is the estimated average depth in Swarm and $c$ is a small integer.

Receiving Trojan messages

The recipient only knows that a chunk is a pss message once they have successfully opened the Trojan message with the private key corresponding to the public key that they advertise as their resident key (see 4.4.2) and perform an integrity check/topic matching. Nodes that want to receive such Trojan Messages will continue attempting to decrypt all messages that they are closest to. Forwarding nodes (or anyone else apart from sender and recipient) have no way to distinguish between a random encrypted chunk and a Trojan message, which means that communication is perfectly obfuscated as generic chunk traffic.

After the recipient has opened the envelope using asymmetric decryption, they proceed with a combined step of integrity check and topic matching step. Knowing the length of the payload (from the first 2 bytes of the message), the recipient takes the payload slice and calculates the Keccak256 hash of it. For each topic the client is subscribed to, the recipient then hashes the payload hash together with the topic. If the resulting segment xor-ed with the topic matches the obfuscated topic ID in the message, then the message is indeed meant as a message with the said topic and the registered handler is called with the payload as argument.

Mailboxing for asynchronous delivery

If the recipient is not online, the Trojan chunk will prevail as any other chunk would, depending on the postage stamp it has. Whenever the recipient node comes online, it pull-syncs the chunks from the closest neighbourhood, including all Trojan chunks and their own unreceived messages. In other words, through Trojan messages, pss automatically provides asynchronous mailboxing functionality, allowing undelivered messages to be preserved and accessible to the recipient when they come online without any additional action needed from the sender. The duration of mailboxing is controlled with postage stamps, just like the storage of chunks, in fact, it is indistinguishable from regular chunk storage.

Mining for proximity

The process of finding a hash close to the recipient address is analogous to mining blocks on the blockchain. The nonce segment in a Trojan chunk also serves exactly the same purpose as a block nonce: it provides sufficient entropy to guarantee a solution. The difficulty of mining corresponds to the length of the destination target: The minimum proximity order required to ensure that the recipient will receive the message needs to be higher than the neighbourhood depth of the recipient⁶ when it comes online, so it is logarithmic in the number of nodes in the network. The expected number of nonces that need to be tried per Trojan message before an appropriate content address is found is exponential in the difficulty, and therefore equal to the number of nodes in the network. In practice, mining a Trojan chunk will never be prohibitively expensive or slow even for a single node, as the expected number of computational cycles needed to find the nonce is equal to the network size. Only a small delay in the second range may occur in a network of a billion nodes, and even that is acceptable given that Trojan messages are meant to be used only for one-off instances such as initiations of a channel. Subsequent real-time exchanges will happen using the previously described bidirectional outbox model using single owner chunks.

Anonymous mailbox

Asynchronous access to pss messages is guaranteed as long as the postage stamp has not expired. The receiver only needs to create a node with an overlay address corresponding to the destination target advertised to be the recipient’s resident address. This allows for the creation of an anonymous mailbox, which can receive pss messages on behalf of a client and then publish them on a separate, private feed. The intended recipient can then read the messages whenever they come back online.

Register for aggregate indexing

As discussed in 4.3.2, aggregate indexing services help nodes in monitoring sporadic feeds. For instance, a forum indexer can aggregate the contribution feeds of registered members, enabling the efficient tracking and access to forum updates. In the case of public forums, off-chain registration is a viable option that allows users to register without directly interacting with the blockchain. This can be achieved by simply sending a pss message to the aggregator.

4.4.2 Initial contact for key exchange

Encrypted communication requires a handshake protocol to establish the initial parameters that are used as inputs to a symmetric key generation scheme. One such protocol is the extended triple Diffie–Hellmann key exchange, or X3DH, is one such protocol (Marlinspike and Perrin 2016). X3DH is used to establish the initial parameters for a post-handshake communication protocol, like the double-ratchet scheme discussed earlier in the section on feeds (see 4.3.5).

To implement the X3DH protocol with pss in a serverless setting. Swarm utilises the same primitives as are customary in Ethereum, i.e. secp256k elliptic curve, Keccak256 hash, and a 64-byte encoding for EC public keys.

The Swarm X3DH protocol facilitates the establishment of a shared secret between two parties, which serves as the input used to determine the encryption keys used during two-way messaging after the handshake. The initiator is the party that initiates a two-way communication with the responder. The responder is supposed to advertise the information necessary to allow parties previously unknown to the responder to initiate contact. Zero-leak communication can be achieved by first performing an X3DH. This protocol is used to establish the seed keys used by the double-ratchet protocol for the encrypting data, as well as the feed indexing methodology used. This will enable the responder to retrieve the updates of the outbox feed.

X3DH uses the following keys:⁷

$K_{r}^{ENS}$: responder long-term public identity key
$K_{r}^{Res}$: responder resident key (aka signed pre-key)
$K_{i}^{ID}$: initiator long-term identity key
$K_{i}^{EPH}$: initiator ephemeral key for the conversation

Figure 4.18: The X3DH pre-key bundle feed update contains the resident key and resident address and is optionally together encrypted with the ENS name hash to prove uniqueness and provide authentication.

A pre-key bundle consists of all information the initiator needs to know about responder. However, rather than storing this information on external servers, it is instead stored in Swarm. For human-friendly identity management, ENS can be optionally used to provide familiar username-based identities. The owner of the ENS resolver represents the authenticated long-term public identity key for the persona. By utilising the long-term identity address, an epoch-based feed with a topic ID can be created, indicating that it provides the pre-key bundle for potential correspondents. When initiating communication with a new identity, the initiator retrieves the latest update from the feed, which contains the current resident key (aka signed pre-key) and current addresses of residence, i.e. the (potentially multiple) overlay destination targets where the persona expects to receive pss messages. The signature within the feed update chunk signs both the resident key (cf. signed pre-key) and the destination targets. The public key that is recovered from this signature gives the long-term identity public key, see Figure 4.18.

**Figure 4.19:** X3DH initial message. Initiator retrieves the ENS owner as well as the latest update of responder’s pre-key bundle feed containing the resident key and resident address. Initiator sends their identity key and an ephemeral key to responder’s resident address using the resident key for encryption.

In order to invite a responder to an outbox-feed based private communication channel, the initiator first looks up the responder’s public pre-key bundle feed and sends an initial message to the responder (see Figure 4.19), indicating their intent to communicate. She then shares the parameters required to initiate the encrypted conversation, including the public key of her long-term identity and the public key of the ephemeral key-pair generated specifically for that conversation. These details are delivered to the potential responder by sending a Trojan pss message addressed to the responder’s current address of residence, which is also advertised in their pre-key bundle feed.

**Figure 4.20:** X3DH secret key. Both parties can calculate the triple Diffie-Hellmann keys and xor them to get the X3DH shared secret key used as the seed for the post-handshake protocol.

After the responder receives this information, both parties have all the ingredients needed to generate the triple Diffie-Hellmann shared secret (see Figure 4.20).⁸ This shared secret constitutes the seed key for the double-ratchet continuous key agreement protocol as used in the signal protocol. The double-ratchet scheme ensures forward secrecy and post-compromise security to the end-to-end encryption. By applying separate key-chains for the outbox feed’s indexing scheme, additional recover security, i.e. resilience to message insertion attack, can be achieved. Most importantly, however, adding forward and backward secrecy to outbox addresses, obfuscates the communication channel, which renders sequential message interception contingent on the same security assumptions as encryption. This eliminates the only known attack surface for double-ratchet encryption. The obfuscation and deniability of the channel based on outbox feeds, together with the initial X3DH message being disguised, indistinguishable from a regular chunk, warrants classifying this communication approach as zero-leak communication.

4.4.3 Addressed envelopes

Mining single owner chunk addresses

The question immediately arises whether it is feasible to mine single owner chunks. Since the address in this case is the hash of a 32-byte identifier and a 20-byte account address, the ID provides sufficient entropy to mine addresses even if the owner account is fixed. So for a particular account, if we discover an ID such that the resulting single owner chunk address is close to a target overlay address, the chunk can be used as a message, similar to Trojan chunks. Importantly, however, since the address can be mined before the chunk content is associated with it, this construct can serve as an addressed envelope.

Let us make explicit the roles relevant to this construct:

issuer ( $I$ ): creates the envelope by mining an address.
poster ( $P$ ): puts the content into the envelope and posts it as a valid single owner chunk to Swarm.
owner ( $O$ ): possesses the private key to the account part of the address and can thus sign off on the association of the payload to the identifier. This effectively decides on the contents of the envelope.
target ( $T$ ): the constraint for mining: a bit sequence that must form the prefix of the mined address. It represents a neighbourhood in the overlay address space where the envelope will be sent. The length of the target sequence corresponds to the difficulty for mining. The longer this target is, the smaller the neighbourhood that the envelope will be able to reach.
recipient ( $R$ ): the party whose overlay address has the target sequence as its prefix and therefore the destination of the message.

The envelope can be perceived as open: since the poster is also the owner of the response single owner chunk, they are able to control what content is placed into the chunk. By constructing such an envelope, the issuer effectively allows the poster to send an arbitrary message to the target without the need for computational resources to mine the chunk. See Figure 4.21.

**Figure 4.21:** Stamped addressed envelopes timeline of events. Issuer $I$ creates the envelope encrypted for $P$ with an identifier such that $P$ as the single owner of the chunk, produces an address that falls in the recipient $R$ ’s neighbourhood. As a result, (only) $P$ can fill the envelope with arbitrary content and then use simple push-syncing to post it to $R$ .

When the poster wants to send a message to the recipient, they simply need to create a Trojan message and sign it against the identifier using the private key of the same account that the issuer used during the address mining process. By doing so, the resulting chunk will be valid. See Figure 4.22.

**Figure 4.22:** A stamped addressed envelope issued for $P$ and addressed to $R$ consists of an identifier which is mined so that when used to create a single owner chunk owned by $P$ , it produces an address which falls within $R$ ’s neighbourhood. This allows $P$ to construct a message, sign it against the identifier and using the postage stamp, post it using the network’s normal push-syncing to $R$ for free.

Pre-paid postage

These chunks exhibit the same behaviour as normal Trojan messages, maintaining their privacy properties, and in some cases, even improving them. The issuer/recipient can associate a random public key for encrypting the message or use symmetric encryption. If a postage stamp is pre-paid for an address and given to someone to post later, they can use push-sync to send the chunk to the target without the poster incurring any additional cost, as the postage has already been covered by the stamped addressed envelope. This construct effectively implements addressed envelopes with pre-paid postage and serves as a base layer solution for various high-level communication needs, such as (1) push notifications to subscribers without any computational or financial postage burden on the sender, (2) free contact vouchers, and (3) zero-delay direct message response.

Issuing a stamped addressed envelope

Issuing a stamped addressed envelope involves the following process:

assume: issuer $I$ , prospective poster $P$ , and prospective recipient $R$ with public keys $K_{I}, K_{P}, K_{R}$ and overlay addresses $A_{I}, A_{P}, A_{R}$ .
mine: $I$ finds a nonce $N_{R}$ such that when used as an identifier to create a single owner chunk, the address of the chunk hashes to $H_{R}$ , which is in the nearest neighbourhood of $A_{R}$ .
pay postage: $I$ signs $H_{R}$ to produce a witness for an appropriate postage payment to produce stamp $P S_{R}$ .
encapsulate: package $N_{R}$ and $P S_{R}$ , which represent the pre-paid envelope pre-addressed to recipient address, and encrypt it with $K_{P}$ then wrap it as a Trojan chunk.
mine: find a nonce $N_{P}$ such that the Trojan chunk hashes to $H_{P}$ , which is in the nearest neighbourhood of $A_{P}$ .

Receiving a stamped addressed envelope

A prospective poster $P$ is assumed to receive a Trojan message that consists of pre-paid envelope $E$ . In order to open it, she carries out the following steps:

decrypt: decrypt message with the private key belonging to $K_{P}$
deserialise: unpack and identify $P S_{R}$ and $N_{R}$ , extract $H_{R}$ from $P S_{R}$
verify: postage stamp $P S_{R}$ and check if $N_{R}$ hashed with the account for $K_{P}$ results in $H_{R}$ to ensure the associated address is in fact owned by $P$ .
store: store $N_{R}$ and $P S_{R}$

Posting a stamped addressed envelope

When the poster wants to use the envelope to send an arbitrary message $M$ to $R$ (with recipient $R$ potentially unknown to the sender), they must follow the following steps:

encrypt: encrypt the message content $M$ with $K_{R}$ to create a payload and wrap it in Trojan message $T$
hash: hash the encrypted Trojan message resulting in $H_{T}$
sign: sign $H_{T}$ against the identifier $N_{R}$ using the private key that belongs to $K_{P}$ , producing signature $W$
encapsulate: include nonce $N_{R}$ as ID, the signature $W$ and the Trojan message $T$ as the payload of a valid single owner chunk with address $H_{R}$
post: post the chunk with the valid stamp $P S_{R}$

Receiving a posted addressed envelope

When $R$ receives chunk with address $H_{R}$

verify: verify postage stamp $P S_{R}$ and validate the chunk as a single owner chunk with payload $T$ .
decrypt: decrypt $T$ with the private key belonging to $K_{R}$ .
deserialise: deserialise the plaintext as a Trojan message, identify the message payload $M$ and check its integrity.
consume: consume $M$ .

4.4.4 Notification requests

This section elaborates on the concept of addressed envelopes and presents three flavours, each implementing different types of notifications.

Direct notification from publisher

If an issuer wants to notify a recipient of the next activity on a feed, she needs to construct a stamped addressed envelope embedded in a regular Trojan message and send it to the publisher, as depicted in Figure 4.23. If the issuer is also the recipient, the same account can be used in both the request and the response envelope.

When the owner of a feed publishes an update, they include a reference to the update chunk in the envelope and send it to the recipient. More formally, the publisher creates a single owner chunk using the identifier of the pre-addressed envelope and signs off on the identifier associated with the feed update content as the payload of the single owner chunk. Subsequently, the publisher, now acting as the poster, push-syncs the chunk to the swarm. This process is known as direct notification from publisher.

**Figure 4.23:** A direct notification request contains a reference to a feed and wraps a pre-paid envelope that is mined for $P$ (the publisher or a known distributor of the feed) and addressed to recipient $R$ . The response follows the same process as generic stamped addressed envelopes, with the only difference being that the message is expected to be the feed update or a reference to its content.

The Trojan message is encrypted using the public key and specifies in its topic that it is a notification. As the address is mined to match the recipient overlay address on a sufficiently long prefix, the message ends up push-synced to the recipient’s neighbourhood. When the recipient comes online and receives the chunk, she detects that it is intended as a message. This can be done either by successfully decrypting the chunk content using the key for the address they had advertised, or by looking it up against the record of the address they saved when it was issued, in the case where the recipient has issued the pre-addressed envelope themselves. See Figure 4.24 for the timeline of events.

**Figure 4.24:** Direct notification from publisher timeline of events. Issuer $I$ constructs a prepaid envelope for the publisher or a known distributor of the feed $P / F$ and addresses to recipient $R$ . Together with a feed topic $I$ , it is sent to $P / F$ wrapped in a pss Trojan message. $P / F$ receives it and stores the request. When they publish the update, they wrap it in the envelope, i.e. sign the identifier received from $I$ against the feed update notification message and post it as a chunk, which $R$ will receive and hence, be notified of the feed update.

Notifications coming directly from publishers allow the poster to include arbitrary content within the envelope. The poster, who is also the owner of the envelope, has the authority to sign off on any content against the identifier when posting the envelope. To create the notification chunk address in advance, the issuer needs to know the account of the prospective poster. However, it is not necessary for the feed update address to be fixed, making this scheme applicable to (sporadic) epoch-based feeds.

Notification from neighbourhood

Consider a scenario where the owner of a feed has chosen not to reveal their overlay (destination target) or has declined to implement notifications. In this case, the feed is updated sporadically using simple sequential indexing, making it impractical to rely on polling for checking the latest update. Furthermore, the consumer may also go offline at any given time. Is there a method to ensure that consumers are still notified?

We introduce another construct, a neighbourhood notification, which works without the issuer of the notification knowing the identity of prospective posters. However, it requires the content or its hash to be known in advance so that it can be signed by the issuer themselves.

To ensure that a recipient is notified of the next update, the issuer can create a neighbourhood notification request by wrapping a message in a Trojan chunk. This message contains an addressed envelope (identifier, signature, and postage stamp) that the poster can use to construct the single owner chunk that serves as the notification. Note that the notification message need not contain any new information; simply receiving it is sufficient for the recipient. The payload of the notification contains the feed update address to indicate what it is a notification of. Upon receiving the notification, the receiver can simply send a regular retrieve request to fetch the actual feed update chunk that contains (or points to) the content of the message. See Figure 4.25 for the structure of neighbourhood notifications and notification requests.

**Figure 4.25:** Neighbourhood notification requests include not only the identifier and postage stamp but also a notification message and its signature, attesting it against the address. Thus, $P$ needs to construct the actual notification and when publisher $F$ posts their update to $P$ ’s neighbour, $P$ can post the notification to recipient $R$ .

If the notification only needs to contain the feed update address as its payload, the association with the identifier can be signed off by the issuers themselves. This signature, along with the identifier, should be considered a necessary component of the envelope and must be included in the notification request. Unlike open envelopes used with publishers directly, neighbourhood notifications can be viewed as closed envelopes, where the content is pre-approved by the issuer.

In this case, the issuer, not the poster, becomes the owner of the chunk, and the signature is already available to the poster when they create and post the notification. As a consequence, no public key or account address information is required from the poster. In fact, the identity of the poster does not need to be fixed, as any peer could qualify as a candidate poster. Issuers can send the notification request to the neighbourhood of the feed update chunk. The nearest neighbours will keep holding the request chunk as specified by the attached postage stamps until they receive the appropriate feed update chunk, which confirms the receipt of the notification. See Figure 4.26 for the timeline of events when using neighbourhood notifications.

**Figure 4.26:** Neighbourhood notification timeline of events. Issuer $I$ mines an identifier for a single owner chunk, with themselves as the owner, such that the chunk address falls within recipient $R$ ’s neighbourhood. The issuer, being the owner, also must sign the identifier against a prefabricated reminder and remember which node should be notified. When $P$ syncs the feed update, the notification is sent to $R$ through the usual push-sync mechanism.

But how can we make sure that notifications are not sent too early or too late? While the integrity of notification chunks is guaranteed by the issuer as their single owner, additional measures are required to prevent nodes that manage the notifications from sending them prematurely. It would be ideal if the notification could not be posted before the arrival of the update, otherwise false alarms could be generated by malicious nodes that service the request.

A simple measure is to symmetrically encrypt the message in the request using a key that is only revealed to the prospective poster upon receiving the feed update. For instance, the hash of the feed update identifier can serve as that key. In order to reveal that the notification request needs to be matched on arrival of the feed update, the topic must be left unencrypted, so here we do not encrypt the pss envelope asymmetrically but only the message and symmetrically.

Note that the feed update address as well as identifier can be known if the feed is public, so neighbourhood notifications are to be used with feeds whose subsequent identifiers are not publicly known.

Targeted chunk delivery

Normally, in Swarm’s DISC model, chunks are obtained using retrieve requests, which are forwarded towards the designated neighbourhood designated based on the requested chunk address (see 2.3.1). The first node on the route that has the chunk will respond and deliver the chunk as a backwarded response travelling back along the same route. In certain cases, however, it may be beneficial to have a mechanism to request a chunk from a specific neighbourhood where it is known to be stored and send it to a different neighbourhood where it is known to be needed. A construct called targeted chunk delivery is meant for such a use case: the request is a Trojan pss message, while the response, the delivery, must be a single owner chunk that wraps the requested chunk. The address of this delivery chunk is mined to fall into the recipient’s neighbourhood.

**Figure 4.27:** Targeted chunk deliveries are similar to neighbourhood notifications in that they are pre-addressed, pre-paid, and pre-signed. Similarly, the identifier is mined so that given issuer $I$ as the owner, it produces an chunk address that falls in recipient $R$ ’s neighbourhood. Here, the issuer signs the identifier against the hash of a content addressed chunk that they want to see posted to $R$ . If the targeted chunk delivery request lands with any node that has the requested chunk, they can use the envelope and the chunk’s content to produce a valid response, which is a single owner chunk wrapping a content addressed chunk.

**Figure 4.28:** Targeted chunk delivery timeline of events. Uploader $U$ uploads a chunk to Swarm, and it lands with a storer node $S$ . Nodes within $P$ replicate and pin this chunk. Now, if issuer $I$ wants this chunk delivered to $R$ , they take the prepaid envelope addressed to $R$ and send it unencrypted to the neighbourhood of a known host. This way, anyone who has the chunk can construct the notification and send or store it for later.

These ’chunk-in-a-soc’ responses are structurally similar to neighbourhood notifications in that the payload’s hash is already known (the content address of the chunk requested). The requestor can then sign off on its association with the identifier and include it, along with the signature, in the request message (see Figure 4.28). When a node receives such a request and has the requested chunk stored, it can construct a valid single owner chunk addressed to the recipient’s neighbourhood as a response. This makes the request generic, i.e. not tied to the identity of the prospective poster, therefore it can be sent to any neighbourhood that requires the content. Even multiple requests can be sent simultaneously, as the uniqueness of the valid response ensures the integrity of chunks (see 2.2.3 and 4.3.3). Targeted delivery is used in missing chunk recovery, see 5.2.3.

The difference is that neighbourhood notifications do not provide any new information, while targeted chunk delivery supplies the chunk data. Additionally, a chunk delivery wrapped in a single owner chunk does not involve message wrapping and does not have a topic. Encryption is not used in either the request or response. A consequence of not using encryption is that if there are multiple targets, it is sufficient for the initial Trojan request to match any one of these targets.

Table 4.1 summarises various properties of the three notification-like constructs.

type	owner	poster	request encryption	notification

direct	poster	publisher	asymmetric on pss	feed update content
neighbourhood	issuer	any	symmetric on envelope	feed update arrival
targeted delivery	issuer	any	none	chunk content

Table 4.1: Requests of and responses to types of feed update notifications and targeted chunk delivery.