Part II Design and architecture

The Swarm project is set out to build a permissionless storage and communication infrastructure for the self-sovereign digital society of tomorrow. From a developer’s perspective, Swarm is best seen as public infrastructure that powers real-time interactive web applications familiar from the Web 2.0 era. It provides a low-level API to primitives that serve as building blocks of complex applications, as well as the basis for the tools and libraries for a Swarm-based Web 3.0 development stack. Designed to allow access from any traditional web browser, the API and tools ensure that Swarm can swiftly offer a private and decentralised alternative to today’s World Wide Web (WWW).

This part details the design and architecture of the system. In accordance with the principles laid out in 1.3.2, we prioritise a modular design approach for Swarm, conceiving it with clearly separable layers, each dependent on the previous one (see Figure 1.1):

• A peer-to-peer network protocol to serve as underlay transport.
• An overlay network with protocols powering a distributed immutable storage of chunks (fixed size data blocks).
• A component providing high-level data access and defining APIs for base-layer features.
• An application layer defining standards and outlining best practices for more elaborate use-cases.

We regard (2) and (3) as the core of Swarm. Since the network layer relies on it, we will also formulate the requirements for (1), but consider the detailed treatment of both (1) and (4) outside the scope of this book.

Central to the design is the architecture of Swarm’s overlay network (layer 2 in Figure 1.1), which is discussed in Chapter 2. Chapter 3 complements this by describing the system of economic incentives which makes Swarm self-sustaining. In Chapter 4, we introduce the algorithms and conventions that allow Swarm to map data concepts onto the chunk layer to enable the high-level functionalities for storage and communication, notably data structures such as filesystems and databases, access control, indexed feeds, and direct messaging which comprise layer 3 of Swarm. In Chapter 5, we present ways to prevent garbage-collected chunks from disappearing from the network, including: erasure codes, pinning and insurance, and also provide ways to monitor, recover and re-upload them using missing chunk notifications and insurance challenges. Finally, in Chapter 6, we will look at functionality from the perspective of the developer who builds on Swarm.