Why filesystems have loose-coupling and your protocol doesn’t
Interfaces are a powerful concept in programming: the idea that you can separate the abstract behaviour from the concrete implementation with a protocol: an agreement on behaviour between components.
Good interfaces can lower cognitive overhead, and encourage code-reuse, but they don’t guarantee it. Merely hiding implementation detail in itself does not make libraries easy to compose, or replace. These properties are commonly known as loose coupling, which comes from a larger focus on decomposition, clear management of dependencies, and separation of concerns.
Unfortunately, loose coupling is one of those things where “I know it when I see it”, and it’s hard to break down the constraints and design choices except through hindsight. Thankfully we have plenty of examples, so let’s look at two interface styles, which I will term “RPC style” and “Filesystem style”.
An RPC style API is one with a focus on verbs, IMAP, POP3 and SMTP all fall into this pattern. Like many application protocols, the client issues textual commands and the server responds with numerical codes, but each of these protocols has a different command set and response set. Although you can swap out implementations, they are hard to extend: each new feature requires a new command to be added, subtly changing the base protocol.
Meanwhile, a filesystem style has a focus on nouns, or filenames, with a comparatively small set of operations (open, close, append). This commonality of interface means it’s quite easy to swap out one filesystem with another, or even use network filesystems without substantial changes to interface or code, but you might be lured into thinking they are less extensible because of the fixed command set. If you fix the verbs, how do you extend the system? Through different types of files, and filesystems.
Plan 9 pioneered this approach, making unix’s ‘most things are files’ into 'everything is a file’, or more precisely, “all services are filesystems”. Instead of adding new commands (or in unix terms, ioctls), operating system services were exposed as regular files.
For example, network protocols became a directory of connections /net/tcp/1, /net/tcp/2. To create a new connection, /net/clone was read, which returned the connection number. Inside the connection directory, there were a number of files, like “ctl” and “data”. To send data you wrote to the data file, and to configure the connection you wrote to the ctl file. This turned out to be far more extensible, without having to write all new commands for each service.
New features don’t change the protocol. New services speak the same basic interfaces. Instead of encoding the behaviour in the protocol, the behaviour is determined by the underlying resources or files and how the client interprets them. You don’t need a separate filesystem API for image editors, text editors, or IDEs.
This design is often called the “Uniform Interface Constraint”, and is present in one of the more popular protocols, HTTP. Although it shares text commands and numeric responses with earlier internet protocols, it is far more like a filesystem than an RPC system—a small set of common verbs. HTTP when used well, can result in amazing amounts of loose coupling: you can use the same caches, proxies and load balances, to speak to a variety of services. Similarly to Plan9, instead of hard coding absolute paths to files, files can link to each other.
By comparison, BEEP (A HTTP competitor), set out to provide a generic and extensible protocol for services. Instead of focusing on a common interface, it focused on a common transport layer. It aimed for extensibility, with workarounds for problems with using TCP (Pipelining, Head of Line blocking, and Multiplexing), unfortunately with a considerable implementation cost.
Although designed as a transport for IMAP, POP3 and the like, most new protocols copy them, or tunnel everything over HTTP. BEEP hasn’t seen the adoption the designers hoped for, possibly because they felt using XML for connection negotiation was reasonable, but URLs were too complex.
Ultimately, extensibility in a protocol doesn’t come from adding new commands and responses, it comes from being able to use existing commands on different resources. Uniform interfaces are at the heart of loose coupling in systems.
Despite this, many people who use HTTP today ram everything through one resource via POST requests. Although Plan 9 gave us UTF-8 and /proc, the idea of exposing services as filesystems has yet to go mainstream. Building in loose-coupling doesn’t mean that developers will take advantage of it.