Modern email can be built from borrowed parts

Let's design the successor to email on top of HTTP, fixing the design flaws SMTP has been dragging along for 40 years. Piece by piece. The goal is not to replace the current mail system, heaven forbid!, but to learn, have fun and discover current technologies to swap out every element: sending, receiving, gateway, keys, and so on. The system will never talk to Gmail or any classic email provider: it only talks to itself. The only thing we are going to keep is the shape of the addresses, user@domain. Everything else gets reinvented.

Since it is a mail system over HTTP, a good name for the protocol would be HMTP: Hypertext Mail Transfer Protocol (the S for Simple in SMTP gives up its seat to the H for HTTP).

Now it's time to prepare the tech stack: HTTP, TLS, WebFinger, ActivityPub, Webmention, Ed25519, HPKE, sigchains, etc.

This design doesn't invent a single technology: everything already exists.

Every piece we are going to use is standardized, deployed and battle-tested at scale by millions of people; the only new thing is the assembly.

Choosing HTTP is not just pragmatism. It solves, right out of the gate, several things any new protocol would have to build at some point:

Let's move to the next level: how do we discover a user, verify their identity, deliver the message, sign it and encrypt it, all over HTTP?

Delegation is solved with a static document:

The inbox can live on another host: that is the MX record, but without touching DNS. A static blog on GitHub Pages can delegate its mail to a provider by serving a JSON file. And it allows something MX never did: per-user delegation (each mailbox of a domain on a different provider). We wouldn't even need to invent the route: WebFinger (RFC 7033) does exactly this, and Mastodon has already proven it scales.

Identity cannot be a key (they get lost, they expire); it has to be something that keys back. A proposal with two anchors:

However, domain-anchored identity has its own Achilles heel: a domain is not owned, it is rented. If you stop paying, it expires and someone registers it, the new owner publishes their keys in your .well-known and from that moment on they receive your mail and sign as you. Nobody can tell the legitimate heir from the squatter. It's the problem ATProto tries to solve by separating identity from the domain with DIDs, at the price of another piece of infrastructure. We know it's a real problem with a known solution. Now let's move on.

Delivery is a POST to the recipient's inbox:

The key is not the request, it's who makes it. Your client doesn't deliver directly to the recipient, but to your own server (an authenticated POST to your outbox), and it is your server that queues, retries with exponential backoff and honors Retry-After. It's admitting that SMTP's MUA/MSA/MTA separation was right. One of email's quiet strokes of genius is that if the destination server is down, your server retries for days and you forget about it.

But let's add an improvement SMTP never had. Every message carries an ID that is the hash of its content, so retries are idempotent. The receiving server deduplicates by ID and the classic "duplicate email because the ACK failed" disappears by construction.

The full cycle of a delivery, with the destination node down on the first attempt:

Notice that the diagram contains the entire protocol: the two GETs to .well-known are discovery and verification, the POST is the delivery, and the queue lives where it should, on the sender's server.

The message is a signed object, not loose text:

Like in any system, this is a complex problem that needs several layers of defense. With HMTP we could start with three:

Email standardized sending (SMTP) and reading (IMAP/POP) as separate worlds. We don't need to invent anything: reading, synchronizing and searching mailboxes over JSON/HTTP is already solved and standardized by the IETF. It's called JMAP. HMTP would define delivery; reading is JMAP with one new object type. Push to the client with SSE or WebPush. The full cycle (send, deliver, read, sync) stays on HTTP.

Every problem of email has a solution deployed and working: discovery in Mastodon, delivery in ActivityPub, verification in the IndieWeb, rotatable identity in Bluesky, reading in Fastmail, consent in Signal. An upgrade to email already exists, but nobody has assembled it.

We solve many problems with elegant, modern solutions:

Talk is cheap, so I implemented a working prototype in Python: github.com/tanrax/hmtp. All in a single file. The prototype covers the full transport: signed delivery, verification at the source, end-to-end encryption, first-contact consent, deduplication, threads, a queue with exponential backoff and key rotation in one command. It deliberately leaves out the pieces at the periphery: the chain of signed rotations (receivers fetch your key live on every delivery instead of pinning it, so the chain only starts paying off once nodes cache keys), the 402 postage, attachments by reference and JMAP reading. The README has the quickstart (your first message in two minutes, mailing yourself), a demo of two nodes exchanging encrypted mail on your machine, and the full production guide, from DNS to systemd. Remember it's a design experiment with unaudited cryptography; don't use it for secrets anyone depends on. Although it could be a lightweight communication system for whatever ecosystem comes to your mind.

I hope you enjoyed the ride. And if you ever send your first HMTP message, I'd love to hear about it.

This work is under a Attribution-NonCommercial-NoDerivatives 4.0 International license.

This is how I keep writing without ads or paywalls.

Source: Hacker News — This article was automatically imported from the source. Read full article at original source →