Building a Go Implementation of the Farcaster Hub

[Ankarrr]

I am excited about building a Go implementation of the Farcaster Hub! While we already have Hubble, which is well-written in TypeScript and Rust, having an additional Hub implementation in Go could increase the hub diversity and enhance the network's reliability. Go is a high-performance programming language with plenty of libraries, including libp2p and CRDT, which are core parts of the Hub. Additionally, the large Go developer community could be invited to contribute to Farcaster through this new implementation.

I've read the protocol specification, and my end goal is to create a fully functional Hub that adheres to the specification and can interoperate with Hubble. My initial plan is to follow the same architecture as Hubble, which consists of three main components: the p2p engine, storage engine, and sync engine. I aim to implement each of these step by step.

1. I think we should start with the p2p engine to establish connections with the peers, receive messages, and store them in a database. We can use libp2p to start a gossip node.
2. Next, we need to build the storage engine to process received messages and implement all necessary CRDTs with general rules, specific validation rules, and conflict resolution rules. We could probably use IPFS CRDT library.
3. Following that, we need to build the sync engine with RPC for performing Diff Sync.
4. Lastly, we should build the RPC server to allow clients to interact with the Hub.

One thing I hope to figure out is what is a minimum viable hub looks like? In other words, what are the basic requirements a Hub needs to meet to connect to the existing Farcaster network and be a valid peer? Understanding these minimum requirements will help me prioritize the development efforts.

This is my rough idea, and I am open to any feedback or ideas. If you are a developer or contributor to Farcaster, I would really appreciate it if you could share your thoughts. Also if you are interested in collaborating on this project, please let me know!

[varunsrin]

Supportive of this!

I'm not sure if starting with p2p is going to be feasible because 1) libp2p has some subtle diffs across implementations that might lead to hard to track bugs and 2) even if it works you can't do anything with the messages because there is no storage engine to process it.

I also don't think you should use a CRDT library.

If I was building it, i'd probably write a simple in memory implementation of the crdts, then write the storage layer and sync layer and finally tackle gossip.