Scope of Work: Neume as a Network

[neatonk]

IMPORTANT:

This scope of work has been delayed in favor of: #58 "Scope of Work: Neume Publisher, SDK, and UI Kit backed by Arweave".

Scope of Work: Neume as a Network

The overall objective of this work is to begin to initiate the progression of Neume into fully decentralized network of nodes. This incorporates aspects of neuIP-5 as well as the proposed neume-pub bounty project. Instead of implementing set reconciliation immediately, the current scope opts to synchronize data using arweave for storage and retrieval. This approach is has the advantage of being significantly easier to implement and ensuring data availability regardless of node availability.

---

1. Implementation of Schema Identification for Monolithic JSON Schema

Objective:
Introduce a unique deterministic identifier for the current monolithic JSON schema in the Neume Network, in line with the neuIP-5 standard.

Approach:

• Get acquainted with the current codebase and it operation, with access to suitable RPC nodes.
• Design and implement the algorithm to derive a unique identifier from the monolithic JSON schema.
• Incorporate the identifier generation mechanism into the neume crawler's startup routines.
• Test schema identifier algo. to be sure it behaves as expected in the face of schema changes.

Duration: 1.5 week

---

2. Specification and Implementation of Data Distribution and Synchronization via Arweave

Objective:
Devise and set in place a system for nodes to distribute and synchronize data via arweave, using the schema identifier and last synchronization timestamp.

Approach:

• Establish an abstraction layer for data storage and retrieval, ensuring that the main application remains agnostic to the data source or destination specifics. This will ensure that Arweave is optional.
• Design a protocol for nodes to exchange data based on the schema identifier and their last sync timestamp, via Arweave.
• Implement strategies to handle scenarios where data might be published out of order.
• Test synchronization between two nodes under various circumstances to confirm repeatable outcomes.

Duration: 1.5 weeks

---

3. Decomposition of Monolithic Schema and Linking Sub-Schemas (Optional)

Objective:
Segment the monolithic schema into distinct, atomic subsets in line with neuIP-5 and introduce a method of linking related objects, such as associating a platform with a track using suitable identifiers.

Approach:

• Analyze the monolithic schema to discern logical segments that can be transformed into atomic subsets.
• Design and implement a method for linking related objects, catering to the current gaps in neuIP-5.
• Decompose the monolithic schema into meaningful atomic subsets.
• Configure a node to consume the monolithic schema and produce the new atomic schema.
• Conduct tests to confirm that the results are functionally equivalent.
• Configure a node to consume and/or produce a subset of the new atomic schema.

Duration: 1 week

---

Total Duration: 3-4 weeks (up to 20 hours per week)

[djfnd]

This looks great, very thorough, and well put together. Thanks @neatonk .

Questions, which came to me as reading:

• Can you foresee any potential roadblocks that may impact the development?
• What would be the final outcome of the work, i.e. what would we have at the end of this tranche of work?
• Can you add any detail at this stage as to any follow-up workstreams that could build on this work?

[neatonk]

Good questions.

Can you foresee any potential roadblocks that may impact the development?

I've limited the scope to keep it simple and minimize unknowns. This will help to avoid potential roadblocks. That said, to test out the changes I'll want to setup a local node and sync the state from the current node. It's possible this may not work as expected, in which case I'll need to start from scratch, which would take a while. I would also need access to rpc endpoints for full/archive nodes.

What would be the final outcome of the work, i.e. what would we have at the end of this tranche of work?

When part 1 and 2 are complete, it will be possible to:

• Access historical data via Arweave with or without running a Neume node.
• Perform "offline" processing of the historical data on Arweave, e.g. to migrate between schema versions.
• Bootstrap a new node from the historical data on Arweave.
• Perform basic filtering of the historical data e.g. by timestamp or block.

When part 3 is complete, it will be possible to:

• Run a node which consumes and produces a small subset of data relevant to the operator.
• Create a node with custom strategies and schemas to produce novel data for different applications.

Can you add any detail at this stage as to any follow-up workstreams that could build on this work?

Future workstreams could include:

• Implementing set reconciliation for efficient coordination between nodes without Arweave.
• A standardized a way of announcing/discovering schema identifiers without adding them to the code and updating each node.
• Auto-assemble ad-hoc strategy graphs to produce a desired output schema.
• Compile a strategy graph e.g. to run in strategies in parallel when possible.
• Run a node that constant publishes new data to Arweave so that it is always up-to-date.

[neatonk]

I recognize that these are incremental steps towards the larger goal of growing neume into a network, but I do not expect the number of projects using neume will increase as a result of these changes.

It may be more compelling to instead focus on distribution via Arweave with a dev friendly SDK, React Hooks, and UI component libraries built on top of that data. This would be much more likely to increase the number of people interested in building on Neume in the short term and justify decentralization efforts in the long term.

@djfnd if this sounds like a more appealing outcome for tranche 3 I can prepare an alternate scope of work that covers it.

[neatonk]

In our discussion today we've agreed that this alternative scope is the way to go. Here's why:

• By making Neume easier to use we hope to bring in more builders.
• By distributing data to Arweave, we hope to gain the benefits of Arweave as a decentralized network and decouple data consumers from the nodes that produce the data.
• If successful, the increased number of users and apps relying on Neume, will justify the larger project of evolving Neume into a proper network of its own.

[djfnd]

100% aligned. And will help with any future hack that we might want to support. What would be the level of work to turn this then into an SDK?

[neatonk]

FYI I've linked to the updated scope of work above.