Video Streaming Interoperability Between Clients

[ericxtang]

Overview

Video streaming is an important functionality for social networking. The ability to upload video & live stream to an audience has become a table-stakes feature for most social applications today. As an example, Twitter shared that Tweets with video attract 10x more engagements than tweets without video. Farcaster needs a great video experience, which requires media metadata formats, and a credibly neutral media processing and serving layer. Here’s a proposal for how we can add this to the ecosystem.

Opportunity: simple schema for multimedia assets

Example metadata for video assets

embed {
  string url = "{ipfs://QmT9CAYUunH2ooZqe1LCmfE4QwF9Q663m879uvyWQAZSQ4 | ar://xiITmZjnMS_vH_NuTKeMwaPNMYhyJWo6KPh903rWynA}";
  CastId cast_id = "{cast_id}";
}

Retrieve metadata

ipfs://QmUnMkaEB5FBMDhjPsEtLyHr4ShcAoAUrwqVryCeuMosNr
ar://xiITmZjnMS_vH_NuTKeMwaPNMYhyJWo6KPh903rWynA
{
	assetCidUri: "https://nftstorage.link/ipfs/QmXkZ9vKwZ6J3sKWynbvdvUMLtVZPmUsQzqoz6Qocnx1Kb",
	type: "video/{mp4|mov|avi|...}",
	coverUri: "https://nftstorage.link/ipfs/QmdVXgXh1eiLtFa5nHtHx7ckUujoqW2ueuGQcs4phqPJUD"
}

To retrieve the video metadata

https://nftstorage.link/ipfs/QmT9CAYUunH2ooZqe1LCmfE4QwF9Q663m879uvyWQAZSQ4

To retrieve & play the video file (subject to streaming quality issues due to network condition variations, especially on mobile)

https://nftstorage.link/ipfs/QmXkZ9vKwZ6J3sKWynbvdvUMLtVZPmUsQzqoz6Qocnx1Kb

To retrieve & play the optimized (transcoded and cached) video

• Option1: Use any cloud streaming API (AWS, Mux, etc), keep track of the map of (cid->HLS)
• Option2: Use a service to upload video to IPFS and transcode/cache in one API call. Livepeer has a service like this:.
  • Use this demo app to see how it works in the background. Note you can sign up for a Livepeer account to create set up your own.

Discussion points

Having a simple schema for multimedia assets can help set better expectations with client developers. Multimedia assets can be stored in decentralized storage networks like Filecoin or Arweave, and the content hash can be included in the cast that references the media content. In the case of video, these assets can be the originally uploaded video files. The client developers can decide to either host the files (incurring cost for good performance), or use the public gateway services from the respective decentralized storage networks (low cost, but no performance guarantee). This preserves the content integrity of the video casts through content hash, and unlocks a few possibilities:

1. Improved interoperability

Decentralized content ID represents the data itself, and sets clear expectations on how to retrieve the data. This simple schema requires the minimum information required to render the video. This makes it easy for any client developer to implement video experiences, even allows common SDKs / UI components to emerge. As a side note, it’s debatable what is the “minimum viable” metadata for a video asset.

2. Client specific optimization

Each client can decide how to optimize user experience based on the video assets. For example, a client targeting short form video can do client-side caching, while a client targeting long form video can transcode the video and attach subtitles in different languages.

3. A Farcaster media pinning service

Using decentralized storage networks doesn’t answer the question of “who pays for the service”, but it makes it is quite possible to create a cost effective data pinning service for the Farcaster ecosystem. This service can limit access to registered developers and doesn’t need to guarantee scalable delivery, thus can be build with low cost (10x-50x cheaper than using S3), low enough to be funded as a public goods service by the community, or be operated as a paid service for clients who want access to archived media data. It can even have a retention policy, similar to how Farcaster Hubs will have to periodically purge data.

4. A Farcaster video content tagging service

Automated video tagging can be a valuable tool for social apps. It can help create more context for content categorization / recommendation, and it can help flag NSFW / Illegal content before it becomes harmful to users. In a decentralized ecosystem like Farcaster, this can be done by cryptographic attestations, so the trust-worthiness of the service can be determined by the ecosystem. Having a common schema makes it easier to create an automated video tagging service, and client developers can decide on what to do given the tags. The tags can be stored in a database, or in a decentralized network like IPFS / Arweave (Arweave’s Momoka seems like an interesting option)

Why not OpenGraph?

OpenGraph is great for linking data for webpages / embedding external videos like Youtube links, but not great for dealing with media uploaded directly into Farcaster.

Relevant Discussions

• @betashop brought up this interoperability issue in the Canonical URL discussion, but no clear solution (supporting video uploads) was proposed in both the discussion thread and the dev call.
• Making Casts More Flexible post by Merkle team.

[danromero]

Thanks for writing this up.

Could this be accomplished with existing URL embeds and the standard set of video filetypes?

example.com/media.mp4
example.com/media.mov

Or do IPFS and Arweave hashes not include filetypes in the URL?

[ericxtang]

Existing URL embeds introduce the challenge where the writing clients has to decide whether to make the file openly accessible. If it is, then the writing clients has to bare the cost of streaming for all other rendering clients. If not, the video is only viewable in the writing client application. In either case, the interoperability story breaks a little.

By using a content addressed ID, the video file is "compressed" into the ID and recorded in the cast by the writing client, allowing the writing client to "be a good citizen" without incurring unbounded cost on video streaming.

Re: do IPFS and Arweave hashes not include filetypes in the URL?

IPFS / Arweave hashes don't include filetypes in URL by default. For example, https://nftstorage.link/ipfs/QmXkZ9vKwZ6J3sKWynbvdvUMLtVZPmUsQzqoz6Qocnx1Kb is an mp4 file, but it's not included in the URL.

[michaelhly]

Curious here. What are the bandwidth comparisons of streaming multimedia from Livepeer/IPFS/Arweave compared to streaming platforms such as YouTube, Cloudflare, etc.?

[ericxtang]

Hi @michaelhly - since Livepeer uses the same video streaming protocols (HLS, webrtc, etc) used by Youtube and Cloudflare, bandwidth usage is very similar. However, because Livepeer is a decentralized marketplace that aggregates compute resources, the COST for the overall video streaming workflow is 50% - 80% less than cloud platforms like Cloudflare. Youtube is technical free, but you pay in giving up data and monetization opportunities.

[varunsrin]

I'm generally in favor of making it easier to embed videos into casts.

Is this a Standards FIP, meant to introduce a new convention for how to embed URI's to decentralized resources like IPFS? Or is this an implementation FIP that requires code changes at the protocol level? I think it's the former, but some of the discussion items imply the latter.

[ericxtang]

I think it's more of a Standards FIP, but also curious to hear from @davidfurlong and @betashop who are building / planning to build video features.

[davidfurlong]

Seems to be a case for canonical urls via #72 for IPFS/arweave assets and clients handling these assets in native ways. I'm not sure whether that FIP includes ipfs or arweave protocol support as standards, but it probably makes sense to support these kinds of URIs

[ericxtang]

Yeah I agree if #72 supports IPFS/arweave links, it'll get pretty close to this proposal.

In addition, this proposal also contains a simple schema for describing media assets, and recommends writing clients to follow such schema when creating new casts. A part of it requires media to be written to IPFS/Arweave in order to get the file CID. From Discove's perspective, do you think it's a reasonable ask?

[varunsrin]

Closing due to inactivity.