python-libp2p dev meeting

[amstocker]

cc: @BrendanBenshoof @JulienPalard @bmcorser @SilverWingedSeraph @mvanveen

Our kick-off meeting went pretty well (#20) and I think we now have a pretty good idea about how to start this project. I think the plan is to build a python port of libp2p first and then build the IPFS protocol on top of that. I was thinking we should probably schedule another meeting to discuss libp2p in more detail and perhaps delegate some tasks. Thoughts?

[JulienPalard]

I tried to start loading the big picture of libp2p in my head, but current spec, typically
abstract-transport opens so many questions for me.

Typically, a few one I still have in mind:

• transport.createListener(options, function (stream) {})
  • what is "options" ?
  • what is this callback for ?
• "This method waits and listens for incoming transports by other peers." OK but
  • listener.listen(options, [callback]) "This method opens the listener to start listening for incoming transports" Wasn't this already done by the createListener ?

Does someone here have answers to those kind of questions or should we open tickets for each of them ?

It also look like it stick a lot to the underlying nodejs network paradign, and if we're using coroutines, I don't think that the "callback everywhere" design pattern is the pythonic way to do it. Yet there's probably almost no difference. But this opens the (already opened ?) subject await vs yield from.

[jbenet]

@JulienPalard the spec was most recently re-written by @diasdavid after making it in node, so yeah it will have a node tint to it. But just check out the go implementation for an example with coroutines.

[jbenet]

• transport.createListener(options, function (stream) {})
  • what is "options" ?
  • what is this callback for ?

This doesnt have to be this way at all, that's just the idiomatic node version. here it is in go:

l := utp.Transport.Listen("/ip4/1.2.3.4/udp/1234/utp") 
// though maybe more options, dont recall. but def no callback.

• "This method waits and listens for incoming transports by other peers." OK but
  • listener.listen(options, [callback]) "This method opens the listener to start listening for incoming transports" Wasn't this already done by the createListener ?

yeah. another idiomatic choice.

@diasdavid, in Go it's not idiomatic to do that btw, the above call (transport.Listen(...)) should leave the listener in a state of already listening. from there,

conn := l.Accept()

is this ok with you @diasdavid? or there a critical reason why one MUST do:

// your way in go
l, err := transport.NewListener()
err := l.Listen(addr)

// the go way
l, err := transport.Listen(addr) // implicitly creates listener

cc @whyrusleeping

[daviddias]

To clarify, an abstract-<something> module is a way to create a spec for a component that has the goal to be swappable, it is not, however, to force any kind of interface bias on other language ecosystems.

One example, which has two different interfaces already, is https://github.com/diasdavid/abstract-stream-muxer . Don't feel that you have to use the Node.js way of doing Asynchronous I/O in Python, just make sure you can create an interface in Python that follows the same expectations.

Also, one goal is to have the battery of tests all in one place, currently only the Node.js ones live in that repo, the Go ones live inside https://github.com/jbenet/go-peerstream, until we have a better way to manage packages in Go ready.

@amstocker I would love to be part of the Python libp2p meetings :D

[jbenet]

@diasdavid btw, the "abstract-" thing is common in node, but most other languages use "interface" to mean this instead. may be worth using that for cross-lang friendliness.

[daviddias]

@jbenet good point, should we get a better name that is abstract/interface agnostic? What about making available on that repo, the file which contains the interface/abstract for the language, such thing would not exist for JS, but for go, we can have the interface.go, and go would import it?

[jbenet]

we could-- but might make it more annoying to work with. i kinda like how multiaddr and multihash work in go/js.

[amstocker]

@JulienPalard I think we should interpret the libp2p specs in terms of data flow between subsystems, and as @diasdavid said we should definitely do whats most fitting for python async i/o (e.g. asyncio coroutines -- for which we can reference the go implementations like @jbenet suggested).

@diasdavid It would be awesome to have you in python libp2p meetings! I'm very excited to get this up and running.

Also @JulienPalard I think if we are going to support python 3 at first we should do 3.3+ so that means no await (which I thought was just the new syntax for yield from?). I think it would be too much to restrict to 3.5. Also, eventually I do want to have some kind of python 2 compatibility, so it would be nice if we had an interface to plug in different async i/o reactors like gevent, twisted, etc., but all this can come later.

[JulienPalard]

@amstocker Yep async/await are almost only syntactic sugar over @asyncio.coroutine and yield from.

They're nicer on some points (more strict, more readable, distinguisable from the real yield from usage, etc...).

But I'm OK to not stick to 3.5 only for the moment, not sure how much person have access to a 3.5 interpreter for the moment.

@jbenet @diasdavid In one hand it's nice to hear we won't have to stick to the specs about interfaces, (mostly for the non-python way to present things). In the other hand, it's probably harder to read a spec when it mixes between itself and current implementation in another language.

For the moment I still have hard times to load the spec in my brain, find an independent, testable module I can write, and write it. (spent a lot of time last time to read the abstract-* wondering what to do in python with it, and what the undocumented parameter meant :-p).

[jbenet]

@JulienPalard take a look at go interfaces too. and yes the specs need cleaning, but it's hard work. help would be appreciated, but note it's bikeshedding central.

[amstocker]

@JulienPalard @jbenet I think I get the gist of the libp2p ideas/architecture, and am planning to do some kind of a roadmap for python-libp2p by next week. As the libp2p spec is refined we will of course have to change things, but as it stands we definitely have a lot of work to do in the first place. @JulienPalard Here are some of the references I've found that you can hopefully use to piece together the spec:

https://github.com/ipfs/js-ipfs#roadmap
https://github.com/diasdavid/js-libp2p
https://github.com/diasdavid/js-libp2p-swarm
https://github.com/ipfs/specs/blob/master/protocol/network/4-architecture.md#41-peer-routing
https://github.com/ipfs/specs/blob/master/protocol/network/6-interfaces.md#62-peer-routing
https://github.com/ipfs/go-ipfs/tree/057f0e281f9c516e8157c28e7542bae9fd4b0821/p2p

[ricmoo]

I was just chatting on the IRC and was wondering if the meeting is still open to join? I've been tinkering with IPFS in Python (mostly as a proof of concept before implementing it in Objective-C), and I'd love to be involved.

[jbenet]

May be useful if some Python folks join the general libp2p hangouts too.
Those are high throughput so prob worth keeping both for now

Btw-- are you keeping minutes like the others? It's useful to be able to
read back on discussion. Feel free to PR any coordination stuff you need in
github.com/ipfs/pm and join the general irc syncs as you please
On Wed, Nov 18, 2015 at 00:00 Richard Moore [email protected]
wrote:

I was just chatting on the IRC and was wondering if the meeting is still
open to join? I've been tinkering with IPFS in Python (mostly as a proof of
concept before implementing it in Objective-C), and I'd love to be involved.

—
Reply to this email directly or view it on GitHub
#21 (comment).

[amstocker]

@jbenet I'm planning on eavesdropping a few of the core dev hangouts as soon as I have time, and I'm sure others would be willing to as well. In regards to minutes, we've only had one meeting so far (#20), this python-libp2p one hasn't happened yet.

[daviddias]

I've been cleaning up, organizing things and going through the process of what I would have to do if I wanted to start building a new libp2p implementation in another language. The end result was a cleaner description and a uniform pattern across available implementations, check:

• https://github.com/diasdavid/go-libp2p/blob/docs/roadmap/README.md
• https://github.com/diasdavid/js-libp2p
• https://github.com/diasdavid/rust-libp2p (pretty much just the landing repo, for now)

Some of the first things that need to be built are the Generic modules, used by every other part:

• PeerInfo
• PeerId
• multihash
• multihashing
• multiaddr
• multistream

Hope this helps :)

[amstocker]

@diasdavid this is awesome! Should give us a lot to work on for the time being.

[daviddias]

glad you liked it :) also, to test multistream against go and node, one way is to follow PDD, for more info and guide check - https://github.com/ipfs/specs/tree/master/pdd

[candeira]

@diasdavid, really nice, thanks. I have a PR with a copy of your roadmap that I intended for the py-ipfs implementation to use as documentation when implementing. I've added your links and @amstocker's from another comment.

[daviddias]

Any update on progress? Let me know if I can provide any help! :)

[amstocker]

@diasdavid I've been busy finishing school so I haven't been around much, but in the next few weeks I'm definitely interested in getting back up to speed and start work on python-libp2p.

[daviddias]

@amstocker sounds great, good luck with school!

[Mec-iS]

@amstocker would be great to help for this implementation. Let me know what I have to read to catch up.

[amstocker]

@Mec-iS I think right now we still need to get the generic modules done (#23), then we can start to focus on the core implementation for py-libp2p (see the js repo here: https://github.com/diasdavid/js-libp2p). Also it would be good to get as familiar as possible with the libp2p spec (see: https://github.com/ipfs/specs/tree/master/libp2p). The specs for libp2p are a little scattered but you should be able to get the gist from the js-libp2p repo and the spec repo, also @candeira has done a good job with the py-ipfs roadmap: https://github.com/candeira/py-ipfs/blob/readme-roadmap/README.md .