Rust IPFS Integration

[aphelionz]

1. Need to create a package called rust-ipfs-dep so it can be npm install-ed and tested via this repo
2. Need to add a IPFS_RUST_EXEC for ad-hoc testing of unreleased versions

[koivunej]

Looking at the README of ipfs/interface-js-ipfs-core the use of js-ipfsd-ctl might not be needed at all but a function could be used to invoke cargo run --bin testing_httpd and then kill it during teardown?

For interop there does not seem to be similar "escape hatch" in https://github.com/ipfs/interop/blob/master/test/utils/daemon-factory.js which will build upon createFactory from js-ipfsd-ctl.

[koivunej]

Should also be noted that while some tests have been written easily extendable (e.g. exchange-files.js) to more node pairs or sequences, others are not (e.g. cid-version-agnostic.js). I wonder what is the general wanted direction here, perhaps I should file an issue.

[koivunej]

Created ipfs#106

[koivunej]

Going over my run of the interop tests, at least the before linked could be the very first test to enable (and refactor).

Phase 1.0: none
Phase 1.1: cid-version-agnostic.js, pubsub.js
Phase 1.2: repo.js

The exchange-files.js looks good but requires unixfsv1 support for adding files and reading those.

[aphelionz]

Phase 1.0: none

I think I agree with this and everything else, so long as "none" includes scaffolding everything so that all that would remain is to enable the cid-version-agnostic.js and pubsub.js in Phase 1.1 once we've enabled that functionality.

[aphelionz]

@vmx ipfs/interop uses js-ipfsd-ctl 0.74.4 but the latest version is 3.0.0. How do you suggest we approach this?

[vmx]

@hugomrdias can you help please?

[aphelionz]

Thanks @vmx and @hugomrdias.

Actually, to further specify it looks like:

• ipfs/interop uses 0.74.4
• ipfs/ipfs-js-interface-core uses 1.0.0
• The latest js-ipfsd-ctl is 3.0.0

[hugomrdias]

ok first we should aim to use js-ipfs-ctl latest version!
Steps

• support ipfs-ctl
  • look at https://github.com/ipfs/js-ipfsd-ctl/blob/master/src/ipfsd-daemon.js and see if rust-ipfs can support those api calls (or ignore without failing)
  • if yes we can just use the go Controller and pass ipfsBin option with the rust-ipfs bin path
  • if not we should make ctl support rust-ipfs and create a new Controller specific to it
• setup an env just like https://github.com/ipfs/js-ipfs/blob/master/packages/ipfs-http-client/test/interface.spec.js where you can enable/disable tests as you need.
• think about interop repo after the previous steps are running properly

Makes sense ? I can help with both steps just do a first pass and create issues and we will go from there.

@vmx did you setup a realtime communication channel ?

[aphelionz]

@hugomrdias Thanks for the reply. The good news is that the code currently in https://github.com/ipfs-rust/ipfs-rust-conformance works, both against our new HTTP endpoints and (I think) with the latest version of js-ipfsd-ctl. Have a look in the /test folder there

[hugomrdias]

that's awesome 👏, did you need to make changes to ctl you are using a local version there?

[vmx]

@vmx did you setup a realtime communication channel?

There's a Matrix channel AFIAK. @aphelionz it would be good to have a link to that in the rust-ipfs README (or at welcome, or wherever you see fit).

[aphelionz]

It should be there now! Let me know if the links don't work

Edit: looks like maybe shields.io is having issues

[hugomrdias]

lol slack, irc, matrix, discord can't we all just use telegram ?

@aphelionz so you'r good with interface-core ? just need help with interop ?

[aphelionz]

@swedneck did us the favor of bridging the chats between Matrix and Discord. Can we add Telegram? 🤣

Yes, thank you @hugomrdias, I think your guidance above is sufficient for now. I think the first priority is getting the interface tests running as interop won't really come into play for another week or two.

[hugomrdias]

Perfect feel free to ping me when you need me and ill join in whatever medium the conversation is happening :)

[swedneck]

We could bridge to telegram using https://talk.feneas.org/t/matrix-telegram-bridge/120

[aphelionz]

Coming back to this conversation, I have some concerns about the proposed tests:

Phase 1.0: none
Phase 1.1: cid-version-agnostic.js, pubsub.js
Phase 1.2: repo.js

Here's what I'm seeing, correct me if I'm wrong:

• cid-version-agnostic.js tests use the add and cat APIs, which are out of devgrant scope due to the usage of unixfs in those endpoints
• pubsub.js looks like we can make this work after Phase 1.1 (/swarm and /pubsub endpoints complete)
• repo.js tests will not pass given the discussion here: IPFS_PATH Folder Compatibility rust-ipfs#88

Either way, I'm comfortable for the time being introducing a very very minor PR here to introduce the rust-ipfs-dep library and adding a rust type to utils/daemon-factory.js, and then moving this issue to Phase 1.1. This issue will likely then again move to Phase 1.2 once decisions are more finalized and more endpoints are supported.

[aphelionz]

@vmx and @hugomrdias please weigh in on the above

[koivunej]

Good that you remembered to comment here.

• cid-version-agnostic.js tests use the add and cat APIs, which are out of devgrant scope due to the usage of unixfs in those endpoints

My initial thought for this was, could we just use the block api here. Then it was pointed out to me that there is no such as raw typed cidv0. I still haven't made sure that add operation yields a unixfs File but a single block files without any bells or whistles might already be doable.

• repo.js tests will not pass

Agreed.

Even getting the two interop tests mentioned going, refactored and somewhat passing will be great.

Earlier I mentioned:

The exchange-files.js looks good but requires unixfsv1 support for adding files and reading those.

This was also a really slow and possibly failing test when I ran the interop's last time. It'd be great to get the cid-version-agnostic.js test refactored and passing (I understand it'll show single block transfers between nodes as is) and then set sights on exchange-files.js for any upcoming milestones.