Webtorrent support

[faddat]

Does this client by any chance support webtorrent?

If no, anyone know of a go torrent solution that does?

thanks!

---

IssueHunt Summary

anacrolix has been rewarded.

Backers (Total: $250.00)

• steelbrain ($50.00)
• $200.00 have been anonymously funded.

Submitted pull Requests

• #393 Webtorrent support

---

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

---

IssueHunt has been backed by the following sponsors. Become a sponsor

[anacrolix]

Do you refer to https://github.com/feross/webtorrent ? Can you link to a spec on bittorrent clients interacting with webrtc clients?

[faddat]

Not offhand, but I'm happy to google for it....

To leverage existing WebRTC implementations a dedicated instance of a Chromium browser is used. This 'bridge' mediates communication between Vuze and WebTorrent peers.
For the original implementation see https://webtorrent.io/

....ouch.

Maybe I can find better in the node.js?

[anacrolix]

That's as much as I found. If you found specifics in the node.js code, that may expedite adding support for this.

[faddat]

@feross

[feross]

Hey! We'd love to help make WebTorrent support happen here. We're working on a spec. I'll share here when it's complete. At the moment you can check out the following files, to see what's going on:

• client: https://github.com/feross/bittorrent-tracker/blob/e5812fdd29eef06e240e2522464a25098e394b52/lib/client/websocket-tracker.js
• server: https://github.com/feross/bittorrent-tracker/blob/e5812fdd29eef06e240e2522464a25098e394b52/server.js#L444-L573

To sum up what it would take to support this:

• webrtc library or bindings in Go
• websocket library

When a ws:// or wss:// tracker is detected, create a websocket connection to that server. There are a few extra announce message types to facilitate establishing webrtc connections "offer" and "answer", but everything else is essentially the same.

Once connected to a peer over WebRTC, you can treat the remote peer exactly the same -- the wire protocol is exactly the same.

[faddat]

Referenced in:

https://github.com/dawn-network/glogchain/issues/#10
https://github.com/dawn-network/glogchain/issues/#11

[mattlepacek]

Webtorrent support here would be amazing. The current node.js implementations are limited by chromes implementation of webrtc causing cpu time to multiply with each torrent being seeded. I'm interested in sponsoring Webtorrent support

[anacrolix]

I'll revisit this when there's a spec.

[anacrolix]

No progress. There's no spec, and WebRTC is abysmal to work with.

[Sean-Der]

Hey @anacrolix !

I have been working on Go native WebRTC https://github.com/pions/webrtc and would love to get it to the point that you could use it!

It currently has full support for Audio/Video (send and recv) and I am maybe two weeks out from finishing DataChannels. We have ~5 developers involved, so things are moving pretty quick! If you are interested would love to chat/figure out what is most important to you. All the devs are on the Gophers slack in the #pion channel

[anacrolix]

That sounds great @Sean-Der . I'm guessing WebRTC is not so simple as a stream of bytes, when data-channels are ready we can look at integrating those. Specifically dial/accept interfaces are required, and a way to send/receive bytes. A spec on how the peer protocol operates over WebRTC is still required.

Unrelated, there was a comment earlier about websocket trackers, I've yet to see any in the wild. Some examples of those will be helpful. Implementing those should be very easy.

[backkem]

Quick update on pions/webrtc: we've finished implementing DTLS and plan to land it in master as soon as December 1st. This enables WebRTC in pure Go!

I reached out in chihaya/chihaya#388 for the tracker side of things.

Regarding the client side. It seems we need to:

1. Introduce a new Webtorrent network type and revise any logic that assumes a hard coded amount of networks.
2. Generalize the Peer struct so it can store Webtorrent peers alongside IP peers. The main difference is that they have a list of SDP offers instead of an IP+Port.
3. Allow the Client to connect using these new type of peers. We have to choose between:
   • Passing these offers to the client as a address string like the current implementation.
   • Modifying the client to accept something more complex than just an address string. The Peer object itself for example.
4. Create a WebRTC socket. The Listener is likely the most tricky part. It seems you have to open a bunch of PeerConnections and advertise their offer via the websocket tracker. TBD if we can use pions/dcnet to handle the connection itself.

After that we should be able to pipe the Bittorrent protocol over the WebRTC socket as usual.

@anacrolix what are your thoughts on this? Am I missing any big steps? How would you favor to handle the Webtorrent peers alongside regular IP-based peers?

[anacrolix]

@backkem It looks like you've covered all the important factors. Can you link to the WebRTC Dial/Addr/Listen equivalents so I can get a feel for how it will sit alongside the other socket types?

[Vladislavik]

Any news about support webrtc? It is https://github.com/gorilla/websocket
That work pretty with websockets on Go

[anacrolix]

@Vladislavik No news on WebRTC.

[jpillora]

https://github.com/pion/webrtc

A pure Go implementation of the WebRTC API

[Sean-Der]

@jpillora I would love to see that happen :)

I wish I had the bandwidth to help, but I am always behind with pion/webrtc. If you start a PR I can answer any questions/make improvements that are need to make it happen.

[anacrolix]

@vijayanandnandam I have Patreon, Liberapay, and GitHub Sponsors, you can view that stuff at the top of the project page when you click the Sponsor button.

@Sean-Der I've created https://issuehunt.io/r/anacrolix/torrent/issues/138, I think that should work? I'm open to suggestions.

Thanks to @backkem for his existing sponsorship.

Sorry everyone else for the noise.

[issuehunt-oss]

An anonymous user has funded $100.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[issuehunt-oss]

An anonymous user has funded $100.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[Sean-Der]

@anacrolix and there it is :)

If you hit any bugs/have questions we are in Slack and would love to help. I am sure you will hit some roadblocks along the way, but will do my best to unblock you!

[anacrolix]

@Sean-Der thanks.

[issuehunt-oss]

@steelbrain has funded $50.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[anacrolix]

The Dialer and Listener interfaces have been abstracted out in #380, which is the first step toward implementing this. I've created https://github.com/anacrolix/torrent/milestone/2 to track any issues and PRs that progress this issue.

[Sean-Der]

Anything I can do to help @anacrolix ? Happy to get involved with any of the WebRTC stuff.

[anacrolix]

@Sean-Der I've created issues the next 2 items we need. I think it may be possible to do them concurrently, so long as the addr string is agreed upon by the tracker and Dialer implementations. See those issues for more information. I've not had time myself to progress further yet due to work and other constraints.

[anacrolix]

I did some more investigating. This looks exceptionally difficult to integrate into the existing code. The tracker and connection management are tightly coupled. It's also probably not appropriate just to maintain state at the Torrent level for trackers, it might be necessary at the Client level in order to scale to many Torrents with the same tracker URLs specified. The potential for disconnection from the trackers is a real kicker, I think there will be need to be many events flowing around the system.

[backkem]

I think disconnecting form the tracker isn't the end of the world. Once established, the WebRTC connections don't need the tracker's Websocket connection to stay alive.
However, losing the connection before signaling is complete does mean you won't be able to complete the setup of a WebRTC connection.

Managing the tracker Websocket connections globally is probably a good idea since they will have more overhead than regular UDP ones.

[backkem]

The WebRTC connection setup is likely even more tricky. Normally WebRTC will allocate random ports for both sides of the connection. In addition, there are two steps: signaling (tracker) and the actual connections (torrent). This would require a bunch of back-and-fort between the tracker and connection logic and it's quite inefficient for this use-case.

However, unlike in the browser we have more control over the network stack. You could potentially address these downsides as follows:

• Create an offer with one IP and a random port (Potentially without actually allocating it but that creates a risk that it will be taken later..) and a random certificate.
• Announce it to the tracker.
• Create answers for incoming offers by always using one IP and the same port but different certificates.
• Send it to the tracker.
• Listen on that one port for incoming connections.
• When a connection comes in, identify it by looking up the IP & certificate in the offers you received. (This may require some DTLS trickery..)
• Establish the connection.

This would mean:

• Decent separation between tracker & torrent connection logic.
• Reduced need to pro-actively open a bunch of listening ports.

The pion/webrtc library should be flexible enough to make this happen.

[anacrolix]

Thanks for the assistance again @backkem. I have a in-progress branch at https://github.com/anacrolix/torrent/tree/webtorrent that's able to download. Within the torrent repo, you can test it with GOPPROF=http go run -race -v ./cmd/torrent -dht=false 'magnet:?xt=urn:btih:08ada5a7a6183aae1e09d831df6748d566095a10&dn=Sintel&tr=wss://tracker.btorrent.xyz&tr=wss://tracker.fastcast.nz&tr=wss://tracker.openwebtorrent.com'. It's very rough for now, but I'll continue to mold it to the torrent architecture, and take suggestions and improvements from people more familiar with WebRTC and webtorrent.

[anacrolix]

That command is now $ GOPPROF=http go run -v ./cmd/torrent download -dht=false -tcpPeers=false -utpPeers=false 'magnet:?xt=urn:btih:08ada5a7a6183aae1e09d831df6748d566095a10&dn=Sintel&tr=wss://tracker.btorrent.xyz&tr=wss://tracker.fastcast.nz&tr=wss://tracker.openwebtorrent.com'.

[issuehunt-oss]

@anacrolix has rewarded $225.00 to @anacrolix. See it on IssueHunt

• 💰 Total deposit: $250.00
• 🎉 Repository reward(0%): $0.00
• 🔧 Service fee(10%): $25.00

[anacrolix]

I'm closing this as done. There are further improvements to be made, but now that Webtorrent is supported on master, those will come over time.

[jpillora]

Nice work Matt!

…

On Thu, 16 Apr 2020 at 9:35 am Matt Joiner ***@***.***> wrote: Closed #138 <#138>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#138 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE2X4234VIIWQYJNC6W6MTRMZAE7ANCNFSM4CZIEDIQ> .