IPFS Integration #1

daviddias · 2017-06-27T17:27:56Z

WIP PR for brave#9556 (comment)

Notes to self (or others that might want to contribute :))

Folder structure

.
├── ...
├── app        # Where the electron's main process things live
│   ├── # ...
│   ├── browser
│   │   ├── # ...
│   │   ├── ipfs.js   # The ipfs instance running on the main process
│   │   ├── # ...
│   │   ├── webtorrent.js  # The webtorrent instance running on the main process
│   │   └── # ...
│   ├── extensions # Where the extensions live, except the actual app that gets rendered to the user
│   │   ├── brave
│   │   ├── ipfs
│   │   └── torrent
│   └── # ...
├── js
│   ├── # ...
│   ├── ipfs # Where the IPFS extension app lives  (renderer process)
│   │   └── entry.js
│   ├── lib # Lot's of utils 
│   │   ├── appUrlUtil.js
│   │   ├── base64.js
│   │   ├── baseDomain.js
│   │   ├── classSet.js
│   │   ├── color.js
│   │   ├── cryptoUtil.js
│   │   ├── debounce.js
│   │   ├── eventUtil.js
│   │   ├── functional.js
│   │   ├── imageUtil.js
│   │   ├── importer.js
│   │   ├── openSearch.js
│   │   ├── promisify.js
│   │   ├── psl.js
│   │   ├── request.js
│   │   ├── text.js
│   │   ├── textCalculator.js
│   │   ├── throttle.js
│   │   ├── urlutil.js
│   │   └── zoom.js
│   └── webtorrent # The WebTorrent app (renderer process)
│       ├── components
│       └── entry.js
└── # ...

daviddias · 2017-06-27T18:08:51Z

app/extensions.js

+    incognito: 'split',
+    // TODO(diasdavid) How to get this key??
+    key: 'MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyWl+wMvL0wZX3JUs7GeZAvxMP+LWEh2bwMV1HyuBra/lGZIq3Fmh0+AFnvFPXz1NpQkbLS3QWyqhdIn/lepGwuc2ma0glPzzmieqwctUurMGSGManApGO1MkcbSPhb+R1mx8tMam5+wbme4WoW37PI3oATgOs2NvHYuP60qol3U7b/zB3IWuqtwtqKe2Q1xY17btvPuz148ygWWIHneedt0jwfr6Zp+CSLARB9Heq/jqGXV4dPSVZ5ebBHLQ452iZkHxS6fm4Z+IxjKdYs3HNj/s8xbfEZ2ydnArGdJ0lpSK9jkDGYyUBugq5Qp3FH6zV89WqBvoV1dqUmL9gxbHsQIDAQAB'
+  }


https://developer.chrome.com/apps/manifest/key

daviddias · 2017-06-27T18:09:02Z

js/constants/config.js

@@ -49,6 +49,8 @@ module.exports = {
  },
  braveExtensionId: 'mnojpmjdmbbfmejpflffifhffcmidifd',
  torrentExtensionId: 'fmdpfempfmekjkcfdehndghogpnpjeno',
+  // TODO(diasdavid) how to these get calculated?
+  ipfsExtensionId: 'asdakdaklsjdaklsjdalsdaslkdjasds',


daviddias · 2017-06-27T18:09:37Z

js/stores/appStore.js

@@ -427,6 +427,8 @@ const handleAppAction = (action) => {
      appState = filtering.init(appState, action, appStore)
      appState = basicAuth.init(appState, action, appStore)
      appState = webtorrent.init(appState, action, appStore)
+      // TODO(diasdavid) - init IPFS in the main process
+      // appState = ipfs.init(appState, action, appStore)


⁉️ Should I do this here?

daviddias · 2017-06-27T19:09:16Z

app/renderer/components/frame/frame.js

@@ -316,7 +323,8 @@ class Frame extends React.Component {
        // In this case both the user display and the user think they're on this.props.location.
        if (this.props.tabUrl !== this.props.location &&
          !this.isAboutPage() &&
-          !isTorrentViewerURL(this.props.location)) {
+          !isTorrentViewerURL(this.props.location) &&
+          !isIPFSPath(this.props.location)) {


I believe I got this code path right (it checks if location starts with dweb: or ipfs://), however, I never managed to get the log on 74 to print.

daviddias · 2017-06-28T09:51:15Z

js/ipfs/entry.js

@@ -0,0 +1,3 @@
+// const ipc = window.chrome.ipcRenderer
+
+console.log('woot! managed to capture the ipfs:// or dweb:// protocols')


🌟 if I get this one to run, I believe I know what to do next :)

daviddias · 2017-06-28T15:35:11Z

app/extensions.js

+  if (getSetting(settings.IPFS_ENABLED)) {
+    extensionInfo.setState(config.ipfsExtensionId, extensionStates.REGISTERED)
+    loadExtension(config.ipfsExtensionId, getExtensionsPath('ipfs'), generateIPFSManifest(), 'component')
+  } else {


I can confirm that getSetting(settings.IPFS_ENABLED)) prints true, which means that the extension should be loaded.

Line 537 and 538 do nothing for some reason.

daviddias · 2017-06-29T22:53:40Z

app/extensions.js

@@ -471,6 +531,17 @@ module.exports.init = () => {
    extensionActions.extensionDisabled(config.torrentExtensionId)
  }

+  // ipfsExtension
+  const isIPFSEnabled = getSetting(settings.IPFS_ENABLED)
+  console.log('IPFS: is extension enabled', isIPFSEnabled, config.ipfsExtensionId, getExtensionsPath('ipfs'))


This prints:
IPFS: is extension enabled true interplanetaryfilesystemuniverse /Users/koruza/code/brave-browser-laptop/app/extensions/ipfs

daviddias · 2017-06-29T23:21:08Z

@jonathansampson @bridiver @NejcZdovc mind taking a look at what I might be failing when loading this extension?

bridiver · 2017-06-30T00:19:31Z

app/extensions.js

+// Returns the Chromium extension manifest for the ipfsExtension
+// The ipfsExtension handles ipfs:// and dweb:
+// Analagous to the PDFJS extension, it shows a special UI for that type of resource
+let generateIPFSManifest = () => {


I don't think you need all this. You can just create a regular manifest file and pass the path to the directory like non-component extensions

daviddias · 2017-06-30T18:36:28Z

Turning on tabbycat just started throwing uncaught exceptions:

Have you seen this behaviour?

jonathansampson · 2017-07-01T05:32:52Z

@diasdavid I have not seen that issue before, but when I ran Tabby Cat in Brave, we didn't have the about:preferences#extensions page. Are you able to share any further information about the Exception itself? Perhaps file/line-number?

You may even be able to break on exception from the Shift+F8 developer tools. Happy to help investigate further if needed.

daviddias · 2017-07-06T06:19:31Z

🗡 I've slain the first 🐉

daviddias · 2017-07-06T06:50:58Z

app/extensions/ipfs/js/main.js

+    callback('hi there!' + test() + IPFS) // eslint-disable-line
+  })
+  */
+  callback('hi there!' + test() + IPFS) // eslint-disable-line


I'm being unable to create an IPFS node as the background process crashes (for some reason) and I have no way to inspect it. Looking for a better way to debug this background extension.

daviddias · 2017-08-22T07:28:10Z

package.json

@@ -45,7 +45,8 @@
    "watch-all": "npm run watch & npm run watch-test",
    "watch-test": "cross-env NODE_ENV=test webpack --watch",
    "webpack": "webpack",
-    "win-renpm": "powershell ./tools/windows/re-npm.ps1"
+    "win-renpm": "powershell ./tools/windows/re-npm.ps1",
+    "ipfs": "wget https://unpkg.com/ipfs/dist/index.min.js -O app/extensions/ipfs/js/ipfs.min.js"


I added this step to make it simpler to reproduce my local set up.

daviddias · 2017-08-22T16:55:30Z

Update - Great news everyone!

We have successfully managed to run IPFS from a background process in a Brave Extension, capturing the ipfs:// protocol and resolving an hash from the IPFS network, see the image below for the full result.

The main issue we were facing was that the CSP was blocking our use of eval, which is exactly that @lidel while developing the Chrome and Firefox Web Extension with js-ipfs-api. For now, we removed that restriction for testing purposes but before we merge and release this feature, we want to:

make sure that we really need to use eval (can we change how the dependency that is injecting it works?)
if we absolutely need to use eval, can we ensure that it doesn't open any security vulnerability

daviddias · 2017-08-22T17:16:51Z

Streaming Responses Issue brave#10629

daviddias · 2017-08-28T05:10:41Z

app/extensions/brave/index-dev.html

@@ -7,7 +7,7 @@
    <!-- TODO: Don't allow img-src *, needed for favicons -->
    <!-- TODO: Refactor away all unsafe-inline content -->
    <!-- TODO: Replace suggestqueries.google.com and ac.duckduckgo.com and other search engines with a single config search engine -->
-    <meta http-equiv="Content-Security-Policy" content="default-src 'none'; form-action http://localhost:*; script-src 'self' http://localhost:*; connect-src 'self' https://s3.amazonaws.com/adblock-data/ https://s3.amazonaws.com/safe-browsing-data/ https://s3.amazonaws.com/tracking-protection-data/ https://s3.amazonaws.com/https-everywhere-data/ http://localhost:* ws://localhost:* https://suggestqueries.google.com https://ac.duckduckgo.com https://completion.amazon.com https://search.yahoo.com https://api.bing.com https://www.startpage.com https://infogalactic.com https://api.qwant.com https://ac.ecosia.org https://searx.me https://www.findx.com https://brave-download.global.ssl.fastly.net https://brave-laptop-updates.global.ssl.fastly.net https://brave-download.global.ssl.fastly.net https://laptop-updates-pre.brave.com https://brave-laptop-updates-pre.brave.com; style-src 'unsafe-inline'; font-src 'self' http://localhost:*; img-src 'self' * data: file: chrome-extension:; object-src 'self'; plugin-types application/browser-plugin">
+    <meta http-equiv="Content-Security-Policy" content="default-src 'none'; form-action http://localhost:*; script-src 'self' 'unsafe-eval' http://localhost:*; connect-src 'self' https://s3.amazonaws.com/adblock-data/ https://s3.amazonaws.com/safe-browsing-data/ https://s3.amazonaws.com/tracking-protection-data/ https://s3.amazonaws.com/https-everywhere-data/ http://localhost:* ws://localhost:* https://suggestqueries.google.com https://ac.duckduckgo.com https://completion.amazon.com https://search.yahoo.com https://api.bing.com https://www.startpage.com https://infogalactic.com https://api.qwant.com https://ac.ecosia.org https://searx.me https://www.findx.com https://brave-download.global.ssl.fastly.net https://brave-laptop-updates.global.ssl.fastly.net https://brave-download.global.ssl.fastly.net https://laptop-updates-pre.brave.com https://brave-laptop-updates-pre.brave.com; style-src 'unsafe-inline'; font-src 'self' http://localhost:*; img-src 'self' * data: file: chrome-extension:; object-src 'self'; plugin-types application/browser-plugin">


added the 'unsafe-eval'

why is that needed? I think @diracdeltas will likely have an issue with it

Unfortunately unsafe-eval removes most of the purpose of having a CSP - please avoid it by refactoring all inline script code into external files.

Removed it from here. It was not necessary

daviddias · 2017-08-28T07:09:34Z

app/extensions/ipfs/ipfs.html

+        wss://nyc-1.bootstrap.libp2p.io
+        wss://nyc-2.bootstrap.libp2p.io
+      ;
+      object-src 'self'; plugin-types application/browser-plugin"/>


In order to support a background page (and not just scripts) had to add manually all the endpoints a js-ipfs node connects on startup. This is a bit of annoyance as it will stop js-ipfs to connect dynamically to other public WebSockets endpoints unless the CSP can be relaxed.

oh wow, it actually takes wildcards. I completely had discarded that option.

bridiver · 2017-08-28T19:52:43Z

app/extensions/ipfs/manifest.json

+    "page": "ipfs.html",
+    "persistent": true
+  },
+  "content_security_policy": "script-src 'self' 'unsafe-eval'"


I believe the csp you have defined in the page above can be defined here instead

if the CSP is just for the IPFS extension, it should be defined here instead of in index-dev.html. i would still encourage refactoring such that unsafe-eval isn't needed.

@bridiver ~~when an extension has an index.html page, the extension loader ignores this CSP and applies the one in index.html. So yes, you are right, the index.html one is the only one necessary.~~ My test yielded a false positive, it actually needs to have this in the manifest too.

@diracdeltas agreed that we should not have the need for unsafe-eval. We are working on it :)

@diasdavid I think the latest version of js-ipfs-api is ready to be tested without evals :)

Here we just use js-ipfs. It seems that there still something. Need to test this more:

that is definitely still a protobuf thing

looks like we missed two modules

npm ls protocol-buffers ipfs@0.25.4 /Users/dignifiedquire/opensource/ipfs/js-ipfs ├─┬ libp2p-floodsub@0.11.0 │ └─┬ libp2p-crypto@0.9.4 │ └── protocol-buffers@3.2.1 deduped └─┬ libp2p-kad-dht@0.5.0 └── protocol-buffers@3.2.1

💡 So it is evident. Nice, I was worried that it was npm false deduping. Thanks for catching that!

Woot 🚀 No more unsafe-eval!

🎉 🎉 🎉

…rave doesnt fully support it yet

kyledrake · 2017-11-03T15:13:16Z

Any updates here? Sounds like this is ready to go now that the eval issue is fixed. It would be nice to see this get merged into Brave and released.

Jacalz · 2017-11-03T15:23:42Z

Yeah, I would love to see this too, have been waiting for a long time now 😅

daviddias · 2017-11-13T10:28:58Z

Good news everyone! @alanshaw and @olizilla are coming to help us polish the last quirks here! \o/

You can expect more activity starting this week this week. Especially when it comes to the UX for the user, cleaning out any last bugs and following up on outstanding tasks such as the needed streaming API.

@olizilla and @alanshaw welcome to this endeavor! Ping me through this thread or IRC if you have any questions! :)

alanshaw · 2017-11-13T11:36:58Z

Just a few of notes:

Webpack needs to be installed globally to build
wget needs to be installed for npm run ipfs to work
Document that npm run ipfs needs to be run!
ad-block@3.1.8 (wanted ad-block@3.0.2) causes error Cannot find module 'ad-block/lib/regions'. I had to install ad-block@3.0.2 to successfully start (maybe this is fixed upstream?)

olizilla · 2017-11-14T10:59:04Z

app/extensions/ipfs/js/main.js

+  }
+})
+
+// TODO: bring back once brave supports registerURIHandlers


@diasdavid do you have any background info on why registerURIHandlers is needed?

because an IPFS hash is, in reality, a URI and not a URL, we should be able to do ipfs:QmHash and not ipfs://QmHash.

This is still an ongoing proposal, read more at:

Background on Address Scheme discussions (dweb, ipfs:// and more) ipfs/specs#152

https://github.com/ipfs/specs/pull/167/files

ghost · 2017-11-14T15:23:10Z

we should be able to do ipfs:QmHash and not ipfs://QmHash.

Mh no, ipfs://QmHash is absolutely fine, and the URI variant is dweb:/ipfs/QmHash

olizilla · 2017-11-14T15:27:15Z

@lgierth i think @diasdavid is saying that in the context of this PR, the current impl can only do ipfs://QmHash style, and if we had the registerURIHandlers api available, we could do dweb:/ipfs/QmHash or ipfs:QmHash uris. (and that dweb is preferred)

daviddias force-pushed the feat/ipfs branch from 6408d45 to e004c23 Compare June 27, 2017 18:06

daviddias commented Jun 27, 2017

View reviewed changes

daviddias force-pushed the feat/ipfs branch from e004c23 to 34d2501 Compare June 27, 2017 18:53

daviddias commented Jun 27, 2017

View reviewed changes

daviddias force-pushed the feat/ipfs branch 3 times, most recently from 4e40ac2 to 9d4381a Compare June 28, 2017 09:47

daviddias commented Jun 28, 2017

View reviewed changes

daviddias force-pushed the feat/ipfs branch from 9d4381a to 5cbc1a4 Compare June 28, 2017 09:52

daviddias commented Jun 28, 2017

View reviewed changes

daviddias force-pushed the feat/ipfs branch 3 times, most recently from e6be4bc to 4737315 Compare June 29, 2017 22:48

daviddias commented Jun 29, 2017

View reviewed changes

bridiver reviewed Jun 30, 2017

View reviewed changes

daviddias force-pushed the feat/ipfs branch from 29d8c35 to 53f9d8b Compare June 30, 2017 22:08

daviddias mentioned this pull request Jun 30, 2017

IPFS Integration - Roadmap and discussion brave/browser-laptop#9556

Closed

daviddias force-pushed the feat/ipfs branch from 6f0be9a to f5691c5 Compare July 5, 2017 14:19

daviddias mentioned this pull request Jul 5, 2017

Tabbycat errors with uncaught exception. brave/browser-laptop#9871

Closed

daviddias commented Jul 6, 2017

View reviewed changes

daviddias force-pushed the feat/ipfs branch 3 times, most recently from a3a481b to 6ea5fdf Compare August 22, 2017 06:49

feat: mix in IPFS

a6d660c

daviddias commented Aug 22, 2017

View reviewed changes

it loads an IPFS node!

08b4f38

This was referenced Aug 22, 2017

IMKInputSession presentFunctionRowItemTextInputViewWithEndpoint:completionHandler brave/browser-laptop#10630

Closed

Brave crashes when a background process of an Extension uses WebRTC brave/browser-laptop#10631

Closed

daviddias commented Aug 28, 2017

View reviewed changes

daviddias added 4 commits August 28, 2017 07:15

rm useIPFS from internationalization messages (not used anywhere)

4d2b5cc

add new ipfs 0.25.2

4ca81cf

IPFS background page

e248f2d

feat: make it work with a background page instead

faa9afe

daviddias commented Aug 28, 2017

View reviewed changes

daviddias mentioned this pull request Aug 28, 2017

CSP for IPFS Integration (can be used for other Brave extensions) brave/browser-laptop#10683

Closed

bridiver reviewed Aug 28, 2017

View reviewed changes

daviddias mentioned this pull request Aug 29, 2017

Using IPFS with electron (or muon) ipfs/notes#256

Open

daviddias added 4 commits September 1, 2017 08:28

rm csp from ipfs ext

3952beb

add csp back again

f2a0df9

identify next steps, wait for node to be ready, remove WebRTC while b…

ca10910

…rave doesnt fully support it yet

only use unsafe-eval where it needs to be explicitely applied

bc43e50

lidel mentioned this pull request Sep 2, 2017

Remove use of unsafe-eval ipfs/ipfs-companion#269

Closed

daviddias mentioned this pull request Sep 7, 2017

Awesome Endeavour - unsafe-eval no more! ipfs/js-ipfs#991

Closed

11 tasks

daviddias mentioned this pull request Oct 16, 2017

How to add google chrome extension to muon? brave/muon#163

Open

olizilla reviewed Nov 14, 2017

View reviewed changes

olizilla mentioned this pull request Nov 29, 2017

Init ipfs-api or js-ipfs ipfs/ipfs-companion#320

Merged

		@@ -0,0 +1,3 @@
		// const ipc = window.chrome.ipcRenderer

		console.log('woot! managed to capture the ipfs:// or dweb:// protocols')

IPFS Integration #1

Are you sure you want to change the base?

IPFS Integration #1

Conversation

daviddias commented Jun 27, 2017 • edited Loading

Folder structure

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias Jun 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Jun 29, 2017

Choose a reason for hiding this comment

daviddias commented Jun 30, 2017

jonathansampson commented Jul 1, 2017

daviddias commented Jul 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Aug 22, 2017

Update - Great news everyone!

daviddias commented Aug 22, 2017

Streaming Responses Issue brave#10629

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias Sep 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyledrake commented Nov 3, 2017

Jacalz commented Nov 3, 2017

daviddias commented Nov 13, 2017

alanshaw commented Nov 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Nov 14, 2017

olizilla commented Nov 14, 2017

daviddias commented Jun 27, 2017 •

edited

Loading

daviddias Jun 27, 2017 •

edited

Loading

daviddias Sep 1, 2017 •

edited

Loading