Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib: improve esm resolve performance #46652

Merged
merged 1 commit into from
Apr 23, 2023

Conversation

anonrig
Copy link
Member

@anonrig anonrig commented Feb 14, 2023

I couldn't find a benchmark or had the time to write it, but in theory, it should be a lot faster.

cc @nodejs/performance @nodejs/loaders

Refs: nodejs/performance#39

@anonrig anonrig added the performance Issues and PRs related to the performance of Node.js. label Feb 14, 2023
@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/loaders
  • @nodejs/modules

@nodejs-github-bot nodejs-github-bot added esm Issues and PRs related to the ECMAScript Modules implementation. needs-ci PRs that need a full CI run. labels Feb 14, 2023
Copy link
Member

@GeoffreyBooth GeoffreyBooth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On quick glance this seems fine, but we need to land #44710 first. Just putting a hold on this until that gets landed.

@GeoffreyBooth GeoffreyBooth added the loaders Issues and PRs related to ES module loaders label Feb 14, 2023
Copy link
Member

@GeoffreyBooth GeoffreyBooth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chatted with @anonrig and they offered to rebase #44710 on top of this if this is ready first, which seems fine to me. I’d still like to ensure this doesn’t slow down #44710 landing though; the rebased version of that branch shouldn’t have any new test failures.

Copy link
Contributor

@aduh95 aduh95 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see a benchmark for this, but the theory looks solid. This can land before or after the off-thread PR, the conflicts should be minimal :)

@arcanis
Copy link
Contributor

arcanis commented Feb 14, 2023

Shouldn't it be factorized with the cjs/loader implementation (note that since it isn't in this PR, the esm implementation doesn't has a cache)?

@anonrig
Copy link
Member Author

anonrig commented Feb 14, 2023

Shouldn't it be factorized with the cjs/loader implementation (note that since it isn't in this PR, the esm implementation doesn't has a cache)?

I agree, but I prefer to do it in a different pull request, and keep this minimal.

@mscdex mscdex added the needs-citgm PRs that need a CITGM CI run. label Feb 14, 2023
Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@anonrig anonrig added the request-ci Add this label to start a Jenkins CI on a PR. label Feb 17, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 17, 2023
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@GeoffreyBooth
Copy link
Member

Before we land this shouldn’t there be some attempt at confirming that this does, in fact, make things faster? And especially not slower. Even just running a benchmark on someone’s local machine.

Copy link
Member

@benjamingr benjamingr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To strengthen @GeoffreyBooth 's point
Making sure benchmarks are run in some form before this lands. Please dismiss my review when that happens @anonrig (or whomever)

@anonrig
Copy link
Member Author

anonrig commented Apr 23, 2023

@GeoffreyBooth Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1332/

Results
                                                                              confidence improvement accuracy (*)    (**)   (***)
esm/esm-loader-defaultResolve.js specifier='./relative-existing.js' n=1000             *      9.41 %       ±8.24% ±10.97% ±14.29%
esm/esm-loader-defaultResolve.js specifier='./relative-nonexistent.js' n=1000        ***     15.64 %       ±6.89%  ±9.20% ±12.03%
esm/esm-loader-defaultResolve.js specifier='node:os' n=1000                                  -0.28 %       ±8.78% ±11.68% ±15.20%
esm/esm-loader-defaultResolve.js specifier='node:prefixed-nonexistent' n=1000                -3.20 %       ±7.41%  ±9.87% ±12.85%
esm/esm-loader-defaultResolve.js specifier='unprefixed-existing' n=1000               **     13.37 %       ±8.00% ±10.65% ±13.88%
esm/esm-loader-defaultResolve.js specifier='unprefixed-nonexistent' n=1000             *      9.01 %       ±7.15%  ±9.52% ±12.39%

@aduh95 aduh95 added the commit-queue Add this label to land a pull request using GitHub Actions. label Apr 23, 2023
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Apr 23, 2023
@nodejs-github-bot nodejs-github-bot merged commit 3e9ed7e into nodejs:main Apr 23, 2023
@nodejs-github-bot
Copy link
Collaborator

Landed in 3e9ed7e

yjl9903 pushed a commit to yjl9903/node that referenced this pull request Apr 28, 2023
PR-URL: nodejs#46652
Refs: nodejs/performance#39
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Erick Wendel <erick.workspace@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
yjl9903 pushed a commit to yjl9903/node that referenced this pull request Apr 28, 2023
PR-URL: nodejs#46652
Refs: nodejs/performance#39
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Erick Wendel <erick.workspace@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
yjl9903 pushed a commit to yjl9903/node that referenced this pull request Apr 29, 2023
PR-URL: nodejs#46652
Refs: nodejs/performance#39
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Erick Wendel <erick.workspace@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
targos pushed a commit that referenced this pull request May 2, 2023
PR-URL: #46652
Refs: nodejs/performance#39
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Erick Wendel <erick.workspace@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
@targos targos mentioned this pull request May 2, 2023
@danielleadams danielleadams added the backport-requested-v18.x PRs awaiting manual backport to the v18.x-staging branch. label Jul 4, 2023
codebytere added a commit to electron/electron that referenced this pull request Nov 6, 2023
* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)
targos pushed a commit that referenced this pull request Nov 10, 2023
PR-URL: #46652
Refs: nodejs/performance#39
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Erick Wendel <erick.workspace@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
codebytere added a commit to electron/electron that referenced this pull request Nov 14, 2023
* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)
codebytere added a commit to electron/electron that referenced this pull request Nov 15, 2023
* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)
codebytere added a commit to electron/electron that referenced this pull request Nov 16, 2023
* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)
codebytere added a commit to electron/electron that referenced this pull request Nov 21, 2023
* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)
codebytere added a commit to electron/electron that referenced this pull request Nov 22, 2023
* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)
codebytere added a commit to electron/electron that referenced this pull request Nov 28, 2023
* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)
codebytere added a commit to electron/electron that referenced this pull request Nov 29, 2023
* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)
jkleinsc pushed a commit to electron/electron that referenced this pull request Nov 30, 2023
* chore: upgrade to Node.js v20

* src: allow embedders to override NODE_MODULE_VERSION

nodejs/node#49279

* src: fix missing trailing ,

nodejs/node#46909

* src,tools: initialize cppgc

nodejs/node#45704

* tools: allow passing absolute path of config.gypi in js2c

nodejs/node#49162

* tools: port js2c.py to C++

nodejs/node#46997

* doc,lib: disambiguate the old term, NativeModule

nodejs/node#45673

* chore: fixup Node.js BSSL tests

* nodejs/node#49492
* nodejs/node#44498

* deps: upgrade to libuv 1.45.0

nodejs/node#48078

* deps: update V8 to 10.7

nodejs/node#44741

* test: use gcUntil() in test-v8-serialize-leak

nodejs/node#49168

* module: make CJS load from ESM loader

nodejs/node#47999

* src: make BuiltinLoader threadsafe and non-global

nodejs/node#45942

* chore: address changes to CJS/ESM loading

* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)

* bootstrap: optimize modules loaded in the built-in snapshot

nodejs/node#45849

* test: mark test-runner-output as flaky

nodejs/node#49854

* lib: lazy-load deps in modules/run_main.js

nodejs/node#45849

* url: use private properties for brand check

nodejs/node#46904

* test: refactor `test-node-output-errors`

nodejs/node#48992

* assert: deprecate callTracker

nodejs/node#47740

* src: cast v8::Object::GetInternalField() return value to v8::Value

nodejs/node#48943

* test: adapt test-v8-stats for V8 update

nodejs/node#45230

* tls: ensure TLS Sockets are closed if the underlying wrap closes

nodejs/node#49327

* test: deflake test-tls-socket-close

nodejs/node#49575

* net: fix crash due to simultaneous close/shutdown on JS Stream Sockets

nodejs/node#49400

* net: use asserts in JS Socket Stream to catch races in future

nodejs/node#49400

* lib: fix BroadcastChannel initialization location

nodejs/node#46864

* src: create BaseObject with node::Realm

nodejs/node#44348

* src: implement DataQueue and non-memory resident Blob

nodejs/node#45258

* sea: add support for V8 bytecode-only caching

nodejs/node#48191

* chore: fixup patch indices

* gyp: put filenames in variables

nodejs/node#46965

* build: modify js2c.py into GN executable

* fix: (WIP) handle string replacement of fs -> original-fs

* [v20.x] backport vm-related memory fixes

nodejs/node#49874

* src: make BuiltinLoader threadsafe and non-global

nodejs/node#45942

* src: avoid copying string in fs_permission

nodejs/node#47746

* look upon my works ye mighty

and dispair

* chore: patch cleanup

* [api] Remove AllCan Read/Write

https://chromium-review.googlesource.com/c/v8/v8/+/5006387

* fix: missing include for NODE_EXTERN

* chore: fixup patch indices

* fix: fail properly when js2c fails in Node.js

* build: fix js2c root_gen_dir

* fix: lib/fs.js -> lib/original-fs.js

* build: fix original-fs file xforms

* fixup! module: make CJS load from ESM loader

* build: get rid of CppHeap for now

* build: add patch to prevent extra fs lookup on esm load

* build: greatly simplify js2c modifications

Moves our original-fs modifications back into a super simple python script action, wires up the output of that action into our call to js2c

* chore: update to handle moved internal/modules/helpers file

* test: update @types/node test

* feat: enable preventing cppgc heap creation

* feat: optionally prevent calling V8::EnableWebAssemblyTrapHandler

* fix: no cppgc initialization in the renderer

* gyp: put filenames in variables

nodejs/node#46965

* test: disable single executable tests

* fix: nan tests failing on node headers missing file

* tls,http2: send fatal alert on ALPN mismatch

nodejs/node#44031

* test: disable snapshot tests

* nodejs/node#47887
* nodejs/node#49684
* nodejs/node#44193

* build: use deps/v8 for v8/tools

Node.js hard depends on these in their builtins

* test: fix edge snapshot stack traces

nodejs/node#49659

* build: remove js2c //base dep

* build: use electron_js2c_toolchain to build node_js2c

* fix: don't create SafeSet outside packageResolve

Fixes failure in parallel/test-require-delete-array-iterator:

=== release test-require-delete-array-iterator ===
Path: parallel/test-require-delete-array-iterator
node:internal/per_context/primordials:426
    constructor(i) { super(i); } // eslint-disable-line no-useless-constructor
                     ^

TypeError: object is not iterable (cannot read property Symbol(Symbol.iterator))
    at new Set (<anonymous>)
    at new SafeSet (node:internal/per_context/primordials:426:22)

* fix: failing crashReporter tests on Linux

These were failing because our change from node::InitializeNodeWithArgs to
node::InitializeOncePerProcess meant that we now inadvertently called
PlatformInit, which reset signal handling. This meant that our intentional
crash function ElectronBindings::Crash no longer worked and the renderer process
no longer crashed when process.crash() was called. We don't want to use Node.js'
default signal handling in the renderer process, so we disable it by passing
kNoDefaultSignalHandling to node::InitializeOncePerProcess.

* build: only create cppgc heap on non-32 bit platforms

* chore: clean up util:CompileAndCall

* src: fix compatility with upcoming V8 12.1 APIs

nodejs/node#50709

* fix: use thread_local BuiltinLoader

* chore: fixup v8 patch indices

---------

Co-authored-by: Keeley Hammond <vertedinde@electronjs.org>
Co-authored-by: Samuel Attard <marshallofsound@electronjs.org>
MrHuangJser pushed a commit to MrHuangJser/electron that referenced this pull request Dec 11, 2023
* chore: upgrade to Node.js v20

* src: allow embedders to override NODE_MODULE_VERSION

nodejs/node#49279

* src: fix missing trailing ,

nodejs/node#46909

* src,tools: initialize cppgc

nodejs/node#45704

* tools: allow passing absolute path of config.gypi in js2c

nodejs/node#49162

* tools: port js2c.py to C++

nodejs/node#46997

* doc,lib: disambiguate the old term, NativeModule

nodejs/node#45673

* chore: fixup Node.js BSSL tests

* nodejs/node#49492
* nodejs/node#44498

* deps: upgrade to libuv 1.45.0

nodejs/node#48078

* deps: update V8 to 10.7

nodejs/node#44741

* test: use gcUntil() in test-v8-serialize-leak

nodejs/node#49168

* module: make CJS load from ESM loader

nodejs/node#47999

* src: make BuiltinLoader threadsafe and non-global

nodejs/node#45942

* chore: address changes to CJS/ESM loading

* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)

* bootstrap: optimize modules loaded in the built-in snapshot

nodejs/node#45849

* test: mark test-runner-output as flaky

nodejs/node#49854

* lib: lazy-load deps in modules/run_main.js

nodejs/node#45849

* url: use private properties for brand check

nodejs/node#46904

* test: refactor `test-node-output-errors`

nodejs/node#48992

* assert: deprecate callTracker

nodejs/node#47740

* src: cast v8::Object::GetInternalField() return value to v8::Value

nodejs/node#48943

* test: adapt test-v8-stats for V8 update

nodejs/node#45230

* tls: ensure TLS Sockets are closed if the underlying wrap closes

nodejs/node#49327

* test: deflake test-tls-socket-close

nodejs/node#49575

* net: fix crash due to simultaneous close/shutdown on JS Stream Sockets

nodejs/node#49400

* net: use asserts in JS Socket Stream to catch races in future

nodejs/node#49400

* lib: fix BroadcastChannel initialization location

nodejs/node#46864

* src: create BaseObject with node::Realm

nodejs/node#44348

* src: implement DataQueue and non-memory resident Blob

nodejs/node#45258

* sea: add support for V8 bytecode-only caching

nodejs/node#48191

* chore: fixup patch indices

* gyp: put filenames in variables

nodejs/node#46965

* build: modify js2c.py into GN executable

* fix: (WIP) handle string replacement of fs -> original-fs

* [v20.x] backport vm-related memory fixes

nodejs/node#49874

* src: make BuiltinLoader threadsafe and non-global

nodejs/node#45942

* src: avoid copying string in fs_permission

nodejs/node#47746

* look upon my works ye mighty

and dispair

* chore: patch cleanup

* [api] Remove AllCan Read/Write

https://chromium-review.googlesource.com/c/v8/v8/+/5006387

* fix: missing include for NODE_EXTERN

* chore: fixup patch indices

* fix: fail properly when js2c fails in Node.js

* build: fix js2c root_gen_dir

* fix: lib/fs.js -> lib/original-fs.js

* build: fix original-fs file xforms

* fixup! module: make CJS load from ESM loader

* build: get rid of CppHeap for now

* build: add patch to prevent extra fs lookup on esm load

* build: greatly simplify js2c modifications

Moves our original-fs modifications back into a super simple python script action, wires up the output of that action into our call to js2c

* chore: update to handle moved internal/modules/helpers file

* test: update @types/node test

* feat: enable preventing cppgc heap creation

* feat: optionally prevent calling V8::EnableWebAssemblyTrapHandler

* fix: no cppgc initialization in the renderer

* gyp: put filenames in variables

nodejs/node#46965

* test: disable single executable tests

* fix: nan tests failing on node headers missing file

* tls,http2: send fatal alert on ALPN mismatch

nodejs/node#44031

* test: disable snapshot tests

* nodejs/node#47887
* nodejs/node#49684
* nodejs/node#44193

* build: use deps/v8 for v8/tools

Node.js hard depends on these in their builtins

* test: fix edge snapshot stack traces

nodejs/node#49659

* build: remove js2c //base dep

* build: use electron_js2c_toolchain to build node_js2c

* fix: don't create SafeSet outside packageResolve

Fixes failure in parallel/test-require-delete-array-iterator:

=== release test-require-delete-array-iterator ===
Path: parallel/test-require-delete-array-iterator
node:internal/per_context/primordials:426
    constructor(i) { super(i); } // eslint-disable-line no-useless-constructor
                     ^

TypeError: object is not iterable (cannot read property Symbol(Symbol.iterator))
    at new Set (<anonymous>)
    at new SafeSet (node:internal/per_context/primordials:426:22)

* fix: failing crashReporter tests on Linux

These were failing because our change from node::InitializeNodeWithArgs to
node::InitializeOncePerProcess meant that we now inadvertently called
PlatformInit, which reset signal handling. This meant that our intentional
crash function ElectronBindings::Crash no longer worked and the renderer process
no longer crashed when process.crash() was called. We don't want to use Node.js'
default signal handling in the renderer process, so we disable it by passing
kNoDefaultSignalHandling to node::InitializeOncePerProcess.

* build: only create cppgc heap on non-32 bit platforms

* chore: clean up util:CompileAndCall

* src: fix compatility with upcoming V8 12.1 APIs

nodejs/node#50709

* fix: use thread_local BuiltinLoader

* chore: fixup v8 patch indices

---------

Co-authored-by: Keeley Hammond <vertedinde@electronjs.org>
Co-authored-by: Samuel Attard <marshallofsound@electronjs.org>
sercher added a commit to sercher/graaljs that referenced this pull request Apr 25, 2024
PR-URL: nodejs/node#46652
Refs: nodejs/performance#39
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Erick Wendel <erick.workspace@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
sercher added a commit to sercher/graaljs that referenced this pull request Apr 25, 2024
PR-URL: nodejs/node#46652
Refs: nodejs/performance#39
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Erick Wendel <erick.workspace@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-requested-v18.x PRs awaiting manual backport to the v18.x-staging branch. esm Issues and PRs related to the ECMAScript Modules implementation. loaders Issues and PRs related to ES module loaders needs-ci PRs that need a full CI run. needs-citgm PRs that need a CITGM CI run. performance Issues and PRs related to the performance of Node.js.
Projects
None yet
Development

Successfully merging this pull request may close these issues.