Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix include path when pointing to Node.js source #1055

Closed
wants to merge 1 commit into from

Conversation

richardlau
Copy link
Member

@richardlau richardlau commented Nov 17, 2016

Refs

When building modules that include headers from Node.js dependencies (that are not libuv or V8) compilation fails when --nodedir is used to point to a Node.js source tree.
e.g. nodereport's node_report.cc includes ares.h and zlib.h

-bash-4.2$ node-gyp rebuild --nodedir /home/riclau/sandbox/scratch/tmp/node-v6.9.1-src/
gyp info it worked if it ends with ok
gyp info using node-gyp@3.4.0
gyp info using node@6.9.1 | linux | x64
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/dev/shm/usenode.riclau/node-v6.9.1-linux-x64/lib/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/home/users/riclau/sandbox/github/nodereport/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/dev/shm/usenode.riclau/node-v6.9.1-linux-x64/lib/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/riclau/sandbox/scratch/tmp/node-v6.9.1-src/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/riclau/sandbox/scratch/tmp/node-v6.9.1-src/',
gyp info spawn args   '-Dnode_gyp_dir=/dev/shm/usenode.riclau/node-v6.9.1-linux-x64/lib/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=node.lib',
gyp info spawn args   '-Dmodule_root_dir=/home/users/riclau/sandbox/github/nodereport',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory `/home/users/riclau/sandbox/github/nodereport/build'
make: Warning: File `nodereport.target.mk' has modification time 893 s in the future
  CXX(target) Release/obj.target/nodereport/src/node_report.o
../src/node_report.cc:6:18: fatal error: ares.h: No such file or directory
 #include "ares.h"
                  ^
compilation terminated.
make: *** [Release/obj.target/nodereport/src/node_report.o] Error 1
make: Leaving directory `/home/users/riclau/sandbox/github/nodereport/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/dev/shm/usenode.riclau/node-v6.9.1-linux-x64/lib/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Linux 3.10.0-327.18.2.el7.x86_64
gyp ERR! command "/dev/shm/usenode.riclau/node-v6.9.1-linux-x64/bin/node" "/dev/shm/usenode.riclau/node-v6.9.1-linux-x64/bin/node-gyp" "rebuild" "--nodedir" "/home/riclau/sandbox/scratch/tmp/node-v6.9.1-src/"
gyp ERR! cwd /home/users/riclau/sandbox/github/nodereport
gyp ERR! node -v v6.9.1
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok
-bash-4.2$

This is because the header files for Node.js dependencies are in a different location in the Node.js source tree compared to the release tarballs (see tools/install.py).

@richardlau
Copy link
Member Author

I'm unsure how to add a test for this in node-gyp without downloading a matching source tarball for the version of Node.js running the test. Any ideas?

@rvagg
Copy link
Member

rvagg commented Nov 18, 2016

I'd be fine with no tests for this, I can't think of a convenient way to do it. This list now matches (more closely) what's in install.py for the headers so LGTM.

@bnoordhuis
Copy link
Member

I don't think it was ever the intent for the bundled c-ares to be consumed by add-ons. It's not very suitable for that.

It's a fork of upstream tweaked so it's easy to integrate with core, no effort was (or is) spent on making it API- and ABI-stable. It's also not exported on Windows.

@richardlau
Copy link
Member Author

Well install.py is explicitly copying out the c-ares headers for the release tarballs -- If they're not intended to be used they probably shouldn't be.

In any case it wasn't just the c-ares headers, e.g. https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/449/nodes=ubuntu1604-64/consoleText failed to find zlib.h:

verbose: nodedir             | Using --nodedir="/home/iojs/build/workspace/citgm-smoker/nodes/ubuntu1604-64"
verbose: npm-install:        | > nodereport@1.0.5 install /tmp/ea17a2df-fe7d-424e-a3aa-ea20d84777ef/nodereport
verbose:                     | > node-gyp rebuild                                                             
verbose: npm-install:        | make: Entering directory '/tmp/ea17a2df-fe7d-424e-a3aa-ea20d84777ef/nodereport/build'
warn: npm-install:        | make: *** No rule to make target '/tmp/ea17a2df-fe7d-424e-a3aa-ea20d84777ef/nodereport/build/Release/nodereport.node', needed by '/tmp/ea17a2df-fe7d-424e-a3aa-ea20d84777ef/nodereport/nodereport.node'.  Stop.
warn:                     | make: *** Waiting for unfinished jobs....                                                                                                                                                                      
verbose: npm-install:        | CXX(target) Release/obj.target/nodereport/src/module.o     
verbose:                     | CXX(target) Release/obj.target/nodereport/src/node_report.o
warn: npm-install:        | ../src/node_report.cc:5:18: fatal error: zlib.h: No such file or directory
warn:                     | compilation terminated.                                                   
verbose: npm-install:        | nodereport.target.mk:103: recipe for target 'Release/obj.target/nodereport/src/node_report.o' failed
warn: npm-install:        | make: *** [Release/obj.target/nodereport/src/node_report.o] Error 1
verbose: npm-install:        | make: Leaving directory '/tmp/ea17a2df-fe7d-424e-a3aa-ea20d84777ef/nodereport/build'
warn: npm-install:        | gyp                 
warn: npm-install:        |                     
warn: npm-install:        | ERR! build error    
warn: npm-install:        | gyp ERR! stack Error: `make` failed with exit code: 2

Although the Node.js Addons docs say

Only the V8 and OpenSSL symbols are purposefully re-exported by Node.js and may be used to various extents by Addons.

zlib symbols are exported on Windows (nodejs/node#7983) and there is an explicit zlib-binding addons test in Node.js (nodejs/node#8039).
Incidentally the same docs say:

Node.js uses a number of statically linked libraries such as V8, libuv and OpenSSL. All Addons are required to link to V8 and may link to any of the other dependencies as well. ...
node-gyp can be run using the --nodedir flag pointing at a local Node.js source image. Using this option, the Addon will have access to the full set of dependencies.

For NodeReport I've currently got an inflight pull request (nodejs/node-report#30) which will remove the dependency on ares.h and zlib.h but this has shown that it's possible to write modules which happily compile against the release tarballs but fail when compiled against the Node.js source tree.

@richardlau
Copy link
Member Author

Updated to correct path to openssl headers.

@richardlau
Copy link
Member Author

Actually this would have shown up in Node.js' own openssl-bindings test if the binding.gyp wasn't explicitly adding the openssl include dir:

{
  'targets': [
    {
      'target_name': 'binding',
      'sources': ['binding.cc'],
      'include_dirs': ['../../../deps/openssl/openssl/include'],
    },
  ]
}

If the 'include_dirs' line is commented out, the build fails on a machine that doesn't have the openssl dev package installed:

make[1]: Entering directory `/home/users/riclau/sandbox/github/node/test/addons/openssl-binding/build'
  CXX(target) Release/obj.target/binding/binding.o
../binding.cc:3:26: fatal error: openssl/rand.h: No such file or directory
 #include <openssl/rand.h>
                          ^
compilation terminated.

Replacing deps/npm/node_modules/node-gyp/addon.gypi with the one in this pull request will then allow the node build to compile the above openssl-binding test (without the 'include_dirs' in binding.gyp).

@gibfahn
Copy link
Member

gibfahn commented Dec 13, 2016

@rvagg

I'd be fine with no tests for this, I can't think of a convenient way to do it.

So as Richard said, the easiest way to test this is probably by making this change in the node tests (and possibly adding equivalents for cares and zlib.

diff --git a/test/addons/openssl-binding/binding.gyp b/test/addons/openssl-binding/binding.gyp
index 672f84b..b83bae3 100644
--- a/test/addons/openssl-binding/binding.gyp
+++ b/test/addons/openssl-binding/binding.gyp
@@ -2,8 +2,7 @@
   'targets': [
     {
       'target_name': 'binding',
-      'sources': ['binding.cc'],
-      'include_dirs': ['../../../deps/openssl/openssl/include'],
+      'sources': ['binding.cc']
     },
   ]
 }

@bnoordhuis

I don't think it was ever the intent for the bundled c-ares to be consumed by add-ons. It's not very suitable for that.

So would your preference be to remove the relevant lines from install.py? I feel we should be consistent about it either way.

@bnoordhuis
Copy link
Member

Yes, let's do that.

@gibfahn
Copy link
Member

gibfahn commented Dec 15, 2016

So @richardlau if you remove the cares and zlib headers from this PR I think it should be good to move forward with.

@richardlau
Copy link
Member Author

I'm waiting for a resolution to nodejs/node#10283 so I can update this PR to match.

gibfahn added a commit to gibfahn/node that referenced this pull request Mar 22, 2017
The bundled c-ares isn't very suitable for consumption by addons,
isn't kept stable, and isn't exported on windows.

PR-URL: nodejs#10283
Refs: nodejs/node-gyp#1055
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
@richardlau
Copy link
Member Author

richardlau commented Apr 4, 2017

Needs fixing up:

@yume-chan
Copy link

Any update? nodejs/node#10283 and nodejs/node#12217 have both been closed.

@bnoordhuis
Copy link
Member

nodejs/node#10283 was merged, nodejs/node#12217 wasn't and I more or less abandoned it because Windows.

@richardlau Can you update this to add only openssl and zlib?

@richardlau
Copy link
Member Author

Updated to add openssl, v8 and zlib to match the current install.py.

Perhaps we should also be executing node and checking the node_shared_* variables (we don't currently for uv).

Header files for deps are in a different location in the Node.js
source tree compared to the release tarballs.
@richardlau
Copy link
Member Author

Update: adding v8 was unnecessary as it was already taken care of by <(node_engine_include_dir).

bnoordhuis pushed a commit that referenced this pull request Jun 8, 2018
Header files for deps are in a different location in the Node.js
source tree compared to the release tarballs.

PR-URL: #1055
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
@bnoordhuis
Copy link
Member

Thanks Richard, landed in fbecb38.

@mildsunrise
Copy link
Member

Hmm, after this change (which landed in Node.JS 10.8.0) shouldn't this wiki page be updated?

@richardlau
Copy link
Member Author

Hmm, after this change (which landed in Node.JS 10.8.0) shouldn't this wiki page be updated?

That wiki page was outdated long before the change in this PR and refers to Node.js versions v0.6.x and v0.8.x which have long been unsupported. I'd be inclined to just remove that wiki page completely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants