Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"cannot open URL" with remotes::install_github() #130

Closed
wibeasley opened this issue Mar 30, 2018 · 18 comments
Closed

"cannot open URL" with remotes::install_github() #130

wibeasley opened this issue Mar 30, 2018 · 18 comments

Comments

@wibeasley
Copy link
Contributor

I have a vanilla installation of Windows (no proxies or anything similar) where remotes::install_github("yihui/testit") won't work (it fails similarly for many other packages that do and don't require compilation). I've tried the two most recent patched versions of R (3.4.3 and 3.4.4) and both the CRAN and GitHub master version of remotes (the CRAN version predates the most recent changes to install-github.R). It's been throwing this error for at least a week.

Furthermore, these same arrangements have been working fine on my other Windows machines. And working fine on this same machine with devtools::install_github("yihui/testit") . The error message is

> remotes::install_github("yihui/testit")
Downloading GitHub repo yihui/testit@master
Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
  cannot open URL 'https://api.github.com/repos/yihui/testit/zipball/master'

Stepping through after declaring debug(utils::download.file) leads to the following misbehaving line, and I don't have the skills to debug once it cross the R-C boundary.

.External(C_download, url, destfile, quiet, mode, cacheOK, method == "wininet")

I'm not seeing anything similar amoung the 18 SO hits. Is this something I should ask SO about instead? I'm not that worried for my own case, because I can keep using devtools::install_github(), but as more people migrate to remotes, this corner case might cause more problems.

traceback
> traceback()
11: utils::download.file(url, path, method = download_method(), quiet = quiet, 
        mode = "wb")
10: withCallingHandlers(expr, warning = function(w) invokeRestart("muffleWarning"))
9: suppressWarnings(status <- utils::download.file(url, path, method = download_method(), 
       quiet = quiet, mode = "wb"))
8: base_download(real_url, path, quiet)
7: download(dest, src, auth_token = x$auth_token)
6: remote_download.github_remote(remote, quiet = quiet)
5: remote_download(remote, quiet = quiet)
4: FUN(X[[i]], ...)
3: vapply(remotes, install_remote, ..., FUN.VALUE = logical(1))
2: install_remotes(remotes, ...)
1: remotes::install_github("yihui/testit")
session info
R version 3.4.4 Patched (2018-03-19 r74446)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.4.4 tools_3.4.4    yaml_2.1.18    remotes_1.1.1 
@gaborcsardi
Copy link
Member

This is coming from utils::download.file so either that does not support HTTPS, or you need a GH API token because your IP address was rate limited, or some other error, like a bad proxy setup.

Unfortunately there is not much we can do about it in remotes, other than maybe trying to a different download method, or try multiple methods.

But for that we would need to be able to recreate your problem, so some questions:

  • does it always fail, or does it sometimes work? You can see your GH rate limit, and when it resets with sg like gh::gh("/rate_limit")
  • does it work if you call utils::download.file() directly, with the same URL?
  • does it work for the various utils::download.file() methods?

@wibeasley
Copy link
Contributor Author

Your intuition was right. It appears to be influenced by the 'method' parameter in utils::download.file().

  1. It always failed -it seems unrelated to rate limits
  2. It failed with the default method (see the first entry under "'method' values that fail")
  3. The method="libcurl" option worked. All others failed.

What would be helpful for me to do next?

source_url <- "https://api.github.com/repos/yihui/testit/zipball/master"
capabilities("libcurl")
# libcurl 
#    TRUE 

utils::download.file(source_url, destfile="aa.zip", method="libcurl")
# trying URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# Content length 16294 bytes (15 KB)
# downloaded 15 KB
'method' values that fail
utils::download.file(source_url, destfile="aa.zip")
# trying URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# cannot open URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# In addition: Warning message:
# In utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  :
# InternetOpenUrl failed: 'A connection with the server could not be established'

utils::download.file(source_url, destfile="aa.zip", method="wininet")
# trying URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  :
# cannot open URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# In addition: Warning message:
# In utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  :
# InternetOpenUrl failed: 'A connection with the server could not be established'

utils::download.file(source_url, destfile="aa.zip", method="internal")
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# scheme not supported in URL 'https://api.github.com/repos/yihui/testit/zipball/master'


utils::download.file(source_url, destfile="aa.zip", method="wget")
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# 'wget' call had nonzero exit status
# In addition: Warning message:
# running command 'wget  "https://api.github.com/repos/yihui/testit/zipball/master" -O "aa.zip"' had status 127 

utils::download.file(source_url, destfile="aa.zip", method="curl")
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# 'curl' call had nonzero exit status
# In addition: Warning message:
# running command 'curl  "https://api.github.com/repos/yihui/testit/zipball/master"  -o "aa.zip"' had status 127 

utils::download.file(source_url, destfile="aa.zip", method="auto")
# trying URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# cannot open URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# In addition: Warning message:
# In utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  :
# InternetOpenUrl failed: 'A connection with the server could not be established'

@cderv
Copy link
Contributor

cderv commented Apr 4, 2018

In remotes::install_github, the download method is determined by remotes::download_method() and set to "auto" except on older R version. On windows, it is not libcurl by defaut but wininet when auto is set.

FWIW, I can't reproduce the issue on my windows computer, but I am on windows 10.

Could it come from a different behaviour of wininet between different version of windows?
I will try to verify that.

One solution could be to check for libcurl availability by default but it was the previous behaviour that did not allowed windows proxy behavior offered by wininet.

Reprex

with defaults
tmp_zip <- tempfile(fileext = ".zip")
source_url <- "https://api.github.com/repos/yihui/testit/zipball/master"
utils::download.file(source_url, destfile=tmp_zip)
file.exists(tmp_zip)
#> [1] TRUE
with wininet explicitly
tmp_zip <- tempfile(fileext = ".zip")
source_url <- "https://api.github.com/repos/yihui/testit/zipball/master"
utils::download.file(source_url, destfile=tmp_zip, method = "wininet")
file.exists(tmp_zip)
#> [1] TRUE
with auto from remotes:::download_method()
tmp_zip <- tempfile(fileext = ".zip")
source_url <- "https://api.github.com/repos/yihui/testit/zipball/master"
utils::download.file(source_url, destfile=tmp_zip, method = remotes:::download_method())
file.exists(tmp_zip)
#> [1] TRUE

Created on 2018-04-04 by the reprex package (v0.2.0).

Session info
devtools::session_info()
#> - Session info ----------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.4.3 (2017-11-30)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  French_France.1252          
#>  tz       Europe/Paris                
#>  date     2018-04-04                  
#> 
#> - Packages --------------------------------------------------------------
#>  package     * version     date      
#>  ansistrings   1.0.0.9000  2018-03-31
#>  assertthat    0.2.0       2017-04-11
#>  backports     1.1.2       2017-12-13
#>  callr         2.0.2.9002  2018-03-31
#>  cli           1.0.0.9002  2018-03-31
#>  clisymbols    1.2.0       2018-02-04
#>  crayon        1.3.4       2017-09-16
#>  debugme       1.1.0       2017-10-22
#>  desc          1.1.1       2017-08-03
#>  devtools      1.13.5.9000 2018-03-31
#>  digest        0.6.15      2018-01-28
#>  evaluate      0.10.1      2017-06-24
#>  glue          1.2.0       2017-10-29
#>  hms           0.4.2       2018-03-10
#>  htmltools     0.3.6       2017-04-28
#>  knitr         1.20        2018-02-20
#>  magrittr      1.5         2014-11-22
#>  memoise       1.1.0       2017-04-21
#>  pkgbuild      1.0.0       2018-03-31
#>  pkgconfig     2.0.1       2017-03-21
#>  pkgload       1.0.0       2018-03-31
#>  prettyunits   1.0.2       2015-07-13
#>  progress      1.1.2.9002  2018-03-31
#>  R6            2.2.2       2017-06-17
#>  Rcpp          0.12.16     2018-03-13
#>  rlang         0.2.0.9001  2018-03-28
#>  rmarkdown     1.9         2018-03-01
#>  rprojroot     1.3-2       2018-01-03
#>  selectr       0.4-0       2018-03-27
#>  sessioninfo   1.0.1.9000  2018-01-11
#>  stringi       1.1.7       2018-03-12
#>  stringr       1.3.0       2018-02-19
#>  testthat      2.0.0.9000  2018-03-31
#>  usethis       1.3.0       2018-02-24
#>  withr         2.1.2       2018-03-28
#>  xml2          1.2.0       2018-01-24
#>  yaml          2.1.18      2018-03-08
#>  source                                 
#>  Github (r-lib/ansistrings@4e4d309)     
#>  CRAN (R 3.4.0)                         
#>  CRAN (R 3.4.3)                         
#>  Github (r-lib/callr@9c38827)           
#>  Github (r-lib/cli@53c129a)             
#>  Github (gaborcsardi/clisymbols@e49b4f5)
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.1)                         
#>  Github (r-lib/devtools@2f7bc84)        
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.3)                         
#>  Github (r-lib/pkgbuild@015f7f6)        
#>  CRAN (R 3.4.0)                         
#>  Github (r-lib/pkgload@f827a27)         
#>  CRAN (R 3.4.0)                         
#>  Github (r-lib/progress@72c9873)        
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.4)                         
#>  Github (tidyverse/rlang@49d7a34)       
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.4)                         
#>  Github (r-lib/sessioninfo@c871d01)     
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.3)                         
#>  Github (hadley/testthat@5ed0bb1)       
#>  CRAN (R 3.4.4)                         
#>  Github (r-lib/withr@79d7b0d)           
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.3)

@gaborcsardi
Copy link
Member

@wibeasley I suspect that this is a TLS issue, GitHub does not support TLS v1 and v1.1

See e.g. https://githubengineering.com/crypto-deprecation-notice/

Older Windows systems only support TLS v1.x, so the system's builtin HTTP lib does not work with GitHub. It is possible that you can enable TLS v1.2 or you can download a patch from MS. E.g. https://cloudblogs.microsoft.com/microsoftsecure/2017/07/20/tls-1-2-support-added-to-windows-server-2008/

It depends on your windows version.

To make this more robust, remotes could try a couple of methods, at least on windows trying libcurl and wininet would make sense.

@wibeasley
Copy link
Contributor Author

It is possible that you can enable TLS v1.2 or you can download a patch...

I don't mind doing that or either of the two other solutions (eg, specifying "libcurl", or calling devtools::install_github()). But these may not be feasible to expect other remotes::install_github() users to do without a lot of problems encountered.

If you think I have a weird computer failing on a trivial issue, I'm happy to drop the issue. Or, if you think I have a canary in a coalmine, I'm happy to provide more info to help make
remotes::install_github() more robust. I'm cool with whatever you feel is the best use of your time.

In case this helps create a more dynamic/robust, here's info on the three machines. Notice the failing one isn't the oldest OS.

I can't find a difference in these 'predictor' variables. Again, I don't care about this particular machine --but more about its value improving the robustness of remotes::install_github() (eg, finding an environmental setting/variable value that determine the best download method).

Windows machine 1 (which fails):

  1. download outcome: utils::download.file(source_url, destfile="aa.zip") see error message above
  2. Windows 8
  3. R.version 3.4.4 Patched
  4. getOption("download.file.method"): "wininet"
  5. capabilities(c("libcurl", "http/ftp")): TRUE TRUE
  6. utils::download.file(source_url, destfile="aa.zip", method="libcurl"): works
  7. devtools::install_github("yihui/testit") : works
  8. request <- httr::GET(source_url); writeBin(httr::content(request, "raw"), con="aa.zip"): works (ie, my interpretation of devtool's internals)

Windows machine 2 (which works):

  1. download outcome: utils::download.file(source_url, destfile="aa.zip") works
  2. Windows 7
  3. R.version 3.4.4 Patched
  4. getOption("download.file.method"): "wininet"
  5. capabilities(c("libcurl", "http/ftp")): TRUE TRUE
  6. utils::download.file(source_url, destfile="aa.zip", method="libcurl"): works
  7. devtools::install_github("yihui/testit") : works
  8. request <- httr::GET(source_url); writeBin(httr::content(request, "raw"), con="aa.zip"): works

Windows machine 3 (which works):

  1. download outcome: utils::download.file(source_url, destfile="aa.zip") works
  2. Windows 10
  3. R.version 3.4.4 Patched
  4. getOption("download.file.method"): "wininet"
  5. capabilities(c("libcurl", "http/ftp")): TRUE TRUE
  6. utils::download.file(source_url, destfile="aa.zip", method="libcurl"): works
  7. devtools::install_github("yihui/testit") : works
  8. request <- httr::GET(source_url); writeBin(httr::content(request, "raw"), con="aa.zip"): works

Follow question:

Ideas why devtools::install_github() doesn't have this problem? Is it because httr::GET() is used instead? (Notice I included that as the 8th characteristic in the list above.)

I see a lot of Google hits for ""InternetOpenUrl failed: 'A connection with the server could not be established'". Is this why you guys avoided utils::download.file() in devtools::install_github()? I understand the appeal of avoiding additional package dependencies (eg httr) and relying on required packages (like utils).

@gaborcsardi
Copy link
Member

Ideas why devtools::install_github() doesn't have this problem? Is it because httr::GET() is used instead?

httr uses curl, which uses its own libcurl.

I am pretty sure that this is the GH TLS issue. If you still have that problematic machine, can you please try to install TLS 1.2 support? Note that the next release of devtools will use remotes, so the devtools workaround will soon stop working.

@wibeasley
Copy link
Contributor Author

This guide recommended enabling TLS 1.2 in IE 10. That worked. But it's not very satisfying because when I then unchecked the box, it still worked. I'm guessing that checking the box the first time set something in the registry that helped utils::download.file()?

image

I didn't see anything to install in that link above that applies to Windows 8. I think part of the complexity was that the machine can't be upgraded to Windows 8.1.

This isn't my main computer, so again, I don't really care for my sake. But if there's something else I can run or do that might help you, please tell me. otherwise, I think this issue can be closed.

@gaborcsardi
Copy link
Member

Thanks! I don't think we can do much more here.

@ShaowenJ
Copy link

Hi guys, if I can use utils::download.file(,method="libcurl") to download the file like wibeasley did, how can I install it?

@Baneeishaque
Copy link

This guide recommended enabling TLS 1.2 in IE 10. That worked. But it's not very satisfying because when I then unchecked the box, it still worked. I'm guessing that checking the box the first time set something in the registry that helped utils::download.file()?

image

I didn't see anything to install in that link above that applies to Windows 8. I think part of the complexity was that the machine can't be upgraded to Windows 8.1.

This isn't my main computer, so again, I don't really care for my sake. But if there's something else I can run or do that might help you, please tell me. otherwise, I think this issue can be closed.

Worked...

@alinakamanina
Copy link

This guide recommended enabling TLS 1.2 in IE 10. That worked. But it's not very satisfying because when I then unchecked the box, it still worked. I'm guessing that checking the box the first time set something in the registry that helped utils::download.file()?

image

I didn't see anything to install in that link above that applies to Windows 8. I think part of the complexity was that the machine can't be upgraded to Windows 8.1.

This isn't my main computer, so again, I don't really care for my sake. But if there's something else I can run or do that might help you, please tell me. otherwise, I think this issue can be closed.

!!It's helped me a lot, thank you! xx !!

@etejin
Copy link

etejin commented May 19, 2020

cool ~ thank you very much

@wangbinkarl
Copy link

Very cool ~ thanks

@fanyang1997
Copy link

just run: options(download.file.method = "libcurl")
it works for me
ref: forestgeo/learn#102

@laresbernardo
Copy link

Hi! I think I'm having this same issue on Github Actions for my lares package, testing on ubuntu-16.04. When the action gets to the "Install system dependencies" section, this is the error I get:

Run Rscript -e "remotes::install_github('r-hub/sysreqs')"
Using bundled GitHub PAT. Please add your own PAT to the env var `GITHUB_PAT`
Skipping install of 'sysreqs' from a github remote, the SHA1 (f068afa9) has not changed since last install.
  Use `force = TRUE` to force installation
Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
  cannot open URL 'https://sysreqs.r-hub.io/pkg/lares,R,dplyr,ggplot2,h2o,httr,...
Calls: cat ... get_cran_sysreqs -> unlist -> download_json -> download -> <Anonymous>
In addition: Warning message:
In utils::download.file(url, path, method = download_method(), quiet = quiet,  :
  cannot open URL 'https://sysreqs.r-hub.io/pkg/lares,R,dplyr,ggplot2,h2o,httr,...
...
Execution halted
Error: Process completed with exit code 1.

@wibeasley
Copy link
Contributor Author

Hi @laresbernardo, consider starting a new issue. The problem I encountered was closely related to how Windows supports/interacts with TLS. If you start a new issue, it won't have all the Windows-specific distractions.

@laresbernardo
Copy link

Sure @wibeasley
Actually just tried "re-run jobs" and it was magically fixed and ran OK. Don't know how could anyone replicate this issue because I didn't even change any code at all. Thanks though!

wibeasley added a commit to wibeasley/lares that referenced this issue Jun 29, 2021
I like this package.  I was poking around from r-lib/remotes#130 (comment)
@starryious
Copy link

just run: options(download.file.method = "libcurl") it works for me ref: forestgeo/learn#102

work for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests