psycopg2-binary: Why?

[jleclanche]

Forgive the blunt question but I cannot find discussion on why the split of psycopg2 and psycopg2-binary is happening. The current (2.7) way of doing wheel+source distribution works exactly how it's supposed to. Splitting into a separate -binary distribution actually breaks any library that depends on psycopg2=>2.x, which includes for example sqlalchemy's postgres bindings, because setuptools will complain the dependency is not matched.

[dvarrazzo]

The discussion is here

The explanation is here

If you have another solution let us know in the mailing list. The solution should involve not segfaulting.

[jleclanche]

Thanks for the context! This is super helpful.
I actually hit the issue specified there (uwsgi+psycopg2 libssl issues on debian). It's tough because I get that this stuff should work out of the box, but at the same time I think the current solution will only work out of the box as far as people won't be using psycopg2-binary.

If you are affected by that particular issue, compiling from source is one of the more appropriate fixes and pip has a command line flag for it (which psycopg2 itself documented in various places including this issue tracker).

I understand the desire to fix the issue (as I said, I'm affected by it), but I don't think this really fixes anything:

• Users who stay on psycopg2 now have a runtimewarning they may or may not understand
• pip install psycopg2 time and requirements changed with no clear explanation why without knowing the background or having seen aforementioned warning
• Users who switch to psycopg2-binary now have issues with various package requires such as the one I mentioned in this issue (sqlalchemy is a big user of psycopg2, this is probably going to surface a lot more once 2.8 is released)
• And for the users who switch to psycopg2-binary, the original issue with libssl still exists. It's just been moved.

I think the "solution" really is the --no-binary pip flag. Unless I completely misunderstand something, that has the same effect as moving the wheels to a different package, except that for this package the wheels will not have added issues.

[Cykooz]

I fully agree with @jleclanche. Creating of psycopg2-binary is not a solution for end users. On the contrary, it is a new problem for users whom uses many third-party libraries depended from psycopg2. Now one half of this libraries depends from psycopg2 (e.g. sqlalchemy), and other half - from psycopg2-binary (e.g. pgcli). Which version of psycopg2 package will be installed and used in that situation?

[Cykooz]

psycopg2-binary has not resolved problem with libssl.

Imagine, what if near future all python libraries will be depends from psycopg2-binary instead of psycopg2. What will this situation be different from when there was only one version of psycopg2?

[dvarrazzo]

@Cykooz use psycopg2 and forget the wheel for your use. Let the other decide for theirs. If they are writing multithread programs using ssh they better use psycopg2. Otherwise they are fine with -wheels. The libssl one is not a solvable problem. If it is a problem for you forget the wheels.

[Cykooz]

But not all depends from me. I use different third-party packages which already has psycopg2 or psycopg2-binary in self dependencies. And I can't control an order of installation of this packages. Sometimes psycopg2 will be installed first, other time - psycopg2-binary will.

By default pip don't use package name from METADATA to check that it already installed. It compare only package names taken from names of wheel files. This means that I can install both packages at the same time.

If I install psycopg2-binary first, and then install psycopg2:

$ pip install psycopg2-binary
$ pip install psycopg2

then if I uninstall psycopg2-binary:

$ pip uninstall psycopg2-binary

this will completely remove the code of psycopg2 from my python:

$ python
>>> import psycopg2
...
ImportError: No module named psycopg2

But pip will still consider that the psycopg2 is installed.

$ pip install psycopg2
Requirement already satisfied: psycopg2 in ./dist-packages

All of this is not very well.

[Cykooz]

Third-party packages should not determine which version of the psycopg2 distributive should be used. Because they do not know in what environment they will be used. Version of distributive of psycopg2 should be determined solely by external tools. For example, with help of the --no-binary option for pip.

[dvarrazzo]

@Cykooz What third party package depend on psycopg2-binary?

[Cykooz]

@dvarrazzo pgcli depends from psycopg2-binary.

[dvarrazzo]

@Cykooz pgcli is an interactive tool: it doesn't strike me as something that has to live 100% in the virtualenv of e.g. a Django webapp using psycopg. But those kind of tools are exactly the reason for which I don't want to get rid of the wheels altogether. It's probably worth a chat with them to understand what's the best strategy.

[Cykooz]

Oh, maybe you are in something right.

I'm just afraid that not only standalone applications will include psycopg2-binary as a dependency, but also libraries.

[dvarrazzo]

@Cykooz I appreciate your concern. And it highlights the fact that if some library started depending on that package it may cause interdependencies problem: it's something to highlight very clearly in the docs. We are a bit into uncharted waters here: we are trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it.

[goodtune]

This is a problematic solution for the reasons already mentioned - psycopg2-binary does not satisfy a psycopg2 dependency, and there are more than a few of these.

Pip already had a way to avoid using the wheel distribution - that should be the mechanism to go for a source installation, not a new package which clobbers the namespace of another.

[mmerickel]

Until PyPA adds the new "Provides" metadata there is just no good solution here that involves splitting the packages. PEP 426 actually touched on this but that PEP was since withdrawn and work on it has stalled. This means we just do not have a way to have two distributions claim the same virtual distribution name.

The options in the context of the current packaging tooling are simply

1. stop providing wheels at all
2. provide wheels and require users to use --no-binary if they have problems.

There are several packages including the sqlalchemy[postgresql] that expect psycopg2 and are incompatible with psycopg2-binary. Right now it's simply unresolvable if any upstream dependency of your project chooses either one... they cannot both be installed. The psycopg2 maintainers seem to think that users have control over which one they install if two upstream packages differ here, but it's actually not the case with the way pip resolves things right now. The only viable solution which puts control in the users hands is to make everything just depend on psycopg2 and have users specify --no-binary if they are having problems with the wheel.

[poswald]

Can the psycopg2[nobinary] format be used to trigger a build with --no-binary or perhaps the opposite with psycopg2[binary] pulling in the bins instead of building? No idea if that's possible but it would satisfy the same dependency for the 2 versions.

[ztane]

However the --no-binary is hardly a solution either, because with long requirements lists many other packages are fine to be installed from wheels except psycopg2. Having seen the SSL error now on Ubuntu 18.04 it seems that it is going to be here to stay.

[rouge8]

However the --no-binary is hardly a solution either, because with long requirements lists many other packages are fine to be installed from wheels except psycopg2

You can specify which packages you don't want to install from wheels, e.g. --no-binary=psycopg2.

For example:

$ PIP_NO_CACHE_DIR=off pip install --no-binary=click click flask
Collecting click
  Downloading https://files.pythonhosted.org/packages/95/d9/c3336b6b5711c3ab9d1d3a80f1a3e2afeb9d8c02a7166462f6cc96570897/click-6.7.tar.gz (279kB)
    100% |████████████████████████████████| 286kB 6.6MB/s
Collecting flask
  Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 13.5MB/s
Collecting itsdangerous>=0.24 (from flask)
  Downloading https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)
    100% |████████████████████████████████| 51kB 12.2MB/s
Collecting Jinja2>=2.10 (from flask)
  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
    100% |████████████████████████████████| 133kB 43.8MB/s
Collecting Werkzeug>=0.14 (from flask)
  Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
    100% |████████████████████████████████| 327kB 11.3MB/s
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask)
  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
Installing collected packages: click, itsdangerous, MarkupSafe, Jinja2, Werkzeug, flask
  Running setup.py install for click ... done
  Running setup.py install for itsdangerous ... done
  Running setup.py install for MarkupSafe ... done
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 flask-1.0.2 itsdangerous-0.24

[Vadorequest]

I don't understand anything about all that stuff neither really want to dive into it, I don't know what wheels are, I just know I had a working project using psycopg2, which now displays warning about being renamed, so I thought I had to migrate to psycopg2-binary, but if I remove psycopg2 then it fails connecting to my postgre DB. Kinda lost as people mentionned earlier...

Should I keep both? Should I not care and keep using psycopg2 and ignore that warning?

How do I automate pip install --no-binary :all: psycopg2 in my requirements.txt? Any way of just saying to install psycopg2 without binary and not think about it anymore?

---

Okay, found a solution using the following in my requirements.txt:
psycopg2==2.7.5 --no-binary psycopg2

[sarusso]

@dvarrazzo you mentioned that you are "trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it", but you never commented on the alternative "--no-binary" approach. What are your thoughts?

[frenzymadness]

A lot of projects are affected by this change and it wasn't clear to me what happened. Thanks to this issue I know what to do.
Our case is that we have a lot of testing projects in containers hosted on OpenShift platform where we don't want to install all dependencies required to build psycopg2 from source so we have to switch all of them to psycopg2-binary.

[boolow5]

On Ubuntu 20.04 this answer worked for me.
Answer summery:

sudo apt-get install gcc libpq-dev -y
sudo apt-get install python-dev  python-pip -y
sudo apt-get install python3-dev python3-pip python3-venv python3-wheel -y
pip3 install wheel

None of the above suggestions fixed my issue.