Increment Interpreter ID on each Py_Initialize / Py_Finalize cycle

[davidhewitt]

Feature or enhancement

Proposal:

At present the "main" interpreter has an ID of 0, and sub-interpreters will have an ID of 1 or greater.

Native extensions which are not subinterpreter safe can check if the interpreter ID has changed to avoid being used from multiple (sub)interpreters. For example, Cython does this and PyO3 is considering adopting the same strategy in PyO3/pyo3#3446

There is a known caveat in using Py_Initialize and Py_Finalize more than once that native extensions may not initialize correctly on subsequent import cycles (see docs for Py_FinalizeEx). This can lead to indeterminate behaviour.

If the main interpreter ID were not fixed at 0 but also incremented on each initalize / finalize cycle, then the check with Cython and PyO3 already use to defend against interpreter changes would also apply to initalize / finalize cycles, making them safer to use.

(I am aware the long-term solution is for all native extensions to support PEP 489 and store no static data, but this is a hard problem that will take a long time and all extension maintainers to audit their dependencies.)

Testing for the main interpreter can be done by PyInterpreterState_Get() == PyInterpreterState_Main(), so it is not strictly necessary for the main interpreter to have ID of 0, although this is a nice property which some may have been relying on.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response