How to keep users safe regarding alpha Pythons?

[davidhewitt]

In e.g. pydantic/pydantic#7942 we have a downstream user who tried to compile and run pydantic-core using Python 3.13 alpha. Unsurprisingly it didn't work (though the failure wasn't clearly due to unsoundness).

Well there's a good argument to be made that if you're using an alpha Python that you should expect stuff to be broken, however I think we could do a better job at preventing users blowing their own feet off.

Some ideas:

• Hard-code a max version (currently 3.12), and refuse to build an unsupported version.
• Same as above, but give an environment variable to opt-in to building on an unsupported version. (I sort of imagine users want it, for this to be safe I think we would need to require that the abi3 feature is enabled.)

In fact, I wonder if we should already be running 3.13 alphas in CI, but just the abi3 feature for now? That might be an interesting initial signal to 3.13 compatibility, but without us having to deal with churn of version-specific C API until later down the road.

[davidhewitt]

#3555 - user built an extremely old PyO3 version against PyPy 3.10.

This kind of stuff is definitely not good, so I will aim to push a PR soon to add a Python version ceiling.

[davidhewitt]

From @adamreichold in #3674 (comment)

I thought about this repeatedly and I am really uneasy about a version ceiling as this could really complicate maintenance of old software systems which should be completely usable, i.e. think of a researcher picking up a simulation model after it was left untouched for a few years.

I'm sympathetic to this perspective too, but the problem IMO is that we hardcode version-specific FFI structure layouts into pyo3-ffi, and we see each year that various pieces need updating for new Python versions. I think without this then the risk is that users unknowingly run into UB by compiling an obsolete PyO3 version with a modern Python and access invalid structure layouts.

That's sort of fine on alpha Pythons but I feel quite strongly that this is not ok in production.

I see two possible mitigations:

• Force abi3 on Pythons above the version ceiling (maybe as an opt-in), which may degrade performance but hopefully guarantee safety.
• Have an alternative build mode for pyo3-ffi which uses bindgen to produce a completely safe set of object layouts for unrecognised Python versions. But I think this will be a lot of work.

[adamreichold]

Force abi3 on Pythons above the version ceiling (maybe as an opt-in), which may degrade performance but hopefully guarantee safety.

This feels useful to me as a targeted measure to actually make the use case I am thinking work reliably instead of "fixing" it by disallowing it! This would ideally be accompanied by a warning in the build logs.