-
-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Polars checks not being evaluated correctly #1662
Comments
https://pandera.readthedocs.io/en/stable/polars.html#how-it-works I think this behaviour is expected. |
So are checks never assessed for I feel like the documentation should make this more explicit or a warning should be issued. The top example comes directly from Pandera documentation and having a check that is never assessed creates a false sense of coverage. |
Checks are assessed for |
This is expected behavior @mxblsdl.
I believe it already does, see https://pandera.readthedocs.io/en/stable/polars.html#how-it-works already linked by @kacper-sellforte.
This is also a good idea. I think a better logging experience here would be helpful. Would you mind opening up a separate issue for this request? The correct way to support this would be if polars has a first-class expression that asserts whether a column contains any False values, in which case pandera can catch the error lazily when the lazyframe is evaluated. I opened up an issue in the polars project: pola-rs/polars#16120 |
Also see https://pandera.readthedocs.io/en/stable/polars.html#data-level-validation-with-lazyframes. You can set the environment variable |
okay thank you for taking a look at this. I guess I was just confused on the limits of lazyframe evaluation. I will experiment with the env variable mentioned above and close the issue. |
Describe the bug
The column checks on polars LazyFrames are not registering errors when they should. Values outside of a defined range pass validation with no warnings or errors. This is not true for polars DataFrame which does register an error.
It looks like this was addressed in a recent PR but I am still seeing the bug in the 0.19.3 release.
Code Sample,
Expected behavior
I would expect a
pandera.errors.SchemaError
to be raised. Note that thepolars.DataFrame
version of this code does raise and error.Desktop (please complete the following information):
The text was updated successfully, but these errors were encountered: