Incorrectly raised SchemaError when validating multiindex DataFrame

[ErikLundin98]

Given this minimal example

import pandas as pd
import pandera as pa

class MultiIndexTestSchema(pa.SchemaModel):
    boolean_index_one: pa.typing.Index[bool] = pa.Field(coerce=True)
    boolean_index_two: pa.typing.Index[bool] = pa.Field(coerce=True)
    value: pa.typing.Series[int] = pa.Field()

df = pd.DataFrame({
    "boolean_index_one": [True, False, True, True, False], 
    "boolean_index_two": [True, True, True, True, True],
    "value": [1, 2, 3, 4, 5],
})
df = df.set_index(keys=["boolean_index_one", "boolean_index_two"])
MultiIndexTestSchema.validate(df)

Expected behaviour is that the validation should pass, since the two index columns contain boolean fields.

However, instead I get the following error:

raise schema_error from original_exc
pandera.errors.SchemaError: expected series 'boolean_index_one' to have type bool, got object

This issue seems to only occur with MultiIndex DataFrames and with boolean fields. Changing type from bool to int magically resolves the issue.

I would like to know if anyone knows a workaround for this, if I am misinterpreting anything about defining the schemas?

Thanks in advance!

I am using pandera version 0.14.4

[cosmicBboy]

hi @ErikLundin98 what version of pandas and python are you using?

[ErikLundin98]

@cosmicBboy, I'm using python 3.10.10 and pandas 1.3.5

[cosmicBboy]

So unfortunately pandas 1.3.5 has a bunch of issues with index data types... see this StringDtype xfail test as an example:

pandera/tests/core/test_schema_components.py

Lines 838 to 856 in fe83c19

	def test_index_validation_pandas_string_dtype():
	"""Test that pandas string type is correctly validated."""
	if pandas_version().release <= (1, 3, 5):
	pytest.xfail(
	"pd.StringDtype is not supported in the pd.Index in pandas<=1.3.5"
	)
	schema = DataFrameSchema(
	columns={"data": Column(int)},
	index=Index(pd.StringDtype(), name="uid"),
	)
	df = pd.DataFrame(
	{"data": range(2)},
	index=pd.Index(["one", "two"], dtype=pd.StringDtype(), name="uid"),
	)
	assert isinstance(schema.validate(df), pd.DataFrame)

This is purely a pandas issue:

In [1]: import pandas as pd

In [2]: pd.Index([True, False])
Out[2]: Index([True, False], dtype='object')

In [3]: pd.Index([True, False], dtype=bool)
Out[3]: Index([True, False], dtype='object')  # it's still an "object"!

Any chance you can update your pandas version?

[ErikLundin98]

Thank you for clarifying that it's a pandas issue! I will see if I can update pandas.