Testbed works in run, but hangs in test

[HalfWhitt]

Describe the bug

In the course of trying to isolate a different bug, I deleted my virtual environment and recreated one anew, installing Toga from a fresh git clone as well. After doing so, I've found that briefcase run --test works, but briefcase dev --test doesn't; it hangs indefinitely after collecting the tests. If I attempt to run any one specific test, I get the following:

[testbed] Running test suite in dev environment...
===========================================================================
Waiting for app to be ready for testing... ready.
============================= test session starts ==============================
collecting ... collected 1 item

tests/widgets/test_selection.py::test_selection_change ERROR             [100%]

==================================== ERRORS ====================================
___________________ ERROR at setup of test_selection_change ____________________
Traceback (most recent call last):
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_callers.py", line 155, in _multicall
    teardown[0].send(outcome)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 751, in pytest_fixture_setup
    warnings.warn(
DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in
/Users/charles/toga_clean/toga/testbed/tests/conftest.py:71
Replacing the event_loop fixture with a custom implementation is deprecated
and will lead to errors in the future.
If you want to request an asyncio event loop with a scope other than function
scope, use the "scope" argument to the asyncio mark when marking the tests.
If you want to return different types of event loops, use the event_loop_policy
fixture.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_callers.py", line 181, in _multicall
    return outcome.get_result()
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_result.py", line 99, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/runner.py", line 157, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/runner.py", line 497, in setup
    raise exc
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/runner.py", line 494, in setup
    col.setup()
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/python.py", line 1795, in setup
    self._request._fillfixtures()
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/fixtures.py", line 566, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/fixtures.py", line 585, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/fixtures.py", line 607, in _get_active_fixturedef
    self._compute_fixture_value(fixturedef)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/fixtures.py", line 693, in _compute_fixture_value
    fixturedef.execute(request=subrequest)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/_pytest/fixtures.py", line 1069, in execute
    result = ihook.pytest_fixture_setup(fixturedef=self, request=request)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_callers.py", line 159, in _multicall
    _warn_teardown_exception(hook_name, teardown[1], e)
  File "/Users/charles/.pyenv/versions/3.10.13/envs/toga_310/lib/python3.10/site-packages/pluggy/_callers.py", line 49, in _warn_teardown_exception
    warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: asyncio, Hook: pytest_fixture_setup
DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in
/Users/charles/toga_clean/toga/testbed/tests/conftest.py:71
Replacing the event_loop fixture with a custom implementation is deprecated
and will lead to errors in the future.
If you want to request an asyncio event loop with a scope other than function
scope, use the "scope" argument to the asyncio mark when marking the tests.
If you want to return different types of event loops, use the event_loop_policy
fixture.

For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
=========================== short test summary info ============================
ERROR tests/widgets/test_selection.py::test_selection_change - pluggy.PluggyT...
=============================== 1 error in 0.02s ===============================

If it's only a deprecation warning that will eventually cause errors, I'm unsure why it's throwing an exception now — or for that matter, why it wasn't in my previous development environment, but is now. I've tried downgrading pytest-asyncio a few versions, but didn't see a change. I've also tried fresh virtual envs of Python 3.8, 3.9, 3.10, 3.11, and 3.12; all get the same result. Have I wandered into the Twilight Zone?

Steps to reproduce

1. Create a fresh development environment (virtual env, git clone, briefcase)
2. From the testbed folder, run briefcase dev --test
3. Testbed app hangs indefinitely

Or, alternatively:

2. From the testbed folder, run briefcase dev --test -- tests/widgets/test_selection.py::test_selection_change (any test, that's just the one I used)
3. Testbed crashes.

Expected behavior

I would expect the testbed to run in dev just as well as it does in run.

Environment

• Operating System: macOS 14.2.1
• Python version: 3.8.18, 3.9.18, 3.10.13, 3.11.6, 3.12.0
• Software versions:
  • Briefcase: 0.3.17
  • Toga: from source, up-to-date with main

[freakboy3742]

I'll bet if you run briefcase dev -r --test the tests will pass.

If I'm right, the problem is that the version of pytest-asyncio that is defined by toga-core isn't the same as the version used by testbed. If you've got a fresh toga venv, you have likely pip installed toga-core, which has picked up the newer version. briefcase dev -r resets the version to the older one.

This is definitely an issue we need to resolve - the problem is that there is a backwards incompatible change (which is hinted at in the warning message) in how async fixtures are collected. The fix might be trivial - we just haven't had a chance to dig into it yet.

I'll leave this ticket open as a marker for that problem.

[HalfWhitt]

Aha, you're right! Thanks, I was starting to wonder if I was going crazy.

[freakboy3742]

I cannot speak to your sanity, only to the issues with Toga's test suite 😝 ...