Capture: copy output when using readouterr() instead of consuming it

[RedHellion]

Issue

Currently pytest's capfd/capsys fixtures (CaptureFixture) that allow stdout capturing via readouterr() consume stdout, meaning that logs are no longer available in the "Captured stdout call" up to the point in the test where readouterr() was called. The documentation indicates that this is expected behaviour ("readouterr() : Read and return the captured output so far, resetting the internal buffer."), however the "How to capture stdout/stderr output" tutorial seems to imply that the output is copied from the stdout/stderr streams instead of consumed and reset ("The readouterr() call snapshots the output so far - and capturing will be continued. After the test function finishes the original streams will be restored.")

Example

Code:

def method_under_test():
    print("Hallo, Welt!")
    return 41

def test_result(capsys):
    result = method_under_test()
    out, err = capsys.readouterr()
    assert out.startswith("Hello")
    assert result == 42

Expected:

================================== FAILURES ===================================
______________________________ test_result _______________________________

...

pytestest.py:9: AssertionError
---------------------------- Captured stdout call -----------------------------
Hallo, Welt!
========================== 1 failed in 0.03 seconds ===========================

Actual (no captured stdout because it was consumed by capsys.readouterr()):

================================== FAILURES ===================================
___________________________ test_result ____________________________

...

pytestest.py:14: AssertionError
========================== 1 failed in 0.03 seconds ===========================

Current workaround

Any tests requiring validation of stdout currently need to manually write-back the captured stdout and stderr immediately after capture in order to have them available for viewing under "Captured stdout call" in pytest results. It is not obvious that this is required from the documentation in order to both capture this output for test validation and have the output available for viewing to troubleshoot in the event of test failure.

Example

Code:

def method_under_test():
    print("Hallo, Welt!")
    return 41

def test_result(capsys):
    result = method_under_test()
    out, err = capsys.readouterr()
    sys.stdout.write(out)
    sys.stderr.write(err)
    assert out.startswith("Hello")
    assert result == 42

Result:

================================== FAILURES ===================================
______________________________ test_result _______________________________

...

pytestest.py:9: AssertionError
---------------------------- Captured stdout call -----------------------------
Hallo, Welt!
========================== 1 failed in 0.03 seconds ===========================

Proposed solution

Either

1. Minor change: have CaptureFixture.readouterr(...) call sys.stdout.write(out) and sys.stderr.write(err) itself after performing the consuming capture. Can use a new method arg capture_writeback or similar which defaults to False in order to preserve backwards-compatibility. Update "How to capture stdout/stderr output" tutorial documentation to make it more clear that by default readouterr destructively consumes the output such that it won't be available in the "Captured stdout call" test results.
2. Larger change: create a new CaptureFixture.copyouterr() method that does not destructively consume the stdout/stderror streams and instead copies them for use in test validation, such that the streams also remain intact for the "Captured stdout call" test results. Update "How to capture stdout/stderr output" tutorial documentation to make it more clear that readouterr destructively consumes the output such that it won't be available in the "Captured stdout call" test results, while copyouterr does the same thing non-destructively.

Additional context

Originally mentioned on Stackoverflow in 2014
https://stackoverflow.com/questions/26561822/pytest-capsys-checking-output-and-getting-it-reported

Logged as an Issue in this repo in 2018, but closed as "completed" for unknown reasons
#3448

[Zac-HD]

I believe that issue was closed as a duplicate of #5997; it predates GitHub adding distinguished issue-close reasons and we don't generally track that anyway.

[RedHellion]

Weird, because the issues are actually different (at least in terms of what the user sees) and not duplicates:

#3448 (similar to my issue here) was an issue about how caplog/capsys seems to "consume" logging/stdout/stderr, and then whatever you're capturing for validation purposes doesn't appear in the "Captured X call" section of the pytest output for failed tests.

#5997 is an issue about how some logging/stdout/stderr apparently shows up in "Captured X call" but can't be captured for validation via caplog/capsys.

Issues with the same system (log/stdout/stderr capturing for test validation), but different actual issues (capsys/caplog entirely removing captured calls from the pytest failed test output, vs calls that can be seen via the pytests failed test output not being able to be captured by capsys/caplog).

[The-Compiler]

I believe it wasn't that - IIRC, the author of that issue closed all their issues after some problematic interactions with the rest of the pytest team.

[nicoddemus]

Back to the original issue, I think the CaptureFixture.copyouterr() (or CaptureFixture.getouterr()?) is the way to go: an explicit new method is better than a feature-flag option IMHO. 👍

[ayjayt]

using capsys fixture also seems to not play nice with --capture=tee-sys, which i thought would allow both. This issue I understand addresses a slightly different issue, but does CaptureFixture also play nice with --capture=tee-sys?

[ayjayt]

I think I solved this issue #12854

if you use fixture capteesys with --coverage=sys, it's like capsys AND you still get the reports.

[ayjayt]

I've created a polyfill to add to conftest.py to use this feature and warn when it is finally released:

@pytest.fixture(scope="function")
def capteesys(request):
    from _pytest import capture
    import warnings
    if hasattr(capture, "capteesys"):
        warnings.warn(( "You are using a polyfill for capteesys, but this"
                        " version of pytest supports it natively- you may"
                        f" remove the polyfill from your {__file__}"),
                        DeprecationWarning)
        # Remove next two lines if you don't want to ever switch to native version
        yield request.getfixturevalue("capteesys")
        return
    capman = request.config.pluginmanager.getplugin("capturemanager")
    capture_fixture = capture.CaptureFixture(capture.SysCapture, request, _ispytest=True)
    def _inject_start():
        self = capture_fixture # closure seems easier than importing Type or Partial
        if self._capture is None:
            self._capture = capture.MultiCapture(
                in_ = None,
                out = self.captureclass(1, tee=True),
                err = self.captureclass(2, tee=True)
            )
            self._capture.start_capturing()
    capture_fixture._start = _inject_start
    capman.set_fixture(capture_fixture)
    capture_fixture._start()
    yield capture_fixture
    capture_fixture.close()
    capman.unset_fixture()

I will post again on this thread and @ anyone who 🎉s if the fixture name changes from capteesys!