Work with files asynchronously

[asvetlov]

There is a very old tech debt in aiohttp: the library uses blocking synchronous calls to work with files.

It should be fixed by replacement such direct calls by loop.run_in_executor().

Affected parts:

• web_fileresponse.py
• payload.py

Splitting the task into two pull requests makes sense in term of easier review.

The issue is pretty easy and straightforward, it waits for a champion.

[aio-libs-bot]

GitMate.io thinks possibly related issues are #764 (Too many open files), #20 (Refactor files handling.), #1067 (will there be huge file upload support ?), #72 (HTTP POST not working), and #44 (Please document UnixConnection work).

[socketpair]

Do not forget to open() files also in a thread.

I think, internal API should be the following:

import aiofiles

file = await aiofiles.open()
result = await file.read()
result = await file.write()
await file.chmod()
await file.fsync()
...

UPD: https://github.com/Tinche/aiofiles buth this solution is overingeneered. I would write my own with implementing each method explicitly.

[asvetlov]

I don't want to use aiofiles.
run_in_executor is just enough for the library needs.
aiohttp calls open() in file_response.py only, client API accepts already opened regular IO objects.
Wrapping them into aiofiles looks more complex than explicit thread pool usage.

[chirgjn]

👋 would love to pick this up, if not fixed already

[asvetlov]

Please go ahead!

[kxepal]

This could raise a holywar, but I'll risk to ask you: why not aiofiles?

[asvetlov]

Did you read messages above?
I can send a ball back: why aiofiles despite the fact that we need only run in executor a couple IO calls?

[kxepal]

Yes, I did, but some abstraction may be useful to not care about that bit if it will eventually have changed. If that's the only reason than it satisfied my curiousity (:

[asvetlov]

Changes will not touch the public API.
Choosing an abstraction layer for async files support is the subject for future discussion, let's do it in another issue if you want to.

[asvetlov]

Done

[samuelcolvin]

Woop woop. This is great. Thanks @Tolmachofof

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.