Provide async StreamWriter coroutine so memory does not go unbounded when write is blocked. #116279
Closed as not planned
Description
Feature or enhancement
Proposal:
For example, this snippet will cause memory consumption to fill up over time.
Writer.write does not block, and the data will be kept in buffer until TCP flow control accepts more.
However, it is common to have large file uploads and unpredictable client input in TCP clients.
Though users can implement the client using callbacks from flow control (see #81162), it is much easier to use and also better fits the design of asyncio to provide easy async write access to sockets.
import asyncio
import psutil
async def simulate_slow_network(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
while True:
b = await reader.read(8192)
if not len(b):
break
await asyncio.sleep(0.5) # note that memory will not bulk up if removing the line
async def main():
serv = await asyncio.start_server(simulate_slow_network, host='127.0.0.1', port=8192)
reader, writer = await asyncio.open_connection(host="127.0.0.1", port=8192)
writer: asyncio.StreamWriter
for i in range(1024 ** 2):
writer.write(b'0' * 8192)
await asyncio.sleep(0)
if i % 1024 == 0:
print(psutil.Process().memory_info().rss / 1024 ** 2)
serv.close()
if __name__ == '__main__':
asyncio.run(main())Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response