You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
subprocess Popen wait() function hangs when stdout is > 20480Β #45597
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=<Date2008-01-06.14:12:46.349>created_at=<Date2007-10-10.17:41:49.517>labels= ['type-bug']
title='subprocess Popen wait() function hangs when stdout is > 20480'updated_at=<Date2008-01-06.14:12:46.347>user='https://bugs.python.org/superwesman'
hi - I noticed some odd behaviour with subprocess when the output
(stdout) is large - I did some sleuthing and found a reproduceable case
I tested this on solaris using python 2.4.2 (sparc) and using 2.5
(opteron) - I've attached a python script that details the problem - I
created 2 files ("small_file" and "large_file") that are 20480 and 20481
bytes, respectively ....
ls -lrt ./tmp/*_file
-rw-r--r-- 1 wtorres staff 20481 Oct 10 12:32 ./tmp/large_file
-rw-r--r-- 1 wtorres staff 20480 Oct 10 12:32 ./tmp/small_file
the script cats each file, thus generating an appropriately sized stdout
the workaround we came up with was to use our own file handle and use
the parameter stdout=my_file_handle.fileno() instead of stdout=PIPE -
you can see this in the attached example as well
> python ./tmp/brent.py
testing with a small file - these should both work
using my own file handle .... ok
using PIPE ... ok
testing with a large file - the second one here should fail
using my own file handle .... ok
using PIPE ...
Traceback (most recent call last):
File "./tmp/brent.py", line 43, in ?
test_popen_bug( large_file )
File "./tmp/brent.py", line 31, in test_popen_bug
rc = my_popen.wait()
File "/tools/python/2.4.2/solaris/lib/python2.4/subprocess.py", line
1007, in wait
pid, sts = os.waitpid(self.pid, 0)
KeyboardInterrupt
based on all this, it seems that the PIPE file handle has some buffer or
something that is filling up and not allowing us to continue... or
something. Thoughts?
-w
PIPE on POSIX are implemented using pipe system call so it will have all
the limitations of the system buffering that is used for implementing
pipes. The buffer size is system dependent. For example, on my Linux, I
hit the hang condition at 64K and not 20480 bytes (as is your case).
What basically happens is that the child fills up the pipe buffer and is
blocked until space is available. Since the parent in your test script
is not reading the bytes, space will never be freed up.
I suggest that the bug is closed as invalid.
Vano commented on Nov 23, 2007
Vanomannequin
Mannequin
More actions
I suggest adding this as a note to the subprocess module documentation instead. The "17.1.2 Popen Objects" section seems to be the right place.
Activity
superwesman commentedon Oct 10, 2007
hi - I noticed some odd behaviour with subprocess when the output
(stdout) is large - I did some sleuthing and found a reproduceable case
(opteron) - I've attached a python script that details the problem - I
created 2 files ("small_file" and "large_file") that are 20480 and 20481
bytes, respectively ....
the script cats each file, thus generating an appropriately sized stdout
the parameter stdout=my_file_handle.fileno() instead of stdout=PIPE -
you can see this in the attached example as well
based on all this, it seems that the PIPE file handle has some buffer or
something that is filling up and not allowing us to continue... or
something. Thoughts?
-w
draghuram commentedon Oct 23, 2007
PIPE on POSIX are implemented using pipe system call so it will have all
the limitations of the system buffering that is used for implementing
pipes. The buffer size is system dependent. For example, on my Linux, I
hit the hang condition at 64K and not 20480 bytes (as is your case).
What basically happens is that the child fills up the pipe buffer and is
blocked until space is available. Since the parent in your test script
is not reading the bytes, space will never be freed up.
I suggest that the bug is closed as invalid.
Vano commentedon Nov 23, 2007
I suggest adding this as a note to the subprocess module documentation instead. The "17.1.2 Popen Objects" section seems to be the right place.
tiran commentedon Jan 6, 2008
The docs should be updated, see bpo-1606