Skip to content

subprocess Popen wait() function hangs when stdout is > 20480Β #45597

Closed

Description

@superwesman
BPO 1256
Nosy @tiran
Superseder
  • bpo-1606: Doc: subprocess wait() may lead to dead lock
  • Files
  • brent.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2008-01-06.14:12:46.349>
    created_at = <Date 2007-10-10.17:41:49.517>
    labels = ['type-bug']
    title = 'subprocess Popen wait() function hangs when stdout is > 20480'
    updated_at = <Date 2008-01-06.14:12:46.347>
    user = 'https://bugs.python.org/superwesman'

    bugs.python.org fields:

    activity = <Date 2008-01-06.14:12:46.347>
    actor = 'christian.heimes'
    assignee = 'none'
    closed = True
    closed_date = <Date 2008-01-06.14:12:46.349>
    closer = 'christian.heimes'
    components = ['None']
    creation = <Date 2007-10-10.17:41:49.517>
    creator = 'superwesman'
    dependencies = []
    files = ['8505']
    hgrepos = []
    issue_num = 1256
    keywords = []
    message_count = 4.0
    messages = ['56317', '56682', '57794', '59371']
    nosy_count = 4.0
    nosy_names = ['superwesman', 'draghuram', 'christian.heimes', 'Vano']
    pr_nums = []
    priority = 'normal'
    resolution = 'wont fix'
    stage = None
    status = 'closed'
    superseder = '1606'
    type = 'behavior'
    url = 'https://bugs.python.org/issue1256'
    versions = ['Python 2.4']

    Activity

    1. superwesman commented on Oct 10, 2007

      superwesmanmannequin
      MannequinAuthor

      hi - I noticed some odd behaviour with subprocess when the output
      (stdout) is large - I did some sleuthing and found a reproduceable case

      • I tested this on solaris using python 2.4.2 (sparc) and using 2.5
        (opteron) - I've attached a python script that details the problem - I
        created 2 files ("small_file" and "large_file") that are 20480 and 20481
        bytes, respectively ....

      ls -lrt ./tmp/*_file
      -rw-r--r-- 1 wtorres staff 20481 Oct 10 12:32 ./tmp/large_file
      -rw-r--r-- 1 wtorres staff 20480 Oct 10 12:32 ./tmp/small_file

      the script cats each file, thus generating an appropriately sized stdout

      • the workaround we came up with was to use our own file handle and use
        the parameter stdout=my_file_handle.fileno() instead of stdout=PIPE -
        you can see this in the attached example as well
      > python ./tmp/brent.py 
      testing with a small file - these should both work
      using my own file handle .... ok
      using PIPE ... ok
      testing with a large file - the second one here should fail
      using my own file handle .... ok
      using PIPE ...
      Traceback (most recent call last):
        File "./tmp/brent.py", line 43, in ?
          test_popen_bug( large_file )
        File "./tmp/brent.py", line 31, in test_popen_bug
          rc = my_popen.wait()
        File "/tools/python/2.4.2/solaris/lib/python2.4/subprocess.py", line
      1007, in wait
          pid, sts = os.waitpid(self.pid, 0)
      KeyboardInterrupt

      based on all this, it seems that the PIPE file handle has some buffer or
      something that is filling up and not allowing us to continue... or
      something. Thoughts?
      -w

    2. draghuram commented on Oct 23, 2007

      draghurammannequin
      Mannequin

      PIPE on POSIX are implemented using pipe system call so it will have all
      the limitations of the system buffering that is used for implementing
      pipes. The buffer size is system dependent. For example, on my Linux, I
      hit the hang condition at 64K and not 20480 bytes (as is your case).
      What basically happens is that the child fills up the pipe buffer and is
      blocked until space is available. Since the parent in your test script
      is not reading the bytes, space will never be freed up.

      I suggest that the bug is closed as invalid.

    3. Vano commented on Nov 23, 2007

      Vanomannequin
      Mannequin

      I suggest adding this as a note to the subprocess module documentation instead. The "17.1.2 Popen Objects" section seems to be the right place.

    4. tiran commented on Jan 6, 2008

      @tiran
      Member

      The docs should be updated, see bpo-1606

    5. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      type-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions