Skip to content

[EASY] subprocess: TypeError: can't concat str to bytes, in _execute_child()Β #75361

Closed

Description

@vstinner
BPO 31178
Nosy @gpshead, @vstinner, @ammaraskar
PRs
  • bpo-31178: Avoid concatenating bytes with str in subprocess errorΒ #3066
  • [3.6] bpo-31178: Avoid concatenating bytes with str in subprocess errorΒ #3388
  • bpo-31178: Mock os.waitpid() in test_subprocessΒ #3896
  • [3.6] bpo-31178: Mock os.waitpid() in test_subprocess (GH-3896)Β #3897
  • Files
  • subprocess_bug.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
    closed_at = <Date 2017-10-09.15:55:17.396>
    created_at = <Date 2017-08-10.15:29:43.999>
    labels = ['easy', 'library']
    title = "[EASY] subprocess: TypeError: can't concat str to bytes, in _execute_child()"
    updated_at = <Date 2017-10-09.15:55:17.396>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2017-10-09.15:55:17.396>
    actor = 'vstinner'
    assignee = 'gregory.p.smith'
    closed = True
    closed_date = <Date 2017-10-09.15:55:17.396>
    closer = 'vstinner'
    components = ['Library (Lib)']
    creation = <Date 2017-08-10.15:29:43.999>
    creator = 'vstinner'
    dependencies = []
    files = ['47077']
    hgrepos = []
    issue_num = 31178
    keywords = ['patch', 'easy']
    message_count = 9.0
    messages = ['300102', '301447', '301448', '301450', '301515', '303762', '303763', '303764', '303980']
    nosy_count = 3.0
    nosy_names = ['gregory.p.smith', 'vstinner', 'ammar2']
    pr_nums = ['3066', '3388', '3896', '3897']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue31178'
    versions = ['Python 3.6']

    Activity

    1. vstinner commented on Aug 10, 2017

      @vstinner
      MemberAuthor

      Lib/subprocess.py contains the following code:

                  try:
                      exception_name, hex_errno, err_msg = (
                              errpipe_data.split(b':', 2))
                  except ValueError:
                      exception_name = b'SubprocessError'
                      hex_errno = b'0'
                      err_msg = (b'Bad exception data from child: ' +
                                 repr(errpipe_data))
      

      b'...' + repr() is wrong: it raises a "TypeError: can't concat str to bytes" when python3 is run with -bb.

      Example with attached subprocess_bug.patch:

      haypo@selma$ ./python -bb -m test -v test_subprocess -m test_invalid_args
      (...)
      ======================================================================
      ERROR: test_invalid_args (test.test_subprocess.ContextManagerTests)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
        File "/home/haypo/prog/python/master/Lib/subprocess.py", line 1309, in _execute_child
          errpipe_data.split(b':', 1))
      ValueError: not enough values to unpack (expected 3, got 2)
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/haypo/prog/python/master/Lib/test/test_subprocess.py", line 2880, in test_invalid_args
          stderr=subprocess.PIPE) as proc:
        File "/home/haypo/prog/python/master/Lib/subprocess.py", line 709, in __init__
          restore_signals, start_new_session)
        File "/home/haypo/prog/python/master/Lib/subprocess.py", line 1314, in _execute_child
          repr(errpipe_data))
      TypeError: can't concat str to bytes

      IMHO err_msg should be decoded using err_msg.decode(errors="surrogatepass") and then use 'Bad ...: %s' % err_msg. It would need to add an "else:" block to the try/except to do the err_msg.decode(errors="surrogatepass") when no error is raised. Well, something like that :-)

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      on Aug 10, 2017
    3. gpshead commented on Sep 6, 2017

      @gpshead
      Member

      New changeset 3fc499b by Gregory P. Smith (Ammar Askar) in branch 'master':
      bpo-31178: Avoid concatenating bytes with str in subprocess error (bpo-3066)
      3fc499b

    4. gpshead commented on Sep 6, 2017

      @gpshead
      Member

      awaiting our new carrot nosed bot the create the backport PR to take care of 3.6 :)

    5. self-assigned this
      on Sep 6, 2017
    6. ammaraskar commented on Sep 6, 2017

      @ammaraskar
      Member

      If it doesn't manage to make it by tomorrow afternoon, I can backport it manually. It doesn't look like its been able to do it yet.

    7. gpshead commented on Sep 6, 2017

      @gpshead
      Member

      New changeset 3bad165 by Gregory P. Smith in branch '3.6':
      [3.6] bpo-31178: Avoid concatenating bytes with str in subprocess error (GH-3066) (bpo-3388)
      3bad165

    8. vstinner commented on Oct 5, 2017

      @vstinner
      MemberAuthor

      A colleague packaging Python for Red Hat Entreprise Linux reported me that tests hang randomly on test_exception_errpipe_bad_data() of test_subprocess. I don't know why exactly, but using strace, I noticed that the "unit test" calls os.waitpid() with the pid returned by the mocked fork_exec(): pid 0. So the test calls os.waitpid(0, 0).

      In a REPL on my Fedora 26, os.waitpid(0, 0) raises "ChildProcessError: [Errno 10] No child processes". I'm not sure that waitpid() is the cause of the hang, but it doesn't seem correct to me to call waitpid() with the result of a mock function, since the mock doesn't create a real child process.

      Attached PR 3896 mocks also os.waitpid() to fix this bug. I reopen the issue to discuss this bug in the new test added in this issue.

    9. vstinner commented on Oct 5, 2017

      @vstinner
      MemberAuthor

      In a REPL on my Fedora 26, os.waitpid(0, 0) raises "ChildProcessError: [Errno 10] No child processes". I'm not sure that waitpid() is the cause of the hang, (...)

      Oh wait, now I understood the full picture.

      Summary:

      • 2 new tests were added to test_subprocess and these tests call waitpid(0, 0) by mistake
      • In general, waitpid(0, 0) returns immediately and the code handles it properly
      • Sometimes, a previous test leaks a child process and so waitpid(0, 0) takes a few seconds or can even block

      --

      Running tests leak randomly child processes. See for example my recent test_socketserver fix in Python 3.6: commit fdcf3e9. This commit is *not* part of the recent Python 3.6.3 release, tested by my colleague.

      This fix is for the bug bpo-31593: test_socketserver leaked *randomly* child processes. Depending on the system load, waitpid() was called or not called to read the child process exit status.

      If you run "./python -m test test_socketserver test_subprocess" and test_socketserver() doesn't call waitpid() on a single process, it's possible that test_subprocess hangs on waitpid(0, 0): waiting on the process spawned by test_socketserver.

      test_socketserver is just one example, I fixed many other bugs in the Python test suite. Running Python tests in subprocesses using "./python -m test -jN ...", at least -j1, reduces the effect of the bug.

      Short script to explain the bug:
      ---

      import subprocess, sys, os, time
      
      args = [sys.executable, '-c', 'import time; time.sleep(2)']
      proc = subprocess.Popen(args)
      t0 = time.monotonic()
      print("waitpid(0, 0)...")
      pid, status = os.waitpid(0, 0)
      dt = time.monotonic() - t0
      print("%.1f sec later: waitpid(0, 0) -> %s" % (dt, (pid, status)))
      proc.wait()

      This script takes 3 seconds, since a test leaked a child process which takes time to complete.

    10. vstinner commented on Oct 5, 2017

      @vstinner
      MemberAuthor

      New changeset 11045c9 by Victor Stinner in branch 'master':
      bpo-31178: Mock os.waitpid() in test_subprocess (bpo-3896)
      11045c9

    11. vstinner commented on Oct 9, 2017

      @vstinner
      MemberAuthor

      commit fae0512
      Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
      Date: Thu Oct 5 07:10:59 2017 -0700

      [3.6] bpo-31178: Mock os.waitpid() in test_subprocess (GH-3896) (bpo-3897)
      
      Fix test_exception_errpipe_bad_data() and
      test_exception_errpipe_normal() of test_subprocess: mock os.waitpid()
      to avoid calling the real os.waitpid(0, 0) which is an unexpected
      side effect of the test.
      (cherry picked from commit 11045c9d8a21dd9bd182a3939189db02815f9783)
      
    12. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    easystdlibStandard Library Python modules in the Lib/ directory

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions