You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[EASY] subprocess: TypeError: can't concat str to bytes, in _execute_child()Β #75361
Traceback (most recent call last):
File "/home/haypo/prog/python/master/Lib/subprocess.py", line 1309, in _execute_child
errpipe_data.split(b':', 1))
ValueError: not enough values to unpack (expected 3, got 2)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/haypo/prog/python/master/Lib/test/test_subprocess.py", line 2880, in test_invalid_args
stderr=subprocess.PIPE) as proc:
File "/home/haypo/prog/python/master/Lib/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/home/haypo/prog/python/master/Lib/subprocess.py", line 1314, in _execute_childrepr(errpipe_data))
TypeError: can't concat str to bytes
IMHO err_msg should be decoded using err_msg.decode(errors="surrogatepass") and then use 'Bad ...: %s' % err_msg. It would need to add an "else:" block to the try/except to do the err_msg.decode(errors="surrogatepass") when no error is raised. Well, something like that :-)
New changeset 3fc499b by Gregory P. Smith (Ammar Askar) in branch 'master': bpo-31178: Avoid concatenating bytes with str in subprocess error (bpo-3066) 3fc499b
New changeset 3bad165 by Gregory P. Smith in branch '3.6':
[3.6] bpo-31178: Avoid concatenating bytes with str in subprocess error (GH-3066) (bpo-3388) 3bad165
A colleague packaging Python for Red Hat Entreprise Linux reported me that tests hang randomly on test_exception_errpipe_bad_data() of test_subprocess. I don't know why exactly, but using strace, I noticed that the "unit test" calls os.waitpid() with the pid returned by the mocked fork_exec(): pid 0. So the test calls os.waitpid(0, 0).
In a REPL on my Fedora 26, os.waitpid(0, 0) raises "ChildProcessError: [Errno 10] No child processes". I'm not sure that waitpid() is the cause of the hang, but it doesn't seem correct to me to call waitpid() with the result of a mock function, since the mock doesn't create a real child process.
Attached PR 3896 mocks also os.waitpid() to fix this bug. I reopen the issue to discuss this bug in the new test added in this issue.
In a REPL on my Fedora 26, os.waitpid(0, 0) raises "ChildProcessError: [Errno 10] No child processes". I'm not sure that waitpid() is the cause of the hang, (...)
Oh wait, now I understood the full picture.
Summary:
2 new tests were added to test_subprocess and these tests call waitpid(0, 0) by mistake
In general, waitpid(0, 0) returns immediately and the code handles it properly
Sometimes, a previous test leaks a child process and so waitpid(0, 0) takes a few seconds or can even block
--
Running tests leak randomly child processes. See for example my recent test_socketserver fix in Python 3.6: commit fdcf3e9. This commit is *not* part of the recent Python 3.6.3 release, tested by my colleague.
This fix is for the bug bpo-31593: test_socketserver leaked *randomly* child processes. Depending on the system load, waitpid() was called or not called to read the child process exit status.
If you run "./python -m test test_socketserver test_subprocess" and test_socketserver() doesn't call waitpid() on a single process, it's possible that test_subprocess hangs on waitpid(0, 0): waiting on the process spawned by test_socketserver.
test_socketserver is just one example, I fixed many other bugs in the Python test suite. Running Python tests in subprocesses using "./python -m test -jN ...", at least -j1, reduces the effect of the bug.
[3.6] bpo-31178: Mock os.waitpid() in test_subprocess (GH-3896) (bpo-3897)
Fix test_exception_errpipe_bad_data() and
test_exception_errpipe_normal() of test_subprocess: mock os.waitpid()
to avoid calling the real os.waitpid(0, 0) which is an unexpected
side effect of the test.
(cherry picked from commit 11045c9d8a21dd9bd182a3939189db02815f9783)
Activity
vstinner commentedon Aug 10, 2017
Lib/subprocess.py contains the following code:
b'...' + repr() is wrong: it raises a "TypeError: can't concat str to bytes" when python3 is run with -bb.
Example with attached subprocess_bug.patch:
haypo@selma$ ./python -bb -m test -v test_subprocess -m test_invalid_args
(...)
======================================================================
ERROR: test_invalid_args (test.test_subprocess.ContextManagerTests)
----------------------------------------------------------------------
IMHO err_msg should be decoded using err_msg.decode(errors="surrogatepass") and then use 'Bad ...: %s' % err_msg. It would need to add an "else:" block to the try/except to do the err_msg.decode(errors="surrogatepass") when no error is raised. Well, something like that :-)
gpshead commentedon Sep 6, 2017
New changeset 3fc499b by Gregory P. Smith (Ammar Askar) in branch 'master':
bpo-31178: Avoid concatenating bytes with str in subprocess error (bpo-3066)
3fc499b
gpshead commentedon Sep 6, 2017
awaiting our new carrot nosed bot the create the backport PR to take care of 3.6 :)
ammaraskar commentedon Sep 6, 2017
If it doesn't manage to make it by tomorrow afternoon, I can backport it manually. It doesn't look like its been able to do it yet.
gpshead commentedon Sep 6, 2017
New changeset 3bad165 by Gregory P. Smith in branch '3.6':
[3.6] bpo-31178: Avoid concatenating bytes with str in subprocess error (GH-3066) (bpo-3388)
3bad165
vstinner commentedon Oct 5, 2017
A colleague packaging Python for Red Hat Entreprise Linux reported me that tests hang randomly on test_exception_errpipe_bad_data() of test_subprocess. I don't know why exactly, but using strace, I noticed that the "unit test" calls os.waitpid() with the pid returned by the mocked fork_exec(): pid 0. So the test calls os.waitpid(0, 0).
In a REPL on my Fedora 26, os.waitpid(0, 0) raises "ChildProcessError: [Errno 10] No child processes". I'm not sure that waitpid() is the cause of the hang, but it doesn't seem correct to me to call waitpid() with the result of a mock function, since the mock doesn't create a real child process.
Attached PR 3896 mocks also os.waitpid() to fix this bug. I reopen the issue to discuss this bug in the new test added in this issue.
vstinner commentedon Oct 5, 2017
Oh wait, now I understood the full picture.
Summary:
--
Running tests leak randomly child processes. See for example my recent test_socketserver fix in Python 3.6: commit fdcf3e9. This commit is *not* part of the recent Python 3.6.3 release, tested by my colleague.
This fix is for the bug bpo-31593: test_socketserver leaked *randomly* child processes. Depending on the system load, waitpid() was called or not called to read the child process exit status.
If you run "./python -m test test_socketserver test_subprocess" and test_socketserver() doesn't call waitpid() on a single process, it's possible that test_subprocess hangs on waitpid(0, 0): waiting on the process spawned by test_socketserver.
test_socketserver is just one example, I fixed many other bugs in the Python test suite. Running Python tests in subprocesses using "./python -m test -jN ...", at least -j1, reduces the effect of the bug.
Short script to explain the bug:
---
This script takes 3 seconds, since a test leaked a child process which takes time to complete.
vstinner commentedon Oct 5, 2017
New changeset 11045c9 by Victor Stinner in branch 'master':
bpo-31178: Mock os.waitpid() in test_subprocess (bpo-3896)
11045c9
vstinner commentedon Oct 9, 2017
commit fae0512
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date: Thu Oct 5 07:10:59 2017 -0700