* [PATCH v2] Rework the jobserver open logic
@ 2026-01-14 16:34 Jonathan Corbet
2026-01-14 20:28 ` Mauro Carvalho Chehab
2026-01-15 2:44 ` duchangbin
0 siblings, 2 replies; 4+ messages in thread
From: Jonathan Corbet @ 2026-01-14 16:34 UTC (permalink / raw)
To: linux-doc; +Cc: Mauro Carvalho Chehab, Changbin Du
The parsing of jobserver options is done in a massive try: block that hides
problems and (perhaps) bugs. Split up that block and make the logic
explicit by moving the initial parsing of MAKEFLAGS out of that block. Add
warnings in the places things can go wrong.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
v2 changes: send warnings to stderr, s/5/len('fifo:')/.
tools/lib/python/jobserver.py | 143 +++++++++++++++++++++-------------
1 file changed, 90 insertions(+), 53 deletions(-)
diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
index 616411087725e..bcaff993d0bec 100755
--- a/tools/lib/python/jobserver.py
+++ b/tools/lib/python/jobserver.py
@@ -35,6 +35,9 @@ import os
import subprocess
import sys
+def warn(text):
+ print(text, file = sys.stderr)
+
class JobserverExec:
"""
Claim all slots from make using POSIX Jobserver.
@@ -58,64 +61,98 @@ class JobserverExec:
if self.is_open:
return
-
- try:
- # Fetch the make environment options.
- flags = os.environ["MAKEFLAGS"]
- # Look for "--jobserver=R,W"
- # Note that GNU Make has used --jobserver-fds and --jobserver-auth
- # so this handles all of them.
- opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
-
- # Parse out R,W file descriptor numbers and set them nonblocking.
- # If the MAKEFLAGS variable contains multiple instances of the
- # --jobserver-auth= option, the last one is relevant.
- fds = opts[-1].split("=", 1)[1]
-
- # Starting with GNU Make 4.4, named pipes are used for reader
- # and writer.
- # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
- _, _, path = fds.partition("fifo:")
-
- if path:
+ self.is_open = True # We only try once
+ self.claim = None
+ #
+ # Check the make flags for "--jobserver=R,W"
+ # Note that GNU Make has used --jobserver-fds and --jobserver-auth
+ # so this handles all of them.
+ #
+ flags = os.environ.get('MAKEFLAGS', '')
+ opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+ if not opts:
+ return
+ #
+ # Separate out the provided file descriptors
+ #
+ split_opt = opts[-1].split('=', 1)
+ if len(split_opt) != 2:
+ warn('WARNING: unparseable option:', opts[-1])
+ return
+ fds = split_opt[1]
+ #
+ # As of GNU Make 4.4, we'll be looking for a named pipe
+ # identified as fifo:path
+ #
+ if fds.startswith('fifo:'):
+ path = fds[len('fifo:':]
+ try:
self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
self.writer = os.open(path, os.O_WRONLY)
- else:
- self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
+ except (OSError, IOError):
+ warn('WARNING: unable to open jobserver pipe', path)
+ return
+ #
+ # Otherwise look for integer file-descriptor numbers.
+ #
+ else:
+ split_fds = fds.split(',')
+ if len(split_fds) != 2:
+ warn('WARNING: malformed jobserver file descriptors:', fds)
+ return
+ try:
+ self.reader = int(split_fds[0])
+ self.writer = int(split_fds[1])
+ except ValueError:
+ warn('WARNING: non-integer jobserver file-descriptors:', fds)
+ return
+ try:
+ #
# Open a private copy of reader to avoid setting nonblocking
# on an unexpecting process with the same reader fd.
- self.reader = os.open("/proc/self/fd/%d" % (self.reader),
+ #
+ self.reader = os.open(f"/proc/self/fd/{self.reader}",
os.O_RDONLY | os.O_NONBLOCK)
-
- # Read out as many jobserver slots as possible
- while True:
- try:
- slot = os.read(self.reader, 8)
- if not slot:
- # Clear self.jobs to prevent us from probably writing incorrect file.
- self.jobs = b""
- raise ValueError("unexpected empty token from jobserver fd, invalid '--jobserver-auth=' setting?")
- self.jobs += slot
- except (OSError, IOError) as e:
- if e.errno == errno.EWOULDBLOCK:
- # Stop at the end of the jobserver queue.
- break
- # If something went wrong, give back the jobs.
- if self.jobs:
- os.write(self.writer, self.jobs)
- raise e
-
- # Add a bump for our caller's reserveration, since we're just going
- # to sit here blocked on our child.
- self.claim = len(self.jobs) + 1
-
- except (KeyError, IndexError, ValueError, OSError, IOError) as e:
- print(f"jobserver: warning: {repr(e)}", file=sys.stderr)
- # Any missing environment strings or bad fds should result in just
- # not being parallel.
- self.claim = None
-
- self.is_open = True
+ except (IOError, OSError):
+ warn('WARNING: Unable to reopen jobserver read-side pipe')
+ return
+ #
+ # OK, we have the channel to the job server; read out as many jobserver
+ # slots as possible.
+ #
+ while True:
+ try:
+ slot = os.read(self.reader, 8)
+ if not slot:
+ #
+ # Something went wrong. Clear self.jobs to avoid writing
+ # weirdness back to the jobserver and give up.
+ self.jobs = b""
+ warn("WARNING: unexpected empty token from jobserver;"
+ " possible invalid '--jobserver-auth=' setting")
+ self.claim = None
+ return
+ except (OSError, IOError) as e:
+ #
+ # If there is nothing more to read then we are done.
+ #
+ if e.errno == errno.EWOULDBLOCK:
+ break
+ #
+ # Anything else says that something went weird; give back
+ # the jobs and give up.
+ #
+ if self.jobs:
+ os.write(self.writer, self.jobs)
+ self.claim = None
+ warn('WARNING: error reading from jobserver pipe', e)
+ return
+ self.jobs += slot
+ #
+ # Add a bump for our caller's reserveration, since we're just going
+ # to sit here blocked on our child.
+ #
+ self.claim = len(self.jobs) + 1
def close(self):
"""Return all reserved slots to Jobserver"""
--
2.52.0
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [PATCH v2] Rework the jobserver open logic
2026-01-14 16:34 [PATCH v2] Rework the jobserver open logic Jonathan Corbet
@ 2026-01-14 20:28 ` Mauro Carvalho Chehab
2026-01-15 2:44 ` duchangbin
1 sibling, 0 replies; 4+ messages in thread
From: Mauro Carvalho Chehab @ 2026-01-14 20:28 UTC (permalink / raw)
To: Jonathan Corbet; +Cc: linux-doc, Mauro Carvalho Chehab, Changbin Du
Em Wed, 14 Jan 2026 09:34:23 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:
> The parsing of jobserver options is done in a massive try: block that hides
> problems and (perhaps) bugs. Split up that block and make the logic
> explicit by moving the initial parsing of MAKEFLAGS out of that block. Add
> warnings in the places things can go wrong.
>
> Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
> v2 changes: send warnings to stderr, s/5/len('fifo:')/.
>
> tools/lib/python/jobserver.py | 143 +++++++++++++++++++++-------------
> 1 file changed, 90 insertions(+), 53 deletions(-)
>
> diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
> index 616411087725e..bcaff993d0bec 100755
> --- a/tools/lib/python/jobserver.py
> +++ b/tools/lib/python/jobserver.py
> @@ -35,6 +35,9 @@ import os
> import subprocess
> import sys
>
> +def warn(text):
> + print(text, file = sys.stderr)
> +
> class JobserverExec:
> """
> Claim all slots from make using POSIX Jobserver.
> @@ -58,64 +61,98 @@ class JobserverExec:
>
> if self.is_open:
> return
> -
> - try:
> - # Fetch the make environment options.
> - flags = os.environ["MAKEFLAGS"]
> - # Look for "--jobserver=R,W"
> - # Note that GNU Make has used --jobserver-fds and --jobserver-auth
> - # so this handles all of them.
> - opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
> -
> - # Parse out R,W file descriptor numbers and set them nonblocking.
> - # If the MAKEFLAGS variable contains multiple instances of the
> - # --jobserver-auth= option, the last one is relevant.
> - fds = opts[-1].split("=", 1)[1]
> -
> - # Starting with GNU Make 4.4, named pipes are used for reader
> - # and writer.
> - # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
> - _, _, path = fds.partition("fifo:")
> -
> - if path:
> + self.is_open = True # We only try once
> + self.claim = None
> + #
> + # Check the make flags for "--jobserver=R,W"
> + # Note that GNU Make has used --jobserver-fds and --jobserver-auth
> + # so this handles all of them.
> + #
> + flags = os.environ.get('MAKEFLAGS', '')
> + opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
> + if not opts:
> + return
> + #
> + # Separate out the provided file descriptors
> + #
> + split_opt = opts[-1].split('=', 1)
> + if len(split_opt) != 2:
> + warn('WARNING: unparseable option:', opts[-1])
> + return
> + fds = split_opt[1]
> + #
> + # As of GNU Make 4.4, we'll be looking for a named pipe
> + # identified as fifo:path
> + #
> + if fds.startswith('fifo:'):
> + path = fds[len('fifo:':]
I would prefer to encapulate "fifo:" on a var.
With or without such change:
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> + try:
> self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
> self.writer = os.open(path, os.O_WRONLY)
> - else:
> - self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
> + except (OSError, IOError):
> + warn('WARNING: unable to open jobserver pipe', path)
> + return
> + #
> + # Otherwise look for integer file-descriptor numbers.
> + #
> + else:
> + split_fds = fds.split(',')
> + if len(split_fds) != 2:
> + warn('WARNING: malformed jobserver file descriptors:', fds)
> + return
> + try:
> + self.reader = int(split_fds[0])
> + self.writer = int(split_fds[1])
> + except ValueError:
> + warn('WARNING: non-integer jobserver file-descriptors:', fds)
> + return
> + try:
> + #
> # Open a private copy of reader to avoid setting nonblocking
> # on an unexpecting process with the same reader fd.
> - self.reader = os.open("/proc/self/fd/%d" % (self.reader),
> + #
> + self.reader = os.open(f"/proc/self/fd/{self.reader}",
> os.O_RDONLY | os.O_NONBLOCK)
> -
> - # Read out as many jobserver slots as possible
> - while True:
> - try:
> - slot = os.read(self.reader, 8)
> - if not slot:
> - # Clear self.jobs to prevent us from probably writing incorrect file.
> - self.jobs = b""
> - raise ValueError("unexpected empty token from jobserver fd, invalid '--jobserver-auth=' setting?")
> - self.jobs += slot
> - except (OSError, IOError) as e:
> - if e.errno == errno.EWOULDBLOCK:
> - # Stop at the end of the jobserver queue.
> - break
> - # If something went wrong, give back the jobs.
> - if self.jobs:
> - os.write(self.writer, self.jobs)
> - raise e
> -
> - # Add a bump for our caller's reserveration, since we're just going
> - # to sit here blocked on our child.
> - self.claim = len(self.jobs) + 1
> -
> - except (KeyError, IndexError, ValueError, OSError, IOError) as e:
> - print(f"jobserver: warning: {repr(e)}", file=sys.stderr)
> - # Any missing environment strings or bad fds should result in just
> - # not being parallel.
> - self.claim = None
> -
> - self.is_open = True
> + except (IOError, OSError):
> + warn('WARNING: Unable to reopen jobserver read-side pipe')
> + return
> + #
> + # OK, we have the channel to the job server; read out as many jobserver
> + # slots as possible.
> + #
> + while True:
> + try:
> + slot = os.read(self.reader, 8)
> + if not slot:
> + #
> + # Something went wrong. Clear self.jobs to avoid writing
> + # weirdness back to the jobserver and give up.
> + self.jobs = b""
> + warn("WARNING: unexpected empty token from jobserver;"
> + " possible invalid '--jobserver-auth=' setting")
> + self.claim = None
> + return
> + except (OSError, IOError) as e:
> + #
> + # If there is nothing more to read then we are done.
> + #
> + if e.errno == errno.EWOULDBLOCK:
> + break
> + #
> + # Anything else says that something went weird; give back
> + # the jobs and give up.
> + #
> + if self.jobs:
> + os.write(self.writer, self.jobs)
> + self.claim = None
> + warn('WARNING: error reading from jobserver pipe', e)
> + return
> + self.jobs += slot
> + #
> + # Add a bump for our caller's reserveration, since we're just going
> + # to sit here blocked on our child.
> + #
> + self.claim = len(self.jobs) + 1
>
> def close(self):
> """Return all reserved slots to Jobserver"""
Thanks,
Mauro
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH v2] Rework the jobserver open logic
2026-01-14 16:34 [PATCH v2] Rework the jobserver open logic Jonathan Corbet
2026-01-14 20:28 ` Mauro Carvalho Chehab
@ 2026-01-15 2:44 ` duchangbin
2026-01-16 17:34 ` Jonathan Corbet
1 sibling, 1 reply; 4+ messages in thread
From: duchangbin @ 2026-01-15 2:44 UTC (permalink / raw)
To: Jonathan Corbet
Cc: linux-doc@vger.kernel.org, Mauro Carvalho Chehab, duchangbin
On Wed, Jan 14, 2026 at 09:34:23AM -0700, Jonathan Corbet wrote:
> The parsing of jobserver options is done in a massive try: block that hides
> problems and (perhaps) bugs. Split up that block and make the logic
> explicit by moving the initial parsing of MAKEFLAGS out of that block. Add
> warnings in the places things can go wrong.
>
> Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
> v2 changes: send warnings to stderr, s/5/len('fifo:')/.
>
> tools/lib/python/jobserver.py | 143 +++++++++++++++++++++-------------
> 1 file changed, 90 insertions(+), 53 deletions(-)
>
> diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
> index 616411087725e..bcaff993d0bec 100755
> --- a/tools/lib/python/jobserver.py
> +++ b/tools/lib/python/jobserver.py
> @@ -35,6 +35,9 @@ import os
> import subprocess
> import sys
>
> +def warn(text):
> + print(text, file = sys.stderr)
Maybe this could be:
print(f"WARNING: {text}", file = sys.stderr)
to unify the warnning messages.
> +
> class JobserverExec:
> """
> Claim all slots from make using POSIX Jobserver.
> @@ -58,64 +61,98 @@ class JobserverExec:
>
> if self.is_open:
> return
> -
> - try:
> - # Fetch the make environment options.
> - flags = os.environ["MAKEFLAGS"]
> - # Look for "--jobserver=R,W"
> - # Note that GNU Make has used --jobserver-fds and --jobserver-auth
> - # so this handles all of them.
> - opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
> -
> - # Parse out R,W file descriptor numbers and set them nonblocking.
> - # If the MAKEFLAGS variable contains multiple instances of the
> - # --jobserver-auth= option, the last one is relevant.
> - fds = opts[-1].split("=", 1)[1]
> -
> - # Starting with GNU Make 4.4, named pipes are used for reader
> - # and writer.
> - # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
> - _, _, path = fds.partition("fifo:")
> -
> - if path:
> + self.is_open = True # We only try once
> + self.claim = None
> + #
> + # Check the make flags for "--jobserver=R,W"
> + # Note that GNU Make has used --jobserver-fds and --jobserver-auth
> + # so this handles all of them.
> + #
> + flags = os.environ.get('MAKEFLAGS', '')
> + opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
> + if not opts:
> + return
> + #
> + # Separate out the provided file descriptors
> + #
> + split_opt = opts[-1].split('=', 1)
> + if len(split_opt) != 2:
> + warn('WARNING: unparseable option:', opts[-1])
Calling the warn() function with two positional arguments, but the function is
defined to accept only one.
> + return
> + fds = split_opt[1]
> + #
> + # As of GNU Make 4.4, we'll be looking for a named pipe
> + # identified as fifo:path
> + #
> + if fds.startswith('fifo:'):
> + path = fds[len('fifo:':]
Missing a parenthesis.
path = fds[len('fifo:'):]
> + try:
> self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
> self.writer = os.open(path, os.O_WRONLY)
> - else:
> - self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
> + except (OSError, IOError):
> + warn('WARNING: unable to open jobserver pipe', path)
Ditto.
> + return
> + #
> + # Otherwise look for integer file-descriptor numbers.
> + #
> + else:
> + split_fds = fds.split(',')
> + if len(split_fds) != 2:
> + warn('WARNING: malformed jobserver file descriptors:', fds)
> + return
> + try:
> + self.reader = int(split_fds[0])
> + self.writer = int(split_fds[1])
> + except ValueError:
> + warn('WARNING: non-integer jobserver file-descriptors:', fds)
> + return
> + try:
> + #
> # Open a private copy of reader to avoid setting nonblocking
> # on an unexpecting process with the same reader fd.
> - self.reader = os.open("/proc/self/fd/%d" % (self.reader),
> + #
> + self.reader = os.open(f"/proc/self/fd/{self.reader}",
> os.O_RDONLY | os.O_NONBLOCK)
> -
> - # Read out as many jobserver slots as possible
> - while True:
> - try:
> - slot = os.read(self.reader, 8)
> - if not slot:
> - # Clear self.jobs to prevent us from probably writing incorrect file.
> - self.jobs = b""
> - raise ValueError("unexpected empty token from jobserver fd, invalid '--jobserver-auth=' setting?")
> - self.jobs += slot
> - except (OSError, IOError) as e:
> - if e.errno == errno.EWOULDBLOCK:
> - # Stop at the end of the jobserver queue.
> - break
> - # If something went wrong, give back the jobs.
> - if self.jobs:
> - os.write(self.writer, self.jobs)
> - raise e
> -
> - # Add a bump for our caller's reserveration, since we're just going
> - # to sit here blocked on our child.
> - self.claim = len(self.jobs) + 1
> -
> - except (KeyError, IndexError, ValueError, OSError, IOError) as e:
> - print(f"jobserver: warning: {repr(e)}", file=sys.stderr)
> - # Any missing environment strings or bad fds should result in just
> - # not being parallel.
> - self.claim = None
> -
> - self.is_open = True
> + except (IOError, OSError):
> + warn('WARNING: Unable to reopen jobserver read-side pipe')
> + return
> + #
> + # OK, we have the channel to the job server; read out as many jobserver
> + # slots as possible.
> + #
> + while True:
> + try:
> + slot = os.read(self.reader, 8)
> + if not slot:
> + #
> + # Something went wrong. Clear self.jobs to avoid writing
> + # weirdness back to the jobserver and give up.
> + self.jobs = b""
> + warn("WARNING: unexpected empty token from jobserver;"
> + " possible invalid '--jobserver-auth=' setting")
> + self.claim = None
> + return
> + except (OSError, IOError) as e:
> + #
> + # If there is nothing more to read then we are done.
> + #
> + if e.errno == errno.EWOULDBLOCK:
> + break
> + #
> + # Anything else says that something went weird; give back
> + # the jobs and give up.
> + #
> + if self.jobs:
> + os.write(self.writer, self.jobs)
> + self.claim = None
> + warn('WARNING: error reading from jobserver pipe', e)
> + return
> + self.jobs += slot
> + #
> + # Add a bump for our caller's reserveration, since we're just going
> + # to sit here blocked on our child.
> + #
> + self.claim = len(self.jobs) + 1
>
> def close(self):
> """Return all reserved slots to Jobserver"""
> --
> 2.52.0
>
--
Cheers,
Changbin Du
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH v2] Rework the jobserver open logic
2026-01-15 2:44 ` duchangbin
@ 2026-01-16 17:34 ` Jonathan Corbet
0 siblings, 0 replies; 4+ messages in thread
From: Jonathan Corbet @ 2026-01-16 17:34 UTC (permalink / raw)
To: duchangbin; +Cc: linux-doc@vger.kernel.org, Mauro Carvalho Chehab, duchangbin
duchangbin <changbin.du@huawei.com> writes:
>> +def warn(text):
>> + print(text, file = sys.stderr)
> Maybe this could be:
> print(f"WARNING: {text}", file = sys.stderr)
> to unify the warnning messages.
Yes, that makes sense.
>> + split_opt = opts[-1].split('=', 1)
>> + if len(split_opt) != 2:
>> + warn('WARNING: unparseable option:', opts[-1])
> Calling the warn() function with two positional arguments, but the function is
> defined to accept only one.
So that is embarrassing. As penance I have now actually tested the
warning paths...
>> + return
>> + fds = split_opt[1]
>> + #
>> + # As of GNU Make 4.4, we'll be looking for a named pipe
>> + # identified as fifo:path
>> + #
>> + if fds.startswith('fifo:'):
>> + path = fds[len('fifo:':]
> Missing a parenthesis.
> path = fds[len('fifo:'):]
...and that is even moreso. I *did* test this before committing it, but
that obviously could never have gotten past the parsing phase. I have
no idea what happened here.
Thanks for taking a look, new version coming shortly.
jon
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-01-16 17:34 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-01-14 16:34 [PATCH v2] Rework the jobserver open logic Jonathan Corbet
2026-01-14 20:28 ` Mauro Carvalho Chehab
2026-01-15 2:44 ` duchangbin
2026-01-16 17:34 ` Jonathan Corbet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox