Re: [PATCH 0/3] elf: load the main program from AT_EXECFD

Re: [PATCH 0/3] elf: load the main program from AT_EXECFD

[Florian Weimer]

Summary for kernel list: We are looking for ways to make executables not
loaded by the kernel more compatible with the rest of the system.  That
includes proper /proc/self/exe and auxv values.  The discussion was
triggered by Christian's binfmt_misc patch, which happens to share many
of the problems when /usr/bin/ld.so is used to load programs (instead of
the kernel).

* Christian Brauner:

> On 2026-07-16 17:37:21+02:00, Florian Weimer wrote:
>> * Christian Brauner:
>> 
>> >> What does /proc/self/exe look like for such processes?  Does GDB work?
>> >
>> > Right, I checked that.
>> >
>> > /proc/self/exe is ld.so. The kernel exec'd ld.so, so it names ld.so
>> > whether the program arrived via AT_EXECFD, via --program-fd, or via a
>> > plain "ld.so PROG" command line.
>> 
>> Yeah, and that causes problems with binaries that try to be relocatable.
>> We don't have a very convenient way to get the correct path in those
>> scenarios, so a lot of code uses /proc/self/exe instead.
>> 
>> I'm wondering if we can use the checkpoint-restore facilities (the
>> restore parts) to fix this: load ld.so another time, transfer control to
>> it, and instruct it to unmap the first copy and then invoke the
>> necessary prctls to make the process look exactly like a directly
>> invoked process.
>
> At first I was very confused about this proposal but I think I
> understand what you are after now. Your point is that we shouldn't just
> fix argv like in my proposal but actually even fix the exe file and
> remap.

Not sure about remap.  But I think to turn this into an it-just-works
solution, we need to fix /proc/self/exe and the auxiliary vector.

> So I think doing this purely in userspace isn't doable. The restore
> parts can fix everything except the exe link. PR_SET_MM_MAP requires
> capabilities in the relevant user namespace for the exe file. That makes
> it pretty useless for us. And dropping that capability requirement isn't
> feasible, I think. LSMs and audit trust the exe link, so an uncapped
> exe file would let any process masquerade as an arbitrary executable.

Is the latter really a problem?

Today, it's possible to stop a process that is SUID on disk before it
runs any code: 

$ ls -l /proc/163710/exe
lrwxrwxrwx. 1 fweimer fweimer 0 Jul 17 10:11 /proc/163710/exe -> /usr/bin/su

(gdb) info thread
  Id   Target Id           Frame 
* 1    process 163710 "su" 0x00007f5ae2f24e40 in _start ()
   from /lib64/ld-linux-x86-64.so.2

This is with kernel.yama.ptrace_scope=1.  I assume this gives me full
control over a process that is nominally running /usr/bin/su.  Even
AT_SECURE is set to 1:

Breakpoint 2, main (argc=1, argv=0x7ffd77cfb2e8) at login-utils/su.c:5
5	{
(gdb) print __libc_enable_secure
$1 = 1
(gdb) print (int) getuid ()
$2 = 1000
(gdb) print (int) geteuid ()
$3 = 1000
(gdb) print (long) getauxval(23)
$4 = 1

Of course, the AT_SECURE transition did not actually happen, and the
process is running with the original user privileges.

> Everything else (the entire saved auxv, the start/end_code/data, brk and
> stack markers) is validated but requires no capability at all. So an
> unprivileged ld.so can already repair /proc/<pid>/auxv and the
> stat/statm code accounting, but never /proc/self/exe.

It's good for us if AT_SECURE does not need protecting.  We would like
to use a fake 1 value in our test suite.

> Unmapping the first copy is also not really feasible. Even with
> privilege making this work would be very ugly: The kernel's
> replace_mm_exe_file() refuses with -EBUSY while any vma still maps the
> old exe file. From my research, CRIU works around exactly this by
> copying its restorer blob into an anonymous mapping before it unmaps the
> old address space.

Unmapping the first copy was only my idea to make this work with the
existing kernel facilities.  It's not required for the binfmt_misc case
and most /usr/bin/ld.so scenarios, and actually drives up complexity
considerably.

> So I think the exe link should be fixed up at exec time.
> begin_new_exec() sets mm->exe_file to bprm->file which after the
> binfmt_misc handoff is the interpreter. But bprm->executable is the file
> the kernel access-checked and kept open for AT_EXECFD. We have it right
> there. This is the file that would_dump() uses for it's decision and
> binfmt_misc's 'C' flag derive credentials from.
>
> We simply need an extension to binfmt_misc that sets mm->exe_file to
> bprm->executable. Then it is correct from the start and there's no
> window and no privilege question and existing 'O'/'C' users (qemu-user
> registrations) are unaffected. It then raises AT_FLAGS_PRESERVE_ARGV.
>
> On the userspace side, ld.so now sees this AT_* flag and in response
> issues one uncapped PR_SET_MM_MAP (that's available completely
> unprivileged) to retarget AT_PHDR/AT_ENTRY/AT_BASE and drop the stale
> AT_EXECFD from saved_auxv.
>
> With both in place, attaching gdb to a dispatched process is fully
> correct and /proc/self/exe-based self-location works. As a bonus the
> program file gets the same exe_file write-denial a directly executed
> binary has. Today it is ld.so that gets pinned and the running program's
> file stays writable while its text is mapped.
>
> After this, only uninteresting differences should be left. This would be
> a follow-up series I'm happy to do. Does that sound reasonable?

The proposal looks quite straightforward.  It's a shame that it does not
fix the /usr/bin/ld.so case.  Is there anything we can do to address
that?

I wouldn't mind if we had a system call that triggered the kernel ELF
loader for the main executable.

Thanks,
Florian

Re: [PATCH 0/3] elf: load the main program from AT_EXECFD

[Christian Brauner]

> > At first I was very confused about this proposal but I think I
> > understand what you are after now. Your point is that we shouldn't just
> > fix argv like in my proposal but actually even fix the exe file and
> > remap.
> 
> Not sure about remap.  But I think to turn this into an it-just-works
> solution, we need to fix /proc/self/exe and the auxiliary vector.

Yes.

> 
> > So I think doing this purely in userspace isn't doable. The restore
> > parts can fix everything except the exe link. PR_SET_MM_MAP requires
> > capabilities in the relevant user namespace for the exe file. That makes
> > it pretty useless for us. And dropping that capability requirement isn't
> > feasible, I think. LSMs and audit trust the exe link, so an uncapped
> > exe file would let any process masquerade as an arbitrary executable.
> 
> Is the latter really a problem?

I remember that I spoke out against this about 5 years ago when I merged
CAP_CHECKPOINT_RESTORE. I think changing it is something we can try but
it would take a while.

Extending binfmt_misc to do it as an option when registering a binfmt
handler is way easier imho (and simpler).

> > Everything else (the entire saved auxv, the start/end_code/data, brk and
> > stack markers) is validated but requires no capability at all. So an
> > unprivileged ld.so can already repair /proc/<pid>/auxv and the
> > stat/statm code accounting, but never /proc/self/exe.
> 
> It's good for us if AT_SECURE does not need protecting.  We would like
> to use a fake 1 value in our test suite.
> 
> > Unmapping the first copy is also not really feasible. Even with
> > privilege making this work would be very ugly: The kernel's
> > replace_mm_exe_file() refuses with -EBUSY while any vma still maps the
> > old exe file. From my research, CRIU works around exactly this by
> > copying its restorer blob into an anonymous mapping before it unmaps the
> > old address space.
> 
> Unmapping the first copy was only my idea to make this work with the
> existing kernel facilities.  It's not required for the binfmt_misc case
> and most /usr/bin/ld.so scenarios, and actually drives up complexity
> considerably.

I mean, it is kinda moving into CRIU territory, so yes, doing it like
that will be complex. :)

> > So I think the exe link should be fixed up at exec time.
> > begin_new_exec() sets mm->exe_file to bprm->file which after the
> > binfmt_misc handoff is the interpreter. But bprm->executable is the file
> > the kernel access-checked and kept open for AT_EXECFD. We have it right
> > there. This is the file that would_dump() uses for it's decision and
> > binfmt_misc's 'C' flag derive credentials from.
> >
> > We simply need an extension to binfmt_misc that sets mm->exe_file to
> > bprm->executable. Then it is correct from the start and there's no
> > window and no privilege question and existing 'O'/'C' users (qemu-user
> > registrations) are unaffected. It then raises AT_FLAGS_PRESERVE_ARGV.
> >
> > On the userspace side, ld.so now sees this AT_* flag and in response
> > issues one uncapped PR_SET_MM_MAP (that's available completely
> > unprivileged) to retarget AT_PHDR/AT_ENTRY/AT_BASE and drop the stale
> > AT_EXECFD from saved_auxv.
> >
> > With both in place, attaching gdb to a dispatched process is fully
> > correct and /proc/self/exe-based self-location works. As a bonus the
> > program file gets the same exe_file write-denial a directly executed
> > binary has. Today it is ld.so that gets pinned and the running program's
> > file stays writable while its text is mapped.
> >
> > After this, only uninteresting differences should be left. This would be
> > a follow-up series I'm happy to do. Does that sound reasonable?
> 
> The proposal looks quite straightforward.  It's a shame that it does not

Ok, I'll work on that once the basic AT_EXECFD stuff is done.

> fix the /usr/bin/ld.so case.  Is there anything we can do to address

You mean the case where it's called in userspace. Yeah, that's a bit
more tricky.

> that?
> 
> I wouldn't mind if we had a system call that triggered the kernel ELF
> loader for the main executable.

Ok, certainly another future patch series to consider. Fwiw, if you have
some far out ideas like that (kernel related) you should always feel
free to just file an issue or a pull request at:

https://github.com/uapi-group/kernel-features/

They appear on this website:

https://uapi-group.org/kernel-features/

and we have multiple people (not just me) that regularly pick items from
this list and implement them. This is a good way to avoid ideas just
being forgotten or invisible.