Re: MAP_FIXED_NOREPLACE appears to break older i386 binaries

[Russell King - ARM Linux admin <linux@armlinux.org.uk>]

On Sat, Oct 05, 2019 at 05:18:05PM -0700, Linus Torvalds wrote:
> Duh.
> 
> I only looked at recent issues in this area, and overlooked your
> sentence in between the two ELF section dumps, and it appears that you
> have already biseced it to something else:

I hadn't - I'd looked at the changes and identified a likely culpret
that fit with the symptoms and the layout of the binary.

What I'm basically trying to do is update my laptop - it was running
an x86_64 4.5.7 kernel but with 32-bit userland.  I've just installed
into a separate partition Debian Stable with the view to seeing
whether I like it, which means migrating stuff over - and I hit a
problem with the newer Evolution not wanting to recognise the
configuration/data from the previous version.

So I thought... I can just chroot into the old setup, run up evolution
there, export its configuration, so I can import it into the newer
version without having to go through a reboot cycle.

The chroot and exec of bin/bash in the old setup was successful, as
was dmesg, but useful tools like ls failed with a segfault.

The difference between working binaries and non-working binaries seems
to be whether the r-x and rw- LOAD sections in the ELF program headers
overlap on a page.  Here's bash:

    LOAD off    0x00000000 vaddr 0x08047000 paddr 0x08047000 align 2**12
         filesz 0x000bbb08 memsz 0x000bbb08 flags r-x
    LOAD off    0x000bc000 vaddr 0x08103000 paddr 0x08103000 align 2**12
         filesz 0x00004864 memsz 0x00009648 flags rw-

So, the r-x load covers 0x08047000-0x08102b08, and the following rw-
load covers 0x08103000 onwards - so next page.  dmesg is similar:

    LOAD off    0x00000000 vaddr 0x08048000 paddr 0x08048000 align 2**12
         filesz 0x00009c64 memsz 0x00009c64 flags r-x
    LOAD off    0x00009e28 vaddr 0x08052e28 paddr 0x08052e28 align 2**12
         filesz 0x000028ce memsz 0x000028ce flags rw-

0x08048000-0x08051c64 vs 0x08052e28 - so next page.  In contrast, ls:

    LOAD off    0x00000000 vaddr 0x08048000 paddr 0x08048000 align 2**12
         filesz 0x0001d620 memsz 0x0001d620 flags r-x
    LOAD off    0x0001d950 vaddr 0x08065950 paddr 0x08065950 align 2**12
         filesz 0x00000a50 memsz 0x000016e4 flags rw-

0x08048000-0x08065620 vs 0x08065950 - so same page, and fails.

Looking at the commit I referred to, what we end up with is:

- Initially, elf_fixed is MAP_FIXED_NOREPLACE and load_addr_set is false
- elf_brk and elf_bss are initially zero
- The first LOAD requests a mapping for 0x08048000 .. 0x08065fff inclusive
- since this is an executable mapping, we use elf_fixed to set the
  MAP_FIXED* flags, so this mapping is established with
  MAP_FIXED_NOREPLACE.
- load_addr_set is now set to true
- elf_bss is set to vaddr + filesz => 0x08065620
- elf_brk is set to vaddr + memsz => 0x08065620
- Moving on to the second LOAD, this is a mapping starting at 0x08065950
- Since elf_brk > elf_bss is false, we don't take that path through the
  code, which _would_ have set elf_fixed to MAP_FIXED (that's the only
  case which we would do - for the BSS.)
- As load_addr_set is true, we again use elf_fixed to set the
  MAP_FIXED* flags.  elf_fixed is still MAP_FIXED_NOREPLACE, so this
  mapping uses MAP_FIXED_NOREPLACE.
- Since this mapping overlaps the previous mapping, it fails with the
  error mentioned.

Since the ELF load_binary() method returns -EEXIST, we end up in this
code path in fs/exec.c:

                if (retval < 0 && !bprm->mm) {
                        /* we got to flush_old_exec() and failed after it */
                        read_unlock(&binfmt_lock);
                        force_sigsegv(SIGSEGV);
                        return retval;
                }

and the program is killed with a SIGSEGV.

So, from a code inspection point of view, it seems that this is likely
the culpret.

I don't yet have the debian stable system setup enough to build kernels;
that may be today's project, but I'd first like to solve the original
issue (migrating the evolution setup) so I can first see whether it's
going to be worth me continuing, or whether I persist with my existing
setup.

However, I think it _is_ worth highlighting that we seem to have broken
binary compatibility with older i386 userspace with newer kernels.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up