[PATCH] mmap PROT_NONE fix (was Re: serious performance regression

[Daniel McNeil <daniel@osdl.org>]

This patch fixes mmap PROT_NONE from elf binaries that do not
have the PT_GNU_STACK so that the do not have execute permission.

Before this fix, calling access() with a PROT_NONE page
was returning ENOENT instead of EFAULT.

BEFORE: 
$ x.gcc322
p2=0x40013000
pid\x1982
 
access 0 ptr (nil) return code -1 errno 14
access result:: Bad address
access 1 ptr 0xffffffff return code -1 errno 14
access result:: Bad address
access 2 ptr 0x40013000 return code -1 errno 2
access result:: No such file or directory

AFTER:

$ ./x.gcc322
p2=0x40017000
pid$92
                                                                                
access 0 ptr (nil) return code -1 errno 14
access result:: Bad address
access 1 ptr 0xffffffff return code -1 errno 14
access result:: Bad address
access 2 ptr 0x40017000 return code -1 errno 14
access result:: Bad address


Program available at
http://developer.osdl.org/daniel/mmap.PROT_NONE.bug/

Here is the patch.

Signed-off-by: Daniel McNeil <daniel@osdl.org>


=== mm/mmap.c 1.133 vs edited ==--- 1.133/mm/mmap.c	2004-06-29 07:43:12 -07:00
+++ edited/mm/mmap.c	2004-07-14 09:56:37 -07:00
@@ -792,6 +792,12 @@
 	vm_flags = calc_vm_prot_bits(prot) | calc_vm_flag_bits(flags) |
 			mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC;
 
+	/*
+	 * mm->def_flags might have VM_EXEC set, which PROT_NONE does NOT want.
+	 */
+	if (prot = PROT_NONE)
+		vm_flags &= ~VM_EXEC;
+
 	if (flags & MAP_LOCKED) {
 		if (!capable(CAP_IPC_LOCK))
 			return -EPERM;