[PATCH 06/14] bsd-user: Fix crazy bug with mmap

[Warner Losh <imp@bsdimp.com>]

So, for some crazy reason, we checked that only the SHARED bit was set
out of all these bits. The only ones that were live was MAP_ANON and
MAP_FIXED. We never called mmap with MAP_FIXED and MAP_SHARED. And the
equivalent from linux-user tested that MAP_ANON was off, but MAP_SHARED
was on. So incorporate the more direct approach from linux-user for
testing this condition. Many of these bits haven't been actually used in
FreeBSD since the very early days....

Signed-off-by: Warner Losh <imp@bsdimp.com>
---
 bsd-user/mmap.c         | 16 ++++++++++------
 bsd-user/syscall_defs.h |  2 +-
 2 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/bsd-user/mmap.c b/bsd-user/mmap.c
index 548b4824df..76ac0340cb 100644
--- a/bsd-user/mmap.c
+++ b/bsd-user/mmap.c
@@ -215,10 +215,13 @@ static int mmap_frag(abi_ulong real_start,
 
     prot_new = prot | prot1;
     if (fd != -1) {
-        /* msync() won't work here, so we return an error if write is
-           possible while it is a shared mapping */
-        if ((flags & TARGET_BSD_MAP_FLAGMASK) == MAP_SHARED &&
-            (prot & PROT_WRITE))
+        /*
+         * msync() won't work here, so we return an error if write is
+         * possible while it is a shared mapping
+         */
+        if (!(flags & MAP_ANON)
+            && (flags & MAP_TYPE) == MAP_SHARED
+            && (prot & PROT_WRITE))
             return -1;
 
         /* adjust protection to be able to read */
@@ -586,8 +589,9 @@ abi_long target_mmap(abi_ulong start, abi_ulong len, int prot,
              * msync() won't work here, so we return an error if write is
              * possible while it is a shared mapping
              */
-            if ((flags & TARGET_BSD_MAP_FLAGMASK) == MAP_SHARED &&
-                (prot & PROT_WRITE)) {
+            if (!(flags & MAP_ANON)
+                && (flags & MAP_TYPE) == MAP_SHARED
+                && (prot & PROT_WRITE)) {
                 errno = EINVAL;
                 goto fail;
             }
diff --git a/bsd-user/syscall_defs.h b/bsd-user/syscall_defs.h
index 9fa16e0863..506aea3e27 100644
--- a/bsd-user/syscall_defs.h
+++ b/bsd-user/syscall_defs.h
@@ -121,7 +121,7 @@ struct target_msgbuf {
 /*
  *  sys/mman.h
  */
-#define TARGET_BSD_MAP_FLAGMASK         0x3ff7
+#define MAP_TYPE                        0xf
 
 /*
  * sys/time.h

-- 
2.55.0