[PULL 05/11] bsd-user: Fix crazy bug with mmap

[Warner Losh <imp@bsdimp.com>]

So, for some crazy reason, we checked that only the SHARED bit was set
out of all these bits. The only ones that were live was MAP_ANON and
MAP_FIXED. We never called mmap with MAP_FIXED and MAP_SHARED. And the
equivalent from linux-user tested that MAP_ANON was off, but MAP_SHARED
was on. So incorporate the more direct approach from linux-user for
testing this condition. Many of these bits haven't been actually used in
FreeBSD since the very early days....

Also fix MAP_TYPE definition. The 0xf I think was copied from the Linux
headers a long time ago, since BSD doesn't define this. In FreeBSD,
there's only two bits of this low nibble that are used for the sharing
type (linux has 2 more types encoded in these low bits). The kernel
rejects bogus flags (0xc part of 0xf mask), so it's better to have just
these two bits here.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Warner Losh <imp@bsdimp.com>
---
 bsd-user/mmap.c         | 17 +++++++++++------
 bsd-user/syscall_defs.h |  9 ++++++++-
 2 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/bsd-user/mmap.c b/bsd-user/mmap.c
index 548b4824df..5068e5c3b7 100644
--- a/bsd-user/mmap.c
+++ b/bsd-user/mmap.c
@@ -215,11 +215,15 @@ static int mmap_frag(abi_ulong real_start,
 
     prot_new = prot | prot1;
     if (fd != -1) {
-        /* msync() won't work here, so we return an error if write is
-           possible while it is a shared mapping */
-        if ((flags & TARGET_BSD_MAP_FLAGMASK) == MAP_SHARED &&
-            (prot & PROT_WRITE))
+        /*
+         * msync() won't work here, so we return an error if write is
+         * possible while it is a shared mapping
+         */
+        if (!(flags & MAP_ANON)
+            && (flags & MAP_TYPE) == MAP_SHARED
+            && (prot & PROT_WRITE)) {
             return -1;
+        }
 
         /* adjust protection to be able to read */
         if (!(prot1 & PROT_WRITE))
@@ -586,8 +590,9 @@ abi_long target_mmap(abi_ulong start, abi_ulong len, int prot,
              * msync() won't work here, so we return an error if write is
              * possible while it is a shared mapping
              */
-            if ((flags & TARGET_BSD_MAP_FLAGMASK) == MAP_SHARED &&
-                (prot & PROT_WRITE)) {
+            if (!(flags & MAP_ANON)
+                && (flags & MAP_TYPE) == MAP_SHARED
+                && (prot & PROT_WRITE)) {
                 errno = EINVAL;
                 goto fail;
             }
diff --git a/bsd-user/syscall_defs.h b/bsd-user/syscall_defs.h
index 9fa16e0863..c4fd5f5589 100644
--- a/bsd-user/syscall_defs.h
+++ b/bsd-user/syscall_defs.h
@@ -121,7 +121,14 @@ struct target_msgbuf {
 /*
  *  sys/mman.h
  */
-#define TARGET_BSD_MAP_FLAGMASK         0x3ff7
+/*
+ * FreeBSD doesn't define this, but Linux does. In Linux, it's only MAP_PRIVATE
+ * and MAP_SHARED and a few Linux-specific types (0xf). FreeBSD allows other
+ * types of mapping with MAP_ANON, MAP_GUARD, and MAP_STACK. The mask on Linux
+ * excludes these: They don't have MAP_GUARD, and MAP_STACK and MAP_ANON have
+ * similar flags with different names that aren't in it's 0xf MAP_TYPE.
+ */
+#define MAP_TYPE (MAP_PRIVATE | MAP_SHARED)
 
 /*
  * sys/time.h
-- 
2.55.0