[PATCH 1/2] linux-user: return ENOMEM from madvise on unmapped ranges

[Roi Klevansky <roiklevansky@gmail.com>]

The kernel fails with ENOMEM when madvise() is called with a partially
or fully unmapped address range. This fix allows target_madvise() to
emulate this behavior by calling page_check_range() upfront while still
allowing advice specific errors to take precedence.

Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/4382
Signed-off-by: Roi Klevansky <roiklevansky@gmail.com>
---
 linux-user/mmap.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/linux-user/mmap.c b/linux-user/mmap.c
index cc0c2ee6c2..e5195cf501 100644
--- a/linux-user/mmap.c
+++ b/linux-user/mmap.c
@@ -1305,8 +1305,14 @@ abi_long target_madvise(abi_ulong start, abi_ulong len_in, int advice)
      * success, which is broken but some userspace programs fail to work
      * otherwise. Completely implementing such emulation is quite complicated
      * though.
+     *
+     * When some (or all) of the pages in the range are not mapped, madvise
+     * should fail with -ENOMEM. If another error occurs, its value is
+     * returned instead.
      */
     mmap_lock();
+    ret = page_check_range(start, len, PAGE_VALID) ? 0 : -TARGET_ENOMEM;
+
     switch (advice) {
     case MADV_NORMAL:
     case MADV_RANDOM:
@@ -1316,7 +1322,6 @@ abi_long target_madvise(abi_ulong start, abi_ulong len_in, int advice)
     case MADV_FREE:
     case MADV_COLD:
     case MADV_PAGEOUT:
-        ret = 0; /* OK */
         break;
     case MADV_REMOVE:
         ret = -EOPNOTSUPP;
-- 
2.55.0