[Qemu-devel] [PATCH] linux-user: do setrlimit selectively

[Max Filippov <jcmvbkbc@gmail.com>]

When running 32-bit guest on 64-bit host setrlimit guest calls that
affect memory resources (RLIMIT_{AS,DATA,STACK}) don't always make sense
as is. They may result in QEMU lockup because mprotect call in
page_unprotect would fail with ENOMEM error code, causing infinite loop
of SIGSEGV. E.g. it happens when running libstdc++ testsuite for xtensa
target on x86_64 host.

Don't call host setrlimit for memory-related resources when running
32-bit guest on 64-bit host.

Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
---
 linux-user/syscall.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index 850b72a0c760..693a6c8aa7bb 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -9272,7 +9272,14 @@ static abi_long do_syscall1(void *cpu_env, int num, abi_long arg1,
             rlim.rlim_cur = target_to_host_rlim(target_rlim->rlim_cur);
             rlim.rlim_max = target_to_host_rlim(target_rlim->rlim_max);
             unlock_user_struct(target_rlim, arg2, 0);
-            return get_errno(setrlimit(resource, &rlim));
+            if (HOST_LONG_BITS <= TARGET_LONG_BITS ||
+                (resource != RLIMIT_DATA &&
+                 resource != RLIMIT_AS &&
+                 resource != RLIMIT_STACK)) {
+                return get_errno(setrlimit(resource, &rlim));
+            } else {
+                return 0;
+            }
         }
 #endif
 #ifdef TARGET_NR_getrlimit
-- 
2.11.0