linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

When running qemu-riscv64 on a riscv64 host executing a ET_EXEC riscv64
binary it cannot allocate memory for the stack:

$ qemu-riscv64 -d page ./hello.riscv64
host mmap_min_addr=0x10000
Locating guest address space @ 0x3ee000
page layout changed following mmap
start            end              size             prot
0000000000010000-0000000000013000 0000000000003000 ---
page layout changed following mmap
start            end              size             prot
0000000000010000-0000000000011000 0000000000001000 r-x
0000000000011000-0000000000013000 0000000000002000 ---
page layout changed following mmap
start            end              size             prot
0000000000010000-0000000000011000 0000000000001000 r-x
0000000000011000-0000000000013000 0000000000002000 rw-
mmap stack: Cannot allocate memory

The issue is that guest_base is non-zero, which turns the target_mmap
call with zero base in setup_arg_pages into a host mmap call with
non-zero base.  On other hosts like x86_64 or aarch64, guest_base
remains zero and the issue does not occur.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Richard Henderson]

On 6/25/24 04:37, Andreas Schwab wrote:
> When running qemu-riscv64 on a riscv64 host executing a ET_EXEC riscv64
> binary it cannot allocate memory for the stack:
> 
> $ qemu-riscv64 -d page ./hello.riscv64
> host mmap_min_addr=0x10000
> Locating guest address space @ 0x3ee000
> page layout changed following mmap
> start            end              size             prot
> 0000000000010000-0000000000013000 0000000000003000 ---
> page layout changed following mmap
> start            end              size             prot
> 0000000000010000-0000000000011000 0000000000001000 r-x
> 0000000000011000-0000000000013000 0000000000002000 ---
> page layout changed following mmap
> start            end              size             prot
> 0000000000010000-0000000000011000 0000000000001000 r-x
> 0000000000011000-0000000000013000 0000000000002000 rw-
> mmap stack: Cannot allocate memory
> 
> The issue is that guest_base is non-zero, which turns the target_mmap
> call with zero base in setup_arg_pages into a host mmap call with
> non-zero base.  On other hosts like x86_64 or aarch64, guest_base
> remains zero and the issue does not occur.

You need to be more precise in your bug reports, because it works for me.
Everything non-PIE, statically linked:

./qemu-riscv64: ELF 64-bit LSB executable, UCB RISC-V, RVC, double-float ABI, version 1 
(SYSV), statically linked, BuildID[sha1]=92e2b4b9a2cbcc91ac029a49ec72eaefe5111f38, for 
GNU/Linux 4.15.0, with debug_info, not stripped

/home/rth/a.out: ELF 64-bit LSB executable, UCB RISC-V, RVC, double-float ABI, version 1 
(SYSV), statically linked, BuildID[sha1]=4c52c576a0452e97d9117b89dd317c88460b0768, for 
GNU/Linux 4.15.0, not stripped

$ ./qemu-riscv64 -d page ~/a.out
host mmap_min_addr=0x1000
Locating guest address space @ 0x3ff000
page layout changed following mmap
start            end              size             prot
0000000000010000-0000000000084000 0000000000074000 ---
...
end_code    0x0000000000078388
start_code  0x0000000000010000
start_data  0x00000000000795b0
end_data    0x000000000007e8a8
start_stack 0x0000003f812224a0
brk         0x0000000000084000
entry       0x000000000001041c
argv_start  0x0000003f812224a8
env_start   0x0000003f812224b8
auxv_start  0x0000003f81222570
...
Hello, World!


I don't doubt that you see a problem, but I need a reproducer, not a guess as to what the 
problem might be.  Certainly guest_base is *not* it.  One can always force the use of a 
non-zero base with -B or -R.


r~

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

On Jun 25 2024, Richard Henderson wrote:

> can always force the use of a non-zero base with -B or -R.

$ qemu-riscv64 -d page -B 0x3ee000 hello.riscv64
host mmap_min_addr=0x1000 (fallback)
qemu-riscv64: /daten/src/test/hello.riscv64: requires virtual address space that is in use (omit the -B option or choose a different value)

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Richard Henderson]

On 6/26/24 01:23, Andreas Schwab wrote:
> On Jun 25 2024, Richard Henderson wrote:
> 
>> can always force the use of a non-zero base with -B or -R.
> 
> $ qemu-riscv64 -d page -B 0x3ee000 hello.riscv64
> host mmap_min_addr=0x1000 (fallback)
> qemu-riscv64: /daten/src/test/hello.riscv64: requires virtual address space that is in use (omit the -B option or choose a different value)
> 

Well, sure, but that obviously is where qemu-riscv64 itself is located.
Still not a valid test case.


r~

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

On Wed, Jun 26, 2024 at 9:48 AM Richard Henderson <
richard.henderson@linaro.org> wrote:

> On 6/26/24 01:23, Andreas Schwab wrote:
> > On Jun 25 2024, Richard Henderson wrote:
> >
> >> can always force the use of a non-zero base with -B or -R.
> >
> > $ qemu-riscv64 -d page -B 0x3ee000 hello.riscv64
> > host mmap_min_addr=0x1000 (fallback)
> > qemu-riscv64: /daten/src/test/hello.riscv64: requires virtual address
> space that is in use (omit the -B option or choose a different value)
> >
>
> Well, sure, but that obviously is where qemu-riscv64 itself is located.
> Still not a valid test case.
>

Yea, what happens if you say -B 0x3ee000000 or something else that won't
conflict?

Warner

[-- Attachment #2: Type: text/html, Size: 1125 bytes --]

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

On Jun 26 2024, Warner Losh wrote:

> On Wed, Jun 26, 2024 at 9:48 AM Richard Henderson <
> richard.henderson@linaro.org> wrote:
>
>> On 6/26/24 01:23, Andreas Schwab wrote:
>> > On Jun 25 2024, Richard Henderson wrote:
>> >
>> >> can always force the use of a non-zero base with -B or -R.
>> >
>> > $ qemu-riscv64 -d page -B 0x3ee000 hello.riscv64
>> > host mmap_min_addr=0x1000 (fallback)
>> > qemu-riscv64: /daten/src/test/hello.riscv64: requires virtual address
>> space that is in use (omit the -B option or choose a different value)
>> >
>>
>> Well, sure, but that obviously is where qemu-riscv64 itself is located.
>> Still not a valid test case.
>>
>
> Yea, what happens if you say -B 0x3ee000000 or something else that won't
> conflict?

I didn't chose that number, qemu did.  If it doesn't work then qemu must
be fixed.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1587 bytes --]

On Thu, Jun 27, 2024, 1:54 AM Andreas Schwab <schwab@suse.de> wrote:

> On Jun 26 2024, Warner Losh wrote:
>
> > On Wed, Jun 26, 2024 at 9:48 AM Richard Henderson <
> > richard.henderson@linaro.org> wrote:
> >
> >> On 6/26/24 01:23, Andreas Schwab wrote:
> >> > On Jun 25 2024, Richard Henderson wrote:
> >> >
> >> >> can always force the use of a non-zero base with -B or -R.
> >> >
> >> > $ qemu-riscv64 -d page -B 0x3ee000 hello.riscv64
> >> > host mmap_min_addr=0x1000 (fallback)
> >> > qemu-riscv64: /daten/src/test/hello.riscv64: requires virtual address
> >> space that is in use (omit the -B option or choose a different value)
> >> >
> >>
> >> Well, sure, but that obviously is where qemu-riscv64 itself is located.
> >> Still not a valid test case.
> >>
> >
> > Yea, what happens if you say -B 0x3ee000000 or something else that won't
> > conflict?
>
> I didn't chose that number, qemu did.  If it doesn't work then qemu must
> be fixed.
>

And when you are diagnosing the root cause of the bug, the submitter of the
bug sometimes needs to do diagnostic tests when requested, not attack the
volunteers who are trying to help. If that's all you do, there will be no
fix. You can't talk to me like that and expect any reaction but "I have
better things to do with my time than deal with this jerk" regardless of
the merits of the original complaint.

Warner

-- 
> Andreas Schwab, SUSE Labs, schwab@suse.de
> GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
> "And now for something completely different."
>

[-- Attachment #2: Type: text/html, Size: 2518 bytes --]

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

Perhaps you should refrain from attacking the volunteers that report
bugs.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Peter Maydell]

On Thu, 27 Jun 2024 at 15:27, Andreas Schwab <schwab@suse.de> wrote:
> Perhaps you should refrain from attacking the volunteers that report
> bugs.

I think the tone of your previous email was extremely
terse and didn't actually answer the question, which is
quite easy to misinterpret as hostility.

We're all trying to achieve the same thing here, and the underlying
problem is that we can't reproduce the fault you're seeing.
So to help us with that, you need to provide more precise
instructions on exactly how we can get the result you see,
or else provide the answers to the questions we're asking
to try to get more diagnostic information.

For instance, this might help:
 * how exactly did you build the binary (what source, what
   compiler version, what OS, etc)
 * a repro built binary
 * exact QEMU command line

Clearly something is different between your setup and
Richard's and Warner's, so we need more detail to narrow
down what that is. Warner's questions are trying to find
out more about what's going on.

thanks
-- PMM

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

[-- Attachment #1: hello.riscv64 --]
[-- Type: application/octet-stream, Size: 13632 bytes --]

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Richard Henderson]

Hi Andreas,

Thanks for the test binary.

Now I need to know about your qemu version and build configuration.

Your test binary still works for me.  My configuration:

Debian GNU/Linux trixie/sid
gcc version 13.2.0 (Debian 13.2.0-25)

commit 60b4f3aff4e39be04f5d73c65a7e8ef838475c9f (HEAD, tag: v9.0.1, origin/stable-9.0)

../src/configure --static --disable-system --disable-tools --disable-docs

qemu-riscv64: ELF 64-bit LSB executable, UCB RISC-V, RVC, double-float ABI, version 1 
(SYSV), statically linked, BuildID[sha1]=86144e2f215344440f3e21b65b2cf2b8cb33c67f, for 
GNU/Linux 4.15.0, with debug_info, not stripped

   LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
                  0x000000000035feae 0x000000000035feae  R E    0x1000
   LOAD           0x0000000000360708 0x0000000000370708 0x0000000000370708
                  0x0000000000070ebc 0x00000000000833f0  RW     0x1000

So the end of .bss is at 0x3f3af8.

With -d page, I get

Locating guest address space @ 0x3f4000

which is uncomfortably close to the end of .bss, but seems to work.


r~

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

On Jul 01 2024, Richard Henderson wrote:

> With -d page, I get
>
> Locating guest address space @ 0x3f4000

Why do you get a different address?

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Richard Henderson]

On 7/2/24 01:09, Andreas Schwab wrote:
> On Jul 01 2024, Richard Henderson wrote:
> 
>> With -d page, I get
>>
>> Locating guest address space @ 0x3f4000
> 
> Why do you get a different address?
> 

Come on, man.  Please answer my question:

> Now I need to know about your qemu version and build configuration. 


r~

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

https://build.opensuse.org/package/live_build_log/openSUSE:Factory:RISCV/qemu:qemu-linux-user/standard/riscv64

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Richard Henderson]

On 7/2/24 01:09, Andreas Schwab wrote:
> On Jul 01 2024, Richard Henderson wrote:
> 
>> With -d page, I get
>>
>> Locating guest address space @ 0x3f4000
> 
> Why do you get a different address?

Is /proc mounted in your environment?
The guest address base selection depends on /proc/self/maps.


r~

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

On Jul 02 2024, Richard Henderson wrote:

> Is /proc mounted in your environment?

Sure, it's a fully running system on real hardware.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Richard Henderson]

On 7/2/24 07:39, Andreas Schwab wrote:
> On Jul 02 2024, Richard Henderson wrote:
> 
>> Is /proc mounted in your environment?
> 
> Sure, it's a fully running system on real hardware.
> 

I mean, you're not running it in some chroot sandbox without /proc.


r~

Re: linux-user cannot allocate stack memory on riscv64 host due to non-zero guest_base

[Andreas Schwab]

Please follow https://en.opensuse.org/openSUSE:RISC-V#QEMU_system_emulation
to start the JeOS-efi image.  When logged in:

# zypper rm qemu-linux-user-riscv
# zypper in qemu-linux-user

Then you can run qemu-riscv64 inside:

# qemu-riscv64 -d page hello.riscv64 
host mmap_min_addr=0x10000
Locating guest address space @ 0x3ee000
page layout changed following mmap
start            end              size             prot
0000000000010000-0000000000013000 0000000000003000 ---
page layout changed following mmap
start            end              size             prot
0000000000010000-0000000000011000 0000000000001000 r-x
0000000000011000-0000000000013000 0000000000002000 ---
page layout changed following mmap
start            end              size             prot
0000000000010000-0000000000011000 0000000000001000 r-x
0000000000011000-0000000000013000 0000000000002000 rw-
mmap stack: Cannot allocate memory

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."