Qemu master crashing on boot when using file backend for memory

Qemu master crashing on boot when using file backend for memory

[Raphael Norwitz]

When I try run master qemu I am hitting a divide by zero error. It seems
to be coming from util/oslib-posix.c in touch_all_pages(). see line 477:

numpages_per_thread = numpages / memset_num_threads;

Poking around the crash dumps, I can see that the smp_cpus parameter
passed in to touch_all_pages() is 0. Going up the stack to
host_memory_backend_memory_complete() I see backend->prealloc_threads is
also 0.

Here’s how I am running qemu

./x86_64-softmmu/qemu-system-x86_64 \
	-kernel /boot/vmlinuz-3.10.0-1062.el7.x86_64  \
 	-netdev user,id=net0,hostfwd=tcp::2250-:22 \
	-device e1000e,netdev=net0 \
	-m 1G \
	-initrd /boot/initramfs-3.10.0-1062.el7.x86_64.img  \
	-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1G \
	-numa node,nodeid=0,cpus=0,memdev=ram-node0 

I don't see this error on a slightly older qemu, as of commit 105b07f1
(January 27th).

Interestingly when I remove the memory-backend-file parameter I don’t
see the error, i.e. this works:

./x86_64-softmmu/qemu-system-x86_64 \
	-kernel /boot/vmlinuz-3.10.0-1062.el7.x86_64  \
 	-netdev user,id=net0,hostfwd=tcp::2250-:22 \
	-device e1000e,netdev=net0 \
	-m 1G \
	-initrd /boot/initramfs-3.10.0-1062.el7.x86_64.img

Looking at the blame data for backends/hostmem.c I see commit ffac16fa
introduced some churn in this part of the code. Has anyone else seen
this issue? Could I be doing something wrong here?

Re: Qemu master crashing on boot when using file backend for memory

[Igor Mammedov]

On Thu, 12 Mar 2020 01:36:48 -0400
Raphael Norwitz <raphael.norwitz@nutanix.com> wrote:

> When I try run master qemu I am hitting a divide by zero error. It seems
> to be coming from util/oslib-posix.c in touch_all_pages(). see line 477:
> 
> numpages_per_thread = numpages / memset_num_threads;
> 
> Poking around the crash dumps, I can see that the smp_cpus parameter
> passed in to touch_all_pages() is 0. Going up the stack to
> host_memory_backend_memory_complete() I see backend->prealloc_threads is
> also 0.
> 
> Here’s how I am running qemu
> 
> ./x86_64-softmmu/qemu-system-x86_64 \
> 	-kernel /boot/vmlinuz-3.10.0-1062.el7.x86_64  \
>  	-netdev user,id=net0,hostfwd=tcp::2250-:22 \
> 	-device e1000e,netdev=net0 \
> 	-m 1G \
> 	-initrd /boot/initramfs-3.10.0-1062.el7.x86_64.img  \
> 	-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1G \
> 	-numa node,nodeid=0,cpus=0,memdev=ram-node0 
> 
> I don't see this error on a slightly older qemu, as of commit 105b07f1
> (January 27th).
> 
> Interestingly when I remove the memory-backend-file parameter I don’t
> see the error, i.e. this works:
> 
> ./x86_64-softmmu/qemu-system-x86_64 \
> 	-kernel /boot/vmlinuz-3.10.0-1062.el7.x86_64  \
>  	-netdev user,id=net0,hostfwd=tcp::2250-:22 \
> 	-device e1000e,netdev=net0 \
> 	-m 1G \
> 	-initrd /boot/initramfs-3.10.0-1062.el7.x86_64.img
> 
> Looking at the blame data for backends/hostmem.c I see commit ffac16fa
> introduced some churn in this part of the code. Has anyone else seen
> this issue? Could I be doing something wrong here?

It's know issue, see
 "[PATCH] oslib-posix: initialize mutex and condition variable
for a fix
> 

Re: Qemu master crashing on boot when using file backend for memory

[Raphael Norwitz]

On Thu, Mar 12, 2020 at 11:54:50AM +0100, Igor Mammedov wrote:
> 
> On Thu, 12 Mar 2020 01:36:48 -0400
> Raphael Norwitz <raphael.norwitz@nutanix.com> wrote:
> 
> > When I try run master qemu I am hitting a divide by zero error. It seems
> > to be coming from util/oslib-posix.c in touch_all_pages(). see line 477:
> > 
> > numpages_per_thread = numpages / memset_num_threads;
> > 
> > Poking around the crash dumps, I can see that the smp_cpus parameter
> > passed in to touch_all_pages() is 0. Going up the stack to
> > host_memory_backend_memory_complete() I see backend->prealloc_threads is
> > also 0.
> > 
> > Here’s how I am running qemu
> > 
> > ./x86_64-softmmu/qemu-system-x86_64 \
> > 	-kernel /boot/vmlinuz-3.10.0-1062.el7.x86_64  \
> >  	-netdev user,id=net0,hostfwd=tcp::2250-:22 \
> > 	-device e1000e,netdev=net0 \
> > 	-m 1G \
> > 	-initrd /boot/initramfs-3.10.0-1062.el7.x86_64.img  \
> > 	-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1G \
> > 	-numa node,nodeid=0,cpus=0,memdev=ram-node0 
> > 
> > I don't see this error on a slightly older qemu, as of commit 105b07f1
> > (January 27th).
> > 
> > Interestingly when I remove the memory-backend-file parameter I don’t
> > see the error, i.e. this works:
> > 
> > ./x86_64-softmmu/qemu-system-x86_64 \
> > 	-kernel /boot/vmlinuz-3.10.0-1062.el7.x86_64  \
> >  	-netdev user,id=net0,hostfwd=tcp::2250-:22 \
> > 	-device e1000e,netdev=net0 \
> > 	-m 1G \
> > 	-initrd /boot/initramfs-3.10.0-1062.el7.x86_64.img
> > 
> > Looking at the blame data for backends/hostmem.c I see commit ffac16fa
> > introduced some churn in this part of the code. Has anyone else seen
> > this issue? Could I be doing something wrong here?
> 
> It's know issue, see
>  "[PATCH] oslib-posix: initialize mutex and condition variable
> for a fix
> > 
>

I'm testing on qemu master now. Looks like this patch has been merged
and I'm still seeing the same crash.

Re: Qemu master crashing on boot when using file backend for memory

[Igor Mammedov]

On Tue, 24 Mar 2020 22:50:46 -0400
Raphael Norwitz <raphael.norwitz@nutanix.com> wrote:

> On Thu, Mar 12, 2020 at 11:54:50AM +0100, Igor Mammedov wrote:
> > 
> > On Thu, 12 Mar 2020 01:36:48 -0400
> > Raphael Norwitz <raphael.norwitz@nutanix.com> wrote:
> >   
> > > When I try run master qemu I am hitting a divide by zero error. It seems
> > > to be coming from util/oslib-posix.c in touch_all_pages(). see line 477:
> > > 
> > > numpages_per_thread = numpages / memset_num_threads;
> > > 
> > > Poking around the crash dumps, I can see that the smp_cpus parameter
> > > passed in to touch_all_pages() is 0. Going up the stack to
> > > host_memory_backend_memory_complete() I see backend->prealloc_threads is
> > > also 0.
> > > 
> > > Here’s how I am running qemu
> > > 
> > > ./x86_64-softmmu/qemu-system-x86_64 \
> > > 	-kernel /boot/vmlinuz-3.10.0-1062.el7.x86_64  \
> > >  	-netdev user,id=net0,hostfwd=tcp::2250-:22 \
> > > 	-device e1000e,netdev=net0 \
> > > 	-m 1G \
> > > 	-initrd /boot/initramfs-3.10.0-1062.el7.x86_64.img  \
> > > 	-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1G \
> > > 	-numa node,nodeid=0,cpus=0,memdev=ram-node0 
> > > 
> > > I don't see this error on a slightly older qemu, as of commit 105b07f1
> > > (January 27th).
> > > 
> > > Interestingly when I remove the memory-backend-file parameter I don’t
> > > see the error, i.e. this works:
> > > 
> > > ./x86_64-softmmu/qemu-system-x86_64 \
> > > 	-kernel /boot/vmlinuz-3.10.0-1062.el7.x86_64  \
> > >  	-netdev user,id=net0,hostfwd=tcp::2250-:22 \
> > > 	-device e1000e,netdev=net0 \
> > > 	-m 1G \
> > > 	-initrd /boot/initramfs-3.10.0-1062.el7.x86_64.img
> > > 
> > > Looking at the blame data for backends/hostmem.c I see commit ffac16fa
> > > introduced some churn in this part of the code. Has anyone else seen
> > > this issue? Could I be doing something wrong here?  
> > 
> > It's know issue, see
> >  "[PATCH] oslib-posix: initialize mutex and condition variable
> > for a fix  
> > >   
> >  
> 
> I'm testing on qemu master now. Looks like this patch has been merged
> and I'm still seeing the same crash.

Thanks for report, fix is on the way to qemu-devel