Hotplug + Reboot is crashing HPT guest with HPT resizing enabled

Hotplug + Reboot is crashing HPT guest with HPT resizing enabled

[Bharata B Rao]

[-- Attachment #1: Type: text/plain, Size: 882 bytes --]

Hi,

Rebooting a hash guest after hotplugging memory to it is crashing the
guest. This is seen only when HPT resizing is enabled. I see guest crashing
at multiple places, but this location is fairly commonly seen:

kernel BUG at mm/slub.c:3912!

Testing with latest guest kernel and ppc-for-2.12 branch of QEMU.

A bit of debugging shows me that when memory is added, the guest kernel
tries to resize HPT to a htab_shift value lesser than the value with which
the guest has booted. For eg. a 8GB guest boots with htab_shift of 26. When
1G is hot-added,
arch/powerpc/mm/hash_utils_64.c:resize_hpt_for_hotplug() ends up assigning
24 to target_hpt_shift. This looks suspicious as we are increasing the
memory, but kernel is asking for shrinking the HPT size. HPT resizing
requests fail though, but next reboot crashes the guest.

Regards,
Bharata.
-- 
http://raobharata.wordpress.com/

[-- Attachment #2: Type: text/html, Size: 1058 bytes --]

Re: Hotplug + Reboot is crashing HPT guest with HPT resizing enabled

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1898 bytes --]

On Fri, Feb 23, 2018 at 03:02:40PM +0530, Bharata B Rao wrote:
> Hi,
> 
> Rebooting a hash guest after hotplugging memory to it is crashing the
> guest. This is seen only when HPT resizing is enabled. I see guest crashing
> at multiple places, but this location is fairly commonly seen:
> 
> kernel BUG at mm/slub.c:3912!
> 
> Testing with latest guest kernel and ppc-for-2.12 branch of QEMU.

Ugh.  We had several bugs along these lines, but I thought I'd fixed
them.  I wonder what this one is.

> A bit of debugging shows me that when memory is added, the guest kernel
> tries to resize HPT to a htab_shift value lesser than the value with which
> the guest has booted. For eg. a 8GB guest boots with htab_shift of 26. When
> 1G is hot-added,
> arch/powerpc/mm/hash_utils_64.c:resize_hpt_for_hotplug() ends up assigning
> 24 to target_hpt_shift. This looks suspicious as we are increasing the
> memory, but kernel is asking for shrinking the HPT size.

So the shrink-HPT-on-add-memory is actually expected and should be
harmless.  It occurs because qemu estimates HPT size on the
traditional HPT == RAM size / 64 formular, which was devised with 4k
pages in mind.  The kernel on the other hand, knows it is using 64k
pages and so estimates a smaller HPT size.  Hot plugging memory always
prompts the guest to re-estimate the required HPT size, but if the
added memory is small enough, that size can still be smaller than
qemu's initial guess.

> HPT resizing
> requests fail though, but next reboot crashes the guest.

As noted the shrink is expected, so we need to debug the crash
separately.  Do you have 9478956794c11239b7c1c3ef9ce95c883bb839a3 in
your tree?

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Hotplug + Reboot is crashing HPT guest with HPT resizing enabled

[Bharata B Rao]

[-- Attachment #1: Type: text/plain, Size: 1950 bytes --]

On Fri, Mar 2, 2018 at 7:51 AM, David Gibson <david@gibson.dropbear.id.au>
wrote:

> On Fri, Feb 23, 2018 at 03:02:40PM +0530, Bharata B Rao wrote:
> > Hi,
> >
> > Rebooting a hash guest after hotplugging memory to it is crashing the
> > guest. This is seen only when HPT resizing is enabled. I see guest
> crashing
> > at multiple places, but this location is fairly commonly seen:
> >
> > kernel BUG at mm/slub.c:3912!
> >
> > Testing with latest guest kernel and ppc-for-2.12 branch of QEMU.
>
> Ugh.  We had several bugs along these lines, but I thought I'd fixed
> them.  I wonder what this one is.
>
> > A bit of debugging shows me that when memory is added, the guest kernel
> > tries to resize HPT to a htab_shift value lesser than the value with
> which
> > the guest has booted. For eg. a 8GB guest boots with htab_shift of 26.
> When
> > 1G is hot-added,
> > arch/powerpc/mm/hash_utils_64.c:resize_hpt_for_hotplug() ends up
> assigning
> > 24 to target_hpt_shift. This looks suspicious as we are increasing the
> > memory, but kernel is asking for shrinking the HPT size.
>
> So the shrink-HPT-on-add-memory is actually expected and should be
> harmless.  It occurs because qemu estimates HPT size on the
> traditional HPT == RAM size / 64 formular, which was devised with 4k
> pages in mind.  The kernel on the other hand, knows it is using 64k
> pages and so estimates a smaller HPT size.  Hot plugging memory always
> prompts the guest to re-estimate the required HPT size, but if the
> added memory is small enough, that size can still be smaller than
> qemu's initial guess.
>

Thanks for the clarification.


>
> > HPT resizing
> > requests fail though, but next reboot crashes the guest.
>
> As noted the shrink is expected, so we need to debug the crash
> separately.  Do you have 9478956794c11239b7c1c3ef9ce95c883bb839a3 in
> your tree?
>

I do now and I no longer see the crash that I was observing last week.

Regards,
Bharata.

[-- Attachment #2: Type: text/html, Size: 2698 bytes --]