[Qemu-devel] Windows balloon driver PFN issue

[Qemu-devel] Windows balloon driver PFN issue

[Peter Xu]

Hi, Michael and the list,

I observed this on windows 8 enterprise guests, when doing memory ballooning:

23892@1517298572.328354:virtio_balloon_to_target balloon target: 0x80000000 num_pages: 524288
23892@1517298638.542819:virtio_balloon_get_config num_pages: 524288 actual: 0
23892@1517298638.542974:virtio_balloon_handle_output section name: pc.ram gpa: 0x174604000
23892@1517298638.543059:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
23892@1517298638.543135:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460a000
23892@1517298638.543140:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
23892@1517298638.543143:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460b000
23892@1517298638.543146:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
23892@1517298638.543148:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460c000
23892@1517298638.543152:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
23892@1517298638.543154:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460d000
23892@1517298638.543159:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
23892@1517298638.543162:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460e000
23892@1517298638.543165:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
23892@1517298638.543167:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460f000
23892@1517298638.543170:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
...

I think it's very possible that these zero addresses (please let me
know what the first 4K page is used for if anyone knows, since IIUC
that's what we throw away now) are half of the 64bit PFN.  Or say, not
sure whether this means a windows guest driver bug that is using
64bits for PFN rather than 32bits (and I suppose the protocol is using
32bit for PFNs).

Michael, do you know what to do with this?

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] Windows balloon driver PFN issue

[Michael S. Tsirkin]

On Wed, Jan 31, 2018 at 05:28:35PM +0800, Peter Xu wrote:
> Hi, Michael and the list,
> 
> I observed this on windows 8 enterprise guests, when doing memory ballooning:
> 
> 23892@1517298572.328354:virtio_balloon_to_target balloon target: 0x80000000 num_pages: 524288
> 23892@1517298638.542819:virtio_balloon_get_config num_pages: 524288 actual: 0
> 23892@1517298638.542974:virtio_balloon_handle_output section name: pc.ram gpa: 0x174604000
> 23892@1517298638.543059:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> 23892@1517298638.543135:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460a000
> 23892@1517298638.543140:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> 23892@1517298638.543143:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460b000
> 23892@1517298638.543146:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> 23892@1517298638.543148:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460c000
> 23892@1517298638.543152:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> 23892@1517298638.543154:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460d000
> 23892@1517298638.543159:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> 23892@1517298638.543162:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460e000
> 23892@1517298638.543165:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> 23892@1517298638.543167:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460f000
> 23892@1517298638.543170:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> ...
> 
> I think it's very possible that these zero addresses (please let me
> know what the first 4K page is used for if anyone knows, since IIUC
> that's what we throw away now) are half of the 64bit PFN.  Or say, not
> sure whether this means a windows guest driver bug that is using
> 64bits for PFN rather than 32bits (and I suppose the protocol is using
> 32bit for PFNs).
> 
> Michael, do you know what to do with this?
> 
> Thanks,

PFN is GPA>>12.  Do you have more than 1<<44 bytes of memory in this VM then?

> -- 
> Peter Xu

Re: [Qemu-devel] Windows balloon driver PFN issue

[Peter Xu]

On Wed, Jan 31, 2018 at 04:03:12PM +0200, Michael S. Tsirkin wrote:
> On Wed, Jan 31, 2018 at 05:28:35PM +0800, Peter Xu wrote:
> > Hi, Michael and the list,
> > 
> > I observed this on windows 8 enterprise guests, when doing memory ballooning:
> > 
> > 23892@1517298572.328354:virtio_balloon_to_target balloon target: 0x80000000 num_pages: 524288
> > 23892@1517298638.542819:virtio_balloon_get_config num_pages: 524288 actual: 0
> > 23892@1517298638.542974:virtio_balloon_handle_output section name: pc.ram gpa: 0x174604000
> > 23892@1517298638.543059:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > 23892@1517298638.543135:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460a000
> > 23892@1517298638.543140:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > 23892@1517298638.543143:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460b000
> > 23892@1517298638.543146:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > 23892@1517298638.543148:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460c000
> > 23892@1517298638.543152:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > 23892@1517298638.543154:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460d000
> > 23892@1517298638.543159:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > 23892@1517298638.543162:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460e000
> > 23892@1517298638.543165:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > 23892@1517298638.543167:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460f000
> > 23892@1517298638.543170:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > ...
> > 
> > I think it's very possible that these zero addresses (please let me
> > know what the first 4K page is used for if anyone knows, since IIUC
> > that's what we throw away now) are half of the 64bit PFN.  Or say, not
> > sure whether this means a windows guest driver bug that is using
> > 64bits for PFN rather than 32bits (and I suppose the protocol is using
> > 32bit for PFNs).
> > 
> > Michael, do you know what to do with this?
> > 
> > Thanks,
> 
> PFN is GPA>>12.  Do you have more than 1<<44 bytes of memory in this VM then?

No.  But isn't it still not good to drop the page at offset zero (and
drop it NNN times)?  And I'm not sure what will happen if guest has
1<<44 bytes; then we'll possibly drop very random addresses since a
real address will be splitted?

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] Windows balloon driver PFN issue

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 10:18:53AM +0800, Peter Xu wrote:
> On Wed, Jan 31, 2018 at 04:03:12PM +0200, Michael S. Tsirkin wrote:
> > On Wed, Jan 31, 2018 at 05:28:35PM +0800, Peter Xu wrote:
> > > Hi, Michael and the list,
> > > 
> > > I observed this on windows 8 enterprise guests, when doing memory ballooning:
> > > 
> > > 23892@1517298572.328354:virtio_balloon_to_target balloon target: 0x80000000 num_pages: 524288
> > > 23892@1517298638.542819:virtio_balloon_get_config num_pages: 524288 actual: 0
> > > 23892@1517298638.542974:virtio_balloon_handle_output section name: pc.ram gpa: 0x174604000
> > > 23892@1517298638.543059:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > 23892@1517298638.543135:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460a000
> > > 23892@1517298638.543140:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > 23892@1517298638.543143:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460b000
> > > 23892@1517298638.543146:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > 23892@1517298638.543148:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460c000
> > > 23892@1517298638.543152:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > 23892@1517298638.543154:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460d000
> > > 23892@1517298638.543159:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > 23892@1517298638.543162:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460e000
> > > 23892@1517298638.543165:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > 23892@1517298638.543167:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460f000
> > > 23892@1517298638.543170:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > ...
> > > 
> > > I think it's very possible that these zero addresses (please let me
> > > know what the first 4K page is used for if anyone knows, since IIUC
> > > that's what we throw away now) are half of the 64bit PFN.  Or say, not
> > > sure whether this means a windows guest driver bug that is using
> > > 64bits for PFN rather than 32bits (and I suppose the protocol is using
> > > 32bit for PFNs).
> > > 
> > > Michael, do you know what to do with this?
> > > 
> > > Thanks,
> > 
> > PFN is GPA>>12.  Do you have more than 1<<44 bytes of memory in this VM then?
> 
> No.  But isn't it still not good to drop the page at offset zero (and
> drop it NNN times)?

Absolutely - looks like a bug. I just don't know why does this happen.

>  And I'm not sure what will happen if guest has
> 1<<44 bytes; then we'll possibly drop very random addresses since a
> real address will be splitted?
> 
> Thanks,

The balloon won't work, period. There was an interface change to fix
that but implementation issues remained and contributor seems to be busy
with page hints.

> -- 
> Peter Xu

Re: [Qemu-devel] Windows balloon driver PFN issue

[Peter Xu]

On Thu, Feb 01, 2018 at 04:24:40AM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 01, 2018 at 10:18:53AM +0800, Peter Xu wrote:
> > On Wed, Jan 31, 2018 at 04:03:12PM +0200, Michael S. Tsirkin wrote:
> > > On Wed, Jan 31, 2018 at 05:28:35PM +0800, Peter Xu wrote:
> > > > Hi, Michael and the list,
> > > > 
> > > > I observed this on windows 8 enterprise guests, when doing memory ballooning:
> > > > 
> > > > 23892@1517298572.328354:virtio_balloon_to_target balloon target: 0x80000000 num_pages: 524288
> > > > 23892@1517298638.542819:virtio_balloon_get_config num_pages: 524288 actual: 0
> > > > 23892@1517298638.542974:virtio_balloon_handle_output section name: pc.ram gpa: 0x174604000
> > > > 23892@1517298638.543059:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > 23892@1517298638.543135:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460a000
> > > > 23892@1517298638.543140:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > 23892@1517298638.543143:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460b000
> > > > 23892@1517298638.543146:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > 23892@1517298638.543148:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460c000
> > > > 23892@1517298638.543152:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > 23892@1517298638.543154:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460d000
> > > > 23892@1517298638.543159:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > 23892@1517298638.543162:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460e000
> > > > 23892@1517298638.543165:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > 23892@1517298638.543167:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460f000
> > > > 23892@1517298638.543170:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > ...
> > > > 
> > > > I think it's very possible that these zero addresses (please let me
> > > > know what the first 4K page is used for if anyone knows, since IIUC
> > > > that's what we throw away now) are half of the 64bit PFN.  Or say, not
> > > > sure whether this means a windows guest driver bug that is using
> > > > 64bits for PFN rather than 32bits (and I suppose the protocol is using
> > > > 32bit for PFNs).
> > > > 
> > > > Michael, do you know what to do with this?
> > > > 
> > > > Thanks,
> > > 
> > > PFN is GPA>>12.  Do you have more than 1<<44 bytes of memory in this VM then?
> > 
> > No.  But isn't it still not good to drop the page at offset zero (and
> > drop it NNN times)?
> 
> Absolutely - looks like a bug. I just don't know why does this happen.

IMHO if we are using a PFN array like this:

   u64 pfn_array[];

In the windows guest driver, then we'll see this (as mentioned
above).  But for sure this is wild guess of mine.

> 
> >  And I'm not sure what will happen if guest has
> > 1<<44 bytes; then we'll possibly drop very random addresses since a
> > real address will be splitted?
> > 
> > Thanks,
> 
> The balloon won't work, period. There was an interface change to fix
> that but implementation issues remained and contributor seems to be busy
> with page hints.

Okay.  So IIUC this is already a known issue for the driver owner.
Then it seems that there's nothing more I can do for now...

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] Windows balloon driver PFN issue

[Michael S. Tsirkin]

On Thu, Feb 01, 2018 at 10:33:50AM +0800, Peter Xu wrote:
> On Thu, Feb 01, 2018 at 04:24:40AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Feb 01, 2018 at 10:18:53AM +0800, Peter Xu wrote:
> > > On Wed, Jan 31, 2018 at 04:03:12PM +0200, Michael S. Tsirkin wrote:
> > > > On Wed, Jan 31, 2018 at 05:28:35PM +0800, Peter Xu wrote:
> > > > > Hi, Michael and the list,
> > > > > 
> > > > > I observed this on windows 8 enterprise guests, when doing memory ballooning:
> > > > > 
> > > > > 23892@1517298572.328354:virtio_balloon_to_target balloon target: 0x80000000 num_pages: 524288
> > > > > 23892@1517298638.542819:virtio_balloon_get_config num_pages: 524288 actual: 0
> > > > > 23892@1517298638.542974:virtio_balloon_handle_output section name: pc.ram gpa: 0x174604000
> > > > > 23892@1517298638.543059:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > > 23892@1517298638.543135:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460a000
> > > > > 23892@1517298638.543140:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > > 23892@1517298638.543143:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460b000
> > > > > 23892@1517298638.543146:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > > 23892@1517298638.543148:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460c000
> > > > > 23892@1517298638.543152:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > > 23892@1517298638.543154:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460d000
> > > > > 23892@1517298638.543159:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > > 23892@1517298638.543162:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460e000
> > > > > 23892@1517298638.543165:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > > 23892@1517298638.543167:virtio_balloon_handle_output section name: pc.ram gpa: 0x17460f000
> > > > > 23892@1517298638.543170:virtio_balloon_handle_output section name: pc.ram gpa: 0x0
> > > > > ...
> > > > > 
> > > > > I think it's very possible that these zero addresses (please let me
> > > > > know what the first 4K page is used for if anyone knows, since IIUC
> > > > > that's what we throw away now) are half of the 64bit PFN.  Or say, not
> > > > > sure whether this means a windows guest driver bug that is using
> > > > > 64bits for PFN rather than 32bits (and I suppose the protocol is using
> > > > > 32bit for PFNs).
> > > > > 
> > > > > Michael, do you know what to do with this?
> > > > > 
> > > > > Thanks,
> > > > 
> > > > PFN is GPA>>12.  Do you have more than 1<<44 bytes of memory in this VM then?
> > > 
> > > No.  But isn't it still not good to drop the page at offset zero (and
> > > drop it NNN times)?
> > 
> > Absolutely - looks like a bug. I just don't know why does this happen.
> 
> IMHO if we are using a PFN array like this:
> 
>    u64 pfn_array[];
> 
> In the windows guest driver, then we'll see this (as mentioned
> above).  But for sure this is wild guess of mine.

I don't see code like this anywhere in the windows balloon
driver. It's here:
https://github.com/virtio-win/kvm-guest-drivers-windows.git

> > 
> > >  And I'm not sure what will happen if guest has
> > > 1<<44 bytes; then we'll possibly drop very random addresses since a
> > > real address will be splitted?
> > > 
> > > Thanks,
> > 
> > The balloon won't work, period. There was an interface change to fix
> > that but implementation issues remained and contributor seems to be busy
> > with page hints.
> 
> Okay.  So IIUC this is already a known issue for the driver owner.
> Then it seems that there's nothing more I can do for now...
> 
> Thanks,
> 
> -- 
> Peter Xu

Re: [Qemu-devel] Windows balloon driver PFN issue

[Peter Xu]

On Thu, Feb 01, 2018 at 02:48:20PM +0200, Michael S. Tsirkin wrote:

[...]

> > > > > PFN is GPA>>12.  Do you have more than 1<<44 bytes of memory in this VM then?
> > > > 
> > > > No.  But isn't it still not good to drop the page at offset zero (and
> > > > drop it NNN times)?
> > > 
> > > Absolutely - looks like a bug. I just don't know why does this happen.
> > 
> > IMHO if we are using a PFN array like this:
> > 
> >    u64 pfn_array[];
> > 
> > In the windows guest driver, then we'll see this (as mentioned
> > above).  But for sure this is wild guess of mine.
> 
> I don't see code like this anywhere in the windows balloon
> driver. It's here:
> https://github.com/virtio-win/kvm-guest-drivers-windows.git

Thanks for the pointer.  I had a quick glance, the PFN array is
defined as:

    PPFN_NUMBER             pfns_table;

But I don't know what's sizeof(PPFN_NUMBER). :(

-- 
Peter Xu

Re: [Qemu-devel] Windows balloon driver PFN issue

[Peter Xu]

On Mon, Feb 26, 2018 at 10:54:11AM +0200, Gal Hammer wrote:
> Hi Peter,
> 
> On Fri, Feb 2, 2018 at 12:11 PM, Peter Xu <peterx@redhat.com> wrote:
> > On Thu, Feb 01, 2018 at 02:48:20PM +0200, Michael S. Tsirkin wrote:
> >
> > [...]
> >
> >> > > > > PFN is GPA>>12.  Do you have more than 1<<44 bytes of memory in this VM then?
> >> > > >
> >> > > > No.  But isn't it still not good to drop the page at offset zero (and
> >> > > > drop it NNN times)?
> >> > >
> >> > > Absolutely - looks like a bug. I just don't know why does this happen.
> >> >
> >> > IMHO if we are using a PFN array like this:
> >> >
> >> >    u64 pfn_array[];
> >> >
> >> > In the windows guest driver, then we'll see this (as mentioned
> >> > above).  But for sure this is wild guess of mine.
> >>
> >> I don't see code like this anywhere in the windows balloon
> >> driver. It's here:
> >> https://github.com/virtio-win/kvm-guest-drivers-windows.git
> >
> > Thanks for the pointer.  I had a quick glance, the PFN array is
> > defined as:
> >
> >     PPFN_NUMBER             pfns_table;
> >
> > But I don't know what's sizeof(PPFN_NUMBER). :(
> 
> sizeof(PPFN_NUMBER) = sizeof(void*)
> 
> PFN_NUMBER is of an unsigned long type. Although it doesn't matter, as
> a pointer to it is always the same size, 4 bytes in a 32-bit CPU and 8
> bytes in 64-bit one.

Ah, it's not really PPFN_NUMBER that matters, it should be
PFN_NUMBER.  This is how the PFNs are copied in windows driver:

    RtlCopyMemory(ctx->pfns_table, MmGetMdlPfnArray(pPageMdl),
        ctx->num_pfns * sizeof(PFN_NUMBER));

I don't know these APIs, but it looks like MmGetMdlPfnArray() is
returning an PFN_NUMBER array.

And I don't know how the balloon spec says, but in QEMU it's always
using uint32_t as PFN.  See virtio_balloon_handle_output():

    while (iov_to_buf(elem->out_sg, elem->out_num, offset, &pfn, 4) == 4)

So I guess only if sizeof(PFN_NUMBER)==4 is true on both 32/64 bits
platforms of windows, otherwise there might be a problem.

Thanks,

-- 
Peter Xu