When would Xen reset page access rights for a non-migrating guest?

When would Xen reset page access rights for a non-migrating guest?

[Razvan Cojocaru]

Hello,

I have an application that uses libxc (specifically
xc_hvm_set_mem_access()) to protect a bunch of interesting pages - I'm
interested in being notified when writes to these pages happen (via
the mem_event mechanism).

However, it seems that once in a great while, after setting access
rights, something happens that allows unnotified writes to these
pages. Working under the assumption that it might have something to do
the the live-migration mechanism, I've set "nomigrate = 1" in the
guest's configuration file. When that did not work, I even called
xc_domain_disable_migrate() from my application, again to no avail.

I also though it might have something to do with balooning, but the
only memory-related keyword in my guest's configuration file is
"memory = 2048" (no maximum memory specified that would hint towards
).

It probably has something to do with the code in tmem.c
(save/restore). What could cause this?

Any thoughts appreciated. This is Xen 4.3.


Thanks,
Razvan

Re: When would Xen reset page access rights for a non-migrating guest?

[Tim Deegan]

Hi,

At 14:42 +0200 on 19 Mar (1395236568), Razvan Cojocaru wrote:
> I have an application that uses libxc (specifically
> xc_hvm_set_mem_access()) to protect a bunch of interesting pages - I'm
> interested in being notified when writes to these pages happen (via
> the mem_event mechanism).
> 
> However, it seems that once in a great while, after setting access
> rights, something happens that allows unnotified writes to these
> pages. Working under the assumption that it might have something to do
> the the live-migration mechanism, I've set "nomigrate = 1" in the
> guest's configuration file. When that did not work, I even called
> xc_domain_disable_migrate() from my application, again to no avail.

Is there some higher-level thing you can do to make sure the VM isn't
getting migrated/saved/restored?  Or conversely, have you any reason
to believe that that's what's happening?

> I also though it might have something to do with balooning, but the
> only memory-related keyword in my guest's configuration file is
> "memory = 2048" (no maximum memory specified that would hint towards
> ).

AFAIK the guest could still be ballooning; in any case the interaction
with ballooning seems like one you should test and fix. :)  It should
be mostly working already, I think: pages added during ballooning will
get the default mem-access settings (but will have undefined contents).

> It probably has something to do with the code in tmem.c
> (save/restore). What could cause this?

That's very unlikely unless you've specifically turned tmem on.

I think the most likely thing is some path that writes to guest memory
without being explicitly a CPU->RAM write from the guest.  E.g. a guest
with PV drivers making a hypercall that copies some results back, or
some emulated DMA, or the guest has granted write access to another VM.

Cheers,

Tim.

Re: When would Xen reset page access rights for a non-migrating guest?

[Razvan Cojocaru]

Hello Tim, thanks for answering!

>> However, it seems that once in a great while, after setting access
>> rights, something happens that allows unnotified writes to these
>> pages. Working under the assumption that it might have something to do
>> the the live-migration mechanism, I've set "nomigrate = 1" in the
>> guest's configuration file. When that did not work, I even called
>> xc_domain_disable_migrate() from my application, again to no avail.
> 
> Is there some higher-level thing you can do to make sure the VM isn't
> getting migrated/saved/restored?  Or conversely, have you any reason
> to believe that that's what's happening?

Well, I'm sure that the guest is not actually being migrated at the
time, and as I've said I've done my best to completely disable migration
for it, but I've ruled out everything else I could think of, and that's
what's left. I don't seem to be losing any events because of the
mem_event mechanism (if I understand it correctly, the VCPU is paused
until the event is handled, and also if the ring buffer is full), the
page access rights are properly being set.

Again, I'm open to (thankful for) suggestions.

>> I also though it might have something to do with balooning, but the
>> only memory-related keyword in my guest's configuration file is
>> "memory = 2048" (no maximum memory specified that would hint towards
>> ).
> 
> AFAIK the guest could still be ballooning; in any case the interaction
> with ballooning seems like one you should test and fix. :)  It should
> be mostly working already, I think: pages added during ballooning will
> get the default mem-access settings (but will have undefined contents).

I'll take a look there, thanks.

>> It probably has something to do with the code in tmem.c
>> (save/restore). What could cause this?
> 
> That's very unlikely unless you've specifically turned tmem on.

Ah, but I'm not using the vanilla Xen 4.3 release. I'm using a Citrix
XenServer development snapshot from late last year (back when they made
the source code freely available). It's got quite a few quirks of it's
own, and I'm afraid active tmem might be one of them.

Running xl does show tmem parameters:

# xl | grep tmem
  tmem-list           List tmem pools
  tmem-freeze         Freeze tmem pools
  tmem-thaw           Thaw tmem pools
  tmem-set            Change tmem settings
  tmem-shared-auth    De/authenticate shared tmem pool
  tmem-freeable       Get information about how much freeable memory
                      (MB) is in-use by tmem

> I think the most likely thing is some path that writes to guest memory
> without being explicitly a CPU->RAM write from the guest.  E.g. a guest
> with PV drivers making a hypercall that copies some results back, or
> some emulated DMA, or the guest has granted write access to another VM.

These were all Windows HVM guests. No PV or special drivers involved
whatsoever.


Thanks,
Razvan

Re: When would Xen reset page access rights for a non-migrating guest?

[Andrew Cooper]

On 20/03/14 18:13, Razvan Cojocaru wrote:
> Hello Tim, thanks for answering!
>
>>> However, it seems that once in a great while, after setting access
>>> rights, something happens that allows unnotified writes to these
>>> pages. Working under the assumption that it might have something to do
>>> the the live-migration mechanism, I've set "nomigrate = 1" in the
>>> guest's configuration file. When that did not work, I even called
>>> xc_domain_disable_migrate() from my application, again to no avail.
>> Is there some higher-level thing you can do to make sure the VM isn't
>> getting migrated/saved/restored?  Or conversely, have you any reason
>> to believe that that's what's happening?
> Well, I'm sure that the guest is not actually being migrated at the
> time, and as I've said I've done my best to completely disable migration
> for it, but I've ruled out everything else I could think of, and that's
> what's left. I don't seem to be losing any events because of the
> mem_event mechanism (if I understand it correctly, the VCPU is paused
> until the event is handled, and also if the ring buffer is full), the
> page access rights are properly being set.
>
> Again, I'm open to (thankful for) suggestions.
>
>>> I also though it might have something to do with balooning, but the
>>> only memory-related keyword in my guest's configuration file is
>>> "memory = 2048" (no maximum memory specified that would hint towards
>>> ).
>> AFAIK the guest could still be ballooning; in any case the interaction
>> with ballooning seems like one you should test and fix. :)  It should
>> be mostly working already, I think: pages added during ballooning will
>> get the default mem-access settings (but will have undefined contents).
> I'll take a look there, thanks.
>
>>> It probably has something to do with the code in tmem.c
>>> (save/restore). What could cause this?
>> That's very unlikely unless you've specifically turned tmem on.
> Ah, but I'm not using the vanilla Xen 4.3 release. I'm using a Citrix
> XenServer development snapshot from late last year (back when they made
> the source code freely available). It's got quite a few quirks of it's
> own, and I'm afraid active tmem might be one of them.

Tmem is not compiled into Xen for XenServer.

What sort of quirks are you on about?

>
> Running xl does show tmem parameters:
>
> # xl | grep tmem
>   tmem-list           List tmem pools
>   tmem-freeze         Freeze tmem pools
>   tmem-thaw           Thaw tmem pools
>   tmem-set            Change tmem settings
>   tmem-shared-auth    De/authenticate shared tmem pool
>   tmem-freeable       Get information about how much freeable memory
>                       (MB) is in-use by tmem

Trying to use any of those should result in an -ENOSYS.

>
>> I think the most likely thing is some path that writes to guest memory
>> without being explicitly a CPU->RAM write from the guest.  E.g. a guest
>> with PV drivers making a hypercall that copies some results back, or
>> some emulated DMA, or the guest has granted write access to another VM.
> These were all Windows HVM guests. No PV or special drivers involved
> whatsoever.
>
>
> Thanks,
> Razvan

Given the default setup, there will almost certainly be PoD, and
probably also be some Ballooning in there.

~Andrew

Re: When would Xen reset page access rights for a non-migrating guest?

[Razvan Cojocaru]

Hello,

> Tmem is not compiled into Xen for XenServer.

Well, it is compiled into Xen for XenServer:

$ find . -name "*tmem*.o"
./tools/libxc/xc_tmem.o
./xen/common/tmem.o
./xen/common/compat/tmem_xen.o
./xen/common/tmem_xen.o

Maybe it's not active.

> What sort of quirks are you on about?

You need to disable seabios for one. There are certain instructions that
are being emulated by the vanilla Xen but emulation code for them is
commented out with XenServer's Xen. I assume other things as well.

>> Running xl does show tmem parameters:
>>
>> # xl | grep tmem
>>   tmem-list           List tmem pools
>>   tmem-freeze         Freeze tmem pools
>>   tmem-thaw           Thaw tmem pools
>>   tmem-set            Change tmem settings
>>   tmem-shared-auth    De/authenticate shared tmem pool
>>   tmem-freeable       Get information about how much freeable memory
>>                       (MB) is in-use by tmem
> 
> Trying to use any of those should result in an -ENOSYS.

OK, I'm sure you're right. Will try it as soon as I get a chance.

> Given the default setup, there will almost certainly be PoD, and
> probably also be some Ballooning in there.

Sounds good, will look into that.


Thanks,
Razvan