page refcount on pages mapped by the toolstack..

page refcount on pages mapped by the toolstack..

[Mukesh Rathor]

Hi,

I observed something that doesn't seem right to me:

PV dom0 booting PV guest (say domid 1). (no PVH).

xl cr vm.cfg.pv

Take some mfn from domid 1. It's refcnt is 1 as expected. Now, the lib
wants to map it via xen_remap_domain_mfn_range(). The call goes thru
do_mmu_update(), and upon returning the refcnt is 2, as expected.

Now, I noticed the refcnt doesn't go back to 1 after the guest is 
created/booted. I'd have expected the process exit somewhere to have
resulted in the refcnt going down to 1 (which is what would happen in case
of PVH dom0).

The guest is up, I notice the refcnt is 2. I shutdown the guest, the
refcnt goes to 0 and the page is freed via relinquish_memory() called
from domain_relinquish_resources(). I would have expected the page
to hang with refcnt 1, what if the user process still has it mapped?

What am I missing?

Thanks,
Mukesh

Re: page refcount on pages mapped by the toolstack..

[Ian Campbell]

On Thu, 2012-12-06 at 01:41 +0000, Mukesh Rathor wrote:
> Hi,
> 
> I observed something that doesn't seem right to me:
> 
> PV dom0 booting PV guest (say domid 1). (no PVH).
> 
> xl cr vm.cfg.pv
> 
> Take some mfn from domid 1. It's refcnt is 1 as expected.

Any arbitrary mfn or some particular mfn?

>  Now, the lib
> wants to map it via xen_remap_domain_mfn_range(). The call goes thru
> do_mmu_update(), and upon returning the refcnt is 2, as expected.
> 
> Now, I noticed the refcnt doesn't go back to 1 after the guest is 
> created/booted. I'd have expected the process exit somewhere to have
> resulted in the refcnt going down to 1 (which is what would happen in case
> of PVH dom0).

Which process exit? xl will daemonize and keep running in the
background. I'm not sure which pages it will keep mapped, might it
actually be xenstored or something similar which has the extra
reference?

Do privcmd mappings show up in /proc/<pid>map?

> The guest is up, I notice the refcnt is 2. I shutdown the guest, the
> refcnt goes to 0 and the page is freed via relinquish_memory() called
> from domain_relinquish_resources(). I would have expected the page
> to hang with refcnt 1, what if the user process still has it mapped?
> 
> What am I missing?

Doesn't the user process exit when the domain shuts down, thereby
releasing the other mapping?

Ian.

Re: page refcount on pages mapped by the toolstack..

[Jan Beulich]

>>> On 06.12.12 at 02:41, Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> Hi,
> 
> I observed something that doesn't seem right to me:
> 
> PV dom0 booting PV guest (say domid 1). (no PVH).
> 
> xl cr vm.cfg.pv
> 
> Take some mfn from domid 1. It's refcnt is 1 as expected. Now, the lib
> wants to map it via xen_remap_domain_mfn_range(). The call goes thru
> do_mmu_update(), and upon returning the refcnt is 2, as expected.
> 
> Now, I noticed the refcnt doesn't go back to 1 after the guest is 
> created/booted. I'd have expected the process exit somewhere to have
> resulted in the refcnt going down to 1 (which is what would happen in case
> of PVH dom0).
> 
> The guest is up, I notice the refcnt is 2. I shutdown the guest, the
> refcnt goes to 0 and the page is freed via relinquish_memory() called
> from domain_relinquish_resources(). I would have expected the page
> to hang with refcnt 1, what if the user process still has it mapped?
> 
> What am I missing?

Did you perhaps not monitor the changes to the refcnt closely
enough? It ought to be 2 when the guest is up (on reference for
the _PGC_allocated bit, and another for it to be mapped
somewhere in the guest). I.e. between Dom0 creating the guest
(and touching its memory) and the guest actually starting, there
could be further adjustments to the refcnt that simply sum up to
zero.

Jan

Re: page refcount on pages mapped by the toolstack..

[Mukesh Rathor]

On Thu, 06 Dec 2012 10:49:54 +0000
"Jan Beulich" <JBeulich@suse.com> wrote:
> 
> Did you perhaps not monitor the changes to the refcnt closely
> enough? It ought to be 2 when the guest is up (on reference for
> the _PGC_allocated bit, and another for it to be mapped
> somewhere in the guest). I.e. between Dom0 creating the guest
> (and touching its memory) and the guest actually starting, there
> could be further adjustments to the refcnt that simply sum up to
> zero.

Correct, it's 2 when the guest is up. Then relinquish_memory() seems to
bring it to 0 when domain destroy is called. 

For PVH, I somehow end up with 3 when guest is up, so when guest is destroyed
the mfn's are around with refcnt of 1. I get refcnt on the page in xen
when doing xen_remap_domain_mfn_range() in dom0. I hold on to the refcnt
until xen_unmap_domain_mfn_range(). Unmap results in call to 
XENMEM_remove_from_physmap where I do put_page if it's not a grant page
and is from foreign domain....  Debugging that right now.

Thanks for the help,
Mukesh