Need advice on timed access to guest pages

Need advice on timed access to guest pages

[Arn]

Hi KVM devs,

I'm trying to write a guest kernel module (or modify kernel code) to
do the following :

Say I have a kvm guest with 512 MB RAM running on a host with 2 GB
RAM. Now I run a process on the host that will cause increasing memory
pressure which will in turn cause the host to swap out some of the
guests pages (without the guest knowing). My guest kernel module will
randomly access some of its physical pages (say read a byte from a
page) and time the accesses.

In theory an attempt by the guest to access a page that has been
swapped out by the host should take much longer than accessing a page
that is in memory.

Since I'm still a novice can someone advise/guide me on this and let
me know if I'm missing something crucial in my assumptions ? I'm using
an amd64 machine with host/guest kernel 2.6.22 and kvm-64

Thanks in advance
a. rn

Re: Need advice on timed access to guest pages

[Dor Laor]

On Wed, 2008-07-02 at 02:11 -0400, Arn wrote:
> Hi KVM devs,
> 
> I'm trying to write a guest kernel module (or modify kernel code) to
> do the following :
> 
> Say I have a kvm guest with 512 MB RAM running on a host with 2 GB
> RAM. Now I run a process on the host that will cause increasing memory
> pressure which will in turn cause the host to swap out some of the
> guests pages (without the guest knowing). My guest kernel module will
> randomly access some of its physical pages (say read a byte from a
> page) and time the accesses.
> 
> In theory an attempt by the guest to access a page that has been
> swapped out by the host should take much longer than accessing a page
> that is in memory.

Your assumption is correct for userspace allocation inside the guest.
Let me explain:

The current kvm without mmu-notifiers, can transparently swap guest
pages only if they are not pointed by the shadow page tables.
One a spte is instantiated the page cannot go to swap. mmu notifiers
will cancel this limitation and we'll get notifications about page
swapping from the host MM.

IIRC guest kernel allocations has the global bit set, thus we optimize
the shadow mmu not to flush global guest pages, so if this assumption is
correct, you won't feel the difference.

Anyway what's your motivation for this module?


> 
> Since I'm still a novice can someone advise/guide me on this and let
> me know if I'm missing something crucial in my assumptions ? I'm using
> an amd64 machine with host/guest kernel 2.6.22 and kvm-64
> 
> Thanks in advance
> a. rn
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Need advice on timed access to guest pages

[Arn]

On Wed, Jul 2, 2008 at 7:41 AM, Dor Laor <dor.laor@qumranet.com> wrote:

>> In theory an attempt by the guest to access a page that has been
>> swapped out by the host should take much longer than accessing a page
>> that is in memory.
>
> Your assumption is correct for userspace allocation inside the guest.
> Let me explain:
>
> The current kvm without mmu-notifiers, can transparently swap guest
> pages only if they are not pointed by the shadow page tables.
> One a spte is instantiated the page cannot go to swap. mmu notifiers
> will cancel this limitation and we'll get notifications about page
> swapping from the host MM.
>
> IIRC guest kernel allocations has the global bit set, thus we optimize
> the shadow mmu not to flush global guest pages, so if this assumption is
> correct, you won't feel the difference.

I didn't quite understand that last part. Does this mean that a
guest's pages can be swapped out by the host but if the guest kernel
tries to access that page it won't notice a difference. That seems a
bit counterintuitive to me.
Is there a difference when the guest kernel maps some of its processes
pages into its kernel space ? Do such pages get treated differently ?

Let me recap on what I expect in steps :

1) KVM Guest is running on host.
2) Host needs physical memory for other purposes so host swaps out
some of guests pages and hands them over to other process.
3) Guest is not aware that some of its pages are swapped out by host.
4) Now guest kernel maps some of these pages into its kernel space
(guest kernel space) (it still doesn't know that pages have been
swapped out)
5) Guest kernel then tries to read a byte from a swapped out page.
6) This causes some kind of trap/interception by host who swaps the
page in and hands control back to guest kernel.
7) Guest kernel then proceeds to read a byte from the page, only now
its taken longer to read the byte than it would have had the page not
been swapped out.

Basically, if a guest's pages have been swapped out by host and used
for another process, then when the guest kernel tries to access that
page, there will be some delay while the host handles the request.

Would appreciate further info on this.

> Anyway what's your motivation for this module?

Its something I'm working on on part of a larger project related to
resource allocation and guarantees to guest VMs.


Thanks
Arn

>> me know if I'm missing something crucial in my assumptions ? I'm using
>> an amd64 machine with host/guest kernel 2.6.22 and kvm-64

Re: Need advice on timed access to guest pages

[Sukanto Ghosh]

> The current kvm without mmu-notifiers, can transparently swap guest
> pages only if they are not pointed by the shadow page tables.
> One a spte is instantiated the page cannot go to swap.

Can you explain why spte's must be unmapped as well ?
The memory region occupied by the structures maintaining the spte's
(kvm_mmu_page) they are also part of Linux VM. So why won't they be
swapped by Linux VM.
I didn't get the problem behind secondary MMU.

( I was going through the slides of presentation "Integrating  with
the KVM Linux Memory Management" by Andrea Arcangeli : KVM Forum 2008
)



--
Regards,
Sukanto Ghosh

Re: Need advice on timed access to guest pages

[Andrea Arcangeli]

Hi Sukanto,

On Thu, Jul 03, 2008 at 08:34:58PM +0530, Sukanto Ghosh wrote:
> > The current kvm without mmu-notifiers, can transparently swap guest
> > pages only if they are not pointed by the shadow page tables.
> > One a spte is instantiated the page cannot go to swap.
> 
> Can you explain why spte's must be unmapped as well ?

Whenever a linux pagetable or a kvm spte is established, the page
mapped by them has its reference count increased by 1. That tells the
VM the page is referenced by someone and it prevents it to be swapped
out.

The page will look like it was swapped because the pte is successfully
removed, the page will go in swapcache, but it will remain in ram as
long as there's any spte still mapping it. mmu notifiers fixes this.

> The memory region occupied by the structures maintaining the spte's
> (kvm_mmu_page) they are also part of Linux VM. So why won't they be
> swapped by Linux VM.
> I didn't get the problem behind secondary MMU.

sptes, ptes and all kernel data structures are never swapped. Only the
userland memory (and in this case the guest physical ram) is entirely
swapped. And any sptes still mapping a swapcache page prevents the
swapcache to be written to disk and later freed.

It's similar to O_DIRECT, when you do zerocopy dma to a page, you
can't allow the page to be swapped out or the DMA will happen on freed
ram. The page pinning mechanism works fine for temporary pins like
O_DIRECT, but for sptes the pin can last forever if they're
successfully cached.

> ( I was going through the slides of presentation "Integrating  with
> the KVM Linux Memory Management" by Andrea Arcangeli : KVM Forum 2008
> )

You can check the picture, it shows the page being unmapped by the
linux pte when linux swaps, but its reference count being still pinned
if there are spte still mapping it.

It also shows that with mmu notifiers the sptes don't take refcount
anymore (or alternatively depending on the implementation the spte pin
is removed at the same time the linux pte is zapped) and that allows
the whole guest physical memory to be swapped as any other malloc.

Re: Need advice on timed access to guest pages

[Sukanto Ghosh]

Hi Andrea,

Thanks a lot :)

-- 
Regards,
Sukanto Ghosh