xenpaing: one way to avoid paging out the page, when the corresponding mfn is in use.

xenpaing: one way to avoid paging out the page, when the corresponding mfn is in use.

[Hongkaixing]

Hello,

   Recently many advanced memory mechanisms are introduced into Xen. One problem we found is the conflict between p2m query and setting.
   For example, backend drivers always map domU’s page to its own space, during the mapping procedure, situations as follow may happen, 
when mfn is obtained by gfn_to_mfn(), this mfn is likely to be paged out.

first case:
   grant mapping                      xenpaing
  mfn = gfn_to_mfn();
                       <-----------  p2m_paging_nominate()
         |                                         |
     Check type ok                     paged out;
         |
     try to map
What we want is: 
When the page (mfn) is accessed by gfn_to_mfn(), this page should never be paged out until the mapping action is end. 

second case:
   grant mapping                            xenpaing 
                                         p2m_paging_nominate()
                                     
                                                 gfn_to_mfn();    
  mfn = gfn_to_mfn();  ------------->     |  
                                             check type ok
         |                                               |
     Check type ok                     paged out;
         |
     try to map
What we want is:
When the gfn_to_mfn() action happens during paging nomination, the nomination should abort immediately.

Our solution prototype is like this :
1. Introduce a new member named last_access in page_info struct to save the last access time and access tag.
2. when the mfn is obtained through gfn_to_mfn(), we save time stamp and access tag in the page_info.
3. Paging nominate procedure use access information as a criterion.

How it works? 
1.Using time stamp to avoid case 1. When the mfn is obtained by mapping process, 
   the time stamp can prevent the page from being selected by paging .
2.Using access tag to avoid case 2. During the paging nomination, if the access tag of page is detected, 
   paging should skip selecting this page. 

The pseudo-code of step 3 can be written as follow:
int p2m_mem_paging_nominate(struct domain *d, unsigned long gfn)
{
   
    mfn = gfn_to_mfn_noreference(d, gfn, &p2mt);   -----> avoid saving timestamp and access tag 
    
    if ( !mfn_valid(mfn) )
        goto out;
    
    clear_access_tag();   ----------> clear the access tag of this page
    if (test_page_hot())
       goto out;           ------> if the page is accessed recently, go to out
    ........
  
    set_p2m_entry(d, gfn, mfn, 0, p2m_ram_paging_out);
    if ( test_access_tag ( mfn ) )
        goto out;  --------> if access tag is set, the gfn_to_mfn must have happened above, abort anyway.
    ret = 0;
 out:
    p2m_unlock(d->arch.p2m);
    return ret;
}
   Maybe this is an imperfect prototype, do you have any good ideas?

  Hong Kaixing
 

Re: xenpaing: one way to avoid paging out the page, when the corresponding mfn is in use.

[Olaf Hering]

On Tue, Nov 01, Hongkaixing wrote:

>    Recently many advanced memory mechanisms are introduced into Xen.
>    One problem we found is the conflict between p2m query and setting.
>    For example, backend drivers always map domU’s page to its own
>    space, during the mapping procedure, situations as follow may
>    happen, when mfn is obtained by gfn_to_mfn(), this mfn is likely to
>    be paged out.
> 
> first case:
>    grant mapping                      xenpaing
>   mfn = gfn_to_mfn();
>                        <-----------  p2m_paging_nominate()
>          |                                         |
>      Check type ok                     paged out;
>          |
>      try to map
> What we want is: 
> When the page (mfn) is accessed by gfn_to_mfn(), this page should
> never be paged out until the mapping action is end. 

The query+map and query+change opterations on p2m entries should be done
under some lock. I have recently updated the p2m_mem_paging* functions
to do their modifications with the p2m_lock held.

Furthermore the change below checks wether something mapped a page
between nominate and evict:

http://xenbits.xen.org/hg/xen-unstable.hg/rev/eda18b27de6e

While this certainly does not fix all possible races, it slightly
reduces the window.


A more complete approach to add locking around p2m modifcations was
recently started by Andres Lagar-Cavilla:

http://lists.xensource.com/archives/html/xen-devel/2011-10/msg01946.html

Olaf

Re: xenpaing: one way to avoid paging out the page, when the corresponding mfn is in use.

[andres]

Page_info structs are carefully laid out to occupy 64 bytes (I think -- or
some other power of two cache-line-friendly thing).

Were you thinking of this new field to be part of a union? which one?
Otherwise you change the size of page_info and hell breaks loose.

The RFC patches I posted a week ago (which are not ready for primetime)
try to ensure serialization of lookups and modification of p2m entries.
That should solve the issue of paging a foreign-mapped page.

Actually the patches attempt to achieve what you state ("When the page
(mfn) is accessed by gfn_to_mfn(), this page should never be paged out
until the mapping action is end"). And hopefully in a much more general
way.

So, stay posted...

Andres

> Date: Tue, 01 Nov 2011 12:23:37 +0000
> From: Hongkaixing <hongkaixing@huawei.com>
> Subject: [Xen-devel] xenpaing: one way to avoid paging out the page,
> 	when the corresponding mfn is in use.
> To: "olaf@aepfle.de" <olaf@aepfle.de>
> Cc: YangXiaowei <xiaowei.yang@huawei.com>,
> 	"Xen-devel@lists.xensource.com" <Xen-devel@lists.xensource.com>,	"Eric
> 	Li\(Zhentao\)" <lizhentao@huawei.com>,	Yanqiangjun
> 	<yanqiangjun@huawei.com>, hanweidong <hanweidong@huawei.com>
> Message-ID:
> 	<E0AF011477D2AE458DC8801E0E570709014C0556@szxeml528-mbs.china.huawei.com>
>
> Content-Type: text/plain; charset=Windows-1252
>
> Hello,
>
>    Recently many advanced memory mechanisms are introduced into Xen. One
> problem we found is the conflict between p2m query and setting.
>    For example, backend drivers always map domU’s page to its own space,
> during the mapping procedure, situations as follow may happen,
> when mfn is obtained by gfn_to_mfn(), this mfn is likely to be paged out.
>
> first case:
>    grant mapping                      xenpaing
>   mfn = gfn_to_mfn();
>                        <-----------  p2m_paging_nominate()
>          |                                         |
>      Check type ok                     paged out;
>          |
>      try to map
> What we want is:
> When the page (mfn) is accessed by gfn_to_mfn(), this page should never be
> paged out until the mapping action is end.
>
> second case:
>    grant mapping                            xenpaing
>                                          p2m_paging_nominate()
>
>                                                  gfn_to_mfn();
>   mfn = gfn_to_mfn();  ------------->     |
>                                              check type ok
>          |                                               |
>      Check type ok                     paged out;
>          |
>      try to map
> What we want is:
> When the gfn_to_mfn() action happens during paging nomination, the
> nomination should abort immediately.
>
> Our solution prototype is like this :
> 1. Introduce a new member named last_access in page_info struct to save
> the last access time and access tag.
> 2. when the mfn is obtained through gfn_to_mfn(), we save time stamp and
> access tag in the page_info.
> 3. Paging nominate procedure use access information as a criterion.
>
> How it works?
> 1.Using time stamp to avoid case 1. When the mfn is obtained by mapping
> process,
>    the time stamp can prevent the page from being selected by paging .
> 2.Using access tag to avoid case 2. During the paging nomination, if the
> access tag of page is detected,
>    paging should skip selecting this page.
>
> The pseudo-code of step 3 can be written as follow:
> int p2m_mem_paging_nominate(struct domain *d, unsigned long gfn)
> {
>
>     mfn = gfn_to_mfn_noreference(d, gfn, &p2mt);   -----> avoid saving
> timestamp and access tag
>
>     if ( !mfn_valid(mfn) )
>         goto out;
>
>     clear_access_tag();   ----------> clear the access tag of this page
>     if (test_page_hot())
>        goto out;           ------> if the page is accessed recently, go to
> out
>     ........
>
>     set_p2m_entry(d, gfn, mfn, 0, p2m_ram_paging_out);
>     if ( test_access_tag ( mfn ) )
>         goto out;  --------> if access tag is set, the gfn_to_mfn must
> have happened above, abort anyway.
>     ret = 0;
>  out:
>     p2m_unlock(d->arch.p2m);
>     return ret;
> }
>    Maybe this is an imperfect prototype, do you have any good ideas?
>
>   Hong Kaixing
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 01 Nov 2011 13:47:15 +0100
> From: Tobias Heinlein <heinlein@okit.de>
> Subject: Re: [Xen-devel] Re: Xen dom0 linux kernel 3.1 boot failure
> 	ptwr_emulate: could not get_page_from_l1e
> To: xen-devel@lists.xensource.com
> Cc: 2013pfoley@tjhsst.edu
> Message-ID: <4EAFEA53.7070009@okit.de>
> Content-Type: text/plain; charset="iso-8859-1"
>
> I'm not sure if it was obvious, but yesterday I noticed that setting the
> "MPS table mode" to 'Disabled' actually made SMP stop working, i.e. the
> kernel only recognized a single CPU. This is of course not an option, so
> I enabled (set to 'Full Table APIC') the setting again and played around
> with my kernel config a bit. The kernel that crashed had
> CONFIG_X86_MPPARSE=y, and if I disable that, it boots fine (with SMP,
> and with the BIOS setting set to 'Full Table APIC').
>
> So, I for one am quite happy now as I finally found a working
> configuration. But I'd still like to know if this is a hardware-specific
> issue, and/or a bug in Xen.
>
> Konrad Rzeszutek Wilk wrote, on 10/31/2011 03:08 PM:
>> Oh nice. What does you /proc/interrupts look like compared to
>> baremetal?
>
> While I was performing all my kernel tests, I saved the outputs of
> `dmesg` and `cat /proc/interrupts`. Sorry for attaching a tarball, but
> I'd like to give you as much information as possible. You'll probably
> only need the latest tests (#5 to #7), but just in case, I also included
> the others.
>
> Contents of the tarball:
>
> Baremetal tests:
> xen-hp/1/: MPS mode 'Full Table APIC', CONFIG_X86_MPPARSE=y, SMP working
> xen-hp/2/: MPS mode 'Full Table APIC', CONFIG_X86_MPPARSE=n, SMP working
> xen-hp/3/: MPS mode 'Disabled', CONFIG_X86_MPPARSE=y, SMP not working
> xen-hp/4/: MPS mode 'Disabled', CONFIG_X86_MPPARSE=n, SMP not working
>
> Xen tests:
> xen-hp/5/: MPS mode 'Disabled', CONFIG_X86_MPPARSE=n, SMP not working
> xen-hp/6/: MPS mode 'Full Table APIC', CONFIG_X86_MPPARSE=n, SMP working
> xen-hp/7/: MPS mode 'Full Table APIC', CONFIG_X86_MPPARSE=y, CRASHES
>
> (Therefore, #6 is the best working solution; #7 is what originally
> triggered the crash.)
>
> Thanks.
>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: xen-hp.tar.bz2
> Type: application/x-bzip
> Size: 97821 bytes
> Desc: not available
> Url :
> http://lists.xensource.com/archives/html/xen-devel/attachments/20111101/4f96b9e7/xen-hp.tar.bin
>
> ------------------------------
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>
> End of Xen-devel Digest, Vol 81, Issue 2
> ****************************************
>