Re: [PATCH 07/16] xenpaging: populate only paged-out pages

[Olaf Hering <olaf@aepfle.de>]

On Wed, Nov 03, Ian Jackson wrote:

> Olaf Hering writes ("[Xen-devel] [PATCH 07/16] xenpaging: populate only paged-out pages"):
> > populdate a paged-out page only once to reduce pressure in the ringbuffer.
> > Several cpus may still request a page at once. xenpaging can handle this.
> >
> > But: maybe this will miss pages in paging-out state?
> 
> I can't answer your second question, but surely this pattern:
> 
> > -        p2m_mem_paging_populate(p2m, ram_gfn);
> > +        if ( p2m_is_paged(p2mt) )
> > +            p2m_mem_paging_populate(p2m, ram_gfn);
> 
> would better be done inside p2m_mem_paging_populate ?  Unless there
> are some calls to _populate which should skip the check ?

Ian,

thanks for bringing this up. After revisiting this patch and the reasons
for it, two different things need changing.

One thing is that p2m_mem_paging_populate needs to be called
unconditionally. The reason is that the vcpu may need stopping and
resuming until the page content is usable again.
A change for grant_table.c:__get_paged_frame is required.

The other thing is that p2m_mem_paging_populate must only invalidate the
mfn if either the page is still in the process of being paged out, or if
the page is already in the process of being paged in. The latter does
not happen right now. The result is that the page may have state
p2m_ram_paging_in and a new valid mfn. Then that new mfn is invalidated,
the gfn content in the xenpaging process was already overwritten and
there is no wakeup of the vcpu. This situation is not recoverable.
p2m.c:p2m_mem_paging_populate needs a change to no call set_p2m_entry if
the p2mt state is already p2m_ram_paging_in

I will send two new patches for grant_table.c and p2m.c

Olaf

PS:

Below is a custom xenalyze debug log for the this. It translates to:
At some point gfn 3bdb7 was paged out, and later requested on vcpu 0. It
got a new mfn 1120b9 in p2m_mem_paging_prep on vcpu 5.
Later vcpu 3 called p2m_mem_paging_populate again, and it reset the new
mfn 1120b9. The xenpaging binary noticed that gfn 3bdb7 was already
populated and therefore could not prep/resume this gfn.
However, the first p2m_mem_paging_resume call ran into the new invalid
mfn and could not set the gfn state back to p2m_ram_rw. I have a change
to check the mfn in p2m_mem_paging_resume.
If p2m_mem_paging_populate had checked the p2mt properly,
p2m_mem_paging_resume would have succeed and everything could proceed.

19699656: 2073.683567637 -x--|--- d0v3 olh p2m_mem_paging_nominate gfn 3bdb7 mfn f5b7 flags 0 p2mt 1 ret 0
19699705: 2073.683601641 -x--|--- d0v3 olh p2m_mem_paging_evict gfn 3bdb7 mfn f5b7 flags 41 p2mt 9 ret 0
24430211: 2107.837821159 x-|||--| d0v0 olh p2m_mem_paging_populate_trace gfn 3bdb7 mfn ffffffff p2mt a line 159
24430212: 2107.837822145 x-|||--| d0v0 olh p2m_mem_paging_populate gfn 3bdb7 mfn ffffffff flags 20 p2mt a vcpu 0
24434493: 2107.845357003 |-x--||| d0v5 olh p2m_mem_paging_prep gfn 3bdb7 mfn 1120b9 ret 0
24434542: 2107.845409334 |-|--||x d0v3 olh p2m_mem_paging_populate_trace gfn 3bdb7 mfn 1120b9 p2mt b line 159
24434544: 2107.845411875 |-|--||x d0v3 olh p2m_mem_paging_populate gfn 3bdb7 mfn 1120b9 flags 20 p2mt b vcpu 3
24435286: 2107.856054450 |||--||x d0v3 olh p2m_mem_paging_populate_trace gfn 3bdb7 mfn ffffffff p2mt c line 159
24435287: 2107.856056334 |||--||x d0v3 olh p2m_mem_paging_populate gfn 3bdb7 mfn ffffffff flags 20 p2mt c vcpu 3
24435461: 2107.860575628 --x--||- d0v5 olh p2m_mem_paging_resume gfn 3bdb7 mfn ffffffff flags 0 p2mt c
24439829: 2107.864952416 -|--|||x d0v3 olh p2m_mem_paging_populate_trace gfn 3bdb7 mfn ffffffff p2mt c line 159
24439840: 2107.864955042 -|--|||x d0v3 olh p2m_mem_paging_populate gfn 3bdb7 mfn ffffffff flags 20 p2mt c vcpu 3
24449379: 2107.873837210 --|---|x d0v3 olh p2m_mem_paging_populate_trace gfn 3bdb7 mfn ffffffff p2mt c line 159
24449389: 2107.873839448 --|----x d0v3 olh p2m_mem_paging_populate gfn 3bdb7 mfn ffffffff flags 20 p2mt c vcpu 3
24455031: 2107.882722451 ---|-|-x d0v3 olh p2m_mem_paging_populate_trace gfn 3bdb7 mfn ffffffff p2mt c line 159
24455033: 2107.882724440 ---|-|-x d0v3 olh p2m_mem_paging_populate gfn 3bdb7 mfn ffffffff flags 20 p2mt c vcpu 3
24462725: 2107.896056767 -|---|-x d0v3 olh p2m_mem_paging_populate_trace gfn 3bdb7 mfn ffffffff p2mt c line 159
24462730: 2107.896059777 -|---|-x d0v3 olh p2m_mem_paging_populate gfn 3bdb7 mfn ffffffff flags 20 p2mt c vcpu 3
...