Odd mapping behavior with map_pages_to_xen

Odd mapping behavior with map_pages_to_xen

[Arne Mejlholm]

Dear all,

I'm experiencing some odd behavior when using the map_pages_to_xen 
function in arch/x86/mm.c.

The set up is as follows:

1. I allocate an entire page and get its virtual address.
2. For each machine frame number (mfn) I map the mfn onto the virtual 
address acquired in step 1.
        a. I read the contents of the page.
        b. If softirqs are pending, exit and do_softirq(); else continue 
with next mfn.

most of the time this procedure works just fine, but once in a while I 
get odd results. In one iteration all the
pages are reported to contain only zeros. This of course cannot be true.

Perhaps it should be noted that step 2 is carried out from within the 
idle_loop. Although it seems safe to
exploit this idle state, could there be some assumptions that does not 
hold? Could it be that for reasons not
evident to me, mapping pages into Xen during this idle time, is a bad 
idea and not safe?

About mapping pages into Xen, if I understand it correctly, Xen's 64MB 
address space is mapped into every
page table for every process in every domain, so the virtual address 
allocated in step 1 should be accessible
in any context. Is this not the case?

Furthermore there seems to be a direct connection with whether the tls 
libraries are disabled or not. When they are not
disabled, the odd results seems to occur more frequently.

Thank you in advance,
Arne Mejlholm

Re: Odd mapping behavior with map_pages_to_xen

[Keir Fraser]

On 16 Mar 2006, at 14:24, Arne Mejlholm wrote:

> Perhaps it should be noted that step 2 is carried out from within the 
> idle_loop. Although it seems safe to
> exploit this idle state, could there be some assumptions that does not 
> hold? Could it be that for reasons not
> evident to me, mapping pages into Xen during this idle time, is a bad 
> idea and not safe?
>
> About mapping pages into Xen, if I understand it correctly, Xen's 64MB 
> address space is mapped into every
> page table for every process in every domain, so the virtual address 
> allocated in step 1 should be accessible
> in any context. Is this not the case?

Unfortunately you'll only be modifying the idle page table mappings. So 
if you are running on someone else's tables then you won't see the new 
mapping. This can easily happen even in the idle loop, because we 
lazily switch pagetables.

In fact, it would work if the xen heap were mapped with 4kB pages, but 
by default we map with 4MB superpages. So when you change a single 4kB 
mapping the superpage mapping needs replacing with a mapping of a new 
pagetable in the idle pgdir, and that update isn;t  reflected in the 
current running pagetables.

If you build Xen with 'debug=y' option then I expect you'll find your 
code works fine. :-)

  -- Keir

Re: Odd mapping behavior with map_pages_to_xen

[Arne Mejlholm]

First of all, thanks for your fast reply :)

Keir Fraser wrote:

>
> Unfortunately you'll only be modifying the idle page table mappings. 
> So if you are running on someone else's tables then you won't see the 
> new mapping. This can easily happen even in the idle loop, because we 
> lazily switch pagetables.

so when the problem appears, we are actually looking at another domains 
page tables or and old superpage mapping. That is a quite important 
detail :)

> In fact, it would work if the xen heap were mapped with 4kB pages, but 
> by default we map with 4MB superpages. So when you change a single 4kB 
> mapping the superpage mapping needs replacing with a mapping of a new 
> pagetable in the idle pgdir, and that update isn;t  reflected in the 
> current running pagetables.

so are these superpage mappings only done in the idle domain? Is the 
reason for using superpages in the idle domain to avoid flushing the TLB 
when running the idle loop or something clever like that?

> If you build Xen with 'debug=y' option then I expect you'll find your 
> code works fine. :-)

You expected correctly indeed. I take it that enabling debug either 
forces the mappings to be replaced (non-lazily) or does it switch 
entirely to 4kb pages?

Re: Odd mapping behavior with map_pages_to_xen

[Keir Fraser]

On 16 Mar 2006, at 19:05, Arne Mejlholm wrote:

>> If you build Xen with 'debug=y' option then I expect you'll find your 
>> code works fine. :-)
>
> You expected correctly indeed. I take it that enabling debug either 
> forces the mappings to be replaced (non-lazily) or does it switch 
> entirely to 4kb pages?

There is no code to replace the mappings in non-idle pagetables. So 
it's the latter -- to support heap guard pages, we map the entire Xen 
heap with 4kB pages in debug builds.

In non-debug builds the superpages are used in all pagetables to map 
Xen heap. It's to improve TLB utilisation.

  -- Keir