Idea: Small Address Spaces

Idea: Small Address Spaces

[Jacob Gorm Hansen]

hi,

in the cases where users (like me) wish to run Xen with MPI on Ethernets 
or similar, and don't care too much about driver isolation, I am 
thinking of trying to apply Jochen Liedtke's old 'small address spaces' 
hack, to see if I can improve domU I/O performance.

My idea is to reserve some additional virtual address space below Xen, 
e.g. at 0xF0000000, and map the kernel part of dom0* there permanently. 
The user space part of dom0 I would map as normal from 0 - 0xC0000000, 
to avoid relinking dom0 applications.  I would use the segments to keep 
the domUs below 0xF0000000. In this way, TLB flushes should only be 
necessary when dom0 exits to user space, not when handling interrupts or 
when domUs are asking for I/O.

Ideally, I would move dom0 to ring0 to save the cost of switching from 
Xen to dom0, but I have no idea how much trouble this will cause.

I have been working a bit on this, and so far I have only managed to 
move dom0 to its new location, and to map the 0xF0000000 - 0xFC000000 
range into the top of domU pgds instead of changing the cr3 (page table 
base) register. I have managed to break a lot of things, I see that as a 
good sign ;-)

I know this loses driver isolation, but I would argue that if it can be 
done relatively cleanly, and if it yields any performance improvements, 
a significant percentage of users would be willing to make that 
tradeoff.  Naturally, this would be a (compile or run-time) option, not 
the default behavior.

Comments are welcome, now is the time to yell stop ;-)

Jacob


* This could probably be made to work with multiple IDDs as well, 
treating the 0xF0000000 slot as a cache of the last run driver-domain 
rather than as a permanent mapping.

Re: Idea: Small Address Spaces

[Jacob Gorm Hansen]

Jacob Gorm Hansen wrote:
> hi,
> 
> in the cases where users (like me) wish to run Xen with MPI on Ethernets 
> or similar, and don't care too much about driver isolation, I am 
> thinking of trying to apply Jochen Liedtke's old 'small address spaces' 
> hack, to see if I can improve domU I/O performance.
> 
> My idea is to reserve some additional virtual address space below Xen, 
> e.g. at 0xF0000000, and map the kernel part of dom0* there permanently. 
> The user space part of dom0 I would map as normal from 0 - 0xC0000000, 
> to avoid relinking dom0 applications.  I would use the segments to keep 
> the domUs below 0xF0000000. In this way, TLB flushes should only be 
> necessary when dom0 exits to user space, not when handling interrupts or 
> when domUs are asking for I/O.

Hmmm turns out this is pretty hard to do without relinking userspace, as 
naturally the linux0 likes to peek and poke user-space addresses itself, 
as does Xen.

I guess what I could do (my original plan actually, before I got too 
clever for my own good), would be to squeeze userspace into the 
permanent mapping as well, and then have a relinked busybox or similar 
in there, instead of a full, standard Linux disto.

Jacob