Idea: Small Address Spaces

[Jacob Gorm Hansen <jacobg@diku.dk>]

hi,

in the cases where users (like me) wish to run Xen with MPI on Ethernets 
or similar, and don't care too much about driver isolation, I am 
thinking of trying to apply Jochen Liedtke's old 'small address spaces' 
hack, to see if I can improve domU I/O performance.

My idea is to reserve some additional virtual address space below Xen, 
e.g. at 0xF0000000, and map the kernel part of dom0* there permanently. 
The user space part of dom0 I would map as normal from 0 - 0xC0000000, 
to avoid relinking dom0 applications.  I would use the segments to keep 
the domUs below 0xF0000000. In this way, TLB flushes should only be 
necessary when dom0 exits to user space, not when handling interrupts or 
when domUs are asking for I/O.

Ideally, I would move dom0 to ring0 to save the cost of switching from 
Xen to dom0, but I have no idea how much trouble this will cause.

I have been working a bit on this, and so far I have only managed to 
move dom0 to its new location, and to map the 0xF0000000 - 0xFC000000 
range into the top of domU pgds instead of changing the cr3 (page table 
base) register. I have managed to break a lot of things, I see that as a 
good sign ;-)

I know this loses driver isolation, but I would argue that if it can be 
done relatively cleanly, and if it yields any performance improvements, 
a significant percentage of users would be willing to make that 
tradeoff.  Naturally, this would be a (compile or run-time) option, not 
the default behavior.

Comments are welcome, now is the time to yell stop ;-)

Jacob


* This could probably be made to work with multiple IDDs as well, 
treating the 0xF0000000 slot as a cache of the last run driver-domain 
rather than as a permanent mapping.