Are linker scripts needed?

Are linker scripts needed?

[Arun Sharma]

In alt.bk.changesets.1.1508 Keir wrote:

> Some more ioemu config hardcoding. Our 'target arch' is always i386.
> Do we really need linker scripts??
> Signed-off-by: Keir Fraser <keir@xensource.com>

Yes, we do need linker scripts to support VMX guests with large memory 
(> 256MB). Without the linker scripts, we found instabilities with a 
512MB guest.

The reason for this is, qemu is mapping all of guest physical in it's 
address space. The linker script moves all of the qemu text/data/stack 
to the upper part of the user virtual to leave a large enough hole for 
mapping guest physical using xc_map_foreign.

Our intention is to use qemu for i386, x86-64 and ia64. The 
"target-i386-dm" is a misnomer and inherited from upstream. I should 
probably kill the concept of a "target architecture" since ioemu is cpu 
architecture independent right now.

	-Arun

Re: Are linker scripts needed?

[Keir Fraser]

On 23 May 2005, at 07:00, Arun Sharma wrote:

> Yes, we do need linker scripts to support VMX guests with large memory 
> (> 256MB). Without the linker scripts, we found instabilities with a 
> 512MB guest.
>
> The reason for this is, qemu is mapping all of guest physical in it's 
> address space. The linker script moves all of the qemu text/data/stack 
> to the upper part of the user virtual to leave a large enough hole for 
> mapping guest physical using xc_map_foreign.

I suppose it improves performance and simplifies the data path, but it 
should probably dynamically map the vmx guest's pages for I/O on 
demand. After all, this approach isn't going to work for bigmem domains 
on i386 (particularly once PAE is working). So it seems a bad idea to 
come to rely on it any more than as a bit of a hack to get you up and 
running.

Best of all (imo) would be for it to act as a 'driver in the middle': 
connecting the fully virt guest through to an ordinary backend driver, 
and not itself touching I/O buffers at all. Perhaps ultimately we can 
push the qemu drivers into the vmx container itself, perhaps in 
'virtual smm mode', so that from outside the container it looks just 
like any other paravirt I/O channel.

> Our intention is to use qemu for i386, x86-64 and ia64. The 
> "target-i386-dm" is a misnomer and inherited from upstream. I should 
> probably kill the concept of a "target architecture" since ioemu is 
> cpu architecture independent right now.

Sounds good.

  -- Keir

Re: Re: Are linker scripts needed?

[Keir Fraser]

On 23 May 2005, at 09:02, Keir Fraser wrote:

>
>> Our intention is to use qemu for i386, x86-64 and ia64. The 
>> "target-i386-dm" is a misnomer and inherited from upstream. I should 
>> probably kill the concept of a "target architecture" since ioemu is 
>> cpu architecture independent right now.
>
> Sounds good.

In fact, I think it would be good to kill the configure script and have 
a clean up of the Makefiles -- it's really hard to see what's going on 
in them right now, and we aren't going to use all that 
flexibility/genericity anyway. I reckon the Makefiles should end up a 
good deal shorter and clearer.

  -- Keir

RE: Re: Are linker scripts needed?

[Tian, Kevin]

(Just some rough idea, since I don't know much detail of DM part. ;-)


>I suppose it improves performance and simplifies the data path, but it
>should probably dynamically map the vmx guest's pages for I/O on
>demand. After all, this approach isn't going to work for bigmem domains
>on i386 (particularly once PAE is working). So it seems a bad idea to
>come to rely on it any more than as a bit of a hack to get you up and
>running.

Another requirement for dynamic map on demand is that guest pfn ->
machine pfn mapping may be changed due to balloon driver. It's not that
perfect for a DM to assume that mapping info never changed, and thus
maps all the guest memory in one request at the start...

>
>Best of all (imo) would be for it to act as a 'driver in the middle':
>connecting the fully virt guest through to an ordinary backend driver,
>and not itself touching I/O buffers at all. Perhaps ultimately we can
>push the qemu drivers into the vmx container itself, perhaps in
>'virtual smm mode', so that from outside the container it looks just
>like any other paravirt I/O channel.
>

Frist, to clarify a term: Do you mean vmx container as "vmx domain" ? :)

Major benefit of your approach, from my thought, is that backend driver
in service OS can service both para-virtualization and
full-firtualization domain then, with a unified channel interface and
logic, right? I'm trying to understand how 'virtual smm mode' you
mentioned can be achieved, and thinking at least following modifications
may be required:
	- DM can not only be user application. A bunch of kernel module
like a collection of existing frontend drivers have to be created for
each component of DM in vmx container.
	- DM itself has to be bound with a faked device (Maybe PCI
device), to be able to receive emulation message coming from HV, after
HV captures unmodified domain's access to MMIO, etc.
	- More backend-frontend drivers need time to be cultivated
gradually, like SCSI support, etc, to behave like a complete DM.s

However the major defect somehow may lie with the maintain effort. Also
take USB port as the example, there'll be around 10 or more maintainers
of different backend-frontend components, who have to keep up with
kernel evolvement both on service OS side and domainN side, since kernel
module is tightly coupled with kernel internal interfaces. However
current DM model has no such problem. With more deployment of xen,
there'll be more such effort. Of course, also with more deployment of
xen, there'll be more people who'd like to contribute to xen then. ;-) 

Thanks,
Kevin

Re: Re: Are linker scripts needed?

[Keir Fraser]

On 23 May 2005, at 11:22, Tian, Kevin wrote:

> Frist, to clarify a term: Do you mean vmx container as "vmx domain" ? 
> :)

I mean the 'guest mode' of VMX, as opposed to code that runs in VMX 
domain context, but within the hypervisor in 'host mode'.

> Major benefit of your approach, from my thought, is that backend driver
> in service OS can service both para-virtualization and
> full-firtualization domain then, with a unified channel interface and
> logic, right? I'm trying to understand how 'virtual smm mode' you
> mentioned can be achieved, and thinking at least following 
> modifications
> may be required:

I thought that VMX provided a virtual equivalent of SMM, where 
management and emulation code can run under the OS's feet without it 
realising? If this is not provided then I do not think the trick can 
work, as you would need to steal some virtual address space in which to 
execute the qemu code.

  -- Keir

Re: Re: Are linker scripts needed?

[Arun Sharma]

On Mon, May 23, 2005 at 01:54:29PM +0100, Keir Fraser wrote:
> 
> I thought that VMX provided a virtual equivalent of SMM, where 
> management and emulation code can run under the OS's feet without it 
> realising? 

Yes, this is how vmxassist implements real mode support.

> If this is not provided then I do not think the trick can 
> work, as you would need to steal some virtual address space in which to 
> execute the qemu code.

The "world switch" can switch to a different address space (CR3 on
x86/RID on ia64), without having to steal guest virtual.

	-Arun

RE: Re: Are linker scripts needed?

[Ian Pratt]

> > Major benefit of your approach, from my thought, is that backend 
> > driver in service OS can service both para-virtualization and 
> > full-firtualization domain then, with a unified channel 
> interface and 
> > logic, right? I'm trying to understand how 'virtual smm mode' you 
> > mentioned can be achieved, and thinking at least following 
> > modifications may be required:
> 
> I thought that VMX provided a virtual equivalent of SMM, 
> where management and emulation code can run under the OS's 
> feet without it realising? If this is not provided then I do 
> not think the trick can work, as you would need to steal some 
> virtual address space in which to execute the qemu code.

I'd be inclined to move to a model where we execute the device emulation
in the root (monitor) VMCS, using the same protection mechanism we use
for para-virtualized guests e.g. segmentation for x86, paging for
x86_64. The device emulation should should work like a normal front-end
driver, connecting via a device channel to a normal backend.

Infact, I really like this approach. It gives good performance, safety,
code reuse, and unifies the control interface. It does require a bit of
hacking of qemu, to give it the execution environment it needs and make
it connect onto the existing back ends.

Arguments against?

Ian

RE: Re: Are linker scripts needed?

[Tian, Kevin]

>> I thought that VMX provided a virtual equivalent of SMM,
>> where management and emulation code can run under the OS's
>> feet without it realising? If this is not provided then I do
>> not think the trick can work, as you would need to steal some
>> virtual address space in which to execute the qemu code.
>
>I'd be inclined to move to a model where we execute the device
emulation
>in the root (monitor) VMCS, using the same protection mechanism we use
>for para-virtualized guests e.g. segmentation for x86, paging for
>x86_64. The device emulation should should work like a normal front-end
>driver, connecting via a device channel to a normal backend.

I like the idea. :) but where is the resource of device emulation
component getting allocation from, like physical frame allocated to that
module? If it's managed by vmx domain's kernel, I'm not sure it's easy
for such type of DM to acquire system resource and also access the user
eco system across multiple space, like:
	- page fault handle
	- Access special files like /proc/privcmd
	- Where are configuration files of qemu saved? In service OS
file system, or in vmx domain's FS?
	- How to sync between DM and corresponding vmx domain?
	...

Also that style of DM will be another type of domain, which has to been
considered by HV specially with resource and schedule...

Just some immature concerns, and like to learn more in this direction.
:)

Thanks,
Kevin

RE: Re: Are linker scripts needed?

[Tian, Kevin]

Then, after some look at vmxassist mechanism and with some discussion
with Susie Li, I'm now aware that my previous understanding completely
bias from your intent. :) Following the 'hack' you mentioned, that DM
may finally be standalone component, without any dependency with upper
vmx domain. Control panel can load it to appropriate position, which has
its own trap table, own page table (maybe fixed), device emulation
logic, simplified event channel interface, similar assist hook in HV for
context save/restore, etc. No need to have memory management, since DMA
buffer will only be touched by backend. En... now I'm clearer about the
feasibility, which is always the first thing for me to care when
learning new things. :)

Regarding performance, it saves many context switches between kernel and
user space, compared to current DM model which runs as application in
service's OS. But the maximum granularity of this model is still
instruction level when vmx domain tries to do mmio access. Instead, for
some specific device, frontend driver module may be more effective to
utilize frontend-backend model, since it's based on transaction/session.
The example is the recent VBD/VNIF driver by Xiaofeng Lin, which can be
installed into vmx domain dynamically and then talk to backend
effectively.

Thanks,
Kevin

>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tian, Kevin
>Sent: Tuesday, May 24, 2005 11:41 AM
>To: Ian Pratt; Keir Fraser
>Cc: Sharma, Arun; xen-devel@lists.xensource.com
>Subject: RE: [Xen-devel] Re: Are linker scripts needed?
>
>>> I thought that VMX provided a virtual equivalent of SMM,
>>> where management and emulation code can run under the OS's
>>> feet without it realising? If this is not provided then I do
>>> not think the trick can work, as you would need to steal some
>>> virtual address space in which to execute the qemu code.
>>
>>I'd be inclined to move to a model where we execute the device
>emulation
>>in the root (monitor) VMCS, using the same protection mechanism we use
>>for para-virtualized guests e.g. segmentation for x86, paging for
>>x86_64. The device emulation should should work like a normal
front-end
>>driver, connecting via a device channel to a normal backend.
>
>I like the idea. :) but where is the resource of device emulation
>component getting allocation from, like physical frame allocated to
that
>module? If it's managed by vmx domain's kernel, I'm not sure it's easy
>for such type of DM to acquire system resource and also access the user
>eco system across multiple space, like:
>	- page fault handle
>	- Access special files like /proc/privcmd
>	- Where are configuration files of qemu saved? In service OS
>file system, or in vmx domain's FS?
>	- How to sync between DM and corresponding vmx domain?
>	...
>
>Also that style of DM will be another type of domain, which has to been
>considered by HV specially with resource and schedule...
>
>Just some immature concerns, and like to learn more in this direction.
>:)
>
>Thanks,
>Kevin
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

RE: Re: Are linker scripts needed?

[Ian Pratt]

> >I'd be inclined to move to a model where we execute the device
> emulation
> >in the root (monitor) VMCS, using the same protection 
> mechanism we use 
> >for para-virtualized guests e.g. segmentation for x86, paging for 
> >x86_64. The device emulation should should work like a 
> normal front-end 
> >driver, connecting via a device channel to a normal backend.
> 
> I like the idea. :) but where is the resource of device 
> emulation component getting allocation from, like physical 
> frame allocated to that module? 

Page frames in use by the device emulation will be on the domain's page
list, just like the 'real guest' running under VT-x.

> If it's managed by vmx 
> Also that style of DM will be another type of domain, which 
> has to been considered by HV specially with resource and schedule...

As far as I can see, the device emulation will be entirely event driven,
driven by VM exits and events coming back from the backend drivers. The
normal scheduling should work just fine.

The device emulation modules could basically be linked against a
revamped mini-os to provide the services they need (e.g printk).

The configuration for qemu could be passed in on the 'command line' of
the mini-os, supplied by the domain builder.

The device emulation shouldn't assume it has all of the 'real' guest's
memory mapped into its address space (it can't for PAE), but can have a
very large map cache.


Ian

RE: Re: Are linker scripts needed?

[Ian Pratt]

 

> Then, after some look at vmxassist mechanism and with some 
> discussion with Susie Li, I'm now aware that my previous 
> understanding completely bias from your intent. :) Following 
> the 'hack' you mentioned, that DM may finally be standalone 
> component, without any dependency with upper vmx domain. 
> Control panel can load it to appropriate position, which has 
> its own trap table, own page table (maybe fixed), device 
> emulation logic, simplified event channel interface, similar 
> assist hook in HV for context save/restore, etc. No need to 
> have memory management, since DMA buffer will only be touched 
> by backend. En... now I'm clearer about the feasibility, 
> which is always the first thing for me to care when learning 
> new things. :)

I don't quite follow your terminiology, but I think we're on the same
wavelength.

I guess the first step would be getting mini-os working on the unstable
tree again -- shouldn't be hard. 
 
> Regarding performance, it saves many context switches between 
> kernel and user space, compared to current DM model which 
> runs as application in service's OS. But the maximum 
> granularity of this model is still instruction level when vmx 
> domain tries to do mmio access. Instead, for some specific 
> device, frontend driver module may be more effective to 
> utilize frontend-backend model, since it's based on 
> transaction/session.
> The example is the recent VBD/VNIF driver by Xiaofeng Lin, 
> which can be installed into vmx domain dynamically and then 
> talk to backend effectively.

Yep, performance won't be as good as the real para-virtualized drivers,
but if we pick the device to emulate carefully it should be possible to
get half decent performance. [We'll probably use the qemu models in the
first instance, but ideally we want to emulate a network device that
supports DMA and cheksum offload. Using the L4/Afterburner team's
DP83820 emulation code might actually be a better starting point than
qemu. ]

Only question is, who's going to do the work? :-)  

Best,
Ian

Re: Re: Are linker scripts needed?

[Arun Sharma]

Keir Fraser wrote:
> 
> In fact, I think it would be good to kill the configure script and have 
> a clean up of the Makefiles -- it's really hard to see what's going on 
> in them right now, and we aren't going to use all that 
> flexibility/genericity anyway. I reckon the Makefiles should end up a 
> good deal shorter and clearer.

Agree that configure isn't adding much value today. The only useful 
thing it's doing is to detect if the build env has libvncserver and 
disabling VNC support if it doesn't have it.

Will send a makefile patch as soon as I have one...

	-Arun