frontend and backend devices and different types of hw - pci for example

frontend and backend devices and different types of hw - pci for example

[Sting Zax]

Hello, 

 In the sparse tree , under drivers/xen, there are frontend and backend drivers
for network (netback and netfront), for block devices (blkfront and
blkback), and for
usb devices (usbfront and usbback). 
What about other devices ? let's say a PCI sound card (or any other PCI device).
 Where is the software that should handle it ? I remember I saw somewhere some
discussion about PCI configuration space, but I don't remember where.

TIA,
Sting

Re: frontend and backend devices and different types of hw - pci for example

[Mark Williamson]

> What about other devices ? let's say a PCI sound card (or any other PCI
> device). Where is the software that should handle it ? I remember I saw
> somewhere some discussion about PCI configuration space, but I don't
> remember where.

That code is in Xen itself in Xen 2.0.  Xen controls access to the PCI 
configuration spaces so that guests can only see the devices they have access 
to.  It also controls the IO memory / ports that domains are allowed to 
access in order to control PCI devices.

Note that giving direct physical access to a PCI device has security 
implications since the guest can potentially use the cards' DMA capabilities 
to access all of physical memory.  The front/back-style devices do not have 
this limitation.

Btw, I've laid some groundwork for a virtual sound device but haven't had much 
time to hack on it yet.

Cheers,
Mark

Re: frontend and backend devices and different types of hw - pci for example

[Sting Zax]

Hello, 

 I had looked at the code of 2.0.*  under xen/arch/x86 saw 
pci-irq.c and pci-pc.c and pci-x86.c which as I understand handle pci devices 
other than net/usb.
However, I did not saw such modules in the unstable version. 
May I ask : is this PCI support for non net/usb PCI devices  removed
(or temporarily removed) from the unstable version? or maybe I simply
missed it ?


>Note that giving direct physical access to a PCI device has security
>implications since the guest can potentially use the cards' DMA capabilities
>to access all of physical memory.  

Will IOMMU support help solving this security problems ? 


Regards,
Sting

On 8/28/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > What about other devices ? let's say a PCI sound card (or any other PCI
> > device). Where is the software that should handle it ? I remember I saw
> > somewhere some discussion about PCI configuration space, but I don't
> > remember where.
> 
> That code is in Xen itself in Xen 2.0.  Xen controls access to the PCI
> configuration spaces so that guests can only see the devices they have access
> to.  It also controls the IO memory / ports that domains are allowed to
> access in order to control PCI devices.
> 
> Note that giving direct physical access to a PCI device has security
> implications since the guest can potentially use the cards' DMA capabilities
> to access all of physical memory.  The front/back-style devices do not have
> this limitation.
> 
> Btw, I've laid some groundwork for a virtual sound device but haven't had much
> time to hack on it yet.
> 
> Cheers,
> Mark
>

Re: frontend and backend devices and different types of hw - pci for example

[Keir Fraser]

On 29 Aug 2005, at 07:57, Sting Zax wrote:

>  I had looked at the code of 2.0.*  under xen/arch/x86 saw
> pci-irq.c and pci-pc.c and pci-x86.c which as I understand handle pci 
> devices
> other than net/usb.
> However, I did not saw such modules in the unstable version.
> May I ask : is this PCI support for non net/usb PCI devices  removed
> (or temporarily removed) from the unstable version? or maybe I simply
> missed it ?

Xen no longer controls PCI devices -- the PCI code in domain 0 now 
accesses PCI devices directly. You can run pretty much any device 
driver you want in domain 0.

  -- Keir

Re: frontend and backend devices and different types of hw - pci for example

[Mark Williamson]

>  I had looked at the code of 2.0.*  under xen/arch/x86 saw
> pci-irq.c and pci-pc.c and pci-x86.c which as I understand handle pci
> devices other than net/usb.
> However, I did not saw such modules in the unstable version.
> May I ask : is this PCI support for non net/usb PCI devices  removed
> (or temporarily removed) from the unstable version? or maybe I simply
> missed it ?

Dom0 now basically owns anything to do with PCI whereas previously Xen 
controlled core PCI stuff and dom0 just accessed devices.  Support for giving 
device access to other domains will need to be implemented in dom0 (this 
hasn't been done yet but it's hoped the feature will be back in time for the 
release).

> >Note that giving direct physical access to a PCI device has security
> >implications since the guest can potentially use the cards' DMA
> > capabilities to access all of physical memory.
>
> Will IOMMU support help solving this security problems ?

Yes but only if it enforces access permissions fully i.e. I don't think the 
IOEMU in AMD64 machines is sufficient.  From the looks of Pacifica it might 
have sufficient support to control the DMA problem, I'm sure Intel have a 
similar solution (although I don't think it's implemented in Vanderpool - 
they'll probably need chipset support).

Cheers,
Mark

>
> Regards,
> Sting
>
> On 8/28/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > > What about other devices ? let's say a PCI sound card (or any other PCI
> > > device). Where is the software that should handle it ? I remember I saw
> > > somewhere some discussion about PCI configuration space, but I don't
> > > remember where.
> >
> > That code is in Xen itself in Xen 2.0.  Xen controls access to the PCI
> > configuration spaces so that guests can only see the devices they have
> > access to.  It also controls the IO memory / ports that domains are
> > allowed to access in order to control PCI devices.
> >
> > Note that giving direct physical access to a PCI device has security
> > implications since the guest can potentially use the cards' DMA
> > capabilities to access all of physical memory.  The front/back-style
> > devices do not have this limitation.
> >
> > Btw, I've laid some groundwork for a virtual sound device but haven't had
> > much time to hack on it yet.
> >
> > Cheers,
> > Mark

Re: frontend and backend devices and different types of hw - pci for example

[Stefan Berger]

xen-devel-bounces@lists.xensource.com wrote on 08/29/2005 06:45:51 AM:

> >  I had looked at the code of 2.0.*  under xen/arch/x86 saw
> > pci-irq.c and pci-pc.c and pci-x86.c which as I understand handle pci
> > devices other than net/usb.
> > However, I did not saw such modules in the unstable version.
> > May I ask : is this PCI support for non net/usb PCI devices  removed
> > (or temporarily removed) from the unstable version? or maybe I simply
> > missed it ?
> 
> Dom0 now basically owns anything to do with PCI whereas previously Xen 
> controlled core PCI stuff and dom0 just accessed devices.  Support for 
giving 
> device access to other domains will need to be implemented in dom0 (this 

> hasn't been done yet but it's hoped the feature will be back in time for 
the 
> release).

Do you think that an emulated PCI layer underneath every domU could be a 
possibility for a solution of moving PCI devices to user domains? I have 
had some success with it and got as far as for example moving a PCI 
ethernet card or the whole USB controller to a user domain and making the 
user domain kernel activate its drivers. I did this by reading the PCI 
config space (256 bytes) from the device and presenting the data to the 
user level in the emulated PCI bus. However I have then encountered a 
couple of problems afterwards when trying to activate the IRQ. There seems 
to be some translation going on of a PCI IRQ number to the actual number 
the system is using (due to APIC I suppose) and so the data exchange with 
the device did not start.

  Stefan
> 
> > >Note that giving direct physical access to a PCI device has security
> > >implications since the guest can potentially use the cards' DMA
> > > capabilities to access all of physical memory.
> >
> > Will IOMMU support help solving this security problems ?
> 
> Yes but only if it enforces access permissions fully i.e. I don't think 
the 
> IOEMU in AMD64 machines is sufficient.  From the looks of Pacifica it 
might 
> have sufficient support to control the DMA problem, I'm sure Intel have 
a 
> similar solution (although I don't think it's implemented in Vanderpool 
- 
> they'll probably need chipset support).
> 
> Cheers,
> Mark
> 
> >
> > Regards,
> > Sting
> >
> > On 8/28/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > > > What about other devices ? let's say a PCI sound card (or any 
other PCI
> > > > device). Where is the software that should handle it ? I remember 
I saw
> > > > somewhere some discussion about PCI configuration space, but I 
don't
> > > > remember where.
> > >
> > > That code is in Xen itself in Xen 2.0.  Xen controls access to the 
PCI
> > > configuration spaces so that guests can only see the devices they 
have
> > > access to.  It also controls the IO memory / ports that domains are
> > > allowed to access in order to control PCI devices.
> > >
> > > Note that giving direct physical access to a PCI device has security
> > > implications since the guest can potentially use the cards' DMA
> > > capabilities to access all of physical memory.  The front/back-style
> > > devices do not have this limitation.
> > >
> > > Btw, I've laid some groundwork for a virtual sound device but 
haven't had
> > > much time to hack on it yet.
> > >
> > > Cheers,
> > > Mark
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: frontend and backend devices and different types of hw - pci for example

[Mark Williamson]

> Do you think that an emulated PCI layer underneath every domU could be a
> possibility for a solution of moving PCI devices to user domains? I have
> had some success with it and got as far as for example moving a PCI
> ethernet card or the whole USB controller to a user domain and making the
> user domain kernel activate its drivers.

Possibly...  I would have been inclined to do it using some sort of 
interdomain communication rather than using an emulator in Xen but I'm 
actually open to persuasion that I'm wrong on this point ;-)

> I did this by reading the PCI 
> config space (256 bytes) from the device and presenting the data to the
> user level in the emulated PCI bus. However I have then encountered a
> couple of problems afterwards when trying to activate the IRQ. There seems
> to be some translation going on of a PCI IRQ number to the actual number
> the system is using (due to APIC I suppose) and so the data exchange with
> the device did not start.

Thanks for the extra detail.

Hmmm.  Sounds a bit weird...  Afraid I'm not sure what might be going on here.

Cheers,
Mark

>   Stefan
>
> > > >Note that giving direct physical access to a PCI device has security
> > > >implications since the guest can potentially use the cards' DMA
> > > > capabilities to access all of physical memory.
> > >
> > > Will IOMMU support help solving this security problems ?
> >
> > Yes but only if it enforces access permissions fully i.e. I don't think
>
> the
>
> > IOEMU in AMD64 machines is sufficient.  From the looks of Pacifica it
>
> might
>
> > have sufficient support to control the DMA problem, I'm sure Intel have
>
> a
>
> > similar solution (although I don't think it's implemented in Vanderpool
>
> -
>
> > they'll probably need chipset support).
> >
> > Cheers,
> > Mark
> >
> > > Regards,
> > > Sting
> > >
> > > On 8/28/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > > > > What about other devices ? let's say a PCI sound card (or any
>
> other PCI
>
> > > > > device). Where is the software that should handle it ? I remember
>
> I saw
>
> > > > > somewhere some discussion about PCI configuration space, but I
>
> don't
>
> > > > > remember where.
> > > >
> > > > That code is in Xen itself in Xen 2.0.  Xen controls access to the
>
> PCI
>
> > > > configuration spaces so that guests can only see the devices they
>
> have
>
> > > > access to.  It also controls the IO memory / ports that domains are
> > > > allowed to access in order to control PCI devices.
> > > >
> > > > Note that giving direct physical access to a PCI device has security
> > > > implications since the guest can potentially use the cards' DMA
> > > > capabilities to access all of physical memory.  The front/back-style
> > > > devices do not have this limitation.
> > > >
> > > > Btw, I've laid some groundwork for a virtual sound device but
>
> haven't had
>
> > > > much time to hack on it yet.
> > > >
> > > > Cheers,
> > > > Mark
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

Re: frontend and backend devices and different types of hw - pci for example

[Stefan Berger]

xen-devel-bounces@lists.xensource.com wrote on 09/03/2005 01:46:24 PM:

> > Do you think that an emulated PCI layer underneath every domU could be 
a
> > possibility for a solution of moving PCI devices to user domains? I 
have
> > had some success with it and got as far as for example moving a PCI
> > ethernet card or the whole USB controller to a user domain and making 
the
> > user domain kernel activate its drivers.
> 
> Possibly...  I would have been inclined to do it using some sort of 
> interdomain communication rather than using an emulator in Xen but I'm 
> actually open to persuasion that I'm wrong on this point ;-)

I thought about interdomain communication for emulating PCI devices. It 
'feels' like this would rip apart both  domain's PCI layers quite a bit.

> 
> > I did this by reading the PCI 
> > config space (256 bytes) from the device and presenting the data to 
the
> > user level in the emulated PCI bus. However I have then encountered a

I wrote this wrong: I am not presenting this to the user level, but to the 
user domain kernel.

> > couple of problems afterwards when trying to activate the IRQ. There 
seems
> > to be some translation going on of a PCI IRQ number to the actual 
number
> > the system is using (due to APIC I suppose) and so the data exchange 
with
> > the device did not start.
> 
> Thanks for the extra detail.
> 
> Hmmm.  Sounds a bit weird...  Afraid I'm not sure what might be going on 
here.

What's going on here is for example that the PCI configuration space says 
the interrupt the Ethernet card is on is IRQ '9', but in reality the 
system is using for example interrupt '21' for this device - this happens 
through translation in other parts where APIC or ACPI-related code plays a 
role. I haven't had the time to find out what part of the system is doing 
that. Another issue is that the code in /xen/.../io_apic.c is not 
activated at all in a user/driver domain. 

I believe the architecture for access to PCI was different in Xen 1.0 
where PCI was in the HV. With 2.0 this changed and PCI access was moved to 
domain 0. Did driver domains work with that? If yes, then what was done in 
domain 0 to prepare a domain's access to a device?

  Stefan
> 
> Cheers,
> Mark
> 
> >   Stefan
> >
> > > > >Note that giving direct physical access to a PCI device has 
security
> > > > >implications since the guest can potentially use the cards' DMA
> > > > > capabilities to access all of physical memory.
> > > >
> > > > Will IOMMU support help solving this security problems ?
> > >
> > > Yes but only if it enforces access permissions fully i.e. I don't 
think
> >
> > the
> >
> > > IOEMU in AMD64 machines is sufficient.  From the looks of Pacifica 
it
> >
> > migh - 
> >
> > > have sufficient support to control the DMA problem, I'm sure Intel 
have
> >
> > a
> >
> > > similar solution (although I don't think it's implemented in 
Vanderpool
> >
> > -
> >
> > > they'll probably need chipset support).
> > >
> > > Cheers,
> > > Mark
> > >
> > > > Regards,
> > > > Sting
> > > >
> > > > On 8/28/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > > > > > What about other devices ? let's say a PCI sound card (or any
> >
> > other PCI
> >
> > > > > > device). Where is the software that should handle it ? I 
remember
> >
> > I saw
> >
> > > > > > somewhere some discussion about PCI configuration space, but I
> >
> > don't
> >
> > > > > > remember where.
> > > > >
> > > > > That code is in Xen itself in Xen 2.0.  Xen controls access to 
the
> >
> > PCI
> >
> > > > > configuration spaces so that guests can only see the devices 
they
> >
> > have
> >
> > > > > access to.  It also controls the IO memory / ports that domains 
are
> > > > > allowed to access in order to control PCI devices.
> > > > >
> > > > > Note that giving direct physical access to a PCI device has 
security
> > > > > implications since the guest can potentially use the cards' DMA
> > > > > capabilities to access all of physical memory.  The 
front/back-style
> > > > > devices do not have this limitation.
> > > > >
> > > > > Btw, I've laid some groundwork for a virtual sound device but
> >
> > haven't had
> >
> > > > > much time to hack on it yet.
> > > > >
> > > > > Cheers,
> > > > > Mark
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: frontend and backend devices and different types of hw - pci for example

[Mark Williamson]

> > Possibly...  I would have been inclined to do it using some sort of
> > interdomain communication rather than using an emulator in Xen but I'm
> > actually open to persuasion that I'm wrong on this point ;-)
>
> I thought about interdomain communication for emulating PCI devices. It
> 'feels' like this would rip apart both  domain's PCI layers quite a bit.

Well...  I think you could hook it into the PCI layer reasonably cleanly by 
using function pointers to ensure that one set of arch functions got called 
for dom0 operation and another set get called for driver domains.

Bear in mind we don't necessarily need to interpose the PCI virtualisation 
code on accesses to the device itself, just those to the PCI bridges.  Access 
to the device itself can be made directly (although see my comment below).
> through translation in other parts where APIC or ACPI-related code plays a
> role. I haven't had the time to find out what part of the system is doing
> that. Another issue is that the code in /xen/.../io_apic.c is not
> activated at all in a user/driver domain.

That sounds like the sort of thing I was expecting but I can't tell you where 
to look off the top of my head...

> I believe the architecture for access to PCI was different in Xen 1.0
> where PCI was in the HV. With 2.0 this changed and PCI access was moved to
> domain 0. Did driver domains work with that? If yes, then what was done in
> domain 0 to prepare a domain's access to a device?

1.x: PCI core code and device drivers in Xen, only virtual devices in guest
2.x: PCI core code in Xen, device drivers in domains
3.x: PCI core code in dom0, no driver domains yet...

In the 1.x architecture there just weren't any driver domains, Xen had all the 
drivers (except the graphics driver, which lived in the X server as usual).  
In 2.x the PCI core was in Xen, with guests' PCI layer making hypercalls 
instead of doing direct hardware access.  Giving a guest a device consisted 
of: allowing it to see *just* that device in config space (reported in the 
hypercall interface), modifying its IO bitmap and memory mapping privileges 
so that it could access these things.

Since all this is yanked out of Xen now, we don't have quite such convenient 
mechanisms for hiding things from the guest; hence config space access needs 
to be able to go through another channel, or (as you have been working 
towards) emulated somewhere.

Had you considered retasking some of the existing "IO packet" stuff as used by 
the Qemu device model to pass requests up to userspace?  Since this is only 
for device discovery the performance hit shouldn't be an issue.  This avoids 
adding code to Xen *and* avoids special-casing in the PCI code.

While I'm on the subject, I'd personally like to see guests granted IO access 
slightly differently.  There are two ways to grant IO access on x86: change 
the IOPL (giving the guest access to all IO ports) or set IO bits in the TSS 
(giving fine grained control).  The problem with the latter is that guest 
*apps* will be able to access the hardware; essentially x86 gives you coarse 
grained control and ring-level protection, or vice-versa.

Since people often like to partition their systems using Xen, I don't really 
like giving apps easy access to the hardware in this way.  I'd like to have 
the option of trapping IO port writes in Xen and verifying the guest's IO 
privileges in software, then emulating the write.  It is my hope that this 
won't hurt too much on decent hardware (e.g. devices that use an in memory 
buffer descriptor queue) and that on less clever hardware it won't matter too 
much...

Thoughts?

Cheers,
Mark


>   Stefan
>
> > Cheers,
> > Mark
> >
> > >   Stefan
> > >
> > > > > >Note that giving direct physical access to a PCI device has
>
> security
>
> > > > > >implications since the guest can potentially use the cards' DMA
> > > > > > capabilities to access all of physical memory.
> > > > >
> > > > > Will IOMMU support help solving this security problems ?
> > > >
> > > > Yes but only if it enforces access permissions fully i.e. I don't
>
> think
>
> > > the
> > >
> > > > IOEMU in AMD64 machines is sufficient.  From the looks of Pacifica
>
> it
>
> > > migh -
> > >
> > > > have sufficient support to control the DMA problem, I'm sure Intel
>
> have
>
> > > a
> > >
> > > > similar solution (although I don't think it's implemented in
>
> Vanderpool
>
> > > -
> > >
> > > > they'll probably need chipset support).
> > > >
> > > > Cheers,
> > > > Mark
> > > >
> > > > > Regards,
> > > > > Sting
> > > > >
> > > > > On 8/28/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > > > > > > What about other devices ? let's say a PCI sound card (or any
> > >
> > > other PCI
> > >
> > > > > > > device). Where is the software that should handle it ? I
>
> remember
>
> > > I saw
> > >
> > > > > > > somewhere some discussion about PCI configuration space, but I
> > >
> > > don't
> > >
> > > > > > > remember where.
> > > > > >
> > > > > > That code is in Xen itself in Xen 2.0.  Xen controls access to
>
> the
>
> > > PCI
> > >
> > > > > > configuration spaces so that guests can only see the devices
>
> they
>
> > > have
> > >
> > > > > > access to.  It also controls the IO memory / ports that domains
>
> are
>
> > > > > > allowed to access in order to control PCI devices.
> > > > > >
> > > > > > Note that giving direct physical access to a PCI device has
>
> security
>
> > > > > > implications since the guest can potentially use the cards' DMA
> > > > > > capabilities to access all of physical memory.  The
>
> front/back-style
>
> > > > > > devices do not have this limitation.
> > > > > >
> > > > > > Btw, I've laid some groundwork for a virtual sound device but
> > >
> > > haven't had
> > >
> > > > > > much time to hack on it yet.
> > > > > >
> > > > > > Cheers,
> > > > > > Mark
> > > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

Re: frontend and backend devices and different types of hw - pci for example

[David Hopwood]

Mark Williamson wrote:
> While I'm on the subject, I'd personally like to see guests granted IO access 
> slightly differently.  There are two ways to grant IO access on x86: change 
> the IOPL (giving the guest access to all IO ports) or set IO bits in the TSS 
> (giving fine grained control).  The problem with the latter is that guest 
> *apps* will be able to access the hardware; essentially x86 gives you coarse 
> grained control and ring-level protection, or vice-versa.

Could XenLinux use a different TSS to run its apps?

-- 
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>

Re: frontend and backend devices and different types of hw - pci for example

[Stefan Berger]

xen-devel-bounces@lists.xensource.com wrote on 09/03/2005 11:23:50 PM:

> > > Possibly...  I would have been inclined to do it using some sort of
> > > interdomain communication rather than using an emulator in Xen but 
I'm
> > > actually open to persuasion that I'm wrong on this point ;-)
> >
> > I thought about interdomain communication for emulating PCI devices. 
It
> > 'feels' like this would rip apart both  domain's PCI layers quite a 
bit.
> 
> Well...  I think you could hook it into the PCI layer reasonably cleanly 
by 
> using function pointers to ensure that one set of arch functions got 
called 
> for dom0 operation and another set get called for driver domains.

All inb/w/l outb/w/l for the PCI ports 0xcf8-0xcff could also be proxied 
in domain 0 and domain 0 could make only those devices visible to the 
driver domain that are to be used by that domain. I think that doing this 
in the hypervisor through IO port or (partial) PCI bus emulation is a 
"cleaner" way, though.

> 
> Bear in mind we don't necessarily need to interpose the PCI 
virtualisation 
> code on accesses to the device itself, just those to the PCI bridges. 
Access 
> to the device itself can be made directly (although see my comment 
below).
> > through translation in other parts where APIC or ACPI-related code 
plays a
> > role. I haven't had the time to find out what part of the system is 
doing
> > that. Another issue is that the code in /xen/.../io_apic.c is not
> > activated at all in a user/driver domain.
> 
> That sounds like the sort of thing I was expecting but I can't tell you 
where 
> to look off the top of my head...

Actually the IRQ translation happens through a function call like 
acpi_register_gsi() which in turn is called by pci_device_enable(). Not 
sure whether that call can work properly anywhere else than in domain 0.

> 
> > I believe the architecture for access to PCI was different in Xen 1.0
> > where PCI was in the HV. With 2.0 this changed and PCI access was 
moved to
> > domain 0. Did driver domains work with that? If yes, then what was 
done in
> > domain 0 to prepare a domain's access to a device?
> 
> 1.x: PCI core code and device drivers in Xen, only virtual devices in 
guest
> 2.x: PCI core code in Xen, device drivers in domains
> 3.x: PCI core code in dom0, no driver domains yet...
> 
> In the 1.x architecture there just weren't any driver domains, Xen had 
all the 
> drivers (except the graphics driver, which lived in the X server as 
usual). 
> In 2.x the PCI core was in Xen, with guests' PCI layer making hypercalls 

> instead of doing direct hardware access.  Giving a guest a device 
consisted 
> of: allowing it to see *just* that device in config space (reported in 
the 
> hypercall interface), modifying its IO bitmap and memory mapping 
privileges 
> so that it could access these things.
> 
> Since all this is yanked out of Xen now, we don't have quite such 
convenient 
> mechanisms for hiding things from the guest; hence config space access 
needs 
> to be able to go through another channel, or (as you have been working 
> towards) emulated somewhere.

What I have done so far is that I have presented a static config space to 
the PCI layer in user domains. This works well for reading out the config 
space, but not for writing to it. Writes that for example are coming 
through the pci_enable_device() call try to activate the device by writing 
into its config space. Now emulation is unfortunately only good as long as 
the writes are passed through to the device's config space. Once you start 
passing writes through to the real device, you can also pass the reads 
through to it - the emulation might then only be useful for presenting an 
emulated bus entry to the PCI layer. Another requirement for properly 
reading out the config space is to have some locking done between writing 
to IO port 0xcf8 and until reading from 0xcfc. I think a proper solution 
for this would require that all domains (including domain 0) IO port 
accesses in the range of 0xcf8 to 0xcff be intercepted and either 
presented emulated parts (bus) or real devices - no domain would be 
treated differently.

> 
> Had you considered retasking some of the existing "IO packet" stuff as 
used by 
> the Qemu device model to pass requests up to userspace?  Since this is 
only 
> for device discovery the performance hit shouldn't be an issue.  This 
avoids 
> adding code to Xen *and* avoids special-casing in the PCI code.

I had not looked at this so far.
The problem I see in any case is the IRQ number translation that is 
happening. I believe that can only be done properly in domain 0. Somewhere 
in the path of the acpi_register_gsi function call (called from 
pci_enable_device) you have to be able to calculate the IRQ number that is 
actually being used in the system. 

> 
> While I'm on the subject, I'd personally like to see guests granted IO 
access 
> slightly differently.  There are two ways to grant IO access on x86: 
change 
> the IOPL (giving the guest access to all IO ports) or set IO bits in the 
TSS 
> (giving fine grained control).  The problem with the latter is that 
guest 
> *apps* will be able to access the hardware; essentially x86 gives you 
coarse 
> grained control and ring-level protection, or vice-versa.
> 
> Since people often like to partition their systems using Xen, I don't 
really 
> like giving apps easy access to the hardware in this way.  I'd like to 
have 
> the option of trapping IO port writes in Xen and verifying the guest's 
IO 
> privileges in software, then emulating the write.  It is my hope that 
this 
> won't hurt too much on decent hardware (e.g. devices that use an in 
memory 
> buffer descriptor queue) and that on less clever hardware it won't 
matter too 
> much...
> 
> Thoughts?

What I thought that could possibly be done is open up the IO port access 
of domain 0 to all ports at the beginning. A driver domain that learns 
through the PCI config space that it needs access to a range of IO ports 
could have those IO ports opened up as well (maybe automatically by the 
PCI emulation layer) and at the same time those ports could be closed in 
domain 0. Also, if the privilege bitmap in the TSS should not be done in 
HW, one could certainly do it in SW in a 'IO port emulation layer'.  2 
pages for this bitmap would be enough.

  Stefan


> 
> Cheers,
> Mark
> 
> 
> >   Stefan
> >
> > > Cheers,
> > > Mark
> > >
> > > >   Stefan
> > > >
> > > > > > >Note that giving direct physical access to a PCI device has
> >
> > security
> >
> > > > > > >implications since the guest can potentially use the cards' 
DMA
> > > > > > > capabilities to access all of physical memory.
> > > > > >
> > > > > > Will IOMMU support help solving this security problems ?
> > > > >
> > > > > Yes but only if it enforces access permissions fully i.e. I 
don't
> >
> > think
> >
> > > > the
> > > >
> > > > > IOEMU in AMD64 machines is sufficient.  From the looks of 
Pacifica
> >
> > it
> >
> > > > migh -
> > > >
> > > > > have sufficient support to control the DMA problem, I'm sure 
Intel
> >
> > have
> >
> > > > a
> > > >
> > > > > similar solution (although I don't think it's implemented in
> >
> > Vanderpool
> >
> > > > -
> > > >
> > > > > they'll probably need chipset support).
> > > > >
> > > > > Cheers,
> > > > > Mark
> > > > >
> > > > > > Regards,
> > > > > > Sting
> > > > > >
> > > > > > On 8/28/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> 
wrote:
> > > > > > > > What about other devices ? let's say a PCI sound card (or 
any
> > > >
> > > > other PCI
> > > >
> > > > > > > > device). Where is the software that should handle it ? I
> >
> > remember
> >
> > > > I saw
> > > >
> > > > > > > > somewhere some discussion about PCI configuration space, 
but I
> > > >
> > > > don't
> > > >
> > > > > > > > remember where.
> > > > > > >
> > > > > > > That code is in Xen itself in Xen 2.0.  Xen controls access 
to
> >
> > the
> >
> > > > PCI
> > > >
> > > > > > > configuration spaces so that guests can only see the devices
> >
> > they
> >
> > > > have
> > > >
> > > > > > > access to.  It also controls the IO memory / ports that 
domains
> >
> > are
> >
> > > > > > > allowed to access in order to control PCI devices.
> > > > > > >
> > > > > > > Note that giving direct physical access to a PCI device has
> >
> > security
> >
> > > > > > > implications since the guest can potentially use the cards' 
DMA
> > > > > > > capabilities to access all of physical memory.  The
> >
> > front/back-style
> >
> > > > > > > devices do not have this limitation.
> > > > > > >
> > > > > > > Btw, I've laid some groundwork for a virtual sound device 
but
> > > >
> > > > haven't had
> > > >
> > > > > > > much time to hack on it yet.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Mark
> > > > >
> > > > > _______________________________________________
> > > > > Xen-devel mailing list
> > > > > Xen-devel@lists.xensource.com
> > > > > http://lists.xensource.com/xen-devel
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel