Other PCI devices to mark mark as read-only for dom0

Other PCI devices to mark mark as read-only for dom0

[Andrew Cooper]

Following Jan's infrastructure to mark certain PCI devices as read only,
I think it wise to now consider what other PCI devices should really be
read only to dom0.

My preliminary thoughts include:

* PCI serial devices which Xen is configured to use
* Chipset devices (AMD IOMMU covered by previous patch)
* Cpu information

Are there any others I have overlooked, or reasons that dom0 should be
able to write to these areas?

On a related note, should there be a mechanism for dom0 to determine
which PCI configuration areas are read only to itself?

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: Other PCI devices to mark mark as read-only for dom0

[Sander Eikelenboom]

Friday, June 22, 2012, 11:04:24 AM, you wrote:

> Following Jan's infrastructure to mark certain PCI devices as read only,
> I think it wise to now consider what other PCI devices should really be
> read only to dom0.

> My preliminary thoughts include:

> * PCI serial devices which Xen is configured to use
> * Chipset devices (AMD IOMMU covered by previous patch)
> * Cpu information

> Are there any others I have overlooked, or reasons that dom0 should be
> able to write to these areas?

Make devices specified for pci passthrough be really hidden and "owned" by the hyperviso ?
     - which can in turn delegate ownership to a domain (including dom0)
     - If a domain is destroyed, the hypervisor resets the device and becomes the owner again instead of dom0 ?


> On a related note, should there be a mechanism for dom0 to determine
> which PCI configuration areas are read only to itself?




-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: Other PCI devices to mark mark as read-only for dom0

[Andrew Cooper]

On 22/06/12 10:24, Sander Eikelenboom wrote:
>
>> Are there any others I have overlooked, or reasons that dom0 should be
>> able to write to these areas?
> Make devices specified for pci passthrough be really hidden and "owned" by the hyperviso ?
>      - which can in turn delegate ownership to a domain (including dom0)
>      - If a domain is destroyed, the hypervisor resets the device and becomes the owner again instead of dom0 ?

Why?

Currently this is covered by unbinding the dom0 driver and binding
pciback to the device.  Well behaved device drivers will not cause problems.

This way, the toolstack has control of which domains own what.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: Other PCI devices to mark mark as read-only for dom0

[Jan Beulich]

>>> On 22.06.12 at 11:04, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> Following Jan's infrastructure to mark certain PCI devices as read only,
> I think it wise to now consider what other PCI devices should really be
> read only to dom0.
> 
> My preliminary thoughts include:
> 
> * PCI serial devices which Xen is configured to use

But only if they're single-function.

> * Chipset devices (AMD IOMMU covered by previous patch)
> * Cpu information

What are you thinking of here specifically.

> Are there any others I have overlooked, or reasons that dom0 should be
> able to write to these areas?
> 
> On a related note, should there be a mechanism for dom0 to determine
> which PCI configuration areas are read only to itself?

Perhaps, but that's not the only thing to deal with here. As
said previously, when we want to add devices with active BARs
here (luckily Wei confirmed that AMD IOMMUs have none),
Dom0 trying to re-configure them would get us into problems.
The issue exists today, but could become worse when we
disallow the updates (as that could lead to two devices sharing
resources they shouldn't share, whereas today a device in use
by Xen and getting re-assigned its resources would merely stop
working).

Jan

Re: Other PCI devices to mark mark as read-only for dom0

[Andrew Cooper]

[-- Attachment #1: Type: text/plain, Size: 2532 bytes --]

On 22/06/12 10:43, Jan Beulich wrote:
>>>> On 22.06.12 at 11:04, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> Following Jan's infrastructure to mark certain PCI devices as read only,
>> I think it wise to now consider what other PCI devices should really be
>> read only to dom0.
>>
>> My preliminary thoughts include:
>>
>> * PCI serial devices which Xen is configured to use
> But only if they're single-function.

Why only single function?  Should Xen not turn all the functions it is
using to read-only ?

>
>> * Chipset devices (AMD IOMMU covered by previous patch)
>> * Cpu information
> What are you thinking of here specifically.

See attached lspci from a new sandybridge machine we have gained.  Quite
a lot of that looks rather dangerous for dom0 to play around with.


>
>> Are there any others I have overlooked, or reasons that dom0 should be
>> able to write to these areas?
>>
>> On a related note, should there be a mechanism for dom0 to determine
>> which PCI configuration areas are read only to itself?
> Perhaps, but that's not the only thing to deal with here. As
> said previously, when we want to add devices with active BARs
> here (luckily Wei confirmed that AMD IOMMUs have none),
> Dom0 trying to re-configure them would get us into problems.
> The issue exists today, but could become worse when we
> disallow the updates (as that could lead to two devices sharing
> resources they shouldn't share, whereas today a device in use
> by Xen and getting re-assigned its resources would merely stop
> working).
>
> Jan
>

It is certainly not an easy problem, and perhaps I am needlessly
complicating the issue.

It occurs that we have 3 possible directions to fix this issue.

1) Continue the current method of fixing things up after they break,
which can cause a hassle for a user encountering the issue.
2) Mark as RO and provide an explicit hypercall interface to remap
BARs.  I don't know how well this would go with upstream Linux.
3) Extend current infrastructure to be able to tell when a write is
affecting the BARs and permit them.  This seems like the best option
going forward, but might be quite hard to implement.

I guess the real question is whether we should continue reactively
fixing problems, or start protectively fixing the root cause.  My gut
feeling is that we are going to start seeing more and more devices on
the PCI bus which Xen should be using, rather than dom0.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com




[-- Attachment #2: sandybridge-lspci-tv.log --]
[-- Type: text/x-log, Size: 19470 bytes --]

-+-[0000:ff]-+-08.0  Intel Corporation Sandy Bridge QPI Link 0
 |           +-09.0  Intel Corporation Sandy Bridge QPI Link 1
 |           +-0a.0  Intel Corporation Sandy Bridge Power Control Unit 0
 |           +-0a.1  Intel Corporation Sandy Bridge Power Control Unit 1
 |           +-0a.2  Intel Corporation Sandy Bridge Power Control Unit 2
 |           +-0a.3  Intel Corporation Sandy Bridge Power Control Unit 3
 |           +-0b.0  Intel Corporation Sandy Bridge Interrupt Control Registers
 |           +-0b.3  Intel Corporation Sandy Bridge Semaphore and Scratchpad Configuration Registers
 |           +-0c.0  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.1  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.2  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.3  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.6  Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 0
 |           +-0c.7  Intel Corporation Sandy Bridge System Address Decoder
 |           +-0d.0  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.1  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.2  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.3  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.6  Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 1
 |           +-0e.0  Intel Corporation Sandy Bridge Processor Home Agent
 |           +-0e.1  Intel Corporation Sandy Bridge Processor Home Agent Performance Monitoring
 |           +-0f.0  Intel Corporation Sandy Bridge Integrated Memory Controller Registers
 |           +-0f.1  Intel Corporation Sandy Bridge Integrated Memory Controller RAS Registers
 |           +-0f.2  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 0
 |           +-0f.3  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 1
 |           +-0f.4  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 2
 |           +-0f.5  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 3
 |           +-0f.6  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 4
 |           +-10.0  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 0
 |           +-10.1  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 1
 |           +-10.2  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 0
 |           +-10.3  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 1
 |           +-10.4  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 2
 |           +-10.5  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 3
 |           +-10.6  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 2
 |           +-10.7  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 3
 |           +-11.0  Intel Corporation Sandy Bridge DDRIO
 |           +-13.0  Intel Corporation Sandy Bridge R2PCIe
 |           +-13.1  Intel Corporation Sandy Bridge Ring to PCI Express Performance Monitor
 |           +-13.4  Intel Corporation Sandy Bridge QuickPath Interconnect Agent Ring Registers
 |           +-13.5  Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 0 Performance Monitor
 |           \-13.6  Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 1 Performance Monitor
 +-[0000:c0]-+-02.0-[c1]--
 |           +-03.0-[c2]--
 |           +-04.0  Intel Corporation Sandy Bridge DMA Channel 0
 |           +-04.1  Intel Corporation Sandy Bridge DMA Channel 1
 |           +-04.2  Intel Corporation Sandy Bridge DMA Channel 2
 |           +-04.3  Intel Corporation Sandy Bridge DMA Channel 3
 |           +-04.4  Intel Corporation Sandy Bridge DMA Channel 4
 |           +-04.5  Intel Corporation Sandy Bridge DMA Channel 5
 |           +-04.6  Intel Corporation Sandy Bridge DMA Channel 6
 |           +-04.7  Intel Corporation Sandy Bridge DMA Channel 7
 |           +-05.0  Intel Corporation Sandy Bridge Address Map, VTd_Misc, System Management
 |           +-05.2  Intel Corporation Sandy Bridge Control Status and Global Errors
 |           \-05.4  Intel Corporation Sandy Bridge I/O APIC
 +-[0000:bf]-+-08.0  Intel Corporation Sandy Bridge QPI Link 0
 |           +-09.0  Intel Corporation Sandy Bridge QPI Link 1
 |           +-0a.0  Intel Corporation Sandy Bridge Power Control Unit 0
 |           +-0a.1  Intel Corporation Sandy Bridge Power Control Unit 1
 |           +-0a.2  Intel Corporation Sandy Bridge Power Control Unit 2
 |           +-0a.3  Intel Corporation Sandy Bridge Power Control Unit 3
 |           +-0b.0  Intel Corporation Sandy Bridge Interrupt Control Registers
 |           +-0b.3  Intel Corporation Sandy Bridge Semaphore and Scratchpad Configuration Registers
 |           +-0c.0  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.1  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.2  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.3  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.6  Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 0
 |           +-0c.7  Intel Corporation Sandy Bridge System Address Decoder
 |           +-0d.0  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.1  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.2  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.3  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.6  Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 1
 |           +-0e.0  Intel Corporation Sandy Bridge Processor Home Agent
 |           +-0e.1  Intel Corporation Sandy Bridge Processor Home Agent Performance Monitoring
 |           +-0f.0  Intel Corporation Sandy Bridge Integrated Memory Controller Registers
 |           +-0f.1  Intel Corporation Sandy Bridge Integrated Memory Controller RAS Registers
 |           +-0f.2  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 0
 |           +-0f.3  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 1
 |           +-0f.4  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 2
 |           +-0f.5  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 3
 |           +-0f.6  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 4
 |           +-10.0  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 0
 |           +-10.1  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 1
 |           +-10.2  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 0
 |           +-10.3  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 1
 |           +-10.4  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 2
 |           +-10.5  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 3
 |           +-10.6  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 2
 |           +-10.7  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 3
 |           +-11.0  Intel Corporation Sandy Bridge DDRIO
 |           +-13.0  Intel Corporation Sandy Bridge R2PCIe
 |           +-13.1  Intel Corporation Sandy Bridge Ring to PCI Express Performance Monitor
 |           +-13.4  Intel Corporation Sandy Bridge QuickPath Interconnect Agent Ring Registers
 |           +-13.5  Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 0 Performance Monitor
 |           \-13.6  Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 1 Performance Monitor
 +-[0000:80]-+-02.0-[81]--
 |           +-03.0-[82]--
 |           +-04.0  Intel Corporation Sandy Bridge DMA Channel 0
 |           +-04.1  Intel Corporation Sandy Bridge DMA Channel 1
 |           +-04.2  Intel Corporation Sandy Bridge DMA Channel 2
 |           +-04.3  Intel Corporation Sandy Bridge DMA Channel 3
 |           +-04.4  Intel Corporation Sandy Bridge DMA Channel 4
 |           +-04.5  Intel Corporation Sandy Bridge DMA Channel 5
 |           +-04.6  Intel Corporation Sandy Bridge DMA Channel 6
 |           +-04.7  Intel Corporation Sandy Bridge DMA Channel 7
 |           +-05.0  Intel Corporation Sandy Bridge Address Map, VTd_Misc, System Management
 |           +-05.2  Intel Corporation Sandy Bridge Control Status and Global Errors
 |           \-05.4  Intel Corporation Sandy Bridge I/O APIC
 +-[0000:7f]-+-08.0  Intel Corporation Sandy Bridge QPI Link 0
 |           +-09.0  Intel Corporation Sandy Bridge QPI Link 1
 |           +-0a.0  Intel Corporation Sandy Bridge Power Control Unit 0
 |           +-0a.1  Intel Corporation Sandy Bridge Power Control Unit 1
 |           +-0a.2  Intel Corporation Sandy Bridge Power Control Unit 2
 |           +-0a.3  Intel Corporation Sandy Bridge Power Control Unit 3
 |           +-0b.0  Intel Corporation Sandy Bridge Interrupt Control Registers
 |           +-0b.3  Intel Corporation Sandy Bridge Semaphore and Scratchpad Configuration Registers
 |           +-0c.0  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.1  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.2  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.3  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.6  Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 0
 |           +-0c.7  Intel Corporation Sandy Bridge System Address Decoder
 |           +-0d.0  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.1  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.2  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.3  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.6  Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 1
 |           +-0e.0  Intel Corporation Sandy Bridge Processor Home Agent
 |           +-0e.1  Intel Corporation Sandy Bridge Processor Home Agent Performance Monitoring
 |           +-0f.0  Intel Corporation Sandy Bridge Integrated Memory Controller Registers
 |           +-0f.1  Intel Corporation Sandy Bridge Integrated Memory Controller RAS Registers
 |           +-0f.2  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 0
 |           +-0f.3  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 1
 |           +-0f.4  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 2
 |           +-0f.5  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 3
 |           +-0f.6  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 4
 |           +-10.0  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 0
 |           +-10.1  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 1
 |           +-10.2  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 0
 |           +-10.3  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 1
 |           +-10.4  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 2
 |           +-10.5  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 3
 |           +-10.6  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 2
 |           +-10.7  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 3
 |           +-11.0  Intel Corporation Sandy Bridge DDRIO
 |           +-13.0  Intel Corporation Sandy Bridge R2PCIe
 |           +-13.1  Intel Corporation Sandy Bridge Ring to PCI Express Performance Monitor
 |           +-13.4  Intel Corporation Sandy Bridge QuickPath Interconnect Agent Ring Registers
 |           +-13.5  Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 0 Performance Monitor
 |           \-13.6  Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 1 Performance Monitor
 +-[0000:40]-+-01.0-[41]--
 |           +-02.0-[42]--
 |           +-04.0  Intel Corporation Sandy Bridge DMA Channel 0
 |           +-04.1  Intel Corporation Sandy Bridge DMA Channel 1
 |           +-04.2  Intel Corporation Sandy Bridge DMA Channel 2
 |           +-04.3  Intel Corporation Sandy Bridge DMA Channel 3
 |           +-04.4  Intel Corporation Sandy Bridge DMA Channel 4
 |           +-04.5  Intel Corporation Sandy Bridge DMA Channel 5
 |           +-04.6  Intel Corporation Sandy Bridge DMA Channel 6
 |           +-04.7  Intel Corporation Sandy Bridge DMA Channel 7
 |           +-05.0  Intel Corporation Sandy Bridge Address Map, VTd_Misc, System Management
 |           +-05.2  Intel Corporation Sandy Bridge Control Status and Global Errors
 |           \-05.4  Intel Corporation Sandy Bridge I/O APIC
 +-[0000:3f]-+-08.0  Intel Corporation Sandy Bridge QPI Link 0
 |           +-09.0  Intel Corporation Sandy Bridge QPI Link 1
 |           +-0a.0  Intel Corporation Sandy Bridge Power Control Unit 0
 |           +-0a.1  Intel Corporation Sandy Bridge Power Control Unit 1
 |           +-0a.2  Intel Corporation Sandy Bridge Power Control Unit 2
 |           +-0a.3  Intel Corporation Sandy Bridge Power Control Unit 3
 |           +-0b.0  Intel Corporation Sandy Bridge Interrupt Control Registers
 |           +-0b.3  Intel Corporation Sandy Bridge Semaphore and Scratchpad Configuration Registers
 |           +-0c.0  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.1  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.2  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.3  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0c.6  Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 0
 |           +-0c.7  Intel Corporation Sandy Bridge System Address Decoder
 |           +-0d.0  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.1  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.2  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.3  Intel Corporation Sandy Bridge Unicast Register 0
 |           +-0d.6  Intel Corporation Sandy Bridge Integrated Memory Controller System Address Decoder 1
 |           +-0e.0  Intel Corporation Sandy Bridge Processor Home Agent
 |           +-0e.1  Intel Corporation Sandy Bridge Processor Home Agent Performance Monitoring
 |           +-0f.0  Intel Corporation Sandy Bridge Integrated Memory Controller Registers
 |           +-0f.1  Intel Corporation Sandy Bridge Integrated Memory Controller RAS Registers
 |           +-0f.2  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 0
 |           +-0f.3  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 1
 |           +-0f.4  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 2
 |           +-0f.5  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 3
 |           +-0f.6  Intel Corporation Sandy Bridge Integrated Memory Controller Target Address Decoder 4
 |           +-10.0  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 0
 |           +-10.1  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 1
 |           +-10.2  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 0
 |           +-10.3  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 1
 |           +-10.4  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 2
 |           +-10.5  Intel Corporation Sandy Bridge Integrated Memory Controller Channel 0-3 Thermal Control 3
 |           +-10.6  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 2
 |           +-10.7  Intel Corporation Sandy Bridge Integrated Memory Controller ERROR Registers 3
 |           +-11.0  Intel Corporation Sandy Bridge DDRIO
 |           +-13.0  Intel Corporation Sandy Bridge R2PCIe
 |           +-13.1  Intel Corporation Sandy Bridge Ring to PCI Express Performance Monitor
 |           +-13.4  Intel Corporation Sandy Bridge QuickPath Interconnect Agent Ring Registers
 |           +-13.5  Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 0 Performance Monitor
 |           \-13.6  Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 1 Performance Monitor
 \-[0000:00]-+-00.0  Intel Corporation Sandy Bridge DMI2
             +-01.0-[01-03]----00.0-[02-03]----08.0-[03]--+-00.0  Intel Corporation Patsburg Dual 4-Port SATA/SAS Storage Control Unit
             |                                            +-00.3  Intel Corporation Patsburg SMBus Controller 0
             |                                            +-00.4  Intel Corporation Patsburg SMBus Controller 1
             |                                            \-00.5  Intel Corporation Patsburg SMBus Controller 2
             +-02.0-[04]--
             +-03.0-[05]--
             +-03.2-[06-07]--+-00.0  Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
             |               \-00.1  Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
             +-04.0  Intel Corporation Sandy Bridge DMA Channel 0
             +-04.1  Intel Corporation Sandy Bridge DMA Channel 1
             +-04.2  Intel Corporation Sandy Bridge DMA Channel 2
             +-04.3  Intel Corporation Sandy Bridge DMA Channel 3
             +-04.4  Intel Corporation Sandy Bridge DMA Channel 4
             +-04.5  Intel Corporation Sandy Bridge DMA Channel 5
             +-04.6  Intel Corporation Sandy Bridge DMA Channel 6
             +-04.7  Intel Corporation Sandy Bridge DMA Channel 7
             +-05.0  Intel Corporation Sandy Bridge Address Map, VTd_Misc, System Management
             +-05.2  Intel Corporation Sandy Bridge Control Status and Global Errors
             +-05.4  Intel Corporation Sandy Bridge I/O APIC
             +-16.0  Intel Corporation Patsburg MEI Controller #1
             +-16.1  Intel Corporation Patsburg MEI Controller #2
             +-1a.0  Intel Corporation Patsburg USB2 Enhanced Host Controller #2
             +-1c.0-[08]--
             +-1c.7-[09]----00.0  Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines (SEP1)
             +-1d.0  Intel Corporation Patsburg USB2 Enhanced Host Controller #1
             +-1e.0-[0a]--
             +-1f.0  Intel Corporation Patsburg LPC Controller
             +-1f.2  Intel Corporation Patsburg 6-Port SATA AHCI Controller
             \-1f.3  Intel Corporation Patsburg SMBus Host Controller

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Other PCI devices to mark mark as read-only for dom0

[Jan Beulich]

>>> On 22.06.12 at 12:08, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> On 22/06/12 10:43, Jan Beulich wrote:
>>>>> On 22.06.12 at 11:04, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>> Following Jan's infrastructure to mark certain PCI devices as read only,
>>> I think it wise to now consider what other PCI devices should really be
>>> read only to dom0.
>>>
>>> My preliminary thoughts include:
>>>
>>> * PCI serial devices which Xen is configured to use
>> But only if they're single-function.
> 
> Why only single function?  Should Xen not turn all the functions it is
> using to read-only ?

Because, just like for normal, non-PCI based serial ones, ports
that Xen doesn't use should remain usable by Dom0. For
example, I have a PCI card with two serial and one parallel
ports, so with Xen using one serial port for itself, there's no
reason not to allow Dom0 to use the other or the parallel one.

>>> * Chipset devices (AMD IOMMU covered by previous patch)
>>> * Cpu information
>> What are you thinking of here specifically.
> 
> See attached lspci from a new sandybridge machine we have gained.  Quite
> a lot of that looks rather dangerous for dom0 to play around with.

But that can't be easily qualified into some rule, the more that
some of these - iirc - are needed e.g. by the EDAC drivers.

Jan

Re: Other PCI devices to mark mark as read-only for dom0

[Andrew Cooper]

On 22/06/12 12:23, Jan Beulich wrote:
>>>> On 22.06.12 at 12:08, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> On 22/06/12 10:43, Jan Beulich wrote:
>>>>>> On 22.06.12 at 11:04, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>>> Following Jan's infrastructure to mark certain PCI devices as read only,
>>>> I think it wise to now consider what other PCI devices should really be
>>>> read only to dom0.
>>>>
>>>> My preliminary thoughts include:
>>>>
>>>> * PCI serial devices which Xen is configured to use
>>> But only if they're single-function.
>> Why only single function?  Should Xen not turn all the functions it is
>> using to read-only ?
> Because, just like for normal, non-PCI based serial ones, ports
> that Xen doesn't use should remain usable by Dom0. For
> example, I have a PCI card with two serial and one parallel
> ports, so with Xen using one serial port for itself, there's no
> reason not to allow Dom0 to use the other or the parallel one.

I apologize.  I originally used the term 'device' when I intended to use
'function', so I think we are arguing for the same point.

>
>>>> * Chipset devices (AMD IOMMU covered by previous patch)
>>>> * Cpu information
>>> What are you thinking of here specifically.
>> See attached lspci from a new sandybridge machine we have gained.  Quite
>> a lot of that looks rather dangerous for dom0 to play around with.
> But that can't be easily qualified into some rule, the more that
> some of these - iirc - are needed e.g. by the EDAC drivers.
>
> Jan
>

Which is why I am asking here, to see if there are some rules which
could help.  I agree that it is a sticky situation.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: Other PCI devices to mark mark as read-only for dom0

[Jan Beulich]

>>> On 22.06.12 at 14:06, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> On 22/06/12 12:23, Jan Beulich wrote:
>>>>> On 22.06.12 at 12:08, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>> On 22/06/12 10:43, Jan Beulich wrote:
>>>>>>> On 22.06.12 at 11:04, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>>>> Following Jan's infrastructure to mark certain PCI devices as read only,
>>>>> I think it wise to now consider what other PCI devices should really be
>>>>> read only to dom0.
>>>>>
>>>>> My preliminary thoughts include:
>>>>>
>>>>> * PCI serial devices which Xen is configured to use
>>>> But only if they're single-function.
>>> Why only single function?  Should Xen not turn all the functions it is
>>> using to read-only ?
>> Because, just like for normal, non-PCI based serial ones, ports
>> that Xen doesn't use should remain usable by Dom0. For
>> example, I have a PCI card with two serial and one parallel
>> ports, so with Xen using one serial port for itself, there's no
>> reason not to allow Dom0 to use the other or the parallel one.
> 
> I apologize.  I originally used the term 'device' when I intended to use
> 'function', so I think we are arguing for the same point.

And I understood you meaning so - from a PCI terminology pov.
Multi-function here, however, means multiple serial/parallel ports
within a single PCI function (PCI_CLASS_COMMUNICATION_MULTISERIAL
or PCI_CLASS_COMMUNICATION_OTHER). Hence we shouldn't hide a
full S:B:D.F just because we use some portion of it.

Jan

Re: Other PCI devices to mark mark as read-only for dom0

[Andrew Cooper]

On 22/06/12 13:20, Jan Beulich wrote:
>>> Because, just like for normal, non-PCI based serial ones, ports
>>> that Xen doesn't use should remain usable by Dom0. For
>>> example, I have a PCI card with two serial and one parallel
>>> ports, so with Xen using one serial port for itself, there's no
>>> reason not to allow Dom0 to use the other or the parallel one.
>> I apologize.  I originally used the term 'device' when I intended to use
>> 'function', so I think we are arguing for the same point.
> And I understood you meaning so - from a PCI terminology pov.
> Multi-function here, however, means multiple serial/parallel ports
> within a single PCI function (PCI_CLASS_COMMUNICATION_MULTISERIAL
> or PCI_CLASS_COMMUNICATION_OTHER). Hence we shouldn't hide a
> full S:B:D.F just because we use some portion of it.
>
> Jan
>

Ah right.  Yes.  Perhaps then a warning on the Xen console if Xen
encounters such a device.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: Other PCI devices to mark mark as read-only for dom0

[Jan Beulich]

>>> On 22.06.12 at 12:08, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> On 22/06/12 10:43, Jan Beulich wrote:
>> Perhaps, but that's not the only thing to deal with here. As
>> said previously, when we want to add devices with active BARs
>> here (luckily Wei confirmed that AMD IOMMUs have none),
>> Dom0 trying to re-configure them would get us into problems.
>> The issue exists today, but could become worse when we
>> disallow the updates (as that could lead to two devices sharing
>> resources they shouldn't share, whereas today a device in use
>> by Xen and getting re-assigned its resources would merely stop
>> working).
> 
> It is certainly not an easy problem, and perhaps I am needlessly
> complicating the issue.
> 
> It occurs that we have 3 possible directions to fix this issue.
> 
> 1) Continue the current method of fixing things up after they break,
> which can cause a hassle for a user encountering the issue.
> 2) Mark as RO and provide an explicit hypercall interface to remap
> BARs.  I don't know how well this would go with upstream Linux.

And what would the hypercall do? You don't expect it to fix up
all the uses of the old address(es) to now use the new one(s),
do you? Leaving aside the difficulty in getting this right, in some
cases this might not even be possible in a seamless manner.

> 3) Extend current infrastructure to be able to tell when a write is
> affecting the BARs and permit them.  This seems like the best option
> going forward, but might be quite hard to implement.

Doing the mechanical trap-and-emulate part of this shouldn't
be that hard, namely on top of that other patch I'm about to
commit.

But the thing is that this has the same problems as the hypercall
one above.

> I guess the real question is whether we should continue reactively
> fixing problems, or start protectively fixing the root cause.

I agree with this position from a theoretical standpoint.

>  My gut
> feeling is that we are going to start seeing more and more devices on
> the PCI bus which Xen should be using, rather than dom0.

That would be bad - surfacing arbitrary devices on the PCI bus
is - in my opinion - rather questionable a design (which is also
why I consider this aspect of the AMD IOMMUs badly designed).

Jan

Re: Other PCI devices to mark mark as read-only for dom0

[Andrew Cooper]

On 22/06/12 12:30, Jan Beulich wrote:
>> It is certainly not an easy problem, and perhaps I am needlessly
>> complicating the issue.
>>
>> It occurs that we have 3 possible directions to fix this issue.
>>
>> 1) Continue the current method of fixing things up after they break,
>> which can cause a hassle for a user encountering the issue.
>> 2) Mark as RO and provide an explicit hypercall interface to remap
>> BARs.  I don't know how well this would go with upstream Linux.
> And what would the hypercall do? You don't expect it to fix up
> all the uses of the old address(es) to now use the new one(s),
> do you? Leaving aside the difficulty in getting this right, in some
> cases this might not even be possible in a seamless manner.
>
>> 3) Extend current infrastructure to be able to tell when a write is
>> affecting the BARs and permit them.  This seems like the best option
>> going forward, but might be quite hard to implement.
> Doing the mechanical trap-and-emulate part of this shouldn't
> be that hard, namely on top of that other patch I'm about to
> commit.
>
> But the thing is that this has the same problems as the hypercall
> one above.

Currently my understanding is that dom0 has fairly free reign on the PCI
devices, meaning that if it decides to remap BARs and ends up with
conflicts, we already have a problem, especially if its a PCI device
that Xen is trying to use.

As Xen avoids using PCI devices as much as possible, it should be easier
(but doubtfully 'easy') to make Xen aware that PCI devices it is using
may move from under its feet.  ( I am just throwing ideas out here - it
might be that this is a stupid idea. )

>
>> I guess the real question is whether we should continue reactively
>> fixing problems, or start protectively fixing the root cause.
> I agree with this position from a theoretical standpoint.
>
>>  My gut
>> feeling is that we are going to start seeing more and more devices on
>> the PCI bus which Xen should be using, rather than dom0.
> That would be bad - surfacing arbitrary devices on the PCI bus
> is - in my opinion - rather questionable a design (which is also
> why I consider this aspect of the AMD IOMMUs badly designed).
>
> Jan

Agreed, but we now have a problem and need a solution.

>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com