PCI-passthrough for 32 bit guests and high MMIO addresses

PCI-passthrough for 32 bit guests and high MMIO addresses

[Juergen Gross]

Hi,

again a fallout from my "linear p2m list" tests:

Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
wrong device to the domain. The MMIO address was too large for a
MFN of a 32-bit system (it was 380003200000-3800036fffff).

Instead of rejecting the operation Xen tried to perform it resulting
in a (quite understandable) failure in the domU.

I think either the hypervisor or the tools should refuse to do
PCI-passthrough in this case.


Juergen

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Andrew Cooper]

On 21/11/14 14:39, Juergen Gross wrote:
> Hi,
>
> again a fallout from my "linear p2m list" tests:
>
> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
> wrong device to the domain. The MMIO address was too large for a
> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>
> Instead of rejecting the operation Xen tried to perform it resulting
> in a (quite understandable) failure in the domU.
>
> I think either the hypervisor or the tools should refuse to do
> PCI-passthrough in this case.
>

What actually goes wrong?  Does the 32bit guest truncate the MMIO region
to 44 bits and try to map that?

I would completely agree that something (libxl?) should refuse this
attempt to do passthrough, but I think pci-front should also kick up a
fuss if it is able to detect this error.

~Andrew

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Juergen Gross]

On 11/21/2014 03:45 PM, Andrew Cooper wrote:
> On 21/11/14 14:39, Juergen Gross wrote:
>> Hi,
>>
>> again a fallout from my "linear p2m list" tests:
>>
>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>> wrong device to the domain. The MMIO address was too large for a
>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>
>> Instead of rejecting the operation Xen tried to perform it resulting
>> in a (quite understandable) failure in the domU.
>>
>> I think either the hypervisor or the tools should refuse to do
>> PCI-passthrough in this case.
>>
>
> What actually goes wrong?  Does the 32bit guest truncate the MMIO region
> to 44 bits and try to map that?

I think it is 42 bits (the top 2 bits of the MFN are the "identity" and
"foreign" bits).

>
> I would completely agree that something (libxl?) should refuse this
> attempt to do passthrough, but I think pci-front should also kick up a
> fuss if it is able to detect this error.

Indeed.


Juergen

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Andrew Cooper]

On 21/11/14 15:04, Juergen Gross wrote:
> On 11/21/2014 03:45 PM, Andrew Cooper wrote:
>> On 21/11/14 14:39, Juergen Gross wrote:
>>> Hi,
>>>
>>> again a fallout from my "linear p2m list" tests:
>>>
>>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>>> wrong device to the domain. The MMIO address was too large for a
>>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>>
>>> Instead of rejecting the operation Xen tried to perform it resulting
>>> in a (quite understandable) failure in the domU.
>>>
>>> I think either the hypervisor or the tools should refuse to do
>>> PCI-passthrough in this case.
>>>
>>
>> What actually goes wrong?  Does the 32bit guest truncate the MMIO region
>> to 44 bits and try to map that?
>
> I think it is 42 bits (the top 2 bits of the MFN are the "identity" and
> "foreign" bits).

This is a very grey area in desperate need of some clarification.

There is nothing (I am aware of) in the API or ABI, not even an
INVALID_PFN/INVALID_MFN constant.  ~0UL is independently defined in
several locations, but I believe that only Linux has the notion of a
foreign bit in the p2m; Xen and the Toolstack don't appear to have this
concept.

When developing migration v2, there was quite a lot of digging through
history to find out what was safe.  The top bit being set in the guests
p2m used to mean "this is an skb frame and can safely be left behind on
migrate"  (This change was subsequently backed out).  This was back in
the days when max_pfn was 0x1000, and there were plenty of "free" bits
in the top of a 32bit unsigned long.

None of the interfaces like this have been re-evaluated as machines have
become larger.

~Andrew

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Jan Beulich]

>>> On 21.11.14 at 15:39, <JGross@suse.com> wrote:
> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
> wrong device to the domain. The MMIO address was too large for a
> MFN of a 32-bit system (it was 380003200000-3800036fffff).
> 
> Instead of rejecting the operation Xen tried to perform it resulting
> in a (quite understandable) failure in the domU.
> 
> I think either the hypervisor or the tools should refuse to do
> PCI-passthrough in this case.

What's wrong with this large an address? 32-bit PV uses PAE, i.e.
can map them. If the kernel isn't capable of that that's not
something to make Xen (or the tools) refuse such assignments. I
would only see an issue if a hypercall interface involved here isn't
using wide enough fields (but these addresses should be read
from the BARs, i.e. no hypercall involved).

Jan

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Andrew Cooper]

On 21/11/14 14:54, Jan Beulich wrote:
>>>> On 21.11.14 at 15:39, <JGross@suse.com> wrote:
>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>> wrong device to the domain. The MMIO address was too large for a
>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>
>> Instead of rejecting the operation Xen tried to perform it resulting
>> in a (quite understandable) failure in the domU.
>>
>> I think either the hypervisor or the tools should refuse to do
>> PCI-passthrough in this case.
> What's wrong with this large an address?

It is wider than 44 bits, so doesn't fit in a 32bit pfn for p2m/m2p
update operations.

~Andrew

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Jan Beulich]

>>> On 21.11.14 at 16:01, <andrew.cooper3@citrix.com> wrote:
> On 21/11/14 14:54, Jan Beulich wrote:
>>>>> On 21.11.14 at 15:39, <JGross@suse.com> wrote:
>>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>>> wrong device to the domain. The MMIO address was too large for a
>>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>>
>>> Instead of rejecting the operation Xen tried to perform it resulting
>>> in a (quite understandable) failure in the domU.
>>>
>>> I think either the hypervisor or the tools should refuse to do
>>> PCI-passthrough in this case.
>> What's wrong with this large an address?
> 
> It is wider than 44 bits, so doesn't fit in a 32bit pfn for p2m/m2p
> update operations.

MMIO regions don't go into these tables.

Jan

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[David Vrabel]

On 21/11/14 15:38, Jan Beulich wrote:
>>>> On 21.11.14 at 16:01, <andrew.cooper3@citrix.com> wrote:
>> On 21/11/14 14:54, Jan Beulich wrote:
>>>>>> On 21.11.14 at 15:39, <JGross@suse.com> wrote:
>>>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>>>> wrong device to the domain. The MMIO address was too large for a
>>>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>>>
>>>> Instead of rejecting the operation Xen tried to perform it resulting
>>>> in a (quite understandable) failure in the domU.
>>>>
>>>> I think either the hypervisor or the tools should refuse to do
>>>> PCI-passthrough in this case.
>>> What's wrong with this large an address?
>>
>> It is wider than 44 bits, so doesn't fit in a 32bit pfn for p2m/m2p
>> update operations.
> 
> MMIO regions don't go into these tables.

They do in upstream kernels.

David

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Jan Beulich]

>>> On 21.11.14 at 16:48, <david.vrabel@citrix.com> wrote:
> On 21/11/14 15:38, Jan Beulich wrote:
>>>>> On 21.11.14 at 16:01, <andrew.cooper3@citrix.com> wrote:
>>> On 21/11/14 14:54, Jan Beulich wrote:
>>>>>>> On 21.11.14 at 15:39, <JGross@suse.com> wrote:
>>>>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>>>>> wrong device to the domain. The MMIO address was too large for a
>>>>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>>>>
>>>>> Instead of rejecting the operation Xen tried to perform it resulting
>>>>> in a (quite understandable) failure in the domU.
>>>>>
>>>>> I think either the hypervisor or the tools should refuse to do
>>>>> PCI-passthrough in this case.
>>>> What's wrong with this large an address?
>>>
>>> It is wider than 44 bits, so doesn't fit in a 32bit pfn for p2m/m2p
>>> update operations.
>> 
>> MMIO regions don't go into these tables.
> 
> They do in upstream kernels.

Which still is no reason to make the hypervisor or tools refuse
anything here. A check like the one Jürgen is asking for should
be added only if something in the public interface prevents this
from working; everything beyond should be dealt with by the
respective kernel.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[David Vrabel]

On 21/11/14 16:00, Jan Beulich wrote:
>>>> On 21.11.14 at 16:48, <david.vrabel@citrix.com> wrote:
>> On 21/11/14 15:38, Jan Beulich wrote:
>>>>>> On 21.11.14 at 16:01, <andrew.cooper3@citrix.com> wrote:
>>>> On 21/11/14 14:54, Jan Beulich wrote:
>>>>>>>> On 21.11.14 at 15:39, <JGross@suse.com> wrote:
>>>>>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>>>>>> wrong device to the domain. The MMIO address was too large for a
>>>>>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>>>>>
>>>>>> Instead of rejecting the operation Xen tried to perform it resulting
>>>>>> in a (quite understandable) failure in the domU.
>>>>>>
>>>>>> I think either the hypervisor or the tools should refuse to do
>>>>>> PCI-passthrough in this case.
>>>>> What's wrong with this large an address?
>>>>
>>>> It is wider than 44 bits, so doesn't fit in a 32bit pfn for p2m/m2p
>>>> update operations.
>>>
>>> MMIO regions don't go into these tables.
>>
>> They do in upstream kernels.
> 
> Which still is no reason to make the hypervisor or tools refuse
> anything here. A check like the one Jürgen is asking for should
> be added only if something in the public interface prevents this
> from working; everything beyond should be dealt with by the
> respective kernel.

I agree.

David

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Juergen Gross]

On 11/21/2014 03:54 PM, Jan Beulich wrote:
>>>> On 21.11.14 at 15:39, <JGross@suse.com> wrote:
>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>> wrong device to the domain. The MMIO address was too large for a
>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>
>> Instead of rejecting the operation Xen tried to perform it resulting
>> in a (quite understandable) failure in the domU.
>>
>> I think either the hypervisor or the tools should refuse to do
>> PCI-passthrough in this case.
>
> What's wrong with this large an address? 32-bit PV uses PAE, i.e.
> can map them. If the kernel isn't capable of that that's not
> something to make Xen (or the tools) refuse such assignments. I
> would only see an issue if a hypercall interface involved here isn't
> using wide enough fields (but these addresses should be read
> from the BARs, i.e. no hypercall involved).

The MFN format is part of the pv-ABI. And a MFN of a 32-bit pv-guest is
only 32 bits (even if don't take the invalid bit into account).

Should a pv-guest really be capable to map an address outside it's
accessible MFN-range?

Are the tools capable of processing such a mapping in case of saving the
domain?


Juergen

Re: PCI-passthrough for 32 bit guests and high MMIO addresses

[Jan Beulich]

>>> On 21.11.14 at 16:17, <JGross@suse.com> wrote:
> On 11/21/2014 03:54 PM, Jan Beulich wrote:
>>>>> On 21.11.14 at 15:39, <JGross@suse.com> wrote:
>>> Trying to do PCI-passthrough with a 32-bit pv-domain I passed the
>>> wrong device to the domain. The MMIO address was too large for a
>>> MFN of a 32-bit system (it was 380003200000-3800036fffff).
>>>
>>> Instead of rejecting the operation Xen tried to perform it resulting
>>> in a (quite understandable) failure in the domU.
>>>
>>> I think either the hypervisor or the tools should refuse to do
>>> PCI-passthrough in this case.
>>
>> What's wrong with this large an address? 32-bit PV uses PAE, i.e.
>> can map them. If the kernel isn't capable of that that's not
>> something to make Xen (or the tools) refuse such assignments. I
>> would only see an issue if a hypercall interface involved here isn't
>> using wide enough fields (but these addresses should be read
>> from the BARs, i.e. no hypercall involved).
> 
> The MFN format is part of the pv-ABI. And a MFN of a 32-bit pv-guest is
> only 32 bits (even if don't take the invalid bit into account).
> 
> Should a pv-guest really be capable to map an address outside it's
> accessible MFN-range?

For MMIO, why not? All that counts there is the page table entry
format.

> Are the tools capable of processing such a mapping in case of saving the
> domain?

MMIO isn't subject to saving/restoring.

Jan