Re: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0

[Gordan Bobic <gordan@bobich.net>]

 On Fri, 26 Jul 2013 01:21:24 +0100, Ian Campbell 
 <ian.campbell@citrix.com> wrote:
> On Thu, 2013-07-25 at 23:23 +0100, Gordan Bobic wrote:
>> Now, if I am understanding the basic nature of the problem 
>> correctly,
>> this _could_ be worked around by ensuring that vBAR = pBAR since in 
>> that
>> case there is no room for the mis-mapped memory overwrites to occur. 
>> Is
>> that correct?
>
> AIUI (which is not very well...) it's not so much vBAR=pBAR but 
> making
> the guest e820 (memory map) have the same MMIO holes as the host so 
> that
> there can't be any clash between v- or p-BAR and RAM in the guest.

 Sure, I understand that - but unless I am overlooking something,
 vBAR=pBAR implicitly ensures that.

 The question, then, is what happens in the null translation instance.
 Specifically, if the PCIe bridge/router is broken (and NF200 is, it
 seems), it would imply that when the driver talks to the device, the
 operation will get sent to the vBAR (=pBAR, i.e. straight to the
 hardware). This then gets translated to the pBAR. But - with a
 broken bridge, and vBAR=pBAR, the MMIO request hits the pBAR
 directly from the guest. Does it then still get intercepted by
 the hypervisor, translated (null operation), and re-transmitted?
 If so, this would lead to the card receiving everything twice,
 resulting either in things outright breaking or going half as
 fast at best.

 Now, all this could be a good thing or a bad thing, depending on
 how exactly you spin it. If the bridge is broken and doesn't
 route all the way back to the root bridge, this could actually be
 a performance optimizing feature. If we set vBAR=pBAR and disable
 any translation thereafter, this avoids the overhead of passing
 everything to/from the root PCIe bridge, and we can just directly
 DMA everything.

 I'm sure there are security implications here, but since NF200
 doesn't do PCIe ACS either, any concept of security goes out
 the window pre-emptively.

 So, my question is:
 1) If vBAR = pBAR, does the hypervisor still do any translation?
 I presume it does because it expects the traffic to pass up
 from the root bridge, to the hypervisor and then back, to
 ensure security. If indeed it does do this, where could I
 optionally disable it, and is there an easy to follow bit of
 example code for how to plumb in a boot parameter option for
 this?

 2) Further, I'm finding myself motivated to write that
 auto-set (as opposed to hard coded) vBAR=pBAR patch discussed
 briefly a week or so ago (have an init script read the BAR
 info from dom0 and put it in xenstore, plus a patch to
 make pBAR=vBAR reservations built dynamically rather than
 statically, based on this data. Now, I'm quite fluent in C,
 but my familiarity with Xen soruce code is nearly non-existant
 (limited to studying an old unsupported patch every now and then
 in order to make it apply to a more recent code release).
 Can anyone help me out with a high level view WRT where
 this would be best plumbed in (which files and the flow of
 control between the affected files)?

 The added bonus of this (if it can be made to work) is that
 it might just make unmodified GeForce cards work, too,
 which probably makes it worthwhile on it's own.

>> I guess I could test this easily enough by applying the vBAR = pBAR 
>> hack.
>
> Does the e820_host=1 option help? That might be PV only though, I 
> can't
> remember...

 Thanks for pointing this one out, I just found this post in the 
 archives:
 http://lists.xen.org/archives/html/xen-users/2012-08/msg00150.html

 With a broken PCIe router, would I also need iommu=soft?

 Gordan