Re: GPU passthrough performance regression in >4GB vms due to XSA-60 changes

[Tomasz Wroblewski <tomasz.wroblewski@gmail.com>]

On 05/16/2014 04:36 PM, Jan Beulich wrote:
>>>> On 16.05.14 at 13:38, <JBeulich@suse.com> wrote:
>>>>> On 16.05.14 at 13:18, <tomasz.wroblewski@gmail.com> wrote:
>>>> If I coded up a patch to deal with this on -unstable, would you be
>>>> able to test that?
>>> Willing to give it a go (xen major version updates are often problematic
>>> to do though so can't promise success). What would your patch be doing?
>>> Adding entries to MTRR for the relocated regions?
>> This and properly declare the region in ACPI's _CRS. For starters I'll
>> probably try keeping the WB default overlaid with UC variable ranges,
>> as that's going to be the less intrusive change.
> Okay here are two patches - the first to deal with the above mentioned
> items, and the second to further increase correctness and at once
> shrink the number of MTRR regions needed.
>
> Afaict they apply equally well to stable-4.3, master, and staging.
>
> But to be honest I don't expect any performance improvement, all
> I'd expect is that BARs relocated above 4Gb would now get treated
> equally to such below 4Gb - UC in all cases.
Thanks Jan. I've tried the patches and you're correct, putting UC in 
MTRR for the relocated region didn't help the issue. However, I had to 
hack that manually - the codepaths to do that in your hvmloader patch 
were not activating. The hvmloader is not programming guest pci bars to 
64bit regions at all, rather still programming them with 32 bit 
regions... upon a look this seems because using_64bar conditon, as well 
as bar64_relocate in hvmloader/pci.c is always false.

So bar relocation to 64bit is not happening, but ram relocation as per 
the code tagged as /* Relocate RAM that overlaps PCI space (in 64k-page 
chunks). */ is happening. This maybe is correct (?), although I think 
the fact that RAM is relocated but not the BAR causes the tools (i.e. 
qemu) to lose sight of what memory is used for mmio and as you mentioned 
in one of the previous posts, the calls which would set it to 
mmio_direct in p2m table are not happening. Our qemu is pretty ancient 
and doesn't support 64bit bars so its not super trivial to verify 
whether relocating bars to 64bit would help. Trying to make sense out of 
this..