Re: GPU passthrough performance regression in >4GB vms due to XSA-60 changes

["Jan Beulich" <JBeulich@suse.com>]

>>> On 15.05.14 at 17:39, <tomasz.wroblewski@gmail.com> wrote:

> On 05/15/2014 06:07 PM, Jan Beulich wrote:
>>>>> On 15.05.14 at 16:56, <tomasz.wroblewski@gmail.com> wrote:
>>> On 05/15/2014 04:34 PM, Tomasz Wroblewski wrote:
>>>> On 05/15/2014 03:39 PM, Tomasz Wroblewski wrote:
>>>>> On 05/15/2014 03:23 PM, Jan Beulich wrote:
>>>>>>>>> On 15.05.14 at 14:10, <tomasz.wroblewski@gmail.com> wrote:
>>>>>>> Not really sure why it only affects 64bit vms but I've just noticed
>>>>>>> the
>>>>>>> pci BARs for the card are being relocated by hvmloader as per some
>>>>>>> logs:
>>>>>>>
>>>>>>> (XEN) HVM3: Relocating guest memory for lowmem MMIO space enabled
>>>>>>> (XEN) HVM3: Relocating 0xffff pages from 0e0001000 to 14dc00000 for
>>>>>>> lowmem MMIO hole
>>>>>>> (XEN) HVM3: Relocating 0x1 pages from 0e0000000 to 15dbff000 for
>>>>>>> lowmem
>>>>>>> MMIO hole
>>>>>>>
>>>>>>> So it might be also related to that.
>>>>>> Indeed it might - what are the (guest) MTRR types for those regions?
>>>>> It's writeback for both the 32bit and 64bit above ranges.
>>>> ... however, after a bit more debugging its uncached at the time
>>>> hvmloader does the relocation so that's why it ends up like that in
>>>> EPT tables. It does go to writeback only soon after. Haven't
>>>> pinpointed the exact time point for that yet nor why it's being
>>>> updated to writeback, but it seems to be before the guest starts
>>>> booting (i.e. still on bios screens).
>>> ... and after even more I see that the type is uncached at the time the
>>> relocation is happening because mtrr is disabled at that time and
>>> get_mtrr_type() function exits with uncached value in the first few
>>> lines of it. Later when guests enabled MTRR, ept is not updated. So
>>> maybe the EPTs should be updated in some way at that time,
>> Which is what -unstable is now doing.
> Right.
>> But the question remains why this region doesn't get marked UC or
>> WC, but WB.
> The region doesn't seem to be marked in any way in mtrr so it just goes 
> off the default type for that mtrr ((struct mtrr_state*)->def_type) 
> which seems to be WB.

I was expecting this for the relocated (above 4Gb) region, but is this
also the case for the one below?

In any event - all MMIO regions of passed through devices absolutely
have to be represented in the MTRRs as long as the regions' types
in the host MTRRs differ from the default type in the guest ones,
though in the end this makes me (once again) question whether
defaulting to WB and setting up exceptions for MMIO isn't the wrong
approach especially when pass-through is being used. This used to
be the other way around until April 2008 (commit a6a82232:
"x86, hvm: Lots of MTRR/PAT emulation cleanup").

If I coded up a patch to deal with this on -unstable, would you be
able to test that?

Jan