Re: [Qemu-devel] [PATCH 4/5] x86: Allow physical address bits to be set

[Marcel Apfelbaum <marcel@redhat.com>]

On 06/17/2016 07:07 PM, Laszlo Ersek wrote:
> On 06/17/16 11:52, Igor Mammedov wrote:
>> On Fri, 17 Jun 2016 11:17:54 +0200
>> Gerd Hoffmann <kraxel@redhat.com> wrote:
>>
>>> On Fr, 2016-06-17 at 10:43 +0200, Paolo Bonzini wrote:
>>>>
>>>> On 17/06/2016 10:15, Dr. David Alan Gilbert wrote:
>>>>> Larger is a problem if the guest tries to map something to a high
>>>>> address that's not addressable.
>>>>
>>>> Right.  It's not a problem for most emulated PCI devices (it would be a
>>>> problem for those that have large RAM BARs, but even our emulated video
>>>> cards do not have 64-bit RAM BARs, I think;
>>>
>>> qxl can be configured to have one, try "-device
>>> qxl-vga,vram64_size_mb=1024"
>>>
>>>>>     2) While we have maxmem settings to tell us the top of VM RAM, do
>>>>>        we have anything that tells us the top of IO space? What happens
>>>>>        when we hotplug a PCI card?
>>>
>>>> (arch/x86/kernel/setup.c) but I agree that (2) is a blocker.
>>>
>>> seabios maps stuff right above ram (possibly with a hole due to
>>> alignment requirements).
>>>
>>> ovmf maps stuff into a 32G-aligned 32G hole.  Which lands at 32G and
>>> therefore is addressable with 36 bits, unless you have tons of ram (>
>>> 30G) assigned to your guest.  A physical host machine where you can plug
>>> in enough ram for such a configuration likely has more than 36 physical
>>> address lines too ...
>>>
>>> qemu checks where the firmware mapped 64bit bars, then adds those ranges
>>> to the root bus pci resources in the acpi tables (see /proc/iomem).
>>>
>>>> You don't know how the guest will assign PCI BAR addresses, and as you
>>>> said there's hotplug too.
>>>
>>> Not sure whenever qemu adds some extra space for hotplug to the 64bit
>>> hole and if so how it calculates the size then.  But the guest os should
>>> stick to those ranges when configuring hotplugged devices.
>> currently firmware would assign 64-bit BARs after reserved-memory-end
>> (not sure about ovmf though)
>
> OVMF does the same as well. It makes sure that the 64-bit PCI MMIO
> aperture is located above "etc/reserved-memory-end", if the latter exists.
>
>> but QEMU on ACPI side will add 64-bit _CRS only
>> for firmware mapped devices (i.e. no space reserved for hotplug).
>> And is I recall correctly ovmf won't map BARs if it doesn't have
>> a driver for it
>
> Yes, that's correct, generally for all UEFI firmware.
>
> More precisely, BARs will be allocated and programmed, but the MMIO
> space decoding bit will not be set (permanently) in the device's command
> register, if there is no matching driver in the firmware (or in the
> device's own oprom).
>
>> so ACPI tables won't even have a space for not mapped
>> 64-bit BARs.
>
> This used to be true, but that's not the case since
> <https://github.com/tianocore/edk2/commit/8f35eb92c419>.
>
> Namely, specifically for conforming to QEMU's ACPI generator, OVMF
> *temporarily* enables, as a platform quirk, all PCI devices present in
> the system, before triggering QEMU to generate the ACPI payload.
>
> Thus, nowadays 64-bit BARs work fine with OVMF, both for virtio-modern
> devices, and assigned physical devices. (This is very easy to test,
> because, unlike SeaBIOS, the edk2 stuff built into OVMF prefers to
> allocate 64-bit BARs outside of the 32-bit address space.)
>
> Devices behind PXBs are a different story, but Marcel's been looking
> into that, see <https://bugzilla.redhat.com/show_bug.cgi?id=1323976>.
>
>> There was another attempt to reserve more space in _CRS
>>    https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg00090.html
>
> That's actually Marcel's first own patch set for addressing RHBZ#1323976
> that I mentioned above (see it linked in
> <https://bugzilla.redhat.com/show_bug.cgi?id=1323976#c2>).
>
> It might have wider effects, but it is entirely motivated, to my
> knowledge, by PXB. If you don't have extra root bridges, and/or you plug
> all your devices with 64-bit MMIO BARs into the "main" (default) root
> bridge, then (I believe) that patch set is not supposed to make any
> difference. (I could be wrong, it's been a while since I looked at
> Marcel's work!)
>

Patch 3 and 4 indeed are for PXB only. but patch 'pci: reserve 64 bit MMIO range for PCI hotplug'
(see https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg00091.html) tries
to reserve [above_4g_mem_size, max_addressable_cpu_bits] range for PCI hotplug.

The implementation is not good enough because the number of addressable bits is hard-coded.
However, we have now David's wrapper I can use.


Thanks,
Marcel







> Thanks
> Laszlo
>