Re: [Qemu-devel] [edk2] Could not add PCI device with big memory to aarch64 VMs

[liang yan <liangy@hpe.com>]

Hi, Laszlo,

On 11/30/2015 06:45 PM, Laszlo Ersek wrote:
> On 12/01/15 01:46, liang yan wrote:
>> Hello, Laszlo,
>>
>> On 11/30/2015 03:05 PM, Laszlo Ersek wrote:
> [snip]
>
>>> If you need more room (with large alignments), then there's no way
>>> around supporting QEMU's 64 bit aperture, VIRT_PCIE_MMIO_HIGH (see my
>>> earlier email).
>> I checked the function create_pcie form pathtoqemu/hw/arm/virt.c, it has
>> a flag value use_highmem(which has default "true" value).
>>
>> It set base_mmio_high and size_mmio_high to device tree by function below,
>>
>>          qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "ranges",
>>                                       1, FDT_PCI_RANGE_IOPORT, 2, 0,
>>                                       2, base_pio, 2, size_pio,
>>                                       1, FDT_PCI_RANGE_MMIO, 2, base_mmio,
>>                                       2, base_mmio, 2, size_mmio,
>>                                       1, FDT_PCI_RANGE_MMIO_64BIT,
>>                                       2, base_mmio_high,
>>                                       2, base_mmio_high, 2, size_mmio_high);
>>
>> So basically, I need to add two UINT64 variables like mmio_high_base and
>> mmio_high_size to PCD under function ProcessPciHost(VirtFdtDxe.c),
>> and try to use this high base address and size as new aperture.
>>
>> Is this correct?
> It is correct, but that's only part of the story.
>
> Parsing the 64-bit aperture from the DTB into new PCDs in
> ArmVirtPkg/VirtFdtDxe is the easy part.
>
> The hard part is modifying ArmVirtPkg/PciHostBridgeDxe, so that BAR
> allocation requests (submitted by the platform independent PCI bus
> driver that resides in "MdeModulePkg/Bus/Pci/PciBusDxe") are actually
> serviced from this high aperture too.
>
>>> Unfortunately I can't readily help with that in the
>>> "ArmVirtPkg/PciHostBridgeDxe" driver; there's no such (open-source)
>>> example in the edk2 tree. Of course, I could experiment with it myself
>>> -- only not right now.
>> If possible, I do want to finish this part or help you finish it. I just
>> work on UEFI recently, and thank you so much for your patient and detail
>> explanation. I really appreciate it.
>>> I guess copying and adapting the TypeMem32 logic to TypeMem64 (currently
>>> short-circuited with EFI_ABORTED) could work.
>> Is the 32 or 64 bit determined by BAR(2-3bit) or by the PCI device
>> memory size? Is there an option from QEMU?
> I can't tell. :)
>
>> Does TypeMem32 still keep  "VIRT_PCIE_MMIO" aperture and TypeMem64 use
>> "VIRT_PCIE_MMIO_HIGH" aperture? or It's more like device property
>> controlled from QEMU device simulation?
> Good question. I don't know. I think in order to answer this question,
> we should understand the whole dance between the PCI root bridge / host
> bridge driver and the generic PCI bus driver.
>
> The documentation I know of is in the Platform Init spec, version 1.4,
> Volume 5, Chapter 10 "PCI Host Bridge". I've taken multiple stabs at
> that chapter earlier, but I've always given up.
>
> Sorry I can't help more, but this is new area for me as well.
No, already a big help, Really appreciate your generous sharing.

I also have a problem from guest vm kernel side.

Even we change the UEFI side, should the guest kernel still need to modify?
Because, I noticed that the kernel will do rescan.  use the example of 
256M below.

UEFI side, has two setup, not sure which is the real one.

PciBus: Discovered PCI @ [00|01|00]
    BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100; Offset 
= 0x10
    BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset 
= 0x14
    BAR[2]: Type = PMem64; Alignment = 0xFFFFFFF;    Length = 
0x10000000;    Offset = 0x18
                              ===== PMem64 here


PciBus: Resource Map for Root Bridge PciRoot(0x0)
Type =  Mem32; Base = 0x20000000;    Length = 0x10100000; Alignment = 
0xFFFFFFF
    Base = 0x20000000;    Length = 0x10000000;    Alignment = 
0xFFFFFFF;    Owner = PCI [00|01|00:18]; Type = PMem32 
============>Mem32 here
    Base = 0x30000000;    Length = 0x1000;    Alignment = 0xFFF; Owner = 
PCI [00|01|00:14]
    Base = 0x30001000;    Length = 0x100;    Alignment = 0xFFF; Owner = 
PCI [00|01|00:10]


but kernel side: it becomes 64bit pref

[    3.005355] pci_bus 0000:00: root bus resource [mem 
0x10000000-0x3efeffff window]
[    3.006028] pci_bus 0000:00: root bus resource [bus 00-0f]
.
.
.
[    3.135847] pci 0000:00:01.0: BAR 2: assigned [mem 
0x10000000-0x1fffffff 64bit pref]
[    3.137099] pci 0000:00:01.0: BAR 1: assigned [mem 0x20000000-0x20000fff]
[    3.137382] pci 0000:00:01.0: BAR 0: assigned [mem 0x20001000-0x200010ff]



Also, I found that [mem 0x8000000000 window] was ignored,

[    2.769608] PCI ECAM: ECAM for domain 0000 [bus 00-0f] at [mem 
0x3f000000-0x3fffffff] (base 0x3f000000)
[    2.962930] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-0f])
[    2.965787] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM 
ClockPM Segments MSI]
[    2.990794] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME 
AER PCIeCapability]
[    2.994520] acpi PNP0A08:00: host bridge window [mem 0x8000000000 
window] (ignored, not CPU addressable)


so when do "cat /proc/iomem"
no "8000000000-8000000000 : PCI Bus 0000:00" shows.
Curious is that it works well and shows in another fedora kernel 
provided by Shanon.

Do you have any idea that why this happen? Thanks.



Best,
Liang
> Laszlo
>