From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35692) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a4BCr-000313-6A for qemu-devel@nongnu.org; Wed, 02 Dec 2015 12:28:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a4BCk-0005Lv-S5 for qemu-devel@nongnu.org; Wed, 02 Dec 2015 12:28:41 -0500 Received: from g1t6220.austin.hp.com ([15.73.96.84]:36657) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a4BCk-0005L2-KR for qemu-devel@nongnu.org; Wed, 02 Dec 2015 12:28:34 -0500 References: <563A8509.4010102@hpe.com> <563AA879.2030301@redhat.com> <565C993E.9000605@hpe.com> <565CC838.7090204@redhat.com> <565CEDE2.3020602@hpe.com> <565CFBAF.3030707@redhat.com> From: liang yan Message-ID: <565F2A39.5000703@hpe.com> Date: Wed, 2 Dec 2015 10:28:25 -0700 MIME-Version: 1.0 In-Reply-To: <565CFBAF.3030707@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [edk2] Could not add PCI device with big memory to aarch64 VMs List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Laszlo Ersek Cc: "Hayes, Bill" , edk2-devel@ml01.01.org, qemu-devel@nongnu.org, "Ramirez, Laura L (HP Labs)" , Ard Biesheuvel Hi, Laszlo, On 11/30/2015 06:45 PM, Laszlo Ersek wrote: > On 12/01/15 01:46, liang yan wrote: >> Hello, Laszlo, >> >> On 11/30/2015 03:05 PM, Laszlo Ersek wrote: > [snip] > >>> If you need more room (with large alignments), then there's no way >>> around supporting QEMU's 64 bit aperture, VIRT_PCIE_MMIO_HIGH (see my >>> earlier email). >> I checked the function create_pcie form pathtoqemu/hw/arm/virt.c, it has >> a flag value use_highmem(which has default "true" value). >> >> It set base_mmio_high and size_mmio_high to device tree by function below, >> >> qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "ranges", >> 1, FDT_PCI_RANGE_IOPORT, 2, 0, >> 2, base_pio, 2, size_pio, >> 1, FDT_PCI_RANGE_MMIO, 2, base_mmio, >> 2, base_mmio, 2, size_mmio, >> 1, FDT_PCI_RANGE_MMIO_64BIT, >> 2, base_mmio_high, >> 2, base_mmio_high, 2, size_mmio_high); >> >> So basically, I need to add two UINT64 variables like mmio_high_base and >> mmio_high_size to PCD under function ProcessPciHost(VirtFdtDxe.c), >> and try to use this high base address and size as new aperture. >> >> Is this correct? > It is correct, but that's only part of the story. > > Parsing the 64-bit aperture from the DTB into new PCDs in > ArmVirtPkg/VirtFdtDxe is the easy part. > > The hard part is modifying ArmVirtPkg/PciHostBridgeDxe, so that BAR > allocation requests (submitted by the platform independent PCI bus > driver that resides in "MdeModulePkg/Bus/Pci/PciBusDxe") are actually > serviced from this high aperture too. > >>> Unfortunately I can't readily help with that in the >>> "ArmVirtPkg/PciHostBridgeDxe" driver; there's no such (open-source) >>> example in the edk2 tree. Of course, I could experiment with it myself >>> -- only not right now. >> If possible, I do want to finish this part or help you finish it. I just >> work on UEFI recently, and thank you so much for your patient and detail >> explanation. I really appreciate it. >>> I guess copying and adapting the TypeMem32 logic to TypeMem64 (currently >>> short-circuited with EFI_ABORTED) could work. >> Is the 32 or 64 bit determined by BAR(2-3bit) or by the PCI device >> memory size? Is there an option from QEMU? > I can't tell. :) > >> Does TypeMem32 still keep "VIRT_PCIE_MMIO" aperture and TypeMem64 use >> "VIRT_PCIE_MMIO_HIGH" aperture? or It's more like device property >> controlled from QEMU device simulation? > Good question. I don't know. I think in order to answer this question, > we should understand the whole dance between the PCI root bridge / host > bridge driver and the generic PCI bus driver. > > The documentation I know of is in the Platform Init spec, version 1.4, > Volume 5, Chapter 10 "PCI Host Bridge". I've taken multiple stabs at > that chapter earlier, but I've always given up. > > Sorry I can't help more, but this is new area for me as well. No, already a big help, Really appreciate your generous sharing. I also have a problem from guest vm kernel side. Even we change the UEFI side, should the guest kernel still need to modify? Because, I noticed that the kernel will do rescan. use the example of 256M below. UEFI side, has two setup, not sure which is the real one. PciBus: Discovered PCI @ [00|01|00] BAR[0]: Type = Mem32; Alignment = 0xFFF; Length = 0x100; Offset = 0x10 BAR[1]: Type = Mem32; Alignment = 0xFFF; Length = 0x1000; Offset = 0x14 BAR[2]: Type = PMem64; Alignment = 0xFFFFFFF; Length = 0x10000000; Offset = 0x18 ===== PMem64 here PciBus: Resource Map for Root Bridge PciRoot(0x0) Type = Mem32; Base = 0x20000000; Length = 0x10100000; Alignment = 0xFFFFFFF Base = 0x20000000; Length = 0x10000000; Alignment = 0xFFFFFFF; Owner = PCI [00|01|00:18]; Type = PMem32 ============>Mem32 here Base = 0x30000000; Length = 0x1000; Alignment = 0xFFF; Owner = PCI [00|01|00:14] Base = 0x30001000; Length = 0x100; Alignment = 0xFFF; Owner = PCI [00|01|00:10] but kernel side: it becomes 64bit pref [ 3.005355] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff window] [ 3.006028] pci_bus 0000:00: root bus resource [bus 00-0f] . . . [ 3.135847] pci 0000:00:01.0: BAR 2: assigned [mem 0x10000000-0x1fffffff 64bit pref] [ 3.137099] pci 0000:00:01.0: BAR 1: assigned [mem 0x20000000-0x20000fff] [ 3.137382] pci 0000:00:01.0: BAR 0: assigned [mem 0x20001000-0x200010ff] Also, I found that [mem 0x8000000000 window] was ignored, [ 2.769608] PCI ECAM: ECAM for domain 0000 [bus 00-0f] at [mem 0x3f000000-0x3fffffff] (base 0x3f000000) [ 2.962930] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-0f]) [ 2.965787] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.990794] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability] [ 2.994520] acpi PNP0A08:00: host bridge window [mem 0x8000000000 window] (ignored, not CPU addressable) so when do "cat /proc/iomem" no "8000000000-8000000000 : PCI Bus 0000:00" shows. Curious is that it works well and shows in another fedora kernel provided by Shanon. Do you have any idea that why this happen? Thanks. Best, Liang > Laszlo >