[Qemu-devel] Could not add PCI device with big memory to aarch64 VMs

[Qemu-devel] Could not add PCI device with big memory to aarch64 VMs

[liang yan]

Hello, Laszlo,

(1)I am trying to add ivshmem device(PCI device with big memory) to my 
aarch64 vm.
So far, I could find device information from vm. But it seems vm did not 
create
correct resource file for this device. Do you have any idea that this 
happens?

I used the upstream EDK2 to build my UEFI firmware.

There are three BARs for this device, and memory map is assigned too, 
but only one
resource file is created.

My qemu supports ACPI 5.1 and the command line is :

   -device ivshmem,size=256M,chardev=ivshmem,msi=on,ioeventfd=on \
   -chardev socket,path=/tmp/ivshmem_socket,id=ivshmem \

The lspci information:

00:00.0 Host bridge: Red Hat, Inc. Device 0008
     Subsystem: Red Hat, Inc Device 1100
     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
     Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-

00:01.0 RAM memory: Red Hat, Inc Inter-VM shared memory
     Subsystem: Red Hat, Inc QEMU Virtual Machine
     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
     Interrupt: pin A routed to IRQ 255
     Region 0: Memory at 20001000 (32-bit, non-prefetchable) [disabled] 
[size=256]
     Region 1: Memory at 20000000 (32-bit, non-prefetchable) [disabled] 
[size=4K]
     Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled] 
[size=256M]
     Capabilities: [40] MSI-X: Enable- Count=1 Masked-
         Vector table: BAR=1 offset=00000000
         PBA: BAR=1 offset=00000800

Boot information:

[    2.380924] pci 0000:00:01.0: BAR 2: assigned [mem 
0x10000000-0x1fffffff 64bit pref]
[    2.382836] pci 0000:00:01.0: BAR 1: assigned [mem 0x20000000-0x20000fff]
[    2.383557] pci 0000:00:01.0: BAR 0: assigned [mem 0x20001000-0x200010ff]


Files under /sys/devices/pci0000:00/0000:00:01.0

broken_parity_status      devspec       local_cpus  resource
class              dma_mask_bits    modalias    subsystem
config              driver_override  msi_bus     subsystem_device
consistent_dma_mask_bits  enable       power       subsystem_vendor
d3cold_allowed          irq           remove      uevent
device              local_cpulist    rescan      vendor

Information for resource:

0x0000000020001000 0x00000000200010ff 0x0000000000040200
0x0000000020000000 0x0000000020000fff 0x0000000000040200
0x0000000010000000 0x000000001fffffff 0x000000000014220c
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000




(2)It also has a problem that once I use a memory bigger than 256M for 
ivshmem, it could not get through UEFI,
the error message is

PciBus: Discovered PCI @ [00|01|00]
    BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100; Offset 
= 0x10
    BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset 
= 0x14
    BAR[2]: Type = PMem64; Alignment = 0x3FFFFFFF;    Length = 
0x40000000;    Offset = 0x18

PciBus: HostBridge->SubmitResources() - Success
ASSERT 
/home/liang/studio/edk2/ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c(449): ((BOOLEAN)(0==1))

I am wandering if there are memory limitation for pcie devices under 
Qemu environment?


Just thank you in advance and any information would be appreciated.



Best,
Liang

Re: [Qemu-devel] Could not add PCI device with big memory to aarch64 VMs

[Laszlo Ersek]

On 11/04/15 23:22, liang yan wrote:
> Hello, Laszlo,
> 
> (1)I am trying to add ivshmem device(PCI device with big memory) to my
> aarch64 vm.
> So far, I could find device information from vm. But it seems vm did not
> create
> correct resource file for this device. Do you have any idea that this
> happens?
> 
> I used the upstream EDK2 to build my UEFI firmware.
> 
> There are three BARs for this device, and memory map is assigned too,
> but only one
> resource file is created.
> 
> My qemu supports ACPI 5.1 and the command line is :
> 
>   -device ivshmem,size=256M,chardev=ivshmem,msi=on,ioeventfd=on \
>   -chardev socket,path=/tmp/ivshmem_socket,id=ivshmem \
> 
> The lspci information:
> 
> 00:00.0 Host bridge: Red Hat, Inc. Device 0008
>     Subsystem: Red Hat, Inc Device 1100
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> 
> 00:01.0 RAM memory: Red Hat, Inc Inter-VM shared memory
>     Subsystem: Red Hat, Inc QEMU Virtual Machine
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>     Interrupt: pin A routed to IRQ 255
>     Region 0: Memory at 20001000 (32-bit, non-prefetchable) [disabled]
> [size=256]
>     Region 1: Memory at 20000000 (32-bit, non-prefetchable) [disabled]
> [size=4K]
>     Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
> [size=256M]
>     Capabilities: [40] MSI-X: Enable- Count=1 Masked-
>         Vector table: BAR=1 offset=00000000
>         PBA: BAR=1 offset=00000800
> 
> Boot information:
> 
> [    2.380924] pci 0000:00:01.0: BAR 2: assigned [mem
> 0x10000000-0x1fffffff 64bit pref]
> [    2.382836] pci 0000:00:01.0: BAR 1: assigned [mem
> 0x20000000-0x20000fff]
> [    2.383557] pci 0000:00:01.0: BAR 0: assigned [mem
> 0x20001000-0x200010ff]
> 
> 
> Files under /sys/devices/pci0000:00/0000:00:01.0
> 
> broken_parity_status      devspec       local_cpus  resource
> class              dma_mask_bits    modalias    subsystem
> config              driver_override  msi_bus     subsystem_device
> consistent_dma_mask_bits  enable       power       subsystem_vendor
> d3cold_allowed          irq           remove      uevent
> device              local_cpulist    rescan      vendor
> 
> Information for resource:
> 
> 0x0000000020001000 0x00000000200010ff 0x0000000000040200
> 0x0000000020000000 0x0000000020000fff 0x0000000000040200
> 0x0000000010000000 0x000000001fffffff 0x000000000014220c
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 
> 
> 
> 
> (2)It also has a problem that once I use a memory bigger than 256M for
> ivshmem, it could not get through UEFI,
> the error message is
> 
> PciBus: Discovered PCI @ [00|01|00]
>    BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100; Offset =
> 0x10
>    BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset
> = 0x14
>    BAR[2]: Type = PMem64; Alignment = 0x3FFFFFFF;    Length =
> 0x40000000;    Offset = 0x18
> 
> PciBus: HostBridge->SubmitResources() - Success
> ASSERT
> /home/liang/studio/edk2/ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c(449): ((BOOLEAN)(0==1))
> 
> 
> I am wandering if there are memory limitation for pcie devices under
> Qemu environment?
> 
> 
> Just thank you in advance and any information would be appreciated.

(CC'ing Ard.)

"Apparently", the firmware-side counterpart of QEMU commit 5125f9cd2532
has never been contributed to edk2.

Therefore the the ProcessPciHost() function in
"ArmVirtPkg/VirtFdtDxe/VirtFdtDxe.c" ignores the
DTB_PCI_HOST_RANGE_MMIO64 type range from the DTB. (Thus only
DTB_PCI_HOST_RANGE_MMIO32 is recognized as PCI MMIO aperture.)

However, even if said driver was extended to parse the new 64-bit
aperture into PCDs (which wouldn't be hard), the
ArmVirtPkg/PciHostBridgeDxe driver would still have to be taught to look
at that aperture (from the PCDs) and to serve MMIO BAR allocation
requests from it. That could be hard.

Please check edk2 commits e48f1f15b0e2^..e5ceb6c9d390, approximately,
for the background on the current code. See also chapter 13 "Protocols -
PCI Bus Support" in the UEFI spec.

Patches welcome. :)

(A separate note on ACPI vs. DT: the firmware forwards *both* from QEMU
to the runtime guest OS. However, the firmware parses only the DT for
its own purposes.)

Thanks
Laszlo

Re: [Qemu-devel] Could not add PCI device with big memory to aarch64 VMs

[liang yan]

On 11/04/2015 05:53 PM, Laszlo Ersek wrote:
> On 11/04/15 23:22, liang yan wrote:
>> Hello, Laszlo,
>>
>>
>> (2)It also has a problem that once I use a memory bigger than 256M for
>> ivshmem, it could not get through UEFI,
>> the error message is
>>
>> PciBus: Discovered PCI @ [00|01|00]
>>     BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100; Offset =
>> 0x10
>>     BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset
>> = 0x14
>>     BAR[2]: Type = PMem64; Alignment = 0x3FFFFFFF;    Length =
>> 0x40000000;    Offset = 0x18
>>
>> PciBus: HostBridge->SubmitResources() - Success
>> ASSERT
>> /home/liang/studio/edk2/ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c(449): ((BOOLEAN)(0==1))
>>
>>
>> I am wandering if there are memory limitation for pcie devices under
>> Qemu environment?
>>
>>
>> Just thank you in advance and any information would be appreciated.
> (CC'ing Ard.)
>
> "Apparently", the firmware-side counterpart of QEMU commit 5125f9cd2532
> has never been contributed to edk2.
>
> Therefore the the ProcessPciHost() function in
> "ArmVirtPkg/VirtFdtDxe/VirtFdtDxe.c" ignores the
> DTB_PCI_HOST_RANGE_MMIO64 type range from the DTB. (Thus only
> DTB_PCI_HOST_RANGE_MMIO32 is recognized as PCI MMIO aperture.)
>
> However, even if said driver was extended to parse the new 64-bit
> aperture into PCDs (which wouldn't be hard), the
> ArmVirtPkg/PciHostBridgeDxe driver would still have to be taught to look
> at that aperture (from the PCDs) and to serve MMIO BAR allocation
> requests from it. That could be hard.
>
> Please check edk2 commits e48f1f15b0e2^..e5ceb6c9d390, approximately,
> for the background on the current code. See also chapter 13 "Protocols -
> PCI Bus Support" in the UEFI spec.
>
> Patches welcome. :)
>
> (A separate note on ACPI vs. DT: the firmware forwards *both* from QEMU
> to the runtime guest OS. However, the firmware parses only the DT for
> its own purposes.)
Hello, Laszlo,

Thanks for your advices above, it's very helpful.

When debugging, I also found some problems for 32 bit PCI devices.
Hope could get some clues from you.

I checked on 512M, 1G, and 2G devices.(4G return invalid parameter 
error, so I think it may be taken as a 64bit devices, is this right?).


First,

All devices start from base address 3EFEFFFF.

ProcessPciHost: Config[0x3F000000+0x1000000) Bus[0x0..0xF] 
Io[0x0+0x10000)@0x3EFF0000 Mem[0x10000000+0x2EFF0000)@0x0

PcdPciMmio32Base is  10000000=====================
PcdPciMmio32Size is  2EFF0000=====================


Second,

It could not get new base address when searching memory space in GCD map.

For 512M devices,

*BaseAddress = (*BaseAddress + 1 - Length) & (~AlignmentMask);

BaseAddress is 3EFEFFFF==========================
new BaseAddress is 1EEF0000==========================
~AlignmentMask is E0000000==========================
Final BaseAddress is 0000

Status = CoreSearchGcdMapEntry (*BaseAddress, Length, &StartLink, 
&EndLink, Map);



For bigger devices:

all stops when searching memory space because below code, Length will 
bigger than MaxAddress(3EFEFFFF)

if ((Entry->BaseAddress + Length) > MaxAddress) {
          continue;
}


I also checked on ArmVirtQemu.dsc which all set to 0.

   gArmPlatformTokenSpaceGuid.PcdPciBusMin|0x0
   gArmPlatformTokenSpaceGuid.PcdPciBusMax|0x0
   gArmPlatformTokenSpaceGuid.PcdPciIoBase|0x0
   gArmPlatformTokenSpaceGuid.PcdPciIoSize|0x0
   gArmPlatformTokenSpaceGuid.PcdPciIoTranslation|0x0
   gArmPlatformTokenSpaceGuid.PcdPciMmio32Base|0x0
   gArmPlatformTokenSpaceGuid.PcdPciMmio32Size|0x0
   gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress|0x0


Do you think I should change from PcdPciMmio32Base and PcdPciMmio32Size, 
or do some change for GCD entry list, so it could allocate resources for 
PCI devices(CoreSearchGcdMapEntry)?


Looking forward to your reply.


Thanks,
Liang

> Thanks
> Laszlo
>

Re: [Qemu-devel] [edk2] Could not add PCI device with big memory to aarch64 VMs

[Laszlo Ersek]

On 11/30/15 19:45, liang yan wrote:
> 
> 
> On 11/04/2015 05:53 PM, Laszlo Ersek wrote:
>> On 11/04/15 23:22, liang yan wrote:
>>> Hello, Laszlo,
>>>
>>>
>>> (2)It also has a problem that once I use a memory bigger than 256M for
>>> ivshmem, it could not get through UEFI,
>>> the error message is
>>>
>>> PciBus: Discovered PCI @ [00|01|00]
>>>     BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100;
>>> Offset =
>>> 0x10
>>>     BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset
>>> = 0x14
>>>     BAR[2]: Type = PMem64; Alignment = 0x3FFFFFFF;    Length =
>>> 0x40000000;    Offset = 0x18
>>>
>>> PciBus: HostBridge->SubmitResources() - Success
>>> ASSERT
>>> /home/liang/studio/edk2/ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c(449):
>>> ((BOOLEAN)(0==1))
>>>
>>>
>>> I am wandering if there are memory limitation for pcie devices under
>>> Qemu environment?
>>>
>>>
>>> Just thank you in advance and any information would be appreciated.
>> (CC'ing Ard.)
>>
>> "Apparently", the firmware-side counterpart of QEMU commit 5125f9cd2532
>> has never been contributed to edk2.
>>
>> Therefore the the ProcessPciHost() function in
>> "ArmVirtPkg/VirtFdtDxe/VirtFdtDxe.c" ignores the
>> DTB_PCI_HOST_RANGE_MMIO64 type range from the DTB. (Thus only
>> DTB_PCI_HOST_RANGE_MMIO32 is recognized as PCI MMIO aperture.)
>>
>> However, even if said driver was extended to parse the new 64-bit
>> aperture into PCDs (which wouldn't be hard), the
>> ArmVirtPkg/PciHostBridgeDxe driver would still have to be taught to look
>> at that aperture (from the PCDs) and to serve MMIO BAR allocation
>> requests from it. That could be hard.
>>
>> Please check edk2 commits e48f1f15b0e2^..e5ceb6c9d390, approximately,
>> for the background on the current code. See also chapter 13 "Protocols -
>> PCI Bus Support" in the UEFI spec.
>>
>> Patches welcome. :)
>>
>> (A separate note on ACPI vs. DT: the firmware forwards *both* from QEMU
>> to the runtime guest OS. However, the firmware parses only the DT for
>> its own purposes.)
> Hello, Laszlo,
> 
> Thanks for your advices above, it's very helpful.
> 
> When debugging, I also found some problems for 32 bit PCI devices.
> Hope could get some clues from you.
> 
> I checked on 512M, 1G, and 2G devices.(4G return invalid parameter
> error, so I think it may be taken as a 64bit devices, is this right?).

I guess so.

> 
> 
> First,
> 
> All devices start from base address 3EFEFFFF.

According to the below:

> ProcessPciHost: Config[0x3F000000+0x1000000) Bus[0x0..0xF]
> Io[0x0+0x10000)@0x3EFF0000 Mem[0x10000000+0x2EFF0000)@0x0

the address you mention (0x3EFEFFFF) is the *highest* inclusive
guest-phys address that an MMIO BAR can take. Not sure if that's what
you meant.

The size of the MMIO aperture for the entire PCI host is 0x2EFF0000
bytes: a little less than 752 MB. So devices that need 1G and 2G MMIO
BARs have no chance.

> PcdPciMmio32Base is  10000000=====================
> PcdPciMmio32Size is  2EFF0000=====================
> 
> 
> Second,
> 
> It could not get new base address when searching memory space in GCD map.
> 
> For 512M devices,
> 
> *BaseAddress = (*BaseAddress + 1 - Length) & (~AlignmentMask);

This seems to be from CoreAllocateSpace()
[MdeModulePkg/Core/Dxe/Gcd/Gcd.c]. AlignmentMask is computed from the
Alignment input parameter.

Which in turn seems to come from the BitsOfAlignment parameter computed
in NotifyPhase(), "ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c".

> 
> BaseAddress is 3EFEFFFF==========================

So this is the highest address (inclusive) where the 512MB BAR could end.

> new BaseAddress is 1EEF0000==========================

This is the highest address (inclusive) where the 512MB BAR could start.

This should be rounded down to an 512MB alignment (I believe), and then
checked if that is still in the MMIO aperture.

512MB is 0x2000_0000.

Rounding 0x1EEF_0000 down to an integral multiple of 0x2000_0000 results
in zero:

> ~AlignmentMask is E0000000==========================
> Final BaseAddress is 0000

And that address does not fall into the MMIO aperture.

In other words, although your size requirement of 512MB could be
theoretically satisfied from the aperture (which extends from exactly
256 MB to a little lower than 1008 MB), if you *also* require the base
address to be aligned at 512MB, then that cannot be satisfied.

> 
> Status = CoreSearchGcdMapEntry (*BaseAddress, Length, &StartLink,
> &EndLink, Map);
> 
> 
> 
> For bigger devices:
> 
> all stops when searching memory space because below code, Length will
> bigger than MaxAddress(3EFEFFFF)
> 
> if ((Entry->BaseAddress + Length) > MaxAddress) {
>          continue;
> }
> 
> 
> I also checked on ArmVirtQemu.dsc which all set to 0.
> 
>   gArmPlatformTokenSpaceGuid.PcdPciBusMin|0x0
>   gArmPlatformTokenSpaceGuid.PcdPciBusMax|0x0
>   gArmPlatformTokenSpaceGuid.PcdPciIoBase|0x0
>   gArmPlatformTokenSpaceGuid.PcdPciIoSize|0x0
>   gArmPlatformTokenSpaceGuid.PcdPciIoTranslation|0x0
>   gArmPlatformTokenSpaceGuid.PcdPciMmio32Base|0x0
>   gArmPlatformTokenSpaceGuid.PcdPciMmio32Size|0x0
>   gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress|0x0

These are just default values for the PCDs, to be set later by
VirtFdtDxe, keyed off of QEMU's DTB.

(In a platform DSC, you cannot mark a PCD as dynamic without providing a
default value too. Hence the actual default values here aren't relevant.)

> 
> 
> Do you think I should change from PcdPciMmio32Base and PcdPciMmio32Size,
> or do some change for GCD entry list, so it could allocate resources for
> PCI devices(CoreSearchGcdMapEntry)?

No, I think the code is working as expected.

For devices with 1GB and larger MMIO BAR requirements, the host bridge
simply doesn't have enough room in its MMIO aperture.

For a device with an 512MB MMIO BAR requirement, there would be enough
room (considering nothing but the size); however the alignment
requirement of the BAR makes the request impossible to satisfy. The only
address aligned at 512MB in the host bridge aperture is the absolute
address 512MB itself, and from that point upwards, you don't have 512MB
space, only approx. 1008 - 512 = 496 MB.

This is all rooted in the specific, numerical boundaries of the
VIRT_PCIE_MMIO entry, in QEMU's "hw/arm/virt.c".

Now, if you changed *those* values (that is, the VIRT_PCIE_MMIO
boundaries in QEMU), then that would trickle down to the firmware as
well, through the generated DTB, and then larger devices might work.
But, of course this is not the right way to go about it; that's why QEMU
introduced VIRT_PCIE_MMIO_HIGH.

... You didn't mention 256MB devices -- I assume that is because those
work already. Namely, two such BARs should be possible in the current
aperture; I think at 256MB and at 512MB.

If you need more room (with large alignments), then there's no way
around supporting QEMU's 64 bit aperture, VIRT_PCIE_MMIO_HIGH (see my
earlier email).

Unfortunately I can't readily help with that in the
"ArmVirtPkg/PciHostBridgeDxe" driver; there's no such (open-source)
example in the edk2 tree. Of course, I could experiment with it myself
-- only not right now.

I guess copying and adapting the TypeMem32 logic to TypeMem64 (currently
short-circuited with EFI_ABORTED) could work.

Thanks
Laszlo

> Looking forward to your reply.
> 
> 
> Thanks,
> Liang
> 
>> Thanks
>> Laszlo
>>
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: [Qemu-devel] [edk2] Could not add PCI device with big memory to aarch64 VMs

[liang yan]

Hello, Laszlo,

On 11/30/2015 03:05 PM, Laszlo Ersek wrote:
> On 11/30/15 19:45, liang yan wrote:
>>
>> On 11/04/2015 05:53 PM, Laszlo Ersek wrote:
>>> On 11/04/15 23:22, liang yan wrote:
>>>> Hello, Laszlo,
>>>>
>>>>
>>>> (2)It also has a problem that once I use a memory bigger than 256M for
>>>> ivshmem, it could not get through UEFI,
>>>> the error message is
>>>>
>>>> PciBus: Discovered PCI @ [00|01|00]
>>>>      BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100;
>>>> Offset =
>>>> 0x10
>>>>      BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset
>>>> = 0x14
>>>>      BAR[2]: Type = PMem64; Alignment = 0x3FFFFFFF;    Length =
>>>> 0x40000000;    Offset = 0x18
>>>>
>>>> PciBus: HostBridge->SubmitResources() - Success
>>>> ASSERT
>>>> /home/liang/studio/edk2/ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c(449):
>>>> ((BOOLEAN)(0==1))
>>>>
>>>>
>>>> I am wandering if there are memory limitation for pcie devices under
>>>> Qemu environment?
>>>>
>>>>
>>>> Just thank you in advance and any information would be appreciated.
>>> (CC'ing Ard.)
>>>
>>> "Apparently", the firmware-side counterpart of QEMU commit 5125f9cd2532
>>> has never been contributed to edk2.
>>>
>>> Therefore the the ProcessPciHost() function in
>>> "ArmVirtPkg/VirtFdtDxe/VirtFdtDxe.c" ignores the
>>> DTB_PCI_HOST_RANGE_MMIO64 type range from the DTB. (Thus only
>>> DTB_PCI_HOST_RANGE_MMIO32 is recognized as PCI MMIO aperture.)
>>>
>>> However, even if said driver was extended to parse the new 64-bit
>>> aperture into PCDs (which wouldn't be hard), the
>>> ArmVirtPkg/PciHostBridgeDxe driver would still have to be taught to look
>>> at that aperture (from the PCDs) and to serve MMIO BAR allocation
>>> requests from it. That could be hard.
>>>
>>> Please check edk2 commits e48f1f15b0e2^..e5ceb6c9d390, approximately,
>>> for the background on the current code. See also chapter 13 "Protocols -
>>> PCI Bus Support" in the UEFI spec.
>>>
>>> Patches welcome. :)
>>>
>>> (A separate note on ACPI vs. DT: the firmware forwards *both* from QEMU
>>> to the runtime guest OS. However, the firmware parses only the DT for
>>> its own purposes.)
>> Hello, Laszlo,
>>
>> Thanks for your advices above, it's very helpful.
>>
>> When debugging, I also found some problems for 32 bit PCI devices.
>> Hope could get some clues from you.
>>
>> I checked on 512M, 1G, and 2G devices.(4G return invalid parameter
>> error, so I think it may be taken as a 64bit devices, is this right?).
> I guess so.
>
>>
>> First,
>>
>> All devices start from base address 3EFEFFFF.
> According to the below:
>
>> ProcessPciHost: Config[0x3F000000+0x1000000) Bus[0x0..0xF]
>> Io[0x0+0x10000)@0x3EFF0000 Mem[0x10000000+0x2EFF0000)@0x0
> the address you mention (0x3EFEFFFF) is the *highest* inclusive
> guest-phys address that an MMIO BAR can take. Not sure if that's what
> you meant.

Yes, you are right, current allocation is 
EfiGcdAllocateMaxAddressSearchTopDown, so base address here is the 
highest inclusive address.

> The size of the MMIO aperture for the entire PCI host is 0x2EFF0000
> bytes: a little less than 752 MB. So devices that need 1G and 2G MMIO
> BARs have no chance.
>
>> PcdPciMmio32Base is  10000000=====================
>> PcdPciMmio32Size is  2EFF0000=====================
>>
>>
>> Second,
>>
>> It could not get new base address when searching memory space in GCD map.
>>
>> For 512M devices,
>>
>> *BaseAddress = (*BaseAddress + 1 - Length) & (~AlignmentMask);
> This seems to be from CoreAllocateSpace()
> [MdeModulePkg/Core/Dxe/Gcd/Gcd.c]. AlignmentMask is computed from the
> Alignment input parameter.
>
> Which in turn seems to come from the BitsOfAlignment parameter computed
> in NotifyPhase(), "ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c".
>
>> BaseAddress is 3EFEFFFF==========================
> So this is the highest address (inclusive) where the 512MB BAR could end.
>
>> new BaseAddress is 1EEF0000==========================
> This is the highest address (inclusive) where the 512MB BAR could start.
>
> This should be rounded down to an 512MB alignment (I believe), and then
> checked if that is still in the MMIO aperture.
>
> 512MB is 0x2000_0000.
>
> Rounding 0x1EEF_0000 down to an integral multiple of 0x2000_0000 results
> in zero:
>
>> ~AlignmentMask is E0000000==========================
>> Final BaseAddress is 0000
> And that address does not fall into the MMIO aperture.
>
> In other words, although your size requirement of 512MB could be
> theoretically satisfied from the aperture (which extends from exactly
> 256 MB to a little lower than 1008 MB), if you *also* require the base
> address to be aligned at 512MB, then that cannot be satisfied.
>
Thanks for the detail explanation above, I will write email with detail 
information too. Really appreciate.

>> Status = CoreSearchGcdMapEntry (*BaseAddress, Length, &StartLink,
>> &EndLink, Map);
>>
>>
>>
>> For bigger devices:
>>
>> all stops when searching memory space because below code, Length will
>> bigger than MaxAddress(3EFEFFFF)
>>
>> if ((Entry->BaseAddress + Length) > MaxAddress) {
>>           continue;
>> }
>>
>>
>> I also checked on ArmVirtQemu.dsc which all set to 0.
>>
>>    gArmPlatformTokenSpaceGuid.PcdPciBusMin|0x0
>>    gArmPlatformTokenSpaceGuid.PcdPciBusMax|0x0
>>    gArmPlatformTokenSpaceGuid.PcdPciIoBase|0x0
>>    gArmPlatformTokenSpaceGuid.PcdPciIoSize|0x0
>>    gArmPlatformTokenSpaceGuid.PcdPciIoTranslation|0x0
>>    gArmPlatformTokenSpaceGuid.PcdPciMmio32Base|0x0
>>    gArmPlatformTokenSpaceGuid.PcdPciMmio32Size|0x0
>>    gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress|0x0
> These are just default values for the PCDs, to be set later by
> VirtFdtDxe, keyed off of QEMU's DTB.
>
> (In a platform DSC, you cannot mark a PCD as dynamic without providing a
> default value too. Hence the actual default values here aren't relevant.)
>
>>
>> Do you think I should change from PcdPciMmio32Base and PcdPciMmio32Size,
>> or do some change for GCD entry list, so it could allocate resources for
>> PCI devices(CoreSearchGcdMapEntry)?
> No, I think the code is working as expected.
>
> For devices with 1GB and larger MMIO BAR requirements, the host bridge
> simply doesn't have enough room in its MMIO aperture.
>
> For a device with an 512MB MMIO BAR requirement, there would be enough
> room (considering nothing but the size); however the alignment
> requirement of the BAR makes the request impossible to satisfy. The only
> address aligned at 512MB in the host bridge aperture is the absolute
> address 512MB itself, and from that point upwards, you don't have 512MB
> space, only approx. 1008 - 512 = 496 MB.
>
> This is all rooted in the specific, numerical boundaries of the
> VIRT_PCIE_MMIO entry, in QEMU's "hw/arm/virt.c".
>
> Now, if you changed *those* values (that is, the VIRT_PCIE_MMIO
> boundaries in QEMU), then that would trickle down to the firmware as
> well, through the generated DTB, and then larger devices might work.
> But, of course this is not the right way to go about it; that's why QEMU
> introduced VIRT_PCIE_MMIO_HIGH.
>
> ... You didn't mention 256MB devices -- I assume that is because those
> work already. Namely, two such BARs should be possible in the current
> aperture; I think at 256MB and at 512MB.

Yes, 256M works correctly, 512M could not get the right base address 
when doing alignment,  just like you said above.

> If you need more room (with large alignments), then there's no way
> around supporting QEMU's 64 bit aperture, VIRT_PCIE_MMIO_HIGH (see my
> earlier email).
I checked the function create_pcie form pathtoqemu/hw/arm/virt.c, it has 
a flag value use_highmem(which has default "true" value).

It set base_mmio_high and size_mmio_high to device tree by function below,

         qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "ranges",
                                      1, FDT_PCI_RANGE_IOPORT, 2, 0,
                                      2, base_pio, 2, size_pio,
                                      1, FDT_PCI_RANGE_MMIO, 2, base_mmio,
                                      2, base_mmio, 2, size_mmio,
                                      1, FDT_PCI_RANGE_MMIO_64BIT,
                                      2, base_mmio_high,
                                      2, base_mmio_high, 2, size_mmio_high);

So basically, I need to add two UINT64 variables like mmio_high_base and 
mmio_high_size to PCD under function ProcessPciHost(VirtFdtDxe.c),
and try to use this high base address and size as new aperture.

Is this correct?

> Unfortunately I can't readily help with that in the
> "ArmVirtPkg/PciHostBridgeDxe" driver; there's no such (open-source)
> example in the edk2 tree. Of course, I could experiment with it myself
> -- only not right now.
If possible, I do want to finish this part or help you finish it. I just 
work on UEFI recently, and thank you so much for your patient and detail 
explanation. I really appreciate it.
> I guess copying and adapting the TypeMem32 logic to TypeMem64 (currently
> short-circuited with EFI_ABORTED) could work.

Is the 32 or 64 bit determined by BAR(2-3bit) or by the PCI device memory size? Is there an option from QEMU?

Does TypeMem32 still keep  "VIRT_PCIE_MMIO" aperture and TypeMem64 use 
"VIRT_PCIE_MMIO_HIGH" aperture? or It's more like device property 
controlled from QEMU device simulation?


Best,
Liang
> Thanks
> Laszlo
>
>> Looking forward to your reply.
>>
>>
>> Thanks,
>> Liang
>>
>>> Thanks
>>> Laszlo
>>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>

Re: [Qemu-devel] [edk2] Could not add PCI device with big memory to aarch64 VMs

[Laszlo Ersek]

On 12/01/15 01:46, liang yan wrote:
> Hello, Laszlo,
> 
> On 11/30/2015 03:05 PM, Laszlo Ersek wrote:

[snip]

>> If you need more room (with large alignments), then there's no way
>> around supporting QEMU's 64 bit aperture, VIRT_PCIE_MMIO_HIGH (see my
>> earlier email).
> I checked the function create_pcie form pathtoqemu/hw/arm/virt.c, it has
> a flag value use_highmem(which has default "true" value).
> 
> It set base_mmio_high and size_mmio_high to device tree by function below,
> 
>         qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "ranges",
>                                      1, FDT_PCI_RANGE_IOPORT, 2, 0,
>                                      2, base_pio, 2, size_pio,
>                                      1, FDT_PCI_RANGE_MMIO, 2, base_mmio,
>                                      2, base_mmio, 2, size_mmio,
>                                      1, FDT_PCI_RANGE_MMIO_64BIT,
>                                      2, base_mmio_high,
>                                      2, base_mmio_high, 2, size_mmio_high);
> 
> So basically, I need to add two UINT64 variables like mmio_high_base and
> mmio_high_size to PCD under function ProcessPciHost(VirtFdtDxe.c),
> and try to use this high base address and size as new aperture.
> 
> Is this correct?

It is correct, but that's only part of the story.

Parsing the 64-bit aperture from the DTB into new PCDs in
ArmVirtPkg/VirtFdtDxe is the easy part.

The hard part is modifying ArmVirtPkg/PciHostBridgeDxe, so that BAR
allocation requests (submitted by the platform independent PCI bus
driver that resides in "MdeModulePkg/Bus/Pci/PciBusDxe") are actually
serviced from this high aperture too.

> 
>> Unfortunately I can't readily help with that in the
>> "ArmVirtPkg/PciHostBridgeDxe" driver; there's no such (open-source)
>> example in the edk2 tree. Of course, I could experiment with it myself
>> -- only not right now.
> If possible, I do want to finish this part or help you finish it. I just
> work on UEFI recently, and thank you so much for your patient and detail
> explanation. I really appreciate it.
>> I guess copying and adapting the TypeMem32 logic to TypeMem64 (currently
>> short-circuited with EFI_ABORTED) could work.
> 
> Is the 32 or 64 bit determined by BAR(2-3bit) or by the PCI device
> memory size? Is there an option from QEMU?

I can't tell. :)

> Does TypeMem32 still keep  "VIRT_PCIE_MMIO" aperture and TypeMem64 use
> "VIRT_PCIE_MMIO_HIGH" aperture? or It's more like device property
> controlled from QEMU device simulation?

Good question. I don't know. I think in order to answer this question,
we should understand the whole dance between the PCI root bridge / host
bridge driver and the generic PCI bus driver.

The documentation I know of is in the Platform Init spec, version 1.4,
Volume 5, Chapter 10 "PCI Host Bridge". I've taken multiple stabs at
that chapter earlier, but I've always given up.

Sorry I can't help more, but this is new area for me as well.

Laszlo

Re: [Qemu-devel] [edk2] Could not add PCI device with big memory to aarch64 VMs

[liang yan]

Hi, Laszlo,

On 11/30/2015 06:45 PM, Laszlo Ersek wrote:
> On 12/01/15 01:46, liang yan wrote:
>> Hello, Laszlo,
>>
>> On 11/30/2015 03:05 PM, Laszlo Ersek wrote:
> [snip]
>
>>> If you need more room (with large alignments), then there's no way
>>> around supporting QEMU's 64 bit aperture, VIRT_PCIE_MMIO_HIGH (see my
>>> earlier email).
>> I checked the function create_pcie form pathtoqemu/hw/arm/virt.c, it has
>> a flag value use_highmem(which has default "true" value).
>>
>> It set base_mmio_high and size_mmio_high to device tree by function below,
>>
>>          qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "ranges",
>>                                       1, FDT_PCI_RANGE_IOPORT, 2, 0,
>>                                       2, base_pio, 2, size_pio,
>>                                       1, FDT_PCI_RANGE_MMIO, 2, base_mmio,
>>                                       2, base_mmio, 2, size_mmio,
>>                                       1, FDT_PCI_RANGE_MMIO_64BIT,
>>                                       2, base_mmio_high,
>>                                       2, base_mmio_high, 2, size_mmio_high);
>>
>> So basically, I need to add two UINT64 variables like mmio_high_base and
>> mmio_high_size to PCD under function ProcessPciHost(VirtFdtDxe.c),
>> and try to use this high base address and size as new aperture.
>>
>> Is this correct?
> It is correct, but that's only part of the story.
>
> Parsing the 64-bit aperture from the DTB into new PCDs in
> ArmVirtPkg/VirtFdtDxe is the easy part.
>
> The hard part is modifying ArmVirtPkg/PciHostBridgeDxe, so that BAR
> allocation requests (submitted by the platform independent PCI bus
> driver that resides in "MdeModulePkg/Bus/Pci/PciBusDxe") are actually
> serviced from this high aperture too.
>
>>> Unfortunately I can't readily help with that in the
>>> "ArmVirtPkg/PciHostBridgeDxe" driver; there's no such (open-source)
>>> example in the edk2 tree. Of course, I could experiment with it myself
>>> -- only not right now.
>> If possible, I do want to finish this part or help you finish it. I just
>> work on UEFI recently, and thank you so much for your patient and detail
>> explanation. I really appreciate it.
>>> I guess copying and adapting the TypeMem32 logic to TypeMem64 (currently
>>> short-circuited with EFI_ABORTED) could work.
>> Is the 32 or 64 bit determined by BAR(2-3bit) or by the PCI device
>> memory size? Is there an option from QEMU?
> I can't tell. :)
>
>> Does TypeMem32 still keep  "VIRT_PCIE_MMIO" aperture and TypeMem64 use
>> "VIRT_PCIE_MMIO_HIGH" aperture? or It's more like device property
>> controlled from QEMU device simulation?
> Good question. I don't know. I think in order to answer this question,
> we should understand the whole dance between the PCI root bridge / host
> bridge driver and the generic PCI bus driver.
>
> The documentation I know of is in the Platform Init spec, version 1.4,
> Volume 5, Chapter 10 "PCI Host Bridge". I've taken multiple stabs at
> that chapter earlier, but I've always given up.
>
> Sorry I can't help more, but this is new area for me as well.
No, already a big help, Really appreciate your generous sharing.

I also have a problem from guest vm kernel side.

Even we change the UEFI side, should the guest kernel still need to modify?
Because, I noticed that the kernel will do rescan.  use the example of 
256M below.

UEFI side, has two setup, not sure which is the real one.

PciBus: Discovered PCI @ [00|01|00]
    BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100; Offset 
= 0x10
    BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset 
= 0x14
    BAR[2]: Type = PMem64; Alignment = 0xFFFFFFF;    Length = 
0x10000000;    Offset = 0x18
                              ===== PMem64 here


PciBus: Resource Map for Root Bridge PciRoot(0x0)
Type =  Mem32; Base = 0x20000000;    Length = 0x10100000; Alignment = 
0xFFFFFFF
    Base = 0x20000000;    Length = 0x10000000;    Alignment = 
0xFFFFFFF;    Owner = PCI [00|01|00:18]; Type = PMem32 
============>Mem32 here
    Base = 0x30000000;    Length = 0x1000;    Alignment = 0xFFF; Owner = 
PCI [00|01|00:14]
    Base = 0x30001000;    Length = 0x100;    Alignment = 0xFFF; Owner = 
PCI [00|01|00:10]


but kernel side: it becomes 64bit pref

[    3.005355] pci_bus 0000:00: root bus resource [mem 
0x10000000-0x3efeffff window]
[    3.006028] pci_bus 0000:00: root bus resource [bus 00-0f]
.
.
.
[    3.135847] pci 0000:00:01.0: BAR 2: assigned [mem 
0x10000000-0x1fffffff 64bit pref]
[    3.137099] pci 0000:00:01.0: BAR 1: assigned [mem 0x20000000-0x20000fff]
[    3.137382] pci 0000:00:01.0: BAR 0: assigned [mem 0x20001000-0x200010ff]



Also, I found that [mem 0x8000000000 window] was ignored,

[    2.769608] PCI ECAM: ECAM for domain 0000 [bus 00-0f] at [mem 
0x3f000000-0x3fffffff] (base 0x3f000000)
[    2.962930] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-0f])
[    2.965787] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM 
ClockPM Segments MSI]
[    2.990794] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME 
AER PCIeCapability]
[    2.994520] acpi PNP0A08:00: host bridge window [mem 0x8000000000 
window] (ignored, not CPU addressable)


so when do "cat /proc/iomem"
no "8000000000-8000000000 : PCI Bus 0000:00" shows.
Curious is that it works well and shows in another fedora kernel 
provided by Shanon.

Do you have any idea that why this happen? Thanks.



Best,
Liang
> Laszlo
>

Re: [Qemu-devel] [edk2] Could not add PCI device with big memory to aarch64 VMs

[Laszlo Ersek]

On 12/02/15 18:28, liang yan wrote:
> Hi, Laszlo,
> 
> On 11/30/2015 06:45 PM, Laszlo Ersek wrote:
>> On 12/01/15 01:46, liang yan wrote:
>>> Hello, Laszlo,
>>>
>>> On 11/30/2015 03:05 PM, Laszlo Ersek wrote:
>> [snip]
>>
>>>> If you need more room (with large alignments), then there's no way
>>>> around supporting QEMU's 64 bit aperture, VIRT_PCIE_MMIO_HIGH (see my
>>>> earlier email).
>>> I checked the function create_pcie form pathtoqemu/hw/arm/virt.c, it has
>>> a flag value use_highmem(which has default "true" value).
>>>
>>> It set base_mmio_high and size_mmio_high to device tree by function
>>> below,
>>>
>>>          qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "ranges",
>>>                                       1, FDT_PCI_RANGE_IOPORT, 2, 0,
>>>                                       2, base_pio, 2, size_pio,
>>>                                       1, FDT_PCI_RANGE_MMIO, 2,
>>> base_mmio,
>>>                                       2, base_mmio, 2, size_mmio,
>>>                                       1, FDT_PCI_RANGE_MMIO_64BIT,
>>>                                       2, base_mmio_high,
>>>                                       2, base_mmio_high, 2,
>>> size_mmio_high);
>>>
>>> So basically, I need to add two UINT64 variables like mmio_high_base and
>>> mmio_high_size to PCD under function ProcessPciHost(VirtFdtDxe.c),
>>> and try to use this high base address and size as new aperture.
>>>
>>> Is this correct?
>> It is correct, but that's only part of the story.
>>
>> Parsing the 64-bit aperture from the DTB into new PCDs in
>> ArmVirtPkg/VirtFdtDxe is the easy part.
>>
>> The hard part is modifying ArmVirtPkg/PciHostBridgeDxe, so that BAR
>> allocation requests (submitted by the platform independent PCI bus
>> driver that resides in "MdeModulePkg/Bus/Pci/PciBusDxe") are actually
>> serviced from this high aperture too.
>>
>>>> Unfortunately I can't readily help with that in the
>>>> "ArmVirtPkg/PciHostBridgeDxe" driver; there's no such (open-source)
>>>> example in the edk2 tree. Of course, I could experiment with it myself
>>>> -- only not right now.
>>> If possible, I do want to finish this part or help you finish it. I just
>>> work on UEFI recently, and thank you so much for your patient and detail
>>> explanation. I really appreciate it.
>>>> I guess copying and adapting the TypeMem32 logic to TypeMem64
>>>> (currently
>>>> short-circuited with EFI_ABORTED) could work.
>>> Is the 32 or 64 bit determined by BAR(2-3bit) or by the PCI device
>>> memory size? Is there an option from QEMU?
>> I can't tell. :)
>>
>>> Does TypeMem32 still keep  "VIRT_PCIE_MMIO" aperture and TypeMem64 use
>>> "VIRT_PCIE_MMIO_HIGH" aperture? or It's more like device property
>>> controlled from QEMU device simulation?
>> Good question. I don't know. I think in order to answer this question,
>> we should understand the whole dance between the PCI root bridge / host
>> bridge driver and the generic PCI bus driver.
>>
>> The documentation I know of is in the Platform Init spec, version 1.4,
>> Volume 5, Chapter 10 "PCI Host Bridge". I've taken multiple stabs at
>> that chapter earlier, but I've always given up.
>>
>> Sorry I can't help more, but this is new area for me as well.
> No, already a big help, Really appreciate your generous sharing.
> 
> I also have a problem from guest vm kernel side.
> 
> Even we change the UEFI side, should the guest kernel still need to modify?

No, if the guest UEFI side is fully fixed / enhanced, then the guest
kernel won't need any changes. (Shouldn't.)

> Because, I noticed that the kernel will do rescan.  use the example of
> 256M below.
> 
> UEFI side, has two setup, not sure which is the real one.

Both of these are "real".

The first snippet bears witness to the enumeration / discovery of the
single PCI device 00:01.0:

> PciBus: Discovered PCI @ [00|01|00]
>    BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100; Offset =
> 0x10
>    BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset
> = 0x14
>    BAR[2]: Type = PMem64; Alignment = 0xFFFFFFF;    Length =
> 0x10000000;    Offset = 0x18
>                              ===== PMem64 here

Whereas the second snippet is the cumulative resource map for the root
bridge that the device is behind.

You get similar resource maps near the end of enumeration for PCI-to-PCI
bridges as well, if you have them. The bridge's resource map depends on
the devices behind it. In this case it could be confusing because you
have only one device (I guess).

> PciBus: Resource Map for Root Bridge PciRoot(0x0)
> Type =  Mem32; Base = 0x20000000;    Length = 0x10100000; Alignment =
> 0xFFFFFFF
>    Base = 0x20000000;    Length = 0x10000000;    Alignment =
> 0xFFFFFFF;    Owner = PCI [00|01|00:18]; Type = PMem32
> ============>Mem32 here
>    Base = 0x30000000;    Length = 0x1000;    Alignment = 0xFFF; Owner =
> PCI [00|01|00:14]
>    Base = 0x30001000;    Length = 0x100;    Alignment = 0xFFF; Owner =
> PCI [00|01|00:10]

So the above tells us that the PCI device would like to have a 64-bit
pmem BAR (or at least that it would be capable of working with one).

However, likely *exactly* because of the limitation that we have in the
resource allocation of the PCI host bridge / root bridge driver, that
256 MB BAR ends up being mapped under 4GB. This outcome is then visible
in the root bridge's resource map.

> but kernel side: it becomes 64bit pref
> 
> [    3.005355] pci_bus 0000:00: root bus resource [mem
> 0x10000000-0x3efeffff window]
> [    3.006028] pci_bus 0000:00: root bus resource [bus 00-0f]
> .
> .
> .
> [    3.135847] pci 0000:00:01.0: BAR 2: assigned [mem
> 0x10000000-0x1fffffff 64bit pref]
> [    3.137099] pci 0000:00:01.0: BAR 1: assigned [mem
> 0x20000000-0x20000fff]
> [    3.137382] pci 0000:00:01.0: BAR 0: assigned [mem
> 0x20001000-0x200010ff]

This difference is due to the facts that:

(a) the guest kernel is allowed to re-enumerate PCI devices, and
(b) the guest kernel has a full-fledged PCI resource allocator :)

> Also, I found that [mem 0x8000000000 window] was ignored,
> 
> [    2.769608] PCI ECAM: ECAM for domain 0000 [bus 00-0f] at [mem
> 0x3f000000-0x3fffffff] (base 0x3f000000)
> [    2.962930] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-0f])
> [    2.965787] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM
> ClockPM Segments MSI]
> [    2.990794] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME
> AER PCIeCapability]
> [    2.994520] acpi PNP0A08:00: host bridge window [mem 0x8000000000
> window] (ignored, not CPU addressable)

This is logged by the acpi_pci_root_validate_resources() kernel function
[drivers/acpi/pci_root.c].

The 0x8000000000 value comes from the ACPI payload that QEMU generates
and the firmware installs:

$ git show --color 5125f9cd -- hw/arm/virt-acpi-build.c

The acpi_pci_root_validate_resources() function complains because the
base address 0x8000000000 is greater than or equal to "iomem_resource.end".

"iomem_resource" looks like a global variable in the kernel. I don't
know how it is computed.

> so when do "cat /proc/iomem"
> no "8000000000-8000000000 : PCI Bus 0000:00" shows.
> Curious is that it works well and shows in another fedora kernel
> provided by Shanon.

Using the same firmware and the same VM configuration? Then guest kernel
changes might be necessary too, after all.

Thanks
Laszlo

> Do you have any idea that why this happen? Thanks.
> 
> 
> 
> Best,
> Liang
>> Laszlo
>>
> 

Re: [Qemu-devel] [edk2] Could not add PCI device with big memory to aarch64 VMs

[Laszlo Ersek]

On 11/04/15 23:22, liang yan wrote:
> Hello, Laszlo,
> 
> (1)I am trying to add ivshmem device(PCI device with big memory) to my
> aarch64 vm.
> So far, I could find device information from vm. But it seems vm did not
> create
> correct resource file for this device. Do you have any idea that this
> happens?
> 
> I used the upstream EDK2 to build my UEFI firmware.
> 
> There are three BARs for this device, and memory map is assigned too,
> but only one
> resource file is created.
> 
> My qemu supports ACPI 5.1 and the command line is :
> 
>   -device ivshmem,size=256M,chardev=ivshmem,msi=on,ioeventfd=on \
>   -chardev socket,path=/tmp/ivshmem_socket,id=ivshmem \
> 
> The lspci information:
> 
> 00:00.0 Host bridge: Red Hat, Inc. Device 0008
>     Subsystem: Red Hat, Inc Device 1100
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> 
> 00:01.0 RAM memory: Red Hat, Inc Inter-VM shared memory
>     Subsystem: Red Hat, Inc QEMU Virtual Machine
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>     Interrupt: pin A routed to IRQ 255
>     Region 0: Memory at 20001000 (32-bit, non-prefetchable) [disabled]
> [size=256]
>     Region 1: Memory at 20000000 (32-bit, non-prefetchable) [disabled]
> [size=4K]
>     Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
> [size=256M]
>     Capabilities: [40] MSI-X: Enable- Count=1 Masked-
>         Vector table: BAR=1 offset=00000000
>         PBA: BAR=1 offset=00000800
> 
> Boot information:
> 
> [    2.380924] pci 0000:00:01.0: BAR 2: assigned [mem
> 0x10000000-0x1fffffff 64bit pref]
> [    2.382836] pci 0000:00:01.0: BAR 1: assigned [mem
> 0x20000000-0x20000fff]
> [    2.383557] pci 0000:00:01.0: BAR 0: assigned [mem
> 0x20001000-0x200010ff]
> 
> 
> Files under /sys/devices/pci0000:00/0000:00:01.0
> 
> broken_parity_status      devspec       local_cpus  resource
> class              dma_mask_bits    modalias    subsystem
> config              driver_override  msi_bus     subsystem_device
> consistent_dma_mask_bits  enable       power       subsystem_vendor
> d3cold_allowed          irq           remove      uevent
> device              local_cpulist    rescan      vendor
> 
> Information for resource:
> 
> 0x0000000020001000 0x00000000200010ff 0x0000000000040200
> 0x0000000020000000 0x0000000020000fff 0x0000000000040200
> 0x0000000010000000 0x000000001fffffff 0x000000000014220c
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 
> 
> 
> 
> (2)It also has a problem that once I use a memory bigger than 256M for
> ivshmem, it could not get through UEFI,
> the error message is
> 
> PciBus: Discovered PCI @ [00|01|00]
>    BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100; Offset =
> 0x10
>    BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset
> = 0x14
>    BAR[2]: Type = PMem64; Alignment = 0x3FFFFFFF;    Length =
> 0x40000000;    Offset = 0x18
> 
> PciBus: HostBridge->SubmitResources() - Success
> ASSERT
> /home/liang/studio/edk2/ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c(449): ((BOOLEAN)(0==1))
> 
> 
> I am wandering if there are memory limitation for pcie devices under
> Qemu environment?
> 
> 
> Just thank you in advance and any information would be appreciated.

Now that Ard fixed
<https://tianocore.acgmultimedia.com/show_bug.cgi?id=65> (commit range
0b09c212a8ae..646a9e5b799b), this use case should work.

Thanks
Laszlo