Re: [Qemu-devel] Expand ECAM region in machvirt 2_13?

[Laszlo Ersek <lersek@redhat.com>]

On 05/01/18 17:59, Auger Eric wrote:
> Hi,
> 
> I would like to resume the discussion on extending the number of PCI
> buses to 256 (as in Q35) as a follow-up of past discussions:
> https://lists.gnu.org/archive/html/qemu-devel/2018-01/msg03631.html.
> 
> With the current 16MB ECAM region we are limited to 16 PCIe busses.
> 
> Could we envision to have a 256MB ECAM region and move it to another
> location beyond 256GB, in virt2_13 machine type?
> 
> Current ECAM range within [0x3f000000, 0x40000000]
> would be kept unchanged for legacy and when vms->highmem is set to false.
> Migration from <2_13 to >=2_13 would be allowed whereas migration from
>> =2.13 to <2.13 wouldn't.

If I understand correctly, the idea is to *move* the current one range,
if the virt machine type is >= 2.13 and highmem is set to true (which is
the default IIUC, from 2.12 onward).

For 64-bit (AARCH64) ArmVirtQemu, that should work fine. The firmware
takes the ECAM base and size from the "pci-host-ecam-generic" DT node,
property "reg", uint64_t elements #0 and #1. (Sorry if this isn't exact
DT lingo, I'm paraphrasing the firmware source code.) If the QEMU patch
just changes the values, that should work transparently.

For 32-bit (ARM) ArmVirtQemu, this change (the new ECAM default) could
be a problem. PCI stuff in the firmware wouldn't work unless people
specified highmem=off on the QEMU command line.

Now, I notice highmen defaults to "on" starting with 2.12 even for
"qemu-system-arm -M virt", not just "qemu-system-aarch64 -M virt", so
why doesn't that already cause a problem with PCI in the 32-bit guest fw?

Because, currently "highmen" only controls the presence of the 64-bit
PCI MMIO aperture for BAR allocation; it has no effect on config space.
And if the 64-bit PCI MMIO aperture is exposed to the 32-bit guest
firmware, the latter simply ignores the former, and works with the
32-bit aperture solely (which is always there).

So, for "qemu-system-arm -M virt" compatibility, I think we might need a
separate machine type property, which should default to "on" only on
qemu-system-aarch64 (if such distinctions are allowed).

Of course, I can't tell if the 32-bit ArmVirtQemu firmware is possible
to run on "qemu-system-aarch64 -M virt". (I think it is; I recall
something something about ARMv8 having ARMv7 compat, but I don't
remember ever trying.) If that's the case, then even the above
suggestion won't work, because it would break 32-bit guest fw that the
user has run (for whatever reason) on "qemu-system-aarch64 -M virt". In
this case, I believe we can't just change the contents of the current
"pci-host-ecam-generic" node, but we should implement some structural
DTB addition that old firmware will simply not notice, while new
(64-bit) firmware will specifically look for (and prefer over the old DT
stuff).

Ard, what's your take? (Sorry if you've already followed up, my email
processing lags.)

Thanks
Laszlo