Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Andrea Bolognani <abologna@redhat.com>]

On Mon, 2016-10-10 at 17:36 +0300, Marcel Apfelbaum wrote:
> > What's the advantage in using ARI to stuff more than eight
> > of anything that's not Endpoint Devices in a single slot?
> > 
> > I mean, if we just fill up all 32 slots in a PCIe Root Bus
> > with 8 PCIe Root Ports each we already end up having 256
> > hotpluggable slots[1]. Why would it be preferable to use
> > ARI, or even PCIe Switches, instead?
> 
> What if you need more devices (functions actually) ?
> 
> If some of the pcie.0 slots are occupied by other Integrated devices
> and you need more than 256 functions you can:
> (1) Add a PCIe Switch - if you need hot-plug support -an you are pretty limited
>      by the bus numbers, but it will give you a few more slots.
> (2) Use multi-function devices per root port if you are not interested in hotplug.
>      In this case ARI will give you up to 256 devices per Root Port.
> 
> Now the question is why ARI? Better utilization of the "problematic"
> resources like Bus numbers and IO space; all that if you need an insane
> number of devices, but we don't judge :).

My point is that AIUI ARI is something you only care about
for endpoint devices that want to have more than 8 functions.

When it comes to controller, there's no advantage that I can
think of in having 1 slot with 256 functions as opposed to 32
slots with 8 functions each; if anything, I expect that at
least some guest OSs would be quite baffled in finding eg. a
network adapter, a SCSI controller and a GPU as separate
functions of a single PCI slot.

> > [1] The last slot will have to be limited to 7 PCIe Root
> >     Ports if we don't want to run out of bus numbers
> 
> I don't follow how this will 'save' us. If all the root ports
> are in use and you leave space for one more, what can you do with it?

Probably my math is off, but if we can only have 256 PCI
buses (0-255) and we plug a PCIe Root Port in each of the
8 functions (0-7) of the 32 ports (0-31) available on the
PCIe Root Bus, we end up with

  0:00.[0-7] -> [001-008]:0.[0-7]
  0:01.[0-7] -> [009-016]:0.[0-7]
  0:02.[0-7] -> [017-024]:0.[0-7]
  ...
  0.30.[0-7] -> [241-248]:0.[0-7]
  0.31.[0-7] -> [249-256]:0.[0-7]

but 256 is not a valid bus number, so we should skip that
last PCIe Root Port and stop at 255.

-- 
Andrea Bolognani / Red Hat / Virtualization