From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34767) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1btZIF-0002Xa-Dl for qemu-devel@nongnu.org; Mon, 10 Oct 2016 08:02:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1btZIB-0005hd-PY for qemu-devel@nongnu.org; Mon, 10 Oct 2016 08:02:54 -0400 Received: from mx1.redhat.com ([209.132.183.28]:58720) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1btZIB-0005h7-GJ for qemu-devel@nongnu.org; Mon, 10 Oct 2016 08:02:51 -0400 Message-ID: <1476100967.3672.3.camel@redhat.com> From: Andrea Bolognani Date: Mon, 10 Oct 2016 14:02:47 +0200 In-Reply-To: <20161004125229.6ac23806@t450s.home> References: <1472736127-18137-1-git-send-email-marcel@redhat.com> <6c9103e7-70ad-55c6-5533-d6d9bbaa39b4@redhat.com> <20161004145911.GA2155@redhat.com> <2ea37764-7787-0b1b-5abb-9f8ba12a7080@redhat.com> <2db6a14f-731d-422d-a928-a856360406fd@redhat.com> <547407ca-e8f8-3327-d1c8-8a5c6317f28f@redhat.com> <20161004125229.6ac23806@t450s.home> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alex Williamson , Laine Stump Cc: qemu-devel@nongnu.org, Laszlo Ersek , "Daniel P. Berrange" , Marcel Apfelbaum , Peter Maydell , Drew Jones , mst@redhat.com, Gerd Hoffmann On Tue, 2016-10-04 at 12:52 -0600, Alex Williamson wrote: > > I'ts all just idle number games, but what I was thinking of was the=C2= =A0 > > difference between plugging=C2=A0=C2=A0a bunch of root-port+upstream+= downstreamxN=C2=A0 > > combos directly into pcie-root (flat), vs. plugging the first into=C2= =A0 > > pcie-root, and then subsequent ones into e.g. the last downstream por= t=C2=A0 > > of the previous set. Take the simplest case of needing 63 hotpluggabl= e=C2=A0 > > slots. In the "flat" case, you have: > >=C2=A0 > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02 x pcie-root-port > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02 x pcie-switch-upstream-port > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A063 x pcie-switch-downstream-port > >=C2=A0 > > In the "nested" or "chained" case you have: > >=C2=A0 > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A01 x pcie-root-port > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A01 x pcie-switch-upstream-port > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A032 x pcie-downstream-port > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A01 x pcie-switch-upstream-port > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A032 x pcie-switch-downstream-port >=C2=A0 > You're not thinking in enough dimensions.=C2=A0=C2=A0A single root port= can host > multiple sub-hierarchies on it's own.=C2=A0=C2=A0We can have a multi-fu= nction > upstream switch, so you can have 8 upstream ports (00.{0-7}).=C2=A0=C2=A0= If we > implemented ARI on the upstream ports, we could have 256 upstream ports > attached to a single root port, but of course then we've run out of > bus numbers before we've even gotten to actual devices buses. >=C2=A0 > Another option, look at the downstream ports, why do they each need to > be in separate slots?=C2=A0=C2=A0We have the address space of an entire= bus to > work with, so we can also create multi-function downstream ports, which > gives us 256 downstream ports per upstream port.=C2=A0=C2=A0Oops, we ju= st ran out > of bus numbers again, but at least actual devices can be attached. What's the advantage in using ARI to stuff more than eight of anything that's not Endpoint Devices in a single slot? I mean, if we just fill up all 32 slots in a PCIe Root Bus with 8 PCIe Root Ports each we already end up having 256 hotpluggable slots[1]. Why would it be preferable to use ARI, or even PCIe Switches, instead? [1] The last slot will have to be limited to 7 PCIe Root =C2=A0=C2=A0=C2=A0=C2=A0Ports if we don't want to run out of bus numbers --=C2=A0 Andrea Bolognani / Red Hat / Virtualization