Archs using generic PCI controller drivers vs. resource policy

Archs using generic PCI controller drivers vs. resource policy

[Benjamin Herrenschmidt]

Hi !

As part of my cleanup and consolidation of the PCI resource assignment
policies, I need to clarify something.

At the moment, unless PCI_PROBE_ONLY is set, a number of
platforms/archs expect Linux to reassign everything rather than honor
what has setup, then (re)assign what's left or broken. This is mostly
the case of embedded platforms. Things like x86 and desktop/server
powerpc will generally honor the firmware setup.

One problem is that this policy decision tend to be sprinkled in some
of the controller drivers themselves in drivers/pci/controller (or the
pci_host_probe helper).

This is wrong. I want to move it to the architecture (initially,
eventually it should be platform driven, but the default will start
with architecture specific to avoid changing the existing behaviours
while consolidating the code).

To do that right, I want to understand which archs can potentially use
the code in drivers/pci/controller today so I can change those archs to
explicitely set the default to "reassign everything" (and take the
policy out of the drivers themselves).

So far I've counted arm, arm64 (DT, not ACPI) and nios2. Any other ?

The remaining archs fall into two categories:

 - Those who have their own existing PCI management code and don't
use the generic controller drivers. I'm handling these already.

 - Those who don't seem to have anything to do with PCI (yet ?) or that
I've missed. Those will default to the x86 model (honor FW setup unless
it has conflicts or is broken, and (re)assign what's left) unless
explicitly changed.

Cheers,
Ben.

Re: Archs using generic PCI controller drivers vs. resource policy

[Benjamin Herrenschmidt]

On Sun, 2019-06-23 at 10:30 +1000, Benjamin Herrenschmidt wrote:
> 
> So far I've counted arm, arm64 (DT, not ACPI) and nios2. Any other ?

Hrm... nios2 doesn't do PCI despite what the Kconfig entry for
PCIE_ALTERA might want to let you believe.

So, anybody other than arm and arm64 using host bridge drivers in
drivers/pci/controller ?

Cheers,
Ben.

Re: Archs using generic PCI controller drivers vs. resource policy

[Christoph Hellwig]

On Sun, Jun 23, 2019 at 10:30:42AM +1000, Benjamin Herrenschmidt wrote:
> This is wrong. I want to move it to the architecture (initially,
> eventually it should be platform driven, but the default will start
> with architecture specific to avoid changing the existing behaviours
> while consolidating the code).

Doing this per arch sounds fundamentally wrong.  At best per firmware
type, but hopefully firmware messing with PCIe setup is slowly going
away, at least outside of x86.

> To do that right, I want to understand which archs can potentially use
> the code in drivers/pci/controller today so I can change those archs to
> explicitely set the default to "reassign everything" (and take the
> policy out of the drivers themselves).
> 
> So far I've counted arm, arm64 (DT, not ACPI) and nios2. Any other ?

riscv at least and probably anything that can be synthesized to common
FPGAs with PCIe support.

Re: Archs using generic PCI controller drivers vs. resource policy

[Benjamin Herrenschmidt]

On Mon, 2019-06-24 at 04:02 -0700, Christoph Hellwig wrote:
> On Sun, Jun 23, 2019 at 10:30:42AM +1000, Benjamin Herrenschmidt wrote:
> > This is wrong. I want to move it to the architecture (initially,
> > eventually it should be platform driven, but the default will start
> > with architecture specific to avoid changing the existing behaviours
> > while consolidating the code).
> 
> Doing this per arch sounds fundamentally wrong.  At best per firmware
> type, but hopefully firmware messing with PCIe setup is slowly going
> away, at least outside of x86.

No it's actually the other way around. UEFI/ACPI is growing outside of
x86 and it *is* essing with PCIe in ways we need to deal with better
(it's somewhat broken on arm64 today).

That said, the policy doesn't belong in the host bridge driver. It's
platform (or firmware) driven. At the moment, those platform/firmware
type decisions are done in the architecture. This can change, but I
won't change everything in one day.

Keep in mind that what I'm doing is a clean up keeping existing
behaviours as much as possible. Once we don't have 36 different ways of
doing the same thing, we can look at whether we want to change the
actual policies for some platforms/firmware/arch combinations.

Today, it's a practical fact that the resource allocation policy is
driven by the arch/platform code.

The fact that some host bridge drivers in drivers/pci/controller
effectively implement a resource allocation policy by diretly calling
stuff they shouldn't be calling is what I'm trying to fix.

So I just want to make the arch (today) set a default policy and have
the controller drivers call a generic function that performs the
resource management according to said policy.

> > To do that right, I want to understand which archs can potentially use
> > the code in drivers/pci/controller today so I can change those archs to
> > explicitely set the default to "reassign everything" (and take the
> > policy out of the drivers themselves).
> > 
> > So far I've counted arm, arm64 (DT, not ACPI) and nios2. Any other ?
> 
> riscv at least and probably anything that can be synthesized to common
> FPGAs with PCIe support.

Ok. I'll set riscv default policy to reassign all like arm and arm64
for now.

Cheers,
Ben.

Re: Archs using generic PCI controller drivers vs. resource policy

[Greg Ungerer]

Hi Ben,

On 23/6/19 10:30 am, Benjamin Herrenschmidt wrote:
> As part of my cleanup and consolidation of the PCI resource assignment
> policies, I need to clarify something.
> 
> At the moment, unless PCI_PROBE_ONLY is set, a number of
> platforms/archs expect Linux to reassign everything rather than honor
> what has setup, then (re)assign what's left or broken. This is mostly
> the case of embedded platforms. Things like x86 and desktop/server
> powerpc will generally honor the firmware setup.
> 
> One problem is that this policy decision tend to be sprinkled in some
> of the controller drivers themselves in drivers/pci/controller (or the
> pci_host_probe helper).
> 
> This is wrong. I want to move it to the architecture (initially,
> eventually it should be platform driven, but the default will start
> with architecture specific to avoid changing the existing behaviours
> while consolidating the code).
> 
> To do that right, I want to understand which archs can potentially use
> the code in drivers/pci/controller today so I can change those archs to
> explicitely set the default to "reassign everything" (and take the
> policy out of the drivers themselves).
> 
> So far I've counted arm, arm64 (DT, not ACPI) and nios2. Any other ?

For the m68k platforms which support PCI (which is only some of
the more modern ColdFire variants) they expect PCI resources to
be assigned by Linux. There is no boot firmware that will do that
before kernel startup.

The PCI root driver complex is already in the arch area though
(arch/m68k/coldfire/pci.c) so that is essentially what you
want to achieve right?

Regards
Greg


> The remaining archs fall into two categories:
> 
>   - Those who have their own existing PCI management code and don't
> use the generic controller drivers. I'm handling these already.
> 
>   - Those who don't seem to have anything to do with PCI (yet ?) or that
> I've missed. Those will default to the x86 model (honor FW setup unless
> it has conflicts or is broken, and (re)assign what's left) unless
> explicitly changed.
> 
> Cheers,
> Ben.
> 
> 
> 

Re: Archs using generic PCI controller drivers vs. resource policy

[Benjamin Herrenschmidt]

On Tue, 2019-07-02 at 23:24 +1000, Greg Ungerer wrote:
> > So far I've counted arm, arm64 (DT, not ACPI) and nios2. Any other
> > ?
> 
> For the m68k platforms which support PCI (which is only some of
> the more modern ColdFire variants) they expect PCI resources to
> be assigned by Linux. There is no boot firmware that will do that
> before kernel startup.
> 
> The PCI root driver complex is already in the arch area though
> (arch/m68k/coldfire/pci.c) so that is essentially what you
> want to achieve right?

Thanks !

So no I'm not trying to move controllers to arch. It makes sense to
have controllers live in generic code when those controller are IP
blocks from mobs like DesignWare that various SoC vendors chose to
integrate, regardless of the CPU architecture.

The issue is more about the resource allocation policy. I want to move
that *out* of the host controller drivers. That policy essentially
depends on whether a given platform/board trusts its firmware or not,
so it's fundamentally arch specific.

There are various reason which I can elaborate if you want wy some
platforms actually need to trust the firmware, at least partially.

Most embedded platforms don't give a damn and are happy for Linux to
reassign everything.

Cheers,
Ben.

Re: Archs using generic PCI controller drivers vs. resource policy

[Bjorn Helgaas]

On Sun, Jun 23, 2019 at 10:30:42AM +1000, Benjamin Herrenschmidt wrote:
> Hi !
> 
> As part of my cleanup and consolidation of the PCI resource assignment
> policies, I need to clarify something.
> 
> At the moment, unless PCI_PROBE_ONLY is set, a number of
> platforms/archs expect Linux to reassign everything rather than honor
> what has setup, then (re)assign what's left or broken.

I don't think "reassign everything" is any different from "honor what
has been setup and (re)assign what's left or broken".

"Reassign everything" is clearly allowed to produce the exact same
result as "honor what has been setup and (re)assign what's left or
broken".

I claim any difference between the two is actually a fragile
dependency on the Linux assignment algorithm that is likely to break
as that algorithm changes.

Or am I missing something?

Bjorn

Re: Archs using generic PCI controller drivers vs. resource policy

[Benjamin Herrenschmidt]

On Tue, 2019-07-02 at 15:19 -0500, Bjorn Helgaas wrote:
> On Sun, Jun 23, 2019 at 10:30:42AM +1000, Benjamin Herrenschmidt
> wrote:
> > Hi !
> > 
> > As part of my cleanup and consolidation of the PCI resource
> > assignment
> > policies, I need to clarify something.
> > 
> > At the moment, unless PCI_PROBE_ONLY is set, a number of
> > platforms/archs expect Linux to reassign everything rather than
> > honor
> > what has setup, then (re)assign what's left or broken.
> 
> I don't think "reassign everything" is any different from "honor what
> has been setup and (re)assign what's left or broken".

No it actually is. The policy on these is to rather explicitely ignore
what was set. If you just switch to honoring it, a good number of those
platforms will break. (We know that happens on arm64 as we are trying
to do just that).

Part of the problem is it's not always easy to figure out whether the
existing setup is "broken". It could just be "very suboptimal" for
example. Or broken in ways we don't detect early enough to do a good
job about it.

I really wouldn't try to change that at this point. Those platforms are
happy with Linux doing the complete setup the way it does, I don't see
areason to change it. In fact, in some cases we can do even better (see
below).

> "Reassign everything" is clearly allowed to produce the exact same
> result as "honor what has been setup and (re)assign what's left or
> broken".

Provided we have some IA that can figure out what "broken" means :)

> I claim any difference between the two is actually a fragile
> dependency on the Linux assignment algorithm that is likely to break
> as that algorithm changes.
> 
> Or am I missing something?

Well, reassign-everything isn't that likely to break when Linux changes
as long as linux doesn't change in ways that are completely busted
anyway. It's a pretty simple process really. And that would be caught
quickly since a LOT of platforms use that method.

As for the reasons, well, this tends to be embedded platforms where the
firmware may have left crap behind that we really don't want to honor
and can't really detect as "broken".

For example, switches sized incorrectly, or too big, sub optimal
placement with everything in 32-bit space and no room left for SR-IOV,
etc...

In fact, I'm thinking that on these platforms (which are basically
*all* arm32, all embedded ppc, and pretty much all other non-
server/desktop archs that aren't x86), we could in fact go one step
further and improve the default algorithm by applying the "distribute
available space" method at the top level (we currently only do it for
hotplug bridges below that, so the top level bridges will get the
default which, for hotplug, isn't likely to be enough).

I plan to toy around once I'm done with the first stage of
consolidation (which I put on the backburner for a little bit as I have
to deal with something else first, but I'll get back to it soon).

Cheers,
Ben.

Re: Archs using generic PCI controller drivers vs. resource policy

[Bjorn Helgaas]

On Wed, Jul 03, 2019 at 10:16:49AM +1000, Benjamin Herrenschmidt wrote:
> On Tue, 2019-07-02 at 15:19 -0500, Bjorn Helgaas wrote:
> > On Sun, Jun 23, 2019 at 10:30:42AM +1000, Benjamin Herrenschmidt
> > wrote:
> > > Hi !
> > > 
> > > As part of my cleanup and consolidation of the PCI resource
> > > assignment
> > > policies, I need to clarify something.
> > > 
> > > At the moment, unless PCI_PROBE_ONLY is set, a number of
> > > platforms/archs expect Linux to reassign everything rather than
> > > honor
> > > what has setup, then (re)assign what's left or broken.
> > 
> > I don't think "reassign everything" is any different from "honor what
> > has been setup and (re)assign what's left or broken".
> 
> No it actually is. The policy on these is to rather explicitely ignore
> what was set. If you just switch to honoring it, a good number of those
> platforms will break. (We know that happens on arm64 as we are trying
> to do just that).

It's only different if you're assuming something about how Linux
allocates things.  That assumption is implicit, which makes this
fragile.

You could make this concrete by supplying an example of the actual
firmware assignments that are broken, and the better ones that Linux
produces.  I'm talking about window and BAR values, not all the
needless differences in how the resource tree is managed.

> Part of the problem is it's not always easy to figure out whether the
> existing setup is "broken". It could just be "very suboptimal" for
> example. Or broken in ways we don't detect early enough to do a good
> job about it.
> 
> I really wouldn't try to change that at this point. Those platforms are
> happy with Linux doing the complete setup the way it does, I don't see
> areason to change it. In fact, in some cases we can do even better (see
> below).
> 
> > "Reassign everything" is clearly allowed to produce the exact same
> > result as "honor what has been setup and (re)assign what's left or
> > broken".
> 
> Provided we have some IA that can figure out what "broken" means :)
>
> > I claim any difference between the two is actually a fragile
> > dependency on the Linux assignment algorithm that is likely to break
> > as that algorithm changes.
> > 
> > Or am I missing something?
> 
> Well, reassign-everything isn't that likely to break when Linux changes
> as long as linux doesn't change in ways that are completely busted
> anyway. It's a pretty simple process really. And that would be caught
> quickly since a LOT of platforms use that method.
> 
> As for the reasons, well, this tends to be embedded platforms where the
> firmware may have left crap behind that we really don't want to honor
> and can't really detect as "broken".
> 
> For example, switches sized incorrectly, or too big, sub optimal
> placement with everything in 32-bit space and no room left for SR-IOV,
> etc...

These are things we should fix by improving the generic assignment
code (this might be what you're alluding to below).  I do not want a
"reassign everything" mode that does things differently than the
"change what's broken" mode.

I know we sort of have that today, but if we're reworking this code, I
don't want to perpetuate that kind of black magic.  If we can make the
difference between "reassign everything" and "change what's broken"
explicit, then we can make progress.

> In fact, I'm thinking that on these platforms (which are basically
> *all* arm32, all embedded ppc, and pretty much all other non-
> server/desktop archs that aren't x86), we could in fact go one step
> further and improve the default algorithm by applying the "distribute
> available space" method at the top level (we currently only do it for
> hotplug bridges below that, so the top level bridges will get the
> default which, for hotplug, isn't likely to be enough).

Re: Archs using generic PCI controller drivers vs. resource policy

[Benjamin Herrenschmidt]

On Tue, 2019-07-02 at 22:08 -0500, Bjorn Helgaas wrote:
> 
> > No it actually is. The policy on these is to rather explicitely ignore
> > what was set. If you just switch to honoring it, a good number of those
> > platforms will break. (We know that happens on arm64 as we are trying
> > to do just that).
> 
> It's only different if you're assuming something about how Linux
> allocates things.  That assumption is implicit, which makes this
> fragile.

I don't understand your argument.

Linux has *always* been responsible for the full assignment on these,
there is no UEFI/ACPI, no runtime firmware involved, I don't see the
point in trying to change that policy. The owners of these platforms
chose to do things that way, effectively assuming that Linux will do a
better job than whatever firmware (if any) did.

I remember cases for example where the firmware would just hard wire a
BAR for a boot device to some random value right in the middle of the
address space. If we started honoring this,  it would effectively have
split the already small available memory space for PCI on that card, it
made no sense to try to keep that setup. This was a case of some
obscure ppc embedded board, but that doesn't matter, I dont' see why we
should even consider changing the policy on these things. It's not like
we have to maintain two different algorithms anyway, we're just
skipping the claim pass, At least with my initial patch series it will
be obvious and done in a single place.

> You could make this concrete by supplying an example of the actual
> firmware assignments that are broken, and the better ones that Linux
> produces.  I'm talking about window and BAR values, not all the
> needless differences in how the resource tree is managed.

Why would I waste time chasing the hundreds of random embedded boards
around to do that ? I really completely fail to see your point and what
benefit we would have in trying to change this.

> > Part of the problem is it's not always easy to figure out whether the
> > existing setup is "broken". It could just be "very suboptimal" for
> > example. Or broken in ways we don't detect early enough to do a good
> > job about it.
> > 
> > I really wouldn't try to change that at this point. Those platforms are
> > happy with Linux doing the complete setup the way it does, I don't see
> > areason to change it. In fact, in some cases we can do even better (see
> > below).
> > 
> > > "Reassign everything" is clearly allowed to produce the exact same
> > > result as "honor what has been setup and (re)assign what's left or
> > > broken".
> > 
> > Provided we have some IA that can figure out what "broken" means :)
> > 
> > > I claim any difference between the two is actually a fragile
> > > dependency on the Linux assignment algorithm that is likely to break
> > > as that algorithm changes.
> > > 
> > > Or am I missing something?
> > 
> > Well, reassign-everything isn't that likely to break when Linux changes
> > as long as linux doesn't change in ways that are completely busted
> > anyway. It's a pretty simple process really. And that would be caught
> > quickly since a LOT of platforms use that method.
> > 
> > As for the reasons, well, this tends to be embedded platforms where the
> > firmware may have left crap behind that we really don't want to honor
> > and can't really detect as "broken".
> > 
> > For example, switches sized incorrectly, or too big, sub optimal
> > placement with everything in 32-bit space and no room left for SR-IOV,
> > etc...
> 
> These are things we should fix by improving the generic assignment
> code (this might be what you're alluding to below).  I do not want a
> "reassign everything" mode that does things differently than the
> "change what's broken" mode.

They don't fundamentally. With my patches, "reassign everything"
basically consists of calling pci_assign_unassigned_* without first
claiming existing resources. It's fundamentally the same algorithm.

The one thing I'm considering changing later is to use the "bridge"
variant instead so we get distribution of the available space, though
I'd have to make it work at the root. But that's separate and we can
discuss it later.

> I know we sort of have that today, but if we're reworking this code, I
> don't want to perpetuate that kind of black magic.  If we can make the
> difference between "reassign everything" and "change what's broken"
> explicit, then we can make progress.

I don't think it's black magic. As I said above, the definition of
"broken" in that context is almost impossible to get right. It could
just be massively sub-optimal. If we let Linux claim, it will and will
quickly run out of space if any hotplug or SR-IOV activity tries to
take place later.

Those platforms have historically chosen to ignore the FW allocation
for whatever reasons, changing that isn't something we should aim to
change, I don't see the point or benefit. And the testing burden would
be astronomical.

If anything, history has chown us that we are a lot better at assigning
resources that most embedded firmware out there.

> > In fact, I'm thinking that on these platforms (which are basically
> > *all* arm32, all embedded ppc, and pretty much all other non-
> > server/desktop archs that aren't x86), we could in fact go one step
> > further and improve the default algorithm by applying the "distribute
> > available space" method at the top level (we currently only do it for
> > hotplug bridges below that, so the top level bridges will get the
> > default which, for hotplug, isn't likely to be enough).

Cheers,
Ben.

Re: Archs using generic PCI controller drivers vs. resource policy

[Bjorn Helgaas]

On Wed, Jul 03, 2019 at 03:31:30PM +1000, Benjamin Herrenschmidt wrote:
> On Tue, 2019-07-02 at 22:08 -0500, Bjorn Helgaas wrote:
> > 
> > > No it actually is. The policy on these is to rather explicitely ignore
> > > what was set. If you just switch to honoring it, a good number of those
> > > platforms will break. (We know that happens on arm64 as we are trying
> > > to do just that).
> > 
> > It's only different if you're assuming something about how Linux
> > allocates things.  That assumption is implicit, which makes this
> > fragile.
> 
> I don't understand your argument.
> 
> Linux has *always* been responsible for the full assignment on these,
> there is no UEFI/ACPI, no runtime firmware involved, I don't see the
> point in trying to change that policy. The owners of these platforms
> chose to do things that way, effectively assuming that Linux will do a
> better job than whatever firmware (if any) did.
> 
> I remember cases for example where the firmware would just hard wire a
> BAR for a boot device to some random value right in the middle of the
> address space. If we started honoring this,  it would effectively have
> split the already small available memory space for PCI on that card, it
> made no sense to try to keep that setup. This was a case of some
> obscure ppc embedded board, but that doesn't matter, I dont' see why we
> should even consider changing the policy on these things. It's not like
> we have to maintain two different algorithms anyway, we're just
> skipping the claim pass, At least with my initial patch series it will
> be obvious and done in a single place.
> 
> > You could make this concrete by supplying an example of the actual
> > firmware assignments that are broken, and the better ones that Linux
> > produces.  I'm talking about window and BAR values, not all the
> > needless differences in how the resource tree is managed.
> 
> Why would I waste time chasing the hundreds of random embedded boards
> around to do that ?

All I asked for was a single example so we could talk about something
specific instead of handwaving, and your example of a device in the
middle of the address space was a good one.

That could happen just as easily on a "reassign if broken" platform
like x86 as on a "reassign everything" platform, so I would rather
make the generic code smart enough to deal with it than have the
platform or driver set a "reassign everything" flag.

But I think we're really talking past each other, and we're not
talking about an actual patch, so I don't think we need to come to any
conclusions yet.

Bjorn