* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-18 9:48 ` Alexander Gordeev
@ 2013-09-18 14:22 ` Tejun Heo
2013-09-18 16:50 ` Alexander Gordeev
2013-10-01 7:35 ` Michael Ellerman
2013-09-26 12:32 ` Mark Lord
2013-10-01 7:51 ` Michael Ellerman
2 siblings, 2 replies; 34+ messages in thread
From: Tejun Heo @ 2013-09-18 14:22 UTC (permalink / raw)
To: Alexander Gordeev
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Jan Beulich, linux-pci@vger.kernel.org,
Bjorn Helgaas, linuxppc-dev, Ingo Molnar
Hello,
On Wed, Sep 18, 2013 at 11:48:00AM +0200, Alexander Gordeev wrote:
> On Wed, Sep 18, 2013 at 12:30:23AM +1000, Michael Ellerman wrote:
> > How about no?
> >
> > We have a small number of MSIs available, limited by hardware &
> > firmware, if we don't impose a quota then the first device that probes
> > will get most/all of the MSIs and other devices miss out.
>
> Out of curiosity - how pSeries has had done it without quotas before
> 448e2ca ("powerpc/pseries: Implement a quota system for MSIs")?
Hmmm... do we need to treat this any differently? If the platform
can't allocate full range of requested MSIs, just failing should be
enough regardless of why such allocation can't be met, no?
> > Anyway I don't see what problem you're trying to solve? I agree the
> > -ve/0/+ve return value pattern is ugly, but it's hardly the end of the
> > world.
>
> Well, the interface recently has been re-classified from "ugly" to
> "unnecessarily complex and actively harmful" in Tejun's words ;)
LOL. :)
> Indeed, I checked most of the drivers and it is incredible how people
> are creative in misusing the interface: from innocent pci_disable_msix()
> calls when if pci_enable_msix() failed to assuming MSI-Xs were enabled
> if pci_enable_msix() returned a positive value (apparently untested).
>
> Roughly third of the drivers just do not care and bail out once
> pci_enable_msix() has not succeeded. Not sure how many of these are
> mandated by the hardware.
Yeah, I mean, this type of interface is a trap. People have to
actively resist to avoid doing silly stuff which is a lot to ask.
> /*
> * Retrieving 'nvec' by means other than pci_msix_table_size()
> */
>
> rc = pci_get_msix_limit(pdev);
> if (rc < 0)
> return rc;
>
> /*
> * nvec = min(rc, nvec);
> */
>
> for (i = 0; i < nvec; i++)
> msix_entry[i].entry = i;
>
> rc = pci_enable_msix(pdev, msix_entry, nvec);
> if (rc)
> return rc;
I really think what we should do is
* Determine the number of MSIs the controller wants. Don't worry
about quotas or limits or anything. Just determine the number
necessary to enable enhanced interrupt handling.
* Try allocating that number of MSIs. If it fails, then just revert
to single interrupt mode. It's not the end of the world and mostly
guaranteed to work. Let's please not even try to do partial
multiple interrupts. I really don't think it's worth the risk or
complexity.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 34+ messages in thread* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-18 14:22 ` Tejun Heo
@ 2013-09-18 16:50 ` Alexander Gordeev
2013-09-20 8:24 ` Alexander Gordeev
2013-09-20 12:26 ` Tejun Heo
2013-10-01 7:35 ` Michael Ellerman
1 sibling, 2 replies; 34+ messages in thread
From: Alexander Gordeev @ 2013-09-18 16:50 UTC (permalink / raw)
To: Tejun Heo
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Jan Beulich, linux-pci@vger.kernel.org,
Bjorn Helgaas, linuxppc-dev, Ingo Molnar
On Wed, Sep 18, 2013 at 09:22:31AM -0500, Tejun Heo wrote:
> > > We have a small number of MSIs available, limited by hardware &
> > > firmware, if we don't impose a quota then the first device that probes
> > > will get most/all of the MSIs and other devices miss out.
> >
> > Out of curiosity - how pSeries has had done it without quotas before
> > 448e2ca ("powerpc/pseries: Implement a quota system for MSIs")?
>
> Hmmm... do we need to treat this any differently? If the platform
> can't allocate full range of requested MSIs, just failing should be
> enough regardless of why such allocation can't be met, no?
That depends from what "full range of requested MSIs" is. If that is
a maximum number of MSIs the controller advertised, then no. As MSI
design essentially allows devices to operate in lower-than-maximum
modes it is responsibility of a caller to decide how many vectors to
request. So in case of pSeries I think it is completely legitimate to
request lessers to overcome the platform limitation and let all devices
work.
> I really think what we should do is
>
> * Determine the number of MSIs the controller wants. Don't worry
> about quotas or limits or anything. Just determine the number
> necessary to enable enhanced interrupt handling.
Actually, I do not see much contradiction with what I proposed. The
key words here "determine the number of MSIs the controller wants".
In general case it is not what pci_msix_table_size() returns (or at
least we should not limit ourselves to it) - there could be non-
standard means to report number of MSIs: hardcoded, version-dependant,
device-specific registers etc.
Next, if we opt to determine the number of MSIs by non-MSI standard
means then there is no reason not to call pci_get_msix_limit() (or
whatever) at this step.
The question how I see it - do we want pci_get_msix_limit() interface
as part of the MSI framework or do we want it pSeries-specific?
> * Try allocating that number of MSIs. If it fails, then just revert
> to single interrupt mode. It's not the end of the world and mostly
> guaranteed to work. Let's please not even try to do partial
> multiple interrupts. I really don't think it's worth the risk or
> complexity.
Being Captain Obvious here, but it is up to the device driver to handle
a failure. There could be no such option as single MSI mode after all :)
> Thanks.
>
> --
> tejun
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-18 16:50 ` Alexander Gordeev
@ 2013-09-20 8:24 ` Alexander Gordeev
2013-09-20 12:27 ` Tejun Heo
2013-09-20 12:26 ` Tejun Heo
1 sibling, 1 reply; 34+ messages in thread
From: Alexander Gordeev @ 2013-09-20 8:24 UTC (permalink / raw)
To: Michael Ellerman
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Jan Beulich, linux-pci@vger.kernel.org,
Bjorn Helgaas, Tejun Heo, linuxppc-dev, Ingo Molnar
Michael et al.
The identifiable options sounded so far were:
* Do not change anything
* Make pci_enable_msix() return 0/-errno
* Make pci_enable_msix() return 0/-errno and introduce arch_get_msix_limit()/
arch_get_msi_limit()
* Make pci_enable_msix() return 0/-errno and introduce pci_get_msix_limit()/
pci_get_msi_limit() and arch_get_msix_limit()/arch_get_msi_limit() so that:
pci_get_msix_limit() = min(arch_get_msix_limit(), pci_msix_table_size())
pci_get_msi_limit() = min(arch_get_msi_limit(), pci_get_msi_cap())
Can we have a conclusion here?
Thanks!
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-20 8:24 ` Alexander Gordeev
@ 2013-09-20 12:27 ` Tejun Heo
2013-09-25 18:02 ` Bjorn Helgaas
0 siblings, 1 reply; 34+ messages in thread
From: Tejun Heo @ 2013-09-20 12:27 UTC (permalink / raw)
To: Alexander Gordeev
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Jan Beulich, linux-pci@vger.kernel.org,
Bjorn Helgaas, linuxppc-dev, Ingo Molnar
On Fri, Sep 20, 2013 at 10:24:59AM +0200, Alexander Gordeev wrote:
> * Make pci_enable_msix() return 0/-errno
My choice would be this one.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-20 12:27 ` Tejun Heo
@ 2013-09-25 18:02 ` Bjorn Helgaas
2013-09-25 20:58 ` Alexander Gordeev
0 siblings, 1 reply; 34+ messages in thread
From: Bjorn Helgaas @ 2013-09-25 18:02 UTC (permalink / raw)
To: Tejun Heo
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Alexander Gordeev, Jan Beulich,
linux-pci@vger.kernel.org, linuxppc-dev, Ingo Molnar
On Fri, Sep 20, 2013 at 07:27:36AM -0500, Tejun Heo wrote:
> On Fri, Sep 20, 2013 at 10:24:59AM +0200, Alexander Gordeev wrote:
> > * Make pci_enable_msix() return 0/-errno
>
> My choice would be this one.
I agree; it sounds like you've identified several bugs related to the
current confusing interface, so fixing that seems like the first step.
I hope we can avoid adding a plethora of interfaces to address unusual
corner cases. But if we do the above and it turns out not to be enough,
we can always extend it later.
Bjorn
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-25 18:02 ` Bjorn Helgaas
@ 2013-09-25 20:58 ` Alexander Gordeev
2013-09-25 21:00 ` Tejun Heo
0 siblings, 1 reply; 34+ messages in thread
From: Alexander Gordeev @ 2013-09-25 20:58 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: linux-pci@vger.kernel.org, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, Tejun Heo, linuxppc-dev, Ingo Molnar
On Wed, Sep 25, 2013 at 12:02:20PM -0600, Bjorn Helgaas wrote:
> On Fri, Sep 20, 2013 at 07:27:36AM -0500, Tejun Heo wrote:
> > On Fri, Sep 20, 2013 at 10:24:59AM +0200, Alexander Gordeev wrote:
> > > * Make pci_enable_msix() return 0/-errno
> >
> > My choice would be this one.
>
> I agree; it sounds like you've identified several bugs related to the
> current confusing interface, so fixing that seems like the first step.
Yeah, I am trying to. Turns out to be a nice exercise ;)
> I hope we can avoid adding a plethora of interfaces to address unusual
> corner cases. But if we do the above and it turns out not to be enough,
> we can always extend it later.
Unfortunately, pSeries is a shows-topper here :( It seems we have to
introduce pci_get_msi{,x}_limit() interfaces to honour the quota
thing. I just hope the hardware set for pSeries is limited and we
won't need to use it for all drivers.
> Bjorn
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-25 20:58 ` Alexander Gordeev
@ 2013-09-25 21:00 ` Tejun Heo
2013-09-26 7:46 ` Alexander Gordeev
0 siblings, 1 reply; 34+ messages in thread
From: Tejun Heo @ 2013-09-25 21:00 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linux-pci@vger.kernel.org, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, Bjorn Helgaas, linuxppc-dev, Ingo Molnar
Hello,
On Wed, Sep 25, 2013 at 10:58:05PM +0200, Alexander Gordeev wrote:
> Unfortunately, pSeries is a shows-topper here :( It seems we have to
> introduce pci_get_msi{,x}_limit() interfaces to honour the quota
> thing. I just hope the hardware set for pSeries is limited and we
> won't need to use it for all drivers.
Can you please go into a bit of detail on that? Why does it matter?
Is it because you're worried you might cause performance regression by
forcing prevoius partial multiple allocations to single interrupt
operation?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 34+ messages in thread* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-25 21:00 ` Tejun Heo
@ 2013-09-26 7:46 ` Alexander Gordeev
2013-09-26 8:58 ` David Laight
2013-09-26 13:11 ` Tejun Heo
0 siblings, 2 replies; 34+ messages in thread
From: Alexander Gordeev @ 2013-09-26 7:46 UTC (permalink / raw)
To: Tejun Heo
Cc: linux-pci@vger.kernel.org, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, Bjorn Helgaas, linuxppc-dev, Ingo Molnar
On Wed, Sep 25, 2013 at 05:00:16PM -0400, Tejun Heo wrote:
> Hello,
>
> On Wed, Sep 25, 2013 at 10:58:05PM +0200, Alexander Gordeev wrote:
> > Unfortunately, pSeries is a shows-topper here :( It seems we have to
> > introduce pci_get_msi{,x}_limit() interfaces to honour the quota
> > thing. I just hope the hardware set for pSeries is limited and we
> > won't need to use it for all drivers.
>
> Can you please go into a bit of detail on that? Why does it matter?
Because otherwise we will re-introduce a problem described by Michael:
"We have a small number of MSIs available, limited by hardware &
firmware, if we don't impose a quota then the first device that probes
will get most/all of the MSIs and other devices miss out."
> Is it because you're worried you might cause performance regression by
> forcing prevoius partial multiple allocations to single interrupt
> operation?
Well, not really. I think it won't be possible to force people not to use
partial allocations anyway. Some controllers just do not care how many MSIs
they are configured with. Some drivers derive the number of MSIs desired
from the number of CPUs online - in such cases allocating more MSIs (i.e.
a number the controller advertised) could cause a performance degradation
even.
So when driver authors will know/measure/believe their hardware performs
better with partial allocations they will stand for it. What we can do is
to prevent those try-and-decrease fallbacks.
> Thanks.
>
> --
> tejun
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread* RE: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 7:46 ` Alexander Gordeev
@ 2013-09-26 8:58 ` David Laight
2013-09-26 10:45 ` Alexander Gordeev
2013-09-26 13:11 ` Tejun Heo
1 sibling, 1 reply; 34+ messages in thread
From: David Laight @ 2013-09-26 8:58 UTC (permalink / raw)
To: Alexander Gordeev, Tejun Heo
Cc: linux-pci, Joerg Roedel, x86, linux-kernel, linux-ide,
Jan Beulich, Bjorn Helgaas, linuxppc-dev, Ingo Molnar
> Subject: Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() =
interface
>=20
> On Wed, Sep 25, 2013 at 05:00:16PM -0400, Tejun Heo wrote:
> > Hello,
> >
> > On Wed, Sep 25, 2013 at 10:58:05PM +0200, Alexander Gordeev wrote:
> > > Unfortunately, pSeries is a shows-topper here :( It seems we have =
to
> > > introduce pci_get_msi{,x}_limit() interfaces to honour the quota
> > > thing. I just hope the hardware set for pSeries is limited and we
> > > won't need to use it for all drivers.
> >
> > Can you please go into a bit of detail on that? Why does it matter?
>=20
> Because otherwise we will re-introduce a problem described by Michael:
> "We have a small number of MSIs available, limited by hardware &
> firmware, if we don't impose a quota then the first device that probes
> will get most/all of the MSIs and other devices miss out."
Would it be possible to do some kind of 2-stage allocation.
In the first pass the driver would pass a minimum and desired
number of MSI-X interrupts, but not actually be given any.
Interrupts could then be allocated after it is known how many
are required and how many are available.
David
^ permalink raw reply [flat|nested] 34+ messages in thread* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 8:58 ` David Laight
@ 2013-09-26 10:45 ` Alexander Gordeev
2013-09-26 11:34 ` David Laight
0 siblings, 1 reply; 34+ messages in thread
From: Alexander Gordeev @ 2013-09-26 10:45 UTC (permalink / raw)
To: David Laight
Cc: linuxppc-dev, linux-pci, Joerg Roedel, x86, linux-kernel,
linux-ide, Jan Beulich, Tejun Heo, Bjorn Helgaas, Ingo Molnar
On Thu, Sep 26, 2013 at 09:58:53AM +0100, David Laight wrote:
> Would it be possible to do some kind of 2-stage allocation.
> In the first pass the driver would pass a minimum and desired
> number of MSI-X interrupts, but not actually be given any.
Repeated calls to msi_enable_msi/msix() is what we are trying to avoid.
> Interrupts could then be allocated after it is known how many
> are required and how many are available.
Yep, that what we are heading to. So basic pattern I see would be like this:
int foo_driver_enable_msix(struct pci_dev *pdev, int nvec)
{
...
rc = pci_msix_table_size(pdev);
if (rc < 0)
return rc;
nvec = min(nvec, rc);
if (nvec < FOO_DRIVER_MINIMUM_NVEC)
goto single_msi;
for (i = 0; i < nvec; i++)
entries[i].entry = i;
rc = pci_enable_msix(pdev, entries, nvec);
if (rc)
goto single_msi;
return 0;
single_msi:
...
}
But this will break pSeries and we might end up with something like this:
int foo_driver_enable_msix(struct pci_dev *pdev, int nvec)
{
...
rc = pci_msix_table_size(pdev);
if (rc < 0)
return rc;
nvec = min(nvec, rc);
if (nvec < FOO_DRIVER_MINIMUM_NVEC)
goto single_msi;
rc = pci_get_msix_limit(pdev, nvec);
if (rc < 0)
return rc;
nvec = min(nvec, rc);
if (nvec < FOO_DRIVER_MINIMUM_NVEC)
goto single_msi;
for (i = 0; i < nvec; i++)
entries[i].entry = i;
rc = pci_enable_msix(pdev, entries, nvec);
if (rc)
goto single_msi;
return 0;
single_msi:
...
}
> David
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread* RE: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 10:45 ` Alexander Gordeev
@ 2013-09-26 11:34 ` David Laight
2013-09-26 12:13 ` Alexander Gordeev
0 siblings, 1 reply; 34+ messages in thread
From: David Laight @ 2013-09-26 11:34 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linuxppc-dev, linux-pci, Joerg Roedel, x86, linux-kernel,
linux-ide, Jan Beulich, Tejun Heo, Bjorn Helgaas, Ingo Molnar
> On Thu, Sep 26, 2013 at 09:58:53AM +0100, David Laight wrote:
> > Would it be possible to do some kind of 2-stage allocation.
> > In the first pass the driver would pass a minimum and desired
> > number of MSI-X interrupts, but not actually be given any.
>=20
> Repeated calls to msi_enable_msi/msix() is what we are trying to =
avoid.
I was thinking that the first call would be done during driver probe
(assuming such a time exists) so that the subsystem could determine
how many interrupts all the drivers would like, so it can then
hand out a smaller number to some of the early drivers in order
to have some left to satisfy the minimum requirement of later
ones.
So all it would do is sum the requirements of all the drivers.
David
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 11:34 ` David Laight
@ 2013-09-26 12:13 ` Alexander Gordeev
0 siblings, 0 replies; 34+ messages in thread
From: Alexander Gordeev @ 2013-09-26 12:13 UTC (permalink / raw)
To: David Laight
Cc: linuxppc-dev, linux-pci, Joerg Roedel, x86, linux-kernel,
linux-ide, Jan Beulich, Tejun Heo, Bjorn Helgaas, Ingo Molnar
On Thu, Sep 26, 2013 at 12:34:36PM +0100, David Laight wrote:
> I was thinking that the first call would be done during driver probe
> (assuming such a time exists) so that the subsystem could determine
> how many interrupts all the drivers would like, so it can then
> hand out a smaller number to some of the early drivers in order
> to have some left to satisfy the minimum requirement of later
> ones.
>
> So all it would do is sum the requirements of all the drivers.
It is already implemented - please see commit 448e2ca ("powerpc/pseries:
Implement a quota system for MSIs")
All other archs do not have MSI vector space limitations.
> David
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 7:46 ` Alexander Gordeev
2013-09-26 8:58 ` David Laight
@ 2013-09-26 13:11 ` Tejun Heo
2013-09-26 14:39 ` Alexander Gordeev
1 sibling, 1 reply; 34+ messages in thread
From: Tejun Heo @ 2013-09-26 13:11 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linux-pci@vger.kernel.org, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, Bjorn Helgaas, linuxppc-dev, Ingo Molnar
Hello,
On Thu, Sep 26, 2013 at 09:46:46AM +0200, Alexander Gordeev wrote:
> > Can you please go into a bit of detail on that? Why does it matter?
>
> Because otherwise we will re-introduce a problem described by Michael:
> "We have a small number of MSIs available, limited by hardware &
> firmware, if we don't impose a quota then the first device that probes
> will get most/all of the MSIs and other devices miss out."
Still not following. Why wouldn't just letting the drivers request
the optimal number they want and falling back to single interrupt mode
work? ie. why can't we just have an all or nothing interface?
> > Is it because you're worried you might cause performance regression by
> > forcing prevoius partial multiple allocations to single interrupt
> > operation?
>
> Well, not really. I think it won't be possible to force people not to use
> partial allocations anyway. Some controllers just do not care how many MSIs
> they are configured with. Some drivers derive the number of MSIs desired
> from the number of CPUs online - in such cases allocating more MSIs (i.e.
> a number the controller advertised) could cause a performance degradation
> even.
Yeah, sure thing but just let the *driver* decide that number without
worrying about how many they can actually get. Ultimately, what we
want is removing this extra variable which can arbitrarily affect the
number of allocated interrupts so that we only have to worry about
either proper multiple MSI mode or single interrupt mode, not
something random inbetween. It is possible that there exists a driver
which absolutely requires partial allocation on certain archs, but
that should be a very special case and the interface should look
accordingly ugly / special. But do we actually have those?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 13:11 ` Tejun Heo
@ 2013-09-26 14:39 ` Alexander Gordeev
2013-09-26 14:42 ` Tejun Heo
2013-10-01 7:19 ` Michael Ellerman
0 siblings, 2 replies; 34+ messages in thread
From: Alexander Gordeev @ 2013-09-26 14:39 UTC (permalink / raw)
To: Tejun Heo
Cc: linux-pci@vger.kernel.org, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, Bjorn Helgaas, linuxppc-dev, Ingo Molnar
On Thu, Sep 26, 2013 at 09:11:47AM -0400, Tejun Heo wrote:
> > Because otherwise we will re-introduce a problem described by Michael:
> > "We have a small number of MSIs available, limited by hardware &
> > firmware, if we don't impose a quota then the first device that probes
> > will get most/all of the MSIs and other devices miss out."
>
> Still not following. Why wouldn't just letting the drivers request
> the optimal number they want and falling back to single interrupt mode
> work? ie. why can't we just have an all or nothing interface?
I can imagine a scenario where the first device probes in, requests its
optimal number, acquires that number and exhausts MSIs in pSeries firmware.
The next few devices possibly end up with single MSI, since no MSIs left
to satisfy their optimal numbers. If one of those single-MSI'ed devices
happened to be a high-performance HBA hitting a degraded performance that
alone would force (IBM) to introduce the quotas. Now, if the same/another
device happened does not support the legacy interrupt mode and no MSI
resources have left in the platform firmware at all...
> tejun
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 14:39 ` Alexander Gordeev
@ 2013-09-26 14:42 ` Tejun Heo
2013-10-01 7:19 ` Michael Ellerman
1 sibling, 0 replies; 34+ messages in thread
From: Tejun Heo @ 2013-09-26 14:42 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linux-pci@vger.kernel.org, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, Bjorn Helgaas, linuxppc-dev, Ingo Molnar
Hello,
On Thu, Sep 26, 2013 at 10:39 AM, Alexander Gordeev <agordeev@redhat.com> wrote:
> I can imagine a scenario where the first device probes in, requests its
Well, we can imagine a lot of thing but usually have to draw the line somewhere.
> optimal number, acquires that number and exhausts MSIs in pSeries firmware.
> The next few devices possibly end up with single MSI, since no MSIs left
> to satisfy their optimal numbers. If one of those single-MSI'ed devices
> happened to be a high-performance HBA hitting a degraded performance that
> alone would force (IBM) to introduce the quotas. Now, if the same/another
> device happened does not support the legacy interrupt mode and no MSI
> resources have left in the platform firmware at all...
If that happens, that's just the platform code being dumb. Quota is
there to prevent that from happening, right? Let's please do something
simple and worry about problems if they actually exist.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 14:39 ` Alexander Gordeev
2013-09-26 14:42 ` Tejun Heo
@ 2013-10-01 7:19 ` Michael Ellerman
1 sibling, 0 replies; 34+ messages in thread
From: Michael Ellerman @ 2013-10-01 7:19 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linux-ide@vger.kernel.org, linux-pci@vger.kernel.org,
Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
Bjorn Helgaas, Jan Beulich, Tejun Heo, linuxppc-dev, Ingo Molnar
On Thu, Sep 26, 2013 at 04:39:02PM +0200, Alexander Gordeev wrote:
> On Thu, Sep 26, 2013 at 09:11:47AM -0400, Tejun Heo wrote:
> > > Because otherwise we will re-introduce a problem described by Michael:
> > > "We have a small number of MSIs available, limited by hardware &
> > > firmware, if we don't impose a quota then the first device that probes
> > > will get most/all of the MSIs and other devices miss out."
> >
> > Still not following. Why wouldn't just letting the drivers request
> > the optimal number they want and falling back to single interrupt mode
> > work? ie. why can't we just have an all or nothing interface?
>
> I can imagine a scenario where the first device probes in, requests its
> optimal number, acquires that number and exhausts MSIs in pSeries firmware.
> The next few devices possibly end up with single MSI, since no MSIs left
> to satisfy their optimal numbers. If one of those single-MSI'ed devices
> happened to be a high-performance HBA hitting a degraded performance that
> alone would force (IBM) to introduce the quotas.
Yes that's exactly the scenario, and I didn't imagine it, our test
people actually hit it and yelled at me.
I don't remember exactly which adapters it was, I might be able to find
the details if I looked hard, a quick search through my mail archive
didn't find it - it might have come in via irc / bugzilla etc.
cheers
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-18 16:50 ` Alexander Gordeev
2013-09-20 8:24 ` Alexander Gordeev
@ 2013-09-20 12:26 ` Tejun Heo
2013-10-01 7:26 ` Michael Ellerman
1 sibling, 1 reply; 34+ messages in thread
From: Tejun Heo @ 2013-09-20 12:26 UTC (permalink / raw)
To: Alexander Gordeev
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Jan Beulich, linux-pci@vger.kernel.org,
Bjorn Helgaas, linuxppc-dev, Ingo Molnar
Hello,
On Wed, Sep 18, 2013 at 06:50:45PM +0200, Alexander Gordeev wrote:
> Actually, I do not see much contradiction with what I proposed. The
> key words here "determine the number of MSIs the controller wants".
>
> In general case it is not what pci_msix_table_size() returns (or at
> least we should not limit ourselves to it) - there could be non-
> standard means to report number of MSIs: hardcoded, version-dependant,
> device-specific registers etc.
>
> Next, if we opt to determine the number of MSIs by non-MSI standard
> means then there is no reason not to call pci_get_msix_limit() (or
> whatever) at this step.
Yeah, that's all fine. My point is that we shouldn't try to use
"degraded" multiple MSI mode where the number of MSIs allocated is
smaller than performing full multiple MSI operation. How that number
is determined doesn't really matter but that number is a property
which is solely decided by the device driver, right? If a device
needs full multiple MSI mode, given specific configuration, it needs
>= X number of MSIs and that's the number it should request.
> Being Captain Obvious here, but it is up to the device driver to handle
> a failure. There could be no such option as single MSI mode after all :)
I don't think there actually is a mainstream device which can't
fallback to single interrupt. Anyways, the point is the same, let's
please not try to create an interface which encourages complex retry
logic in its users which are likely to involve less traveled and
tested paths in both the driver and firmware.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-20 12:26 ` Tejun Heo
@ 2013-10-01 7:26 ` Michael Ellerman
0 siblings, 0 replies; 34+ messages in thread
From: Michael Ellerman @ 2013-10-01 7:26 UTC (permalink / raw)
To: Tejun Heo
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Alexander Gordeev, Jan Beulich,
linux-pci@vger.kernel.org, Bjorn Helgaas, linuxppc-dev,
Ingo Molnar
On Fri, Sep 20, 2013 at 07:26:03AM -0500, Tejun Heo wrote:
> Hello,
>
> On Wed, Sep 18, 2013 at 06:50:45PM +0200, Alexander Gordeev wrote:
> > Actually, I do not see much contradiction with what I proposed. The
> > key words here "determine the number of MSIs the controller wants".
> >
> > In general case it is not what pci_msix_table_size() returns (or at
> > least we should not limit ourselves to it) - there could be non-
> > standard means to report number of MSIs: hardcoded, version-dependant,
> > device-specific registers etc.
> >
> > Next, if we opt to determine the number of MSIs by non-MSI standard
> > means then there is no reason not to call pci_get_msix_limit() (or
> > whatever) at this step.
>
> Yeah, that's all fine. My point is that we shouldn't try to use
> "degraded" multiple MSI mode where the number of MSIs allocated is
> smaller than performing full multiple MSI operation. How that number
> is determined doesn't really matter but that number is a property
> which is solely decided by the device driver, right? If a device
> needs full multiple MSI mode, given specific configuration, it needs
> >= X number of MSIs and that's the number it should request.
Sure, the driver is in full control. If it can ONLY work with N MSIs
then it should try for N, else fallback to 1.
But some drivers are able to work with a range of values for N, and
performance is improved vs using a single MSI.
> > Being Captain Obvious here, but it is up to the device driver to handle
> > a failure. There could be no such option as single MSI mode after all :)
>
> I don't think there actually is a mainstream device which can't
> fallback to single interrupt. Anyways, the point is the same, let's
> please not try to create an interface which encourages complex retry
> logic in its users which are likely to involve less traveled and
> tested paths in both the driver and firmware.
Why support > 1 MSI at all? It just adds complex logic and less travelled
paths in the driver and firmware.
cheers
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-18 14:22 ` Tejun Heo
2013-09-18 16:50 ` Alexander Gordeev
@ 2013-10-01 7:35 ` Michael Ellerman
2013-10-01 11:55 ` Tejun Heo
1 sibling, 1 reply; 34+ messages in thread
From: Michael Ellerman @ 2013-10-01 7:35 UTC (permalink / raw)
To: Tejun Heo
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Alexander Gordeev, Jan Beulich,
linux-pci@vger.kernel.org, Bjorn Helgaas, linuxppc-dev,
Ingo Molnar
On Wed, Sep 18, 2013 at 09:22:31AM -0500, Tejun Heo wrote:
> Hello,
>
> On Wed, Sep 18, 2013 at 11:48:00AM +0200, Alexander Gordeev wrote:
> > On Wed, Sep 18, 2013 at 12:30:23AM +1000, Michael Ellerman wrote:
> > > How about no?
> > >
> > > We have a small number of MSIs available, limited by hardware &
> > > firmware, if we don't impose a quota then the first device that probes
> > > will get most/all of the MSIs and other devices miss out.
> >
> > Out of curiosity - how pSeries has had done it without quotas before
> > 448e2ca ("powerpc/pseries: Implement a quota system for MSIs")?
>
> Hmmm... do we need to treat this any differently? If the platform
> can't allocate full range of requested MSIs, just failing should be
> enough regardless of why such allocation can't be met, no?
>
> > > Anyway I don't see what problem you're trying to solve? I agree the
> > > -ve/0/+ve return value pattern is ugly, but it's hardly the end of the
> > > world.
> >
> > Well, the interface recently has been re-classified from "ugly" to
> > "unnecessarily complex and actively harmful" in Tejun's words ;)
>
> LOL. :)
>
> > Indeed, I checked most of the drivers and it is incredible how people
> > are creative in misusing the interface: from innocent pci_disable_msix()
> > calls when if pci_enable_msix() failed to assuming MSI-Xs were enabled
> > if pci_enable_msix() returned a positive value (apparently untested).
> >
> > Roughly third of the drivers just do not care and bail out once
> > pci_enable_msix() has not succeeded. Not sure how many of these are
> > mandated by the hardware.
>
> Yeah, I mean, this type of interface is a trap. People have to
> actively resist to avoid doing silly stuff which is a lot to ask.
I really think you're overstating the complexity here.
Functions typically return a boolean -> nothing to see here
This function returns a tristate value -> brain explosion!
> > /*
> > * Retrieving 'nvec' by means other than pci_msix_table_size()
> > */
> >
> > rc = pci_get_msix_limit(pdev);
> > if (rc < 0)
> > return rc;
> >
> > /*
> > * nvec = min(rc, nvec);
> > */
> >
> > for (i = 0; i < nvec; i++)
> > msix_entry[i].entry = i;
> >
> > rc = pci_enable_msix(pdev, msix_entry, nvec);
> > if (rc)
> > return rc;
>
> I really think what we should do is
>
> * Determine the number of MSIs the controller wants. Don't worry
> about quotas or limits or anything. Just determine the number
> necessary to enable enhanced interrupt handling.
>
> * Try allocating that number of MSIs. If it fails, then just revert
> to single interrupt mode. It's not the end of the world and mostly
> guaranteed to work. Let's please not even try to do partial
> multiple interrupts. I really don't think it's worth the risk or
> complexity.
It will potentially break existing setups on our hardware.
Can I make that any clearer?
cheers
^ permalink raw reply [flat|nested] 34+ messages in thread* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-10-01 7:35 ` Michael Ellerman
@ 2013-10-01 11:55 ` Tejun Heo
2013-10-02 2:33 ` Michael Ellerman
0 siblings, 1 reply; 34+ messages in thread
From: Tejun Heo @ 2013-10-01 11:55 UTC (permalink / raw)
To: Michael Ellerman
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Alexander Gordeev, Jan Beulich,
linux-pci@vger.kernel.org, Bjorn Helgaas, linuxppc-dev,
Ingo Molnar
Hello,
On Tue, Oct 01, 2013 at 05:35:48PM +1000, Michael Ellerman wrote:
> > > Roughly third of the drivers just do not care and bail out once
> > > pci_enable_msix() has not succeeded. Not sure how many of these are
> > > mandated by the hardware.
> >
> > Yeah, I mean, this type of interface is a trap. People have to
> > actively resist to avoid doing silly stuff which is a lot to ask.
>
> I really think you're overstating the complexity here.
>
> Functions typically return a boolean -> nothing to see here
> This function returns a tristate value -> brain explosion!
It is an interface which forces the driver writers to write
complicated fallback code which won't usually be excercised. The same
goes for the hardware. In isolation, it doesn't look like much but
things like this are bound to lead to subtle bugs which are diffiuclt
to trigger.
> > * Determine the number of MSIs the controller wants. Don't worry
> > about quotas or limits or anything. Just determine the number
> > necessary to enable enhanced interrupt handling.
> >
> > * Try allocating that number of MSIs. If it fails, then just revert
> > to single interrupt mode. It's not the end of the world and mostly
> > guaranteed to work. Let's please not even try to do partial
> > multiple interrupts. I really don't think it's worth the risk or
> > complexity.
>
> It will potentially break existing setups on our hardware.
I think it'd be much better to have a separate interface for the
drivers which actually require it *in practice* rather than forcing
everyone to go "oh this interface supports that, I don't know if I
need it but let's implement fallback logic which I won't and have no
means of testing". Are we talking about some limited number of device
drivers here? Also, is the quota still necessary for machines in
production today?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-10-01 11:55 ` Tejun Heo
@ 2013-10-02 2:33 ` Michael Ellerman
2013-10-02 3:23 ` Tejun Heo
0 siblings, 1 reply; 34+ messages in thread
From: Michael Ellerman @ 2013-10-02 2:33 UTC (permalink / raw)
To: Tejun Heo
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Alexander Gordeev, Jan Beulich,
linux-pci@vger.kernel.org, Bjorn Helgaas, linuxppc-dev,
Ingo Molnar
On Tue, Oct 01, 2013 at 07:55:03AM -0400, Tejun Heo wrote:
> Hello,
>
> On Tue, Oct 01, 2013 at 05:35:48PM +1000, Michael Ellerman wrote:
> > > > Roughly third of the drivers just do not care and bail out once
> > > > pci_enable_msix() has not succeeded. Not sure how many of these are
> > > > mandated by the hardware.
> > >
> > > Yeah, I mean, this type of interface is a trap. People have to
> > > actively resist to avoid doing silly stuff which is a lot to ask.
> >
> > I really think you're overstating the complexity here.
> >
> > Functions typically return a boolean -> nothing to see here
> > This function returns a tristate value -> brain explosion!
>
> It is an interface which forces the driver writers to write
> complicated fallback code which won't usually be excercised.
It does not force anyone to do anything. That's just bull.
Code which is unwilling or unable to cope with the extra complexity
can simply do:
if (pci_enable_msix(..))
goto fail;
It's as simple as that.
> > > * Determine the number of MSIs the controller wants. Don't worry
> > > about quotas or limits or anything. Just determine the number
> > > necessary to enable enhanced interrupt handling.
> > >
> > > * Try allocating that number of MSIs. If it fails, then just revert
> > > to single interrupt mode. It's not the end of the world and mostly
> > > guaranteed to work. Let's please not even try to do partial
> > > multiple interrupts. I really don't think it's worth the risk or
> > > complexity.
> >
> > It will potentially break existing setups on our hardware.
>
> I think it'd be much better to have a separate interface for the
> drivers which actually require it *in practice* rather than forcing
> everyone to go "oh this interface supports that, I don't know if I
> need it but let's implement fallback logic which I won't and have no
> means of testing".
Sure, that's easy:
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index d5f90d6..48d0252 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -988,6 +988,18 @@ int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec)
}
EXPORT_SYMBOL(pci_enable_msix);
+int pci_enable_msix_or_fail(struct pci_dev *dev, struct msix_entry *entries,
+ int nvec)
+{
+ int rc;
+
+ rc = pci_enable_msix(dev, entries, nvec);
+ if (rc > 0)
+ rc = -ENOSPC;
+
+ return rc;
+}
+
void pci_msix_shutdown(struct pci_dev *dev)
{
struct msi_desc *entry;
> Are we talking about some limited number of device drivers here?
I don't have a list, but yeah there are certain drivers that folks care about.
> Also, is the quota still necessary for machines in production today?
As far as I know yes. The number of MSIs is growing on modern systems, but so
is the number of cpus and devices.
cheers
^ permalink raw reply related [flat|nested] 34+ messages in thread* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-10-02 2:33 ` Michael Ellerman
@ 2013-10-02 3:23 ` Tejun Heo
0 siblings, 0 replies; 34+ messages in thread
From: Tejun Heo @ 2013-10-02 3:23 UTC (permalink / raw)
To: Michael Ellerman
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
linux-ide@vger.kernel.org, Alexander Gordeev, Jan Beulich,
linux-pci@vger.kernel.org, Bjorn Helgaas, linuxppc-dev,
Ingo Molnar
On Wed, Oct 02, 2013 at 12:33:38PM +1000, Michael Ellerman wrote:
> > It is an interface which forces the driver writers to write
> > complicated fallback code which won't usually be excercised.
>
> It does not force anyone to do anything. That's just bull.
Yeah, sure, we don't have shitty code in drivers which don't need any
of that retry logic, right? What the hell is up with the gratuituous
escalation? You really wanna go that way?
> Code which is unwilling or unable to cope with the extra complexity
> can simply do:
>
> if (pci_enable_msix(..))
> goto fail;
>
> It's as simple as that.
You apparently have no clue how people behave. You give a function
which indicates complex failure mode, driver writers *will* try to
handle that whether they actually understand the implication or not.
That's a natural and correct behavior too because any half-competent
software eng would design API so that it receives and returns
information which is relevant to its users. If there are special
cases to handle, make the damn interface for it special too so that it
doesn't confuse the common case.
Driver codes already have generally lower quality than core code, if
for nothing else, due to the sheer volume, and there are many driver
writers who aren't too privvy with various kernel subsystems. They
usually just copy whatever other similar driver is doing, and this one
is a lot worse - this thing affects hardware directly. If you expect
all the shitty implementations of ahci to handle the different
variations of multiple MSI config cases, you just don't have any
experience dealing with cheap commodity hardware.
Driver APIs should be intuitive, clear in its intentions, and don't
tempt fate with hairy configs for vast majority of cases.
> +int pci_enable_msix_or_fail(struct pci_dev *dev, struct msix_entry *entries,
> + int nvec)
> +{
> + int rc;
> +
> + rc = pci_enable_msix(dev, entries, nvec);
> + if (rc > 0)
> + rc = -ENOSPC;
> +
> + return rc;
> +}
Make the *default* case simple and give clearly special interface for
the special cases. What's so hard about that?
> > Are we talking about some limited number of device drivers here?
>
> I don't have a list, but yeah there are certain drivers that folks care about.
And here's another problem with the current interface. Because the
default interface is the unnecessrily complicated one, now we can't
tell which ones actually need the complicated treatment.
> > Also, is the quota still necessary for machines in production today?
>
> As far as I know yes. The number of MSIs is growing on modern systems, but so
> is the number of cpus and devices.
That's a bummer, but let's please make the default interface simple.
I really don't wanna see partial allocations for ahci.
--
tejun
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-18 9:48 ` Alexander Gordeev
2013-09-18 14:22 ` Tejun Heo
@ 2013-09-26 12:32 ` Mark Lord
2013-09-26 13:03 ` Alexander Gordeev
2013-12-18 18:26 ` Bjorn Helgaas
2013-10-01 7:51 ` Michael Ellerman
2 siblings, 2 replies; 34+ messages in thread
From: Mark Lord @ 2013-09-26 12:32 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linuxppc-dev, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, Bjorn Helgaas,
Ingo Molnar
On 13-09-18 05:48 AM, Alexander Gordeev wrote:
>
> The last pattern makes most of sense to me and could be updated with a more
> clear sequence - a call to (bit modified) pci_msix_table_size() followed
> by a call to pci_enable_msix(). I think this pattern can effectively
> supersede the currently recommended "loop" practice.
The loop is still necessary, because there's a race between those two calls,
so that pci_enable_msix() can still fail due to lack of MSIX slots.
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 12:32 ` Mark Lord
@ 2013-09-26 13:03 ` Alexander Gordeev
2013-10-02 2:46 ` Mark Lord
2013-12-18 18:26 ` Bjorn Helgaas
1 sibling, 1 reply; 34+ messages in thread
From: Alexander Gordeev @ 2013-09-26 13:03 UTC (permalink / raw)
To: Mark Lord
Cc: linuxppc-dev, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, Bjorn Helgaas,
Ingo Molnar
On Thu, Sep 26, 2013 at 08:32:53AM -0400, Mark Lord wrote:
> On 13-09-18 05:48 AM, Alexander Gordeev wrote:
> > The last pattern makes most of sense to me and could be updated with a more
> > clear sequence - a call to (bit modified) pci_msix_table_size() followed
> > by a call to pci_enable_msix(). I think this pattern can effectively
> > supersede the currently recommended "loop" practice.
>
> The loop is still necessary, because there's a race between those two calls,
> so that pci_enable_msix() can still fail due to lack of MSIX slots.
Moreover, the existing loop pattern is racy and could fail just as easily ;)
But (1) that is something drivers should expect and (2) there is basically
nothing to race against - that is probably the reason it has not been a
problem for pSeries. So I think we should not care about this.
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 13:03 ` Alexander Gordeev
@ 2013-10-02 2:46 ` Mark Lord
2013-10-02 7:26 ` Alexander Gordeev
0 siblings, 1 reply; 34+ messages in thread
From: Mark Lord @ 2013-10-02 2:46 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linuxppc-dev, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, Bjorn Helgaas,
Ingo Molnar
On 13-09-26 09:03 AM, Alexander Gordeev wrote:
> On Thu, Sep 26, 2013 at 08:32:53AM -0400, Mark Lord wrote:
>> On 13-09-18 05:48 AM, Alexander Gordeev wrote:
>>> The last pattern makes most of sense to me and could be updated with a more
>>> clear sequence - a call to (bit modified) pci_msix_table_size() followed
>>> by a call to pci_enable_msix(). I think this pattern can effectively
>>> supersede the currently recommended "loop" practice.
>>
>> The loop is still necessary, because there's a race between those two calls,
>> so that pci_enable_msix() can still fail due to lack of MSIX slots.
>
> Moreover, the existing loop pattern is racy and could fail just as easily ;)
Yes, but it then loops again to correct things.
> But (1) that is something drivers should expect and (2) there is basically
> nothing to race against - that is probably the reason it has not been a
> problem for pSeries. So I think we should not care about this.
I always care about race conditions.
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-10-02 2:46 ` Mark Lord
@ 2013-10-02 7:26 ` Alexander Gordeev
0 siblings, 0 replies; 34+ messages in thread
From: Alexander Gordeev @ 2013-10-02 7:26 UTC (permalink / raw)
To: Mark Lord
Cc: linuxppc-dev, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, Bjorn Helgaas,
Ingo Molnar
On Tue, Oct 01, 2013 at 10:46:32PM -0400, Mark Lord wrote:
> >>> The last pattern makes most of sense to me and could be updated with a more
> >>> clear sequence - a call to (bit modified) pci_msix_table_size() followed
> >>> by a call to pci_enable_msix(). I think this pattern can effectively
> >>> supersede the currently recommended "loop" practice.
> >>
> >> The loop is still necessary, because there's a race between those two calls,
> >> so that pci_enable_msix() can still fail due to lack of MSIX slots.
> >
> > Moreover, the existing loop pattern is racy and could fail just as easily ;)
>
> Yes, but it then loops again to correct things.
No. If it failed it should exit the loop.
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-26 12:32 ` Mark Lord
2013-09-26 13:03 ` Alexander Gordeev
@ 2013-12-18 18:26 ` Bjorn Helgaas
1 sibling, 0 replies; 34+ messages in thread
From: Bjorn Helgaas @ 2013-12-18 18:26 UTC (permalink / raw)
To: Mark Lord
Cc: Joerg Roedel, x86@kernel.org, linux-kernel@vger.kernel.org,
Michael Ellerman, linux-ide@vger.kernel.org, Alexander Gordeev,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, linuxppc-dev,
Ingo Molnar
On Thu, Sep 26, 2013 at 08:32:53AM -0400, Mark Lord wrote:
> On 13-09-18 05:48 AM, Alexander Gordeev wrote:
> >
> > The last pattern makes most of sense to me and could be updated with a more
> > clear sequence - a call to (bit modified) pci_msix_table_size() followed
> > by a call to pci_enable_msix(). I think this pattern can effectively
> > supersede the currently recommended "loop" practice.
>
> The loop is still necessary, because there's a race between those two calls,
> so that pci_enable_msix() can still fail due to lack of MSIX slots.
Hi Mark,
Can you elaborate on this race? My understanding is that
pci_msix_table_size() depends only on the "Table Size" field in the MSI-X
Message Control register.
So if there's a concurrency problem here, it would have to be something
like "pci_enable_msix() may not be able to configure the requested number
of vectors because it has to allocate from a shared pool."
If that's the case, pci_msix_table_size() wouldn't be involved at all, and
the only question is how to coordinate between several drivers that each
call pci_enable_msix(). I think that would have to be resolved in some
arch hook used by the PCI core.
Maybe this is already taken care of; I just want to make sure we don't
overlook an issue here.
Bjorn
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-09-18 9:48 ` Alexander Gordeev
2013-09-18 14:22 ` Tejun Heo
2013-09-26 12:32 ` Mark Lord
@ 2013-10-01 7:51 ` Michael Ellerman
2013-10-01 10:35 ` Alexander Gordeev
2 siblings, 1 reply; 34+ messages in thread
From: Michael Ellerman @ 2013-10-01 7:51 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linuxppc-dev, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, Bjorn Helgaas,
Ingo Molnar
On Wed, Sep 18, 2013 at 11:48:00AM +0200, Alexander Gordeev wrote:
> On Wed, Sep 18, 2013 at 12:30:23AM +1000, Michael Ellerman wrote:
> > How about no?
> >
> > We have a small number of MSIs available, limited by hardware &
> > firmware, if we don't impose a quota then the first device that probes
> > will get most/all of the MSIs and other devices miss out.
>
> Out of curiosity - how pSeries has had done it without quotas before
> 448e2ca ("powerpc/pseries: Implement a quota system for MSIs")?
>
> > Anyway I don't see what problem you're trying to solve? I agree the
> > -ve/0/+ve return value pattern is ugly, but it's hardly the end of the
> > world.
>
> Well, the interface recently has been re-classified from "ugly" to
> "unnecessarily complex and actively harmful" in Tejun's words ;)
>
> Indeed, I checked most of the drivers and it is incredible how people
> are creative in misusing the interface: from innocent pci_disable_msix()
> calls when if pci_enable_msix() failed to assuming MSI-Xs were enabled
> if pci_enable_msix() returned a positive value (apparently untested).
OK, but we have the source to the drivers, we could just fix them.
We could even add:
pci_enable_msix_i_am_stupid()
Which swallows the positive return and just gives back -ve/0.
> Roughly third of the drivers just do not care and bail out once
> pci_enable_msix() has not succeeded. Not sure how many of these are
> mandated by the hardware.
Sure, that's fine if those drivers do that, it's up to the drivers after
all.
> Another quite common pattern is a call to pci_enable_msix() to figure out
> the number of MSI-Xs available and a repeated call of pci_enable_msix()
> to enable those MSI-Xs, this time.
Also fine, though as the documentation suggests a loop is the best
construct rather than two explicit calls.
> The recommended practice would be:
>
> /*
> * Retrieving 'nvec' by means other than pci_msix_table_size()
> */
>
> rc = pci_get_msix_limit(pdev);
> if (rc < 0)
> return rc;
>
> /*
> * nvec = min(rc, nvec);
> */
>
> for (i = 0; i < nvec; i++)
> msix_entry[i].entry = i;
>
> rc = pci_enable_msix(pdev, msix_entry, nvec);
> if (rc)
> return rc;
>
> Thoughts?
We could probably make that work.
The disadvantage is that any restriction imposed on us above the quota
can only be reported as an error from pci_enable_msix().
The quota code, called from pci_get_msix_limit(), can only do so much to
interogate firmware about the limitations. The ultimate way to check if
firmware will give us enough MSIs is to try and allocate them. But we
can't do that from pci_get_msix_limit() because the driver is not asking
us to enable MSIs, just query them.
You'll also need to add another arch hook, for the quota check, and
we'll have to add it to our per-platform indirection as well.
All a lot of bother for no real gain IMHO.
cheers
^ permalink raw reply [flat|nested] 34+ messages in thread* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-10-01 7:51 ` Michael Ellerman
@ 2013-10-01 10:35 ` Alexander Gordeev
2013-10-02 2:43 ` Michael Ellerman
0 siblings, 1 reply; 34+ messages in thread
From: Alexander Gordeev @ 2013-10-01 10:35 UTC (permalink / raw)
To: Michael Ellerman
Cc: linuxppc-dev, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, Bjorn Helgaas,
Ingo Molnar
On Tue, Oct 01, 2013 at 05:51:33PM +1000, Michael Ellerman wrote:
> The disadvantage is that any restriction imposed on us above the quota
> can only be reported as an error from pci_enable_msix().
>
> The quota code, called from pci_get_msix_limit(), can only do so much to
> interogate firmware about the limitations. The ultimate way to check if
> firmware will give us enough MSIs is to try and allocate them. But we
> can't do that from pci_get_msix_limit() because the driver is not asking
> us to enable MSIs, just query them.
If things are this way then pci_enable_msix() already exposed to this
problem internally on pSeries.
I see that even successful quota checks in rtas_msi_check_device() and
rtas_setup_msi_irqs() do not guarantee (as you say) that firmware will
give enough MSIs. Hence, pci_enable_msix() might fail even though the
its quota checks succeeded.
Therefore, nothing will really change if we make pci_get_msix_limit() check
quota and hope the follow-up call to pci_enable_msix() succeeded.
(Of course, we could allocate-deallocate MSIs at check time, but I think it
is an overkill).
> You'll also need to add another arch hook, for the quota check, and
> we'll have to add it to our per-platform indirection as well.
Already, in a branch, hidden from Bjorn & Tejun eyes ;)
> All a lot of bother for no real gain IMHO.
Well, I do not have a strong opinion here. I leave it to the ones who have :)
But few drivers have became clearer as result of this change (and messy ones
are still messy).
> cheers
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-10-01 10:35 ` Alexander Gordeev
@ 2013-10-02 2:43 ` Michael Ellerman
2013-10-02 7:10 ` Alexander Gordeev
0 siblings, 1 reply; 34+ messages in thread
From: Michael Ellerman @ 2013-10-02 2:43 UTC (permalink / raw)
To: Alexander Gordeev
Cc: linuxppc-dev, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, Bjorn Helgaas,
Ingo Molnar
On Tue, Oct 01, 2013 at 12:35:27PM +0200, Alexander Gordeev wrote:
> On Tue, Oct 01, 2013 at 05:51:33PM +1000, Michael Ellerman wrote:
> > The disadvantage is that any restriction imposed on us above the quota
> > can only be reported as an error from pci_enable_msix().
> >
> > The quota code, called from pci_get_msix_limit(), can only do so much to
> > interogate firmware about the limitations. The ultimate way to check if
> > firmware will give us enough MSIs is to try and allocate them. But we
> > can't do that from pci_get_msix_limit() because the driver is not asking
> > us to enable MSIs, just query them.
>
> If things are this way then pci_enable_msix() already exposed to this
> problem internally on pSeries.
>
> I see that even successful quota checks in rtas_msi_check_device() and
> rtas_setup_msi_irqs() do not guarantee (as you say) that firmware will
> give enough MSIs. Hence, pci_enable_msix() might fail even though the
> its quota checks succeeded.
Yes, but it can report that failure to the caller, which can then retry.
> Therefore, nothing will really change if we make pci_get_msix_limit() check
> quota and hope the follow-up call to pci_enable_msix() succeeded.
No that's not equivalent. Under your scheme if pci_enable_msix() fails
then the caller just bails, it will never try again with a lower number.
> (Of course, we could allocate-deallocate MSIs at check time, but I think it
> is an overkill).
It's not only overkill, it's messing with the device behind the drivers
back, which is definitely a no-no in my opinion.
> > You'll also need to add another arch hook, for the quota check, and
> > we'll have to add it to our per-platform indirection as well.
>
> Already, in a branch, hidden from Bjorn & Tejun eyes ;)
>
> > All a lot of bother for no real gain IMHO.
>
> Well, I do not have a strong opinion here. I leave it to the ones who have :)
> But few drivers have became clearer as result of this change (and messy ones
> are still messy).
Amen.
cheers
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH v2 2/6] PCI/MSI: Factor out pci_get_msi_cap() interface
2013-10-02 2:43 ` Michael Ellerman
@ 2013-10-02 7:10 ` Alexander Gordeev
0 siblings, 0 replies; 34+ messages in thread
From: Alexander Gordeev @ 2013-10-02 7:10 UTC (permalink / raw)
To: Michael Ellerman
Cc: linuxppc-dev, Joerg Roedel, x86@kernel.org,
linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org,
Jan Beulich, linux-pci@vger.kernel.org, Tejun Heo, Bjorn Helgaas,
Ingo Molnar
On Wed, Oct 02, 2013 at 12:43:24PM +1000, Michael Ellerman wrote:
> On Tue, Oct 01, 2013 at 12:35:27PM +0200, Alexander Gordeev wrote:
> > On Tue, Oct 01, 2013 at 05:51:33PM +1000, Michael Ellerman wrote:
> > > The disadvantage is that any restriction imposed on us above the quota
> > > can only be reported as an error from pci_enable_msix().
> > >
> > > The quota code, called from pci_get_msix_limit(), can only do so much to
> > > interogate firmware about the limitations. The ultimate way to check if
> > > firmware will give us enough MSIs is to try and allocate them. But we
> > > can't do that from pci_get_msix_limit() because the driver is not asking
> > > us to enable MSIs, just query them.
> >
> > If things are this way then pci_enable_msix() already exposed to this
> > problem internally on pSeries.
> >
> > I see that even successful quota checks in rtas_msi_check_device() and
> > rtas_setup_msi_irqs() do not guarantee (as you say) that firmware will
> > give enough MSIs. Hence, pci_enable_msix() might fail even though the
> > its quota checks succeeded.
>
> Yes, but it can report that failure to the caller, which can then retry.
If a driver wants to retry after a failure it is up to the driver (but why?).
The current guidlines state:
"If this function returns a negative number, it indicates an error and
the driver should not attempt to allocate any more MSI-X interrupts for
this device."
Anyway, what number could the driver retry with after it got a negative errno?
> > Therefore, nothing will really change if we make pci_get_msix_limit() check
> > quota and hope the follow-up call to pci_enable_msix() succeeded.
>
> No that's not equivalent. Under your scheme if pci_enable_msix() fails
> then the caller just bails, it will never try again with a lower number.
Currently under the very same circumstances (the quota check within
rtas_setup_msi_irqs() returned Q vectors while the firmware has only F
vectors to allocate and Q > F) rtas_setup_msi_irqs() fails, pci_enable_msix()
fails, the caller bails and never try again with a lower number.
Am I missing something here?
> cheers
--
Regards,
Alexander Gordeev
agordeev@redhat.com
^ permalink raw reply [flat|nested] 34+ messages in thread