Re: [PATCH 1/6] Rewrite MSI-HOWTO

Matthew Wilcox
2009-02-27T12:14:56Z

On Thu, Feb 26, 2009 at 11:15:25PM -0700, Grant Grundler wrote:
> ...
> > +3. Why use MSIs?
> > +
> > +There are three reasons why using MSIs can give an advantage over
> > +traditional pin-based interrupts.
> ...
> > +PCI devices can only support a single pin-based interrupt per function.
>
> Related to this is a 4th reason: distribute workload across CPUs
> and enables construction of efficient, multi-queue devices.
> Care to mention that?

That's true for MSI-X, but not for MSIs in general. Workload is already
distributed across CPUs with round-robin interrupts. I'm inclined to
leave out this level of detail.

> > +The MSI-X capability is much more flexible than the MSI capability.
> > +It supports up to 2048 interrupts, each of which can be separately
> > +assigned.
>
> Nothing describes "assignment" below or what is meant by "assigned".
> My guess is you wanted to differentiate MSIX from MSI with:
> ... and each MSIX can be directed at a different CPU.

I think 'each of which can be controlled separately' might work better.
For example, they're individually maskable which isn't (necessarily)
true of plain MSI.

> > +4.5 Considerations when using MSIs
> > +
> > +4.5.1 Choosing between MSI-X and MSI
> > +
> > +If your device supports both MSI-X and MSI capabilities, you should use
> > +the MSI-X facilities in preference to the MSI facilities. As mentioned
> > +above, MSI-X supports any number of interrupts between 1 and 2048.
> > +In constrast, MSI is restricted to a maximum of 32 interrupts (and
> > +must be a power of two). In addition, the MSI interrupt vectors must
> > +be allocated consecutively, so the system may not be able to allocate
> > +as many vectors for MSI as it could for MSI-X. On some platforms, MSI
> > +interrupts must all be targetted at the same set of CPUs whereas MSI-X
> > +interrupts can all be targetted at different CPUs.
>
> The description for MSI is correct. But Linux will only allocate one
> MSI as noted in an earlier section. This section implies more could
> be allocated when using MSI and that won't happen.
>
> IIRC, for AHCI perf you were working on a patch to change that and
> it should probably update this text at the same time when the
> behavior changes.

Did you see this is patch 1/6? ;-) I removed the description of
pci_enable_msi_block() from this patch, but missed updating this
paragraph. By patch 6/6, this paragraph is true.

> > +5. MSI quirks
> > +
> > +Several PCI chipsets or devices are known not to support MSIs.
> > +The PCI stack provides three ways to disable MSIs:
> > +
> > +1. globally
> > +2. on all devices behind a specific bridge
> > +3. on a single device
> ...
> > +5.3. Disabling MSIs on a single device
> > +
> > +Some devices are known to have faulty MSI implementations. Usually this
> > +is handled in the individual device driver but occasionally it's necessary
> > +to handle this with a quirk. Some drivers have an option to disable MSIs;
> > +this is deprecated.
>
> "this" is ambiguous. My guess is "quirks to disable MSI for a device is
> deprecated" since recently some drivers have added module parameters to
> disable MSI.

Having an option to disable MSI is deprecated. That doesn't mean that
individual driver authors aren't selfish and short-sighted.

> > +5.4. Finding why MSIs are disabled on a device
> > +
> > +From the above three sections, you can see that there are many reasons
> > +why MSIs may not be enabled for a given device. Your first step should
> > +be to examine your dmesg carefully to determine whether MSIs are enabled
> > +for your machine. You should also check your .config to be sure you
> > +have enabled CONFIG_PCI_MSI.
>
> Should mention "fgrep MSI /proc/interrupts" to see if any devices have
> MSI in use?

Yes, you're right.

> > +Then, lspci -t gives the list of bridges above a device. Reading
> > +/sys/bus/pci/devices/*/msi_bus will tell you whether MSI are enabled (1)
> > +or disabled (0). If 0 is found in any of the msi_bus files belonging
> > +to bridges between the PCI root and the device, MSIs are disabled.
> > +
> > +It is also worth checking whether the device driver supports MSIs.
>
> Suggestions on how to check?

'eg has calls to pci_enable_msi(), pci_enable_msix() or
pci_enable_msi_block()'?

> Conversely, one can easily check if the driver has MSI disabled by default
> and MSI can be enabled. e.g. use "modinfo mvsas" to check driver parameters.

I'm not going to give examples of bad practise.

> Reviewed-by: Grant Grundler <grundler@parisc-linunx.org>

Thanks.

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."