Re: [PATCH for-4.21] vpci/msix: improve handling of bogus MSI-X capabilities

[Alejandro Vallejo <alejandro.garciavallejo@amd.com>]

On Tue Sep 30, 2025 at 2:57 PM CEST, Roger Pau Monné wrote:
> On Tue, Sep 30, 2025 at 11:15:01AM +0200, Alejandro Vallejo wrote:
>> On Mon Sep 29, 2025 at 10:41 AM CEST, Roger Pau Monne wrote:
>> > I've had the luck to come across a PCI card that exposes a MSI-X capability
>> > where the BIR of the vector and PBA tables points at a BAR that has 0 size.
>> >
>> > This doesn't play nice with the code in vpci_make_msix_hole(), as it would
>> > still use the address of such empty BAR (0) and attempt to crave a hole in
>> > the p2m.  This leads to errors like the one below being reported by Xen:
>> >
>> > d0v0 0000:22:00.0: existing mapping (mfn: 181c4300 type: 0) at 0 clobbers MSIX MMIO area
>> >
>> > And the device left unable to enable memory decoding due to the failure
>> > reported by vpci_make_msix_hole().
>> >
>> > Introduce checking in init_msix() to ensure the BARs containing the MSI-X
>> > tables are usable.  This requires checking that the BIR points to a
>> > non-empty BAR, and the offset and size of the MSI-X tables can fit in the
>> > target BAR.
>> >
>> > This fixes booting PVH dom0 on Supermicro AS -2126HS-TN severs with AMD
>> > EPYC 9965 processors.  The broken device is:
>> >
>> > 22:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 93)
>> >
>> > There are multiple of those integrated controllers in the system, all
>> > broken in the same way.
>> >
>> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
>> > ---
>> > Cc: Stewart Hildebrand <stewart.hildebrand@amd.com>
>> > Cc: Jan Beulich <jbeulich@suse.com>
>> > Cc: Oleksii Kurochko <oleksii.kurochko@gmail.com>
>> >
>> > While not strictly a bugfix, I consider this a worthy improvement so that
>> > PVH dom0 has a chance to boot on hardware that exposes such broken MSI-X
>> > capabilities.  Hence I think this change should be considered for inclusion
>> > into 4.21.  There a risk of regressing on hardware that was already working
>> > with PVH, but given enough testing that should be minimal.
>> > ---
>> >  xen/drivers/vpci/msix.c | 50 ++++++++++++++++++++++++++++++++++++-----
>> >  1 file changed, 45 insertions(+), 5 deletions(-)
>> >
>> > diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c
>> > index 54a5070733aa..8458955d5bbb 100644
>> > --- a/xen/drivers/vpci/msix.c
>> > +++ b/xen/drivers/vpci/msix.c
>> > @@ -675,6 +675,51 @@ static int cf_check init_msix(struct pci_dev *pdev)
>> >      if ( !msix )
>> >          return -ENOMEM;
>> >  
>> > +    msix->tables[VPCI_MSIX_TABLE] =
>> > +        pci_conf_read32(pdev->sbdf, msix_table_offset_reg(msix_offset));
>> > +    msix->tables[VPCI_MSIX_PBA] =
>> > +        pci_conf_read32(pdev->sbdf, msix_pba_offset_reg(msix_offset));
>> > +
>> > +    /* Check that the provided BAR is valid. */
>> > +    for ( i = 0; i < ARRAY_SIZE(msix->tables); i++ )
>> > +    {
>> > +        const char *name = (i == VPCI_MSIX_TABLE) ? "vector" : "PBA";
>> > +        const struct vpci_bar *bars = pdev->vpci->header.bars;
>> > +        unsigned int bir = msix->tables[i] & PCI_MSIX_BIRMASK;
>> > +        unsigned int type;
>> > +        unsigned int offset = msix->tables[i] & ~PCI_MSIX_BIRMASK;
>> > +        unsigned int size =
>> > +            (i == VPCI_MSIX_TABLE) ? max_entries * PCI_MSIX_ENTRY_SIZE
>> > +                                   : ROUNDUP(DIV_ROUND_UP(max_entries, 8), 8);
>> > +
>> > +        if ( bir >= ARRAY_SIZE(pdev->vpci->header.bars) )
>> > +        {
>> > +            printk(XENLOG_ERR "%pp: MSI-X %s table with out of range BIR %u\n",
>> > +                   &pdev->sbdf, name, bir);
>> 
>> Would it be worth adding something here such that a device vendor testing their
>> hardware under Xen can trivially grep for device bugs?
>> 
>> Something akin to "[Firmware bug]" on Linux, like "[Device bug]" or some such.
>> 
>> It would also let anyone not very knowledgeable about PCI know that a device
>> they own is being unreasonable. Same below in the other XENLOG_ERR messages.
>
> We could add indeed.  I don't think we haven't done so in the past.
> If we go that route I would suggest that I add a:
>
> #define DEVICE_BUG_PREFIX "[Device bug] "
>
> in lib.h or similar, to make sure we use the same prefix uniformly.
> TBH

That works. As would DEV_BUG_PREFIX, XENLOG_DEV_BUG or even just DEV_BUG.

LGTM in any form or shape that makes it patently clear the admin is running
on buggy hardware.

> I think vendors care little about the output of Xen, as long as it boots.

They might care more once they realise they can "grep" lines of interest. That's
what happened on Linux, after all.

Also, it's not just for vendors. Users and developers alike might be interested
in using these message as a "taint" to know when they bought known-bad hardware.

Maybe to avoid buying it twice.

>
> The downside of this is that it makes those messages longer, which
> will require more time to print if using a slow UART.

Only the reporting of known bugs. Which isn't a high throughput operation.

Hopefully.

Cheers,
Alejandro