known MSI errata?

known MSI errata?

[Lucas Stach]

Hi,

recently we tried to enable MSI interrupts with nouveau. Unfortunately
there have been some reports of things failing with certain cards, where
it isn't entirely clear if this is a GPU errata or some other component
in the PCIe chain failing.

Could you perhaps investigate if there are any known Nvidia GPU erratas
with regard to MSI interrupts, or maybe tell us the generations of cards
that are generally safe to enable MSIs with?

Thanks,
Lucas

Re: known MSI errata?

[Robert Morell]

On Mon, Sep 30, 2013 at 10:44:12AM -0700, Lucas Stach wrote:
> Hi,
> 
> recently we tried to enable MSI interrupts with nouveau. Unfortunately
> there have been some reports of things failing with certain cards, where
> it isn't entirely clear if this is a GPU errata or some other component
> in the PCIe chain failing.
> 
> Could you perhaps investigate if there are any known Nvidia GPU erratas
> with regard to MSI interrupts, or maybe tell us the generations of cards
> that are generally safe to enable MSIs with?

Sorry for the slow reply, Lucas.

We enabled MSI interrupts by default relatively recently in the proprietary
driver (version 325.08, released 2013-07-01 according to our public
changelog).  This was enabled across the board for all of the GPUs supported
by the latest release series, so NV50 and up.  We believe it should be safe to
enable MSI on those GPUs.  We never enabled MSI by default on earlier GPUs so
I can't comment there.

I investigated our internal documentation and source code, and found a couple
of things that are probably interesting to you:
- For all pre-Fermi GPUs, we use a write through PCI config space to the "EOI"
  register to rearm the MSI interrupt after servicing it, rather than a write
  through the MMIO pcicfg shadow region in the GPU's PCI BAR0 window at offset
  0x88000.  (This was actually originally implemented for NV4x, so you'll
  probably want to do that there as well.)  It seems that this was done to
  avoid a hardware bug that may cause the EOI write to be dropped, blocking
  all further interrupts from being serviced.
- For GF100 and GF104, an alternate MSI EOI/rearm mechanism is used: instead
  of writing to the MSI rearm register at offset 0x68 in the pcicfg window,
  we write to offset 0x704.


I hope this information helps.

- Robert

Re: known MSI errata?

[Ben Skeggs]

On Fri, Oct 25, 2013 at 7:43 AM, Robert Morell <rmorell-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> On Mon, Sep 30, 2013 at 10:44:12AM -0700, Lucas Stach wrote:
>> Hi,
>>
>> recently we tried to enable MSI interrupts with nouveau. Unfortunately
>> there have been some reports of things failing with certain cards, where
>> it isn't entirely clear if this is a GPU errata or some other component
>> in the PCIe chain failing.
>>
>> Could you perhaps investigate if there are any known Nvidia GPU erratas
>> with regard to MSI interrupts, or maybe tell us the generations of cards
>> that are generally safe to enable MSIs with?
>
> Sorry for the slow reply, Lucas.
>
> We enabled MSI interrupts by default relatively recently in the proprietary
> driver (version 325.08, released 2013-07-01 according to our public
> changelog).  This was enabled across the board for all of the GPUs supported
> by the latest release series, so NV50 and up.  We believe it should be safe to
> enable MSI on those GPUs.  We never enabled MSI by default on earlier GPUs so
> I can't comment there.
>
> I investigated our internal documentation and source code, and found a couple
> of things that are probably interesting to you:
> - For all pre-Fermi GPUs, we use a write through PCI config space to the "EOI"
>   register to rearm the MSI interrupt after servicing it, rather than a write
>   through the MMIO pcicfg shadow region in the GPU's PCI BAR0 window at offset
>   0x88000.  (This was actually originally implemented for NV4x, so you'll
>   probably want to do that there as well.)
Hm, I recently discovered this while looking for the errata myself,
but it seems you guys only do it on G80/G84/G86, and use 0x88068 at
some point after that, before Fermi (G96/GT200, for example).

>  It seems that this was done to
>   avoid a hardware bug that may cause the EOI write to be dropped, blocking
>   all further interrupts from being serviced.
That was our theory also.  Thanks for clarifying.

> - For GF100 and GF104, an alternate MSI EOI/rearm mechanism is used: instead
>   of writing to the MSI rearm register at offset 0x68 in the pcicfg window,
>   we write to offset 0x704.
Thanks for this.  Maarten discovered this for GF100, I'll modify our
patch to cover GF104 also.

>
>
> I hope this information helps.
>
> - Robert
> _______________________________________________
> Nouveau mailing list
> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: known MSI errata?

[Robert Morell]

On Thu, Oct 24, 2013 at 04:03:12PM -0700, Ben Skeggs wrote:
> On Fri, Oct 25, 2013 at 7:43 AM, Robert Morell <rmorell-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> > On Mon, Sep 30, 2013 at 10:44:12AM -0700, Lucas Stach wrote:
> >> Hi,
> >>
> >> recently we tried to enable MSI interrupts with nouveau. Unfortunately
> >> there have been some reports of things failing with certain cards, where
> >> it isn't entirely clear if this is a GPU errata or some other component
> >> in the PCIe chain failing.
> >>
> >> Could you perhaps investigate if there are any known Nvidia GPU erratas
> >> with regard to MSI interrupts, or maybe tell us the generations of cards
> >> that are generally safe to enable MSIs with?
> >
> > Sorry for the slow reply, Lucas.
> >
> > We enabled MSI interrupts by default relatively recently in the proprietary
> > driver (version 325.08, released 2013-07-01 according to our public
> > changelog).  This was enabled across the board for all of the GPUs supported
> > by the latest release series, so NV50 and up.  We believe it should be safe to
> > enable MSI on those GPUs.  We never enabled MSI by default on earlier GPUs so
> > I can't comment there.
> >
> > I investigated our internal documentation and source code, and found a couple
> > of things that are probably interesting to you:
> > - For all pre-Fermi GPUs, we use a write through PCI config space to the "EOI"
> >   register to rearm the MSI interrupt after servicing it, rather than a write
> >   through the MMIO pcicfg shadow region in the GPU's PCI BAR0 window at offset
> >   0x88000.  (This was actually originally implemented for NV4x, so you'll
> >   probably want to do that there as well.)
> Hm, I recently discovered this while looking for the errata myself,
> but it seems you guys only do it on G80/G84/G86, and use 0x88068 at
> some point after that, before Fermi (G96/GT200, for example).

Yes, you're right; I was reading this wrong.  The config space path
described above is used on G8x and G92.  Newer GPUs (other than GF100
and GF104 as mentioned previously) use the "normal" path.

- Robert

Re: known MSI errata?

[Ben Skeggs]

On Fri, Oct 25, 2013 at 9:10 AM, Robert Morell <rmorell-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> On Thu, Oct 24, 2013 at 04:03:12PM -0700, Ben Skeggs wrote:
>> On Fri, Oct 25, 2013 at 7:43 AM, Robert Morell <rmorell-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
>> > On Mon, Sep 30, 2013 at 10:44:12AM -0700, Lucas Stach wrote:
>> >> Hi,
>> >>
>> >> recently we tried to enable MSI interrupts with nouveau. Unfortunately
>> >> there have been some reports of things failing with certain cards, where
>> >> it isn't entirely clear if this is a GPU errata or some other component
>> >> in the PCIe chain failing.
>> >>
>> >> Could you perhaps investigate if there are any known Nvidia GPU erratas
>> >> with regard to MSI interrupts, or maybe tell us the generations of cards
>> >> that are generally safe to enable MSIs with?
>> >
>> > Sorry for the slow reply, Lucas.
>> >
>> > We enabled MSI interrupts by default relatively recently in the proprietary
>> > driver (version 325.08, released 2013-07-01 according to our public
>> > changelog).  This was enabled across the board for all of the GPUs supported
>> > by the latest release series, so NV50 and up.  We believe it should be safe to
>> > enable MSI on those GPUs.  We never enabled MSI by default on earlier GPUs so
>> > I can't comment there.
>> >
>> > I investigated our internal documentation and source code, and found a couple
>> > of things that are probably interesting to you:
>> > - For all pre-Fermi GPUs, we use a write through PCI config space to the "EOI"
>> >   register to rearm the MSI interrupt after servicing it, rather than a write
>> >   through the MMIO pcicfg shadow region in the GPU's PCI BAR0 window at offset
>> >   0x88000.  (This was actually originally implemented for NV4x, so you'll
>> >   probably want to do that there as well.)
>> Hm, I recently discovered this while looking for the errata myself,
>> but it seems you guys only do it on G80/G84/G86, and use 0x88068 at
>> some point after that, before Fermi (G96/GT200, for example).
>
> Yes, you're right; I was reading this wrong.  The config space path
> described above is used on G8x and G92.  Newer GPUs (other than GF100
> and GF104 as mentioned previously) use the "normal" path.
Ah, excellent.  I probably should've guessed G92 would be involved here too.

Thanks again,
Ben.

>
> - Robert