[Xenomai-help] Stuck MSI in normal Linux driver

[Xenomai-help] Stuck MSI in normal Linux driver

[Richard Cochran]

I am running Xenomai 2.5 git master on a P2020 with kernel 2.6.36.

We have a custom PCIe card that raise a MSI, with a normal (not rtdm)
driver. It appears that the interrupt comes just once and then is
stuck. It work under plain Linux, but I cannot rule out a HW timing
bug either.

The wiki pages 

   http://www.xenomai.org/index.php/FAQs
   http://www.xenomai.org/index.php/Configuring_x86_kernels

say not to enable MSI on x86, with a link to a very old (2008)
discussion. I found a newer mail

   https://mail.gna.org/public/xenomai-help/2010-01/msg00095.html

claiming that MSI is okay.

Is there a known problem with MSI and Xenomai?

If so, where/how can I start to get working on fixing it?

(I don't need the MSI in a rtdm or adeos context.)

Thanks,

Richard

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Gilles Chanteperdrix]

Richard Cochran wrote:
> I am running Xenomai 2.5 git master on a P2020 with kernel 2.6.36.
> 
> We have a custom PCIe card that raise a MSI, with a normal (not rtdm)
> driver. It appears that the interrupt comes just once and then is
> stuck. It work under plain Linux, but I cannot rule out a HW timing
> bug either.
> 
> The wiki pages 
> 
>    http://www.xenomai.org/index.php/FAQs
>    http://www.xenomai.org/index.php/Configuring_x86_kernels
> 
> say not to enable MSI on x86, with a link to a very old (2008)
> discussion. I found a newer mail
> 
>    https://mail.gna.org/public/xenomai-help/2010-01/msg00095.html
> 
> claiming that MSI is okay.

No. This is my fault. I simply had forgotten that MSI were not OK on x86.

> 
> Is there a known problem with MSI and Xenomai?

There is a known problem with MSI on x86 (explained by the FAQ link).

> 
> If so, where/how can I start to get working on fixing it?
> 
> (I don't need the MSI in a rtdm or adeos context.)

Given that the problem is that the interrupt controller ack/mask
functions (if I recall correctly) need, on x86, to use some Linux
primitives which can not be called from Xenomai domain, I am not sure it
is easy to fix. And the fact that you do not need the MSI in Xenomai
domain does not change anything, since ack/mask routines are called
ahead of the pipeline anyway.

-- 
					    Gilles.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Richard Cochran]

On Wed, Mar 02, 2011 at 04:26:46PM +0100, Gilles Chanteperdrix wrote:
> Richard Cochran wrote:
> > If so, where/how can I start to get working on fixing it?
> > 
> > (I don't need the MSI in a rtdm or adeos context.)
> 
> Given that the problem is that the interrupt controller ack/mask
> functions (if I recall correctly) need, on x86, to use some Linux
> primitives which can not be called from Xenomai domain, I am not sure it
> is easy to fix. And the fact that you do not need the MSI in Xenomai
> domain does not change anything, since ack/mask routines are called
> ahead of the pipeline anyway.

Okay, but how about on PowerPC, can I get to work there?

Thanks,

Richard

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Gilles Chanteperdrix]

Richard Cochran wrote:
> On Wed, Mar 02, 2011 at 04:26:46PM +0100, Gilles Chanteperdrix wrote:
>> Richard Cochran wrote:
>>> If so, where/how can I start to get working on fixing it?
>>>
>>> (I don't need the MSI in a rtdm or adeos context.)
>> Given that the problem is that the interrupt controller ack/mask
>> functions (if I recall correctly) need, on x86, to use some Linux
>> primitives which can not be called from Xenomai domain, I am not sure it
>> is easy to fix. And the fact that you do not need the MSI in Xenomai
>> domain does not change anything, since ack/mask routines are called
>> ahead of the pipeline anyway.
> 
> Okay, but how about on PowerPC, can I get to work there?

I do not think powerpc has the same issue. Check that you do not have an
edge/level issue (MSI are edge triggered I think).


-- 
					    Gilles.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Jan Kiszka]

On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
> Richard Cochran wrote:
>> I am running Xenomai 2.5 git master on a P2020 with kernel 2.6.36.
>>
>> We have a custom PCIe card that raise a MSI, with a normal (not rtdm)
>> driver. It appears that the interrupt comes just once and then is
>> stuck. It work under plain Linux, but I cannot rule out a HW timing
>> bug either.
>>
>> The wiki pages 
>>
>>    http://www.xenomai.org/index.php/FAQs
>>    http://www.xenomai.org/index.php/Configuring_x86_kernels
>>
>> say not to enable MSI on x86, with a link to a very old (2008)
>> discussion. I found a newer mail
>>
>>    https://mail.gna.org/public/xenomai-help/2010-01/msg00095.html
>>
>> claiming that MSI is okay.
> 
> No. This is my fault. I simply had forgotten that MSI were not OK on x86.
> 
>>
>> Is there a known problem with MSI and Xenomai?
> 
> There is a known problem with MSI on x86 (explained by the FAQ link).
> 
>>
>> If so, where/how can I start to get working on fixing it?
>>
>> (I don't need the MSI in a rtdm or adeos context.)
> 
> Given that the problem is that the interrupt controller ack/mask
> functions (if I recall correctly) need, on x86, to use some Linux
> primitives which can not be called from Xenomai domain, I am not sure it
> is easy to fix. And the fact that you do not need the MSI in Xenomai
> domain does not change anything, since ack/mask routines are called
> ahead of the pipeline anyway.

The current situation looks like this:
 - MSI[-X] usage for plain Linux drivers is fine, we are using it on
   machines where you can hardly avoid MSI today
 - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
   you avoid enabling/disabling from RT context (also used for quite
   a while here, no known issues under this restriction)

Fixing that enabling/disabling (i.e. mask/unmask) is indeed tricky for
classic MSI as we can't rely on direct hardware (too slow). It's easier
for MSI-X - as long as the addressed device is not delaying the mask
operation in firmware or wherever (seen only with MSI devices so far,
but might be possible with MSI-X as well).

Note that much of this is arch independent, only the PCI config space
access method required for classic MSI may vary (likely positively).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Richard Cochran]

On Wed, Mar 02, 2011 at 07:18:46PM +0100, Jan Kiszka wrote:
> On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
> > Richard Cochran wrote:
> >> The wiki pages 
> >>
> >>    http://www.xenomai.org/index.php/FAQs
> >>    http://www.xenomai.org/index.php/Configuring_x86_kernels
> >>
> >> say not to enable MSI on x86, with a link to a very old (2008)
> >> discussion.

...

> The current situation looks like this:
>  - MSI[-X] usage for plain Linux drivers is fine, we are using it on
>    machines where you can hardly avoid MSI today
>  - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
>    you avoid enabling/disabling from RT context (also used for quite
>    a while here, no known issues under this restriction)

Okay, so I understand from this:

1. The wiki warning is out of date.

2. My own PCIe MSI driver/card *should* work.
   (It is not a current known limitation of adeos/xenomai.)

Thanks,

Richard

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Philippe Gerum]

On Thu, 2011-03-03 at 08:21 +0100, Richard Cochran wrote:
> On Wed, Mar 02, 2011 at 07:18:46PM +0100, Jan Kiszka wrote:
> > On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
> > > Richard Cochran wrote:
> > >> The wiki pages 
> > >>
> > >>    http://www.xenomai.org/index.php/FAQs
> > >>    http://www.xenomai.org/index.php/Configuring_x86_kernels
> > >>
> > >> say not to enable MSI on x86, with a link to a very old (2008)
> > >> discussion.
> 
> ...
> 
> > The current situation looks like this:
> >  - MSI[-X] usage for plain Linux drivers is fine, we are using it on
> >    machines where you can hardly avoid MSI today
> >  - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
> >    you avoid enabling/disabling from RT context (also used for quite
> >    a while here, no known issues under this restriction)
> 
> Okay, so I understand from this:
> 
> 1. The wiki warning is out of date.
> 
> 2. My own PCIe MSI driver/card *should* work.
>    (It is not a current known limitation of adeos/xenomai.)

The situation is not that clear. Enabling CONFIG_PCI_MSI on some 83xx
board did cause a lock up at boot for me, so this may point the finger
at the pipeline core for ppc, and not to Xenomai which was barely active
at that stage.

> 
> Thanks,
> 
> Richard
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help

-- 
Philippe.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Wolfgang Grandegger]

Hi Philippe,

On 03/03/2011 08:32 AM, Philippe Gerum wrote:
> On Thu, 2011-03-03 at 08:21 +0100, Richard Cochran wrote:
>> On Wed, Mar 02, 2011 at 07:18:46PM +0100, Jan Kiszka wrote:
>>> On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
>>>> Richard Cochran wrote:
>>>>> The wiki pages 
>>>>>
>>>>>    http://www.xenomai.org/index.php/FAQs
>>>>>    http://www.xenomai.org/index.php/Configuring_x86_kernels
>>>>>
>>>>> say not to enable MSI on x86, with a link to a very old (2008)
>>>>> discussion.
>>
>> ...
>>
>>> The current situation looks like this:
>>>  - MSI[-X] usage for plain Linux drivers is fine, we are using it on
>>>    machines where you can hardly avoid MSI today
>>>  - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
>>>    you avoid enabling/disabling from RT context (also used for quite
>>>    a while here, no known issues under this restriction)
>>
>> Okay, so I understand from this:
>>
>> 1. The wiki warning is out of date.
>>
>> 2. My own PCIe MSI driver/card *should* work.
>>    (It is not a current known limitation of adeos/xenomai.)
> 
> The situation is not that clear. Enabling CONFIG_PCI_MSI on some 83xx
> board did cause a lock up at boot for me, so this may point the finger
> at the pipeline core for ppc, and not to Xenomai which was barely active
> at that stage.

Now I remember vaguely that we have seen a similar issue on the mpc8308
in autumn last year. Here is what you wrote that time:

On 09/29/2010 04:12 PM, Philippe Gerum wrote:
> Ok, I have a good news: the issue is spotted; and a bad one: this is no
> trivial stuff to fix. In short, CONFIG_PCI_MSI causes the PIC code to
> read PCI config memory on the fly during operations (which is intensely
> insane but that is how it goes). Of course, it does that via the regular
> PCI helpers, which are stuffed with normal locking constructs we may
> _not_ call from any real-time/non-regular context.
> 
> To sum up, we need to call those helpers to ack/mask interrupts within
> the I-pipe, but we are not allowed to do this because this is inherently
> unsafe.
> 
> Work-around: disable CONFIG_PCI_MSI in your configuration.
> The real fix would require much more thought, to make the PCI helpers
> Adeos-aware without inducing high latency.

Wolfgang.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Jan Kiszka]

On 2011-03-03 13:25, Wolfgang Grandegger wrote:
> Hi Philippe,
> 
> On 03/03/2011 08:32 AM, Philippe Gerum wrote:
>> On Thu, 2011-03-03 at 08:21 +0100, Richard Cochran wrote:
>>> On Wed, Mar 02, 2011 at 07:18:46PM +0100, Jan Kiszka wrote:
>>>> On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
>>>>> Richard Cochran wrote:
>>>>>> The wiki pages 
>>>>>>
>>>>>>    http://www.xenomai.org/index.php/FAQs
>>>>>>    http://www.xenomai.org/index.php/Configuring_x86_kernels
>>>>>>
>>>>>> say not to enable MSI on x86, with a link to a very old (2008)
>>>>>> discussion.
>>>
>>> ...
>>>
>>>> The current situation looks like this:
>>>>  - MSI[-X] usage for plain Linux drivers is fine, we are using it on
>>>>    machines where you can hardly avoid MSI today
>>>>  - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
>>>>    you avoid enabling/disabling from RT context (also used for quite
>>>>    a while here, no known issues under this restriction)
>>>
>>> Okay, so I understand from this:
>>>
>>> 1. The wiki warning is out of date.
>>>
>>> 2. My own PCIe MSI driver/card *should* work.
>>>    (It is not a current known limitation of adeos/xenomai.)
>>
>> The situation is not that clear. Enabling CONFIG_PCI_MSI on some 83xx
>> board did cause a lock up at boot for me, so this may point the finger
>> at the pipeline core for ppc, and not to Xenomai which was barely active
>> at that stage.
> 
> Now I remember vaguely that we have seen a similar issue on the mpc8308
> in autumn last year. Here is what you wrote that time:
> 
> On 09/29/2010 04:12 PM, Philippe Gerum wrote:
>> Ok, I have a good news: the issue is spotted; and a bad one: this is no
>> trivial stuff to fix. In short, CONFIG_PCI_MSI causes the PIC code to
>> read PCI config memory on the fly during operations (which is intensely
>> insane but that is how it goes). Of course, it does that via the regular
>> PCI helpers, which are stuffed with normal locking constructs we may
>> _not_ call from any real-time/non-regular context.
>>
>> To sum up, we need to call those helpers to ack/mask interrupts within
>> the I-pipe, but we are not allowed to do this because this is inherently
>> unsafe.
>>
>> Work-around: disable CONFIG_PCI_MSI in your configuration.
>> The real fix would require much more thought, to make the PCI helpers
>> Adeos-aware without inducing high latency.

As MSIs are edge-triggered, only requiring ack at the interrupt
controller level, could it be that PPC is using the wrong flow handler here?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Philippe Gerum]

On Thu, 2011-03-03 at 13:30 +0100, Jan Kiszka wrote:
> On 2011-03-03 13:25, Wolfgang Grandegger wrote:
> > Hi Philippe,
> > 
> > On 03/03/2011 08:32 AM, Philippe Gerum wrote:
> >> On Thu, 2011-03-03 at 08:21 +0100, Richard Cochran wrote:
> >>> On Wed, Mar 02, 2011 at 07:18:46PM +0100, Jan Kiszka wrote:
> >>>> On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
> >>>>> Richard Cochran wrote:
> >>>>>> The wiki pages 
> >>>>>>
> >>>>>>    http://www.xenomai.org/index.php/FAQs
> >>>>>>    http://www.xenomai.org/index.php/Configuring_x86_kernels
> >>>>>>
> >>>>>> say not to enable MSI on x86, with a link to a very old (2008)
> >>>>>> discussion.
> >>>
> >>> ...
> >>>
> >>>> The current situation looks like this:
> >>>>  - MSI[-X] usage for plain Linux drivers is fine, we are using it on
> >>>>    machines where you can hardly avoid MSI today
> >>>>  - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
> >>>>    you avoid enabling/disabling from RT context (also used for quite
> >>>>    a while here, no known issues under this restriction)
> >>>
> >>> Okay, so I understand from this:
> >>>
> >>> 1. The wiki warning is out of date.
> >>>
> >>> 2. My own PCIe MSI driver/card *should* work.
> >>>    (It is not a current known limitation of adeos/xenomai.)
> >>
> >> The situation is not that clear. Enabling CONFIG_PCI_MSI on some 83xx
> >> board did cause a lock up at boot for me, so this may point the finger
> >> at the pipeline core for ppc, and not to Xenomai which was barely active
> >> at that stage.
> > 
> > Now I remember vaguely that we have seen a similar issue on the mpc8308
> > in autumn last year. Here is what you wrote that time:
> > 
> > On 09/29/2010 04:12 PM, Philippe Gerum wrote:
> >> Ok, I have a good news: the issue is spotted; and a bad one: this is no
> >> trivial stuff to fix. In short, CONFIG_PCI_MSI causes the PIC code to
> >> read PCI config memory on the fly during operations (which is intensely
> >> insane but that is how it goes). Of course, it does that via the regular
> >> PCI helpers, which are stuffed with normal locking constructs we may
> >> _not_ call from any real-time/non-regular context.
> >>
> >> To sum up, we need to call those helpers to ack/mask interrupts within
> >> the I-pipe, but we are not allowed to do this because this is inherently
> >> unsafe.
> >>
> >> Work-around: disable CONFIG_PCI_MSI in your configuration.
> >> The real fix would require much more thought, to make the PCI helpers
> >> Adeos-aware without inducing high latency.
> 
> As MSIs are edge-triggered, only requiring ack at the interrupt
> controller level, could it be that PPC is using the wrong flow handler here?

The pipeline forces mask/ack to prevent adverse effect of deferred
delivery to downstream handlers. Some chips require this.

> 
> Jan
> 

-- 
Philippe.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3186 bytes --]

On 2011-03-03 13:54, Philippe Gerum wrote:
> On Thu, 2011-03-03 at 13:30 +0100, Jan Kiszka wrote:
>> On 2011-03-03 13:25, Wolfgang Grandegger wrote:
>>> Hi Philippe,
>>>
>>> On 03/03/2011 08:32 AM, Philippe Gerum wrote:
>>>> On Thu, 2011-03-03 at 08:21 +0100, Richard Cochran wrote:
>>>>> On Wed, Mar 02, 2011 at 07:18:46PM +0100, Jan Kiszka wrote:
>>>>>> On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
>>>>>>> Richard Cochran wrote:
>>>>>>>> The wiki pages 
>>>>>>>>
>>>>>>>>    http://www.xenomai.org/index.php/FAQs
>>>>>>>>    http://www.xenomai.org/index.php/Configuring_x86_kernels
>>>>>>>>
>>>>>>>> say not to enable MSI on x86, with a link to a very old (2008)
>>>>>>>> discussion.
>>>>>
>>>>> ...
>>>>>
>>>>>> The current situation looks like this:
>>>>>>  - MSI[-X] usage for plain Linux drivers is fine, we are using it on
>>>>>>    machines where you can hardly avoid MSI today
>>>>>>  - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
>>>>>>    you avoid enabling/disabling from RT context (also used for quite
>>>>>>    a while here, no known issues under this restriction)
>>>>>
>>>>> Okay, so I understand from this:
>>>>>
>>>>> 1. The wiki warning is out of date.
>>>>>
>>>>> 2. My own PCIe MSI driver/card *should* work.
>>>>>    (It is not a current known limitation of adeos/xenomai.)
>>>>
>>>> The situation is not that clear. Enabling CONFIG_PCI_MSI on some 83xx
>>>> board did cause a lock up at boot for me, so this may point the finger
>>>> at the pipeline core for ppc, and not to Xenomai which was barely active
>>>> at that stage.
>>>
>>> Now I remember vaguely that we have seen a similar issue on the mpc8308
>>> in autumn last year. Here is what you wrote that time:
>>>
>>> On 09/29/2010 04:12 PM, Philippe Gerum wrote:
>>>> Ok, I have a good news: the issue is spotted; and a bad one: this is no
>>>> trivial stuff to fix. In short, CONFIG_PCI_MSI causes the PIC code to
>>>> read PCI config memory on the fly during operations (which is intensely
>>>> insane but that is how it goes). Of course, it does that via the regular
>>>> PCI helpers, which are stuffed with normal locking constructs we may
>>>> _not_ call from any real-time/non-regular context.
>>>>
>>>> To sum up, we need to call those helpers to ack/mask interrupts within
>>>> the I-pipe, but we are not allowed to do this because this is inherently
>>>> unsafe.
>>>>
>>>> Work-around: disable CONFIG_PCI_MSI in your configuration.
>>>> The real fix would require much more thought, to make the PCI helpers
>>>> Adeos-aware without inducing high latency.
>>
>> As MSIs are edge-triggered, only requiring ack at the interrupt
>> controller level, could it be that PPC is using the wrong flow handler here?
> 
> The pipeline forces mask/ack to prevent adverse effect of deferred
> delivery to downstream handlers. Some chips require this.

You mean PPC irqchips? I'm a bit confused as MSIs are immediately acked
by the hardware on the bus, independently of the processor delivering or
deferring the event. What prevents immediate termination (up to EOI if
required) of this IRQ type on PPC?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Philippe Gerum]

On Fri, 2011-03-04 at 00:45 +0100, Jan Kiszka wrote:
> On 2011-03-03 13:54, Philippe Gerum wrote:
> > On Thu, 2011-03-03 at 13:30 +0100, Jan Kiszka wrote:
> >> On 2011-03-03 13:25, Wolfgang Grandegger wrote:
> >>> Hi Philippe,
> >>>
> >>> On 03/03/2011 08:32 AM, Philippe Gerum wrote:
> >>>> On Thu, 2011-03-03 at 08:21 +0100, Richard Cochran wrote:
> >>>>> On Wed, Mar 02, 2011 at 07:18:46PM +0100, Jan Kiszka wrote:
> >>>>>> On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
> >>>>>>> Richard Cochran wrote:
> >>>>>>>> The wiki pages 
> >>>>>>>>
> >>>>>>>>    http://www.xenomai.org/index.php/FAQs
> >>>>>>>>    http://www.xenomai.org/index.php/Configuring_x86_kernels
> >>>>>>>>
> >>>>>>>> say not to enable MSI on x86, with a link to a very old (2008)
> >>>>>>>> discussion.
> >>>>>
> >>>>> ...
> >>>>>
> >>>>>> The current situation looks like this:
> >>>>>>  - MSI[-X] usage for plain Linux drivers is fine, we are using it on
> >>>>>>    machines where you can hardly avoid MSI today
> >>>>>>  - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
> >>>>>>    you avoid enabling/disabling from RT context (also used for quite
> >>>>>>    a while here, no known issues under this restriction)
> >>>>>
> >>>>> Okay, so I understand from this:
> >>>>>
> >>>>> 1. The wiki warning is out of date.
> >>>>>
> >>>>> 2. My own PCIe MSI driver/card *should* work.
> >>>>>    (It is not a current known limitation of adeos/xenomai.)
> >>>>
> >>>> The situation is not that clear. Enabling CONFIG_PCI_MSI on some 83xx
> >>>> board did cause a lock up at boot for me, so this may point the finger
> >>>> at the pipeline core for ppc, and not to Xenomai which was barely active
> >>>> at that stage.
> >>>
> >>> Now I remember vaguely that we have seen a similar issue on the mpc8308
> >>> in autumn last year. Here is what you wrote that time:
> >>>
> >>> On 09/29/2010 04:12 PM, Philippe Gerum wrote:
> >>>> Ok, I have a good news: the issue is spotted; and a bad one: this is no
> >>>> trivial stuff to fix. In short, CONFIG_PCI_MSI causes the PIC code to
> >>>> read PCI config memory on the fly during operations (which is intensely
> >>>> insane but that is how it goes). Of course, it does that via the regular
> >>>> PCI helpers, which are stuffed with normal locking constructs we may
> >>>> _not_ call from any real-time/non-regular context.
> >>>>
> >>>> To sum up, we need to call those helpers to ack/mask interrupts within
> >>>> the I-pipe, but we are not allowed to do this because this is inherently
> >>>> unsafe.
> >>>>
> >>>> Work-around: disable CONFIG_PCI_MSI in your configuration.
> >>>> The real fix would require much more thought, to make the PCI helpers
> >>>> Adeos-aware without inducing high latency.
> >>
> >> As MSIs are edge-triggered, only requiring ack at the interrupt
> >> controller level, could it be that PPC is using the wrong flow handler here?
> > 
> > The pipeline forces mask/ack to prevent adverse effect of deferred
> > delivery to downstream handlers. Some chips require this.
> 
> You mean PPC irqchips? I'm a bit confused as MSIs are immediately acked
> by the hardware on the bus, independently of the processor delivering or
> deferring the event. What prevents immediate termination (up to EOI if
> required) of this IRQ type on PPC?

It is not about msi chips in particular, it is about masking all
interrupt sources during their propagation in the pipeline being
generally used for all irqs, to prevent interrupt storm from
edge-triggered sources because of the incurred delay.

To sum up: as far as I'm concerned, I still have to address the issue of
dealing with MSI, because I never did so for any of the pipeline
releases, for any arch, I've rolled out in the past. My perception is
that MSIs are at odds with some assumptions/decisions of the ppc
pipeline, and this went unnoticed until people start using MSI-enabled
hardware there too. This was confirmed by experimenting on 83xx that
something was going wrong with CONFIG_PCI_MSI enabled.

I'm not saying this would be impossible to enable MSIs there, I'm just
saying that so far: nobody cared. Terrae incognitae.

> 
> Jan
> 

-- 
Philippe.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4873 bytes --]

On 2011-03-04 08:51, Philippe Gerum wrote:
> On Fri, 2011-03-04 at 00:45 +0100, Jan Kiszka wrote:
>> On 2011-03-03 13:54, Philippe Gerum wrote:
>>> On Thu, 2011-03-03 at 13:30 +0100, Jan Kiszka wrote:
>>>> On 2011-03-03 13:25, Wolfgang Grandegger wrote:
>>>>> Hi Philippe,
>>>>>
>>>>> On 03/03/2011 08:32 AM, Philippe Gerum wrote:
>>>>>> On Thu, 2011-03-03 at 08:21 +0100, Richard Cochran wrote:
>>>>>>> On Wed, Mar 02, 2011 at 07:18:46PM +0100, Jan Kiszka wrote:
>>>>>>>> On 2011-03-02 16:26, Gilles Chanteperdrix wrote:
>>>>>>>>> Richard Cochran wrote:
>>>>>>>>>> The wiki pages 
>>>>>>>>>>
>>>>>>>>>>    http://www.xenomai.org/index.php/FAQs
>>>>>>>>>>    http://www.xenomai.org/index.php/Configuring_x86_kernels
>>>>>>>>>>
>>>>>>>>>> say not to enable MSI on x86, with a link to a very old (2008)
>>>>>>>>>> discussion.
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>>> The current situation looks like this:
>>>>>>>>  - MSI[-X] usage for plain Linux drivers is fine, we are using it on
>>>>>>>>    machines where you can hardly avoid MSI today
>>>>>>>>  - MSI[-X] usage for RTDM (ie. RT) drivers is basically fine as long as
>>>>>>>>    you avoid enabling/disabling from RT context (also used for quite
>>>>>>>>    a while here, no known issues under this restriction)
>>>>>>>
>>>>>>> Okay, so I understand from this:
>>>>>>>
>>>>>>> 1. The wiki warning is out of date.
>>>>>>>
>>>>>>> 2. My own PCIe MSI driver/card *should* work.
>>>>>>>    (It is not a current known limitation of adeos/xenomai.)
>>>>>>
>>>>>> The situation is not that clear. Enabling CONFIG_PCI_MSI on some 83xx
>>>>>> board did cause a lock up at boot for me, so this may point the finger
>>>>>> at the pipeline core for ppc, and not to Xenomai which was barely active
>>>>>> at that stage.
>>>>>
>>>>> Now I remember vaguely that we have seen a similar issue on the mpc8308
>>>>> in autumn last year. Here is what you wrote that time:
>>>>>
>>>>> On 09/29/2010 04:12 PM, Philippe Gerum wrote:
>>>>>> Ok, I have a good news: the issue is spotted; and a bad one: this is no
>>>>>> trivial stuff to fix. In short, CONFIG_PCI_MSI causes the PIC code to
>>>>>> read PCI config memory on the fly during operations (which is intensely
>>>>>> insane but that is how it goes). Of course, it does that via the regular
>>>>>> PCI helpers, which are stuffed with normal locking constructs we may
>>>>>> _not_ call from any real-time/non-regular context.
>>>>>>
>>>>>> To sum up, we need to call those helpers to ack/mask interrupts within
>>>>>> the I-pipe, but we are not allowed to do this because this is inherently
>>>>>> unsafe.
>>>>>>
>>>>>> Work-around: disable CONFIG_PCI_MSI in your configuration.
>>>>>> The real fix would require much more thought, to make the PCI helpers
>>>>>> Adeos-aware without inducing high latency.
>>>>
>>>> As MSIs are edge-triggered, only requiring ack at the interrupt
>>>> controller level, could it be that PPC is using the wrong flow handler here?
>>>
>>> The pipeline forces mask/ack to prevent adverse effect of deferred
>>> delivery to downstream handlers. Some chips require this.
>>
>> You mean PPC irqchips? I'm a bit confused as MSIs are immediately acked
>> by the hardware on the bus, independently of the processor delivering or
>> deferring the event. What prevents immediate termination (up to EOI if
>> required) of this IRQ type on PPC?
> 
> It is not about msi chips in particular, it is about masking all
> interrupt sources during their propagation in the pipeline being
> generally used for all irqs, to prevent interrupt storm from
> edge-triggered sources because of the incurred delay.

At least for MSI, such storms would be very abnormal. The spec states
that devices can't assume anything about multiple IRQ messages sent
before the driver communicated with the device after the first one. So
they /should/ refrain from sending those. Of course, there can always be
exceptions (guess we've all seen buggy firmware...).

> 
> To sum up: as far as I'm concerned, I still have to address the issue of
> dealing with MSI, because I never did so for any of the pipeline
> releases, for any arch, I've rolled out in the past. My perception is
> that MSIs are at odds with some assumptions/decisions of the ppc
> pipeline, and this went unnoticed until people start using MSI-enabled
> hardware there too. This was confirmed by experimenting on 83xx that
> something was going wrong with CONFIG_PCI_MSI enabled.
> 
> I'm not saying this would be impossible to enable MSIs there, I'm just
> saying that so far: nobody cared. Terrae incognitae.

I'm not disagreeing. I'm just betting that MSI will quickly gain
relevance there as well - up to the point where first devices no longer
include with INTx support (perfectly legal, but I haven't seen one yet).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Richard Cochran]

On Fri, Mar 04, 2011 at 08:51:00AM +0100, Philippe Gerum wrote:
> 
> I'm not saying this would be impossible to enable MSIs there, I'm just
> saying that so far: nobody cared. Terrae incognitae.

I care!

When the first MSI interrupt from my PCIe card should arrive, running
on a P2020RBD, the machine locks up. My BDI shows me that both cores
are going round in circles.

* core 1
  __ipipe_check_percpu_access ()  at kernel/ipipe/core.c:2067
  ipipe_suspend_domain ()         at kernel/ipipe/core.c:826
  __ipipe_check_percpu_access ()  at arch/powerpc/include/asm/ipipe_hwirq.h:57
  ipipe_suspend_domain ()         at kernel/ipipe/core.c:860
  cpu_idle ()                     at arch/powerpc/kernel/idle.c:64
  ipipe_suspend_domain ()         at arch/powerpc/include/asm/ipipe_hwirq.h:57

* core 2
  ipipe_suspend_domain ()         at kernel/ipipe/core.c:858
  cpu_idle ()                     at arch/powerpc/kernel/idle.c:64
  ipipe_suspend_domain ()         at arch/powerpc/include/asm/ipipe_hwirq.h:57

Any thoughts?

Thanks,

Richard

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

On 2011-03-04 09:11, Richard Cochran wrote:
> On Fri, Mar 04, 2011 at 08:51:00AM +0100, Philippe Gerum wrote:
>>
>> I'm not saying this would be impossible to enable MSIs there, I'm just
>> saying that so far: nobody cared. Terrae incognitae.
> 
> I care!
> 
> When the first MSI interrupt from my PCIe card should arrive, running
> on a P2020RBD, the machine locks up. My BDI shows me that both cores
> are going round in circles.
> 
> * core 1
>   __ipipe_check_percpu_access ()  at kernel/ipipe/core.c:2067
>   ipipe_suspend_domain ()         at kernel/ipipe/core.c:826
>   __ipipe_check_percpu_access ()  at arch/powerpc/include/asm/ipipe_hwirq.h:57
>   ipipe_suspend_domain ()         at kernel/ipipe/core.c:860
>   cpu_idle ()                     at arch/powerpc/kernel/idle.c:64
>   ipipe_suspend_domain ()         at arch/powerpc/include/asm/ipipe_hwirq.h:57
> 
> * core 2
>   ipipe_suspend_domain ()         at kernel/ipipe/core.c:858
>   cpu_idle ()                     at arch/powerpc/kernel/idle.c:64
>   ipipe_suspend_domain ()         at arch/powerpc/include/asm/ipipe_hwirq.h:57
> 
> Any thoughts?

Well, if PPC masks MSIs on ack, you may run on a bug check or on an
already acquired Linux spin lock.

Independent of the need to add RT-safe mask/unmask support for MSIs
midterm, you likely first of all want to get rid of this masking in the
fast-path. I suspect it's not needed, also on PPC.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Richard Cochran]

On Fri, Mar 04, 2011 at 09:09:57AM +0100, Jan Kiszka wrote:
> 
> I'm not disagreeing. I'm just betting that MSI will quickly gain
> relevance there as well - up to the point where first devices no longer
> include with INTx support (perfectly legal, but I haven't seen one yet).

IFAICT, PCIe cards are allowed, not required to offer INTx.

In my case, I already asked HW if they could enable INTx in the
design. The IP core would offer this, but there seems not to be
possible in combination with the particular SoC core attached.

So, for now, it looks like I will have to find a way to support MSI...

Richard

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

On 2011-03-04 09:30, Richard Cochran wrote:
> On Fri, Mar 04, 2011 at 09:09:57AM +0100, Jan Kiszka wrote:
>>
>> I'm not disagreeing. I'm just betting that MSI will quickly gain
>> relevance there as well - up to the point where first devices no longer
>> include with INTx support (perfectly legal, but I haven't seen one yet).
> 
> IFAICT, PCIe cards are allowed, not required to offer INTx.

Yep.

> 
> In my case, I already asked HW if they could enable INTx in the
> design. The IP core would offer this, but there seems not to be
> possible in combination with the particular SoC core attached.
> 
> So, for now, it looks like I will have to find a way to support MSI...

I see, you found that first example. :)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Richard Cochran]

On Fri, Mar 04, 2011 at 08:51:00AM +0100, Philippe Gerum wrote:
> I'm not saying this would be impossible to enable MSIs there, I'm just
> saying that so far: nobody cared. Terrae incognitae.

Philippe,

I read adeos-ipipe-2.6.36-powerpc-2.12-02.patch in order to
re-acquaint myself with adeos. While I do not pretend to really
understand it, I did notice an incongruity in the interrupt handling
for my particular board.

The function, fsl_msi_cascade, in arch/powerpc/sysdev/fsl_msi.c:185
has in the non-ipipe version two different actions

	desc->chip->unmask(irq);
	desc->chip->eoi(irq);

depending on whether the controller is an ipic or mpic. The ipipe
code unconditionally executes ipic/unmask.

I noticed that the function 

	cpm_cascade() in arch/powerpc/platforms/8xx/m8xx_setup.c

does call eoi(), so I guessed that the eoi() in fsl_msi_cascade()
might have been overlooked. Putting the eoi() into the ipipe version
lets the machine handle the interrupts, and my test programs will even
run with the PCIe card and driver.

However, I am really making a grab in the dark, and I see strange
warnings like unexpected (in_irq() || irqs_disabled()) from
kernel/softirq.c:143 and so on.

Also, more interrupts appear for my card in /proc/interrupts than
expected.

Is this any kind of hint, or am I just completely lost?

Thanks,

Richard

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Philippe Gerum]

On Mon, 2011-03-07 at 10:26 +0100, Richard Cochran wrote:
> On Fri, Mar 04, 2011 at 08:51:00AM +0100, Philippe Gerum wrote:
> > I'm not saying this would be impossible to enable MSIs there, I'm just
> > saying that so far: nobody cared. Terrae incognitae.
> 
> Philippe,
> 
> I read adeos-ipipe-2.6.36-powerpc-2.12-02.patch in order to
> re-acquaint myself with adeos. While I do not pretend to really
> understand it, I did notice an incongruity in the interrupt handling
> for my particular board.

Yes, that code is broken (untested actually).

> 
> The function, fsl_msi_cascade, in arch/powerpc/sysdev/fsl_msi.c:185
> has in the non-ipipe version two different actions
> 
> 	desc->chip->unmask(irq);
> 	desc->chip->eoi(irq);
> 
> depending on whether the controller is an ipic or mpic. The ipipe
> code unconditionally executes ipic/unmask.
> 
> I noticed that the function 
> 
> 	cpm_cascade() in arch/powerpc/platforms/8xx/m8xx_setup.c
> 
> does call eoi(), so I guessed that the eoi() in fsl_msi_cascade()
> might have been overlooked. Putting the eoi() into the ipipe version
> lets the machine handle the interrupts, and my test programs will even
> run with the PCIe card and driver.
> 
> However, I am really making a grab in the dark, and I see strange
> warnings like unexpected (in_irq() || irqs_disabled()) from
> kernel/softirq.c:143 and so on.

Possibly because the decoded cascade IRQ should be routed to the
I-pipe's low level handler, not through generic_handle_irq(). It
probably re-enters the linux domain context in the middle of nowhere.

> 
> Also, more interrupts appear for my card in /proc/interrupts than
> expected.
> 
> Is this any kind of hint, or am I just completely lost?
> 

No, your logic is right. The fsl_msi handler is utterly wrong there.
Please try the following patch (not even compiled, but should be mostly
ok -- famous last words). You should scrap all I-pipe changes on
arch/powerpc/sysdev/fsl_msi.c before applying it, it is based on the
vanilla 2.6.36 code.

A bit of context:

The pipeline installs all chained IRQ handlers as primary ack handlers
(fired from __ipipe_handle_irq()), so we run fsl_msi_handler() from the
lowest I-pipe core code, for all incoming msi (multiplex) IRQs. When the
pipeline is enabled, that routine is expected to mask[+eoi] the
multiplex IRQ, decode the cascaded one(s), tell the pipeline about that
IRQ, then unmask the multiplex IRQ.

Basically, IRQs routed through the mpic are fasteoi ones, so we should
run the chip's ->eoi() handler for the multiplexed IRQ in such a case,
to do mask+eoi (I-pipe specific behavior to lower the interrupt priority
early). ipic-routed IRQs are either level/edge ones, but we don't care
since there won't be any deferral in acknowledging the cascaded IRQ, so
hopefully, the interrupt source should not freak out badly. Of course,
it might decide to freak out in case there is a deferral in running the
cascaded IRQ handler, but so far, it has never been an issue.

Eventually, in both cases (mpic/ipic), we should unmask the IRQ once we
propagated the cascaded one to the I-pipe primary handler via
ipipe_handle_chained_irq().

diff --git a/arch/powerpc/sysdev/fsl_msi.c b/arch/powerpc/sysdev/fsl_msi.c
index 87991d3..af106ee 100644
--- a/arch/powerpc/sysdev/fsl_msi.c
+++ b/arch/powerpc/sysdev/fsl_msi.c
@@ -182,6 +182,63 @@ out_free:
 	return rc;
 }
 
+#ifdef CONFIG_IPIPE
+
+static void fsl_msi_cascade(unsigned int irq, struct irq_desc *desc)
+{
+	unsigned int cascade_irq;
+	struct fsl_msi *msi_data;
+	int msir_index = -1;
+	u32 msir_value = 0;
+	u32 intr_index;
+	u32 have_shift = 0;
+	struct fsl_msi_cascade_data *cascade_data;
+
+	cascade_data = (struct fsl_msi_cascade_data *)get_irq_data(irq);
+	msi_data = cascade_data->msi_data;
+
+	if ((msi_data->feature &  FSL_PIC_IP_MASK) == FSL_PIC_IP_IPIC) {
+		if (desc->chip->mask_ack)
+			desc->chip->mask_ack(irq);
+		else {
+			desc->chip->mask(irq);
+			desc->chip->ack(irq);
+		}
+	} else
+		desc->chip->eoi(irq);
+
+	msir_index = cascade_data->index;
+
+	if (msir_index >= NR_MSI_REG)
+		cascade_irq = NO_IRQ;
+
+	switch (msi_data->feature & FSL_PIC_IP_MASK) {
+	case FSL_PIC_IP_MPIC:
+		msir_value = fsl_msi_read(msi_data->msi_regs,
+			msir_index * 0x10);
+		break;
+	case FSL_PIC_IP_IPIC:
+		msir_value = fsl_msi_read(msi_data->msi_regs, msir_index * 0x4);
+		break;
+	}
+
+	while (msir_value) {
+		intr_index = ffs(msir_value) - 1;
+
+		cascade_irq = irq_linear_revmap(msi_data->irqhost,
+				msir_index * IRQS_PER_MSI_REG +
+					intr_index + have_shift);
+		if (cascade_irq != NO_IRQ)
+			ipipe_handle_chained_irq(cascade_irq);
+		have_shift += intr_index + 1;
+		msir_value = msir_value >> (intr_index + 1);
+	}
+
+	desc->chip->unmask(irq);
+}
+
+#else /* !CONFIG_IPIPE */
+
 static void fsl_msi_cascade(unsigned int irq, struct irq_desc *desc)
 {
 	unsigned int cascade_irq;
@@ -250,6 +307,8 @@ unlock:
 	raw_spin_unlock(&desc->lock);
 }
 
+#endif /* !CONFIG_IPIPE */
+
 static int fsl_of_msi_remove(struct platform_device *ofdev)
 {
 	struct fsl_msi *msi = ofdev->dev.platform_data;

> Thanks,
> 
> Richard
> 

-- 
Philippe, cursed with IRQs.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Richard Cochran]

On Mon, Mar 07, 2011 at 05:37:17PM +0100, Philippe Gerum wrote:
> No, your logic is right. The fsl_msi handler is utterly wrong there.
> Please try the following patch (not even compiled, but should be mostly
> ok -- famous last words). You should scrap all I-pipe changes on
> arch/powerpc/sysdev/fsl_msi.c before applying it, it is based on the
> vanilla 2.6.36 code.

Yes! It works!

The only strange thing is that many more interrupts appear in
/proc/interrupts than expected, but that is way better than freezing
the box. I'll be testing this box a lot more in the coming days.

Thanks so much for the patch and explanation!

Richard

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> +static void fsl_msi_cascade(unsigned int irq, struct irq_desc *desc)
> +{
> (...)
> +	if ((msi_data->feature &  FSL_PIC_IP_MASK) == FSL_PIC_IP_IPIC) {
> +		if (desc->chip->mask_ack)
> +			desc->chip->mask_ack(irq);
> +		else {
> +			desc->chip->mask(irq);
> +			desc->chip->ack(irq);
> +		}
> +	} else
> +		desc->chip->eoi(irq);
> +
> (...)
> +
> +	desc->chip->unmask(irq);

Do we have to unmask here if we just eoi'ed on entry?

-- 
					    Gilles.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Philippe Gerum]

On Mon, 2011-03-07 at 19:39 +0100, Gilles Chanteperdrix wrote:
> Philippe Gerum wrote:
> > +static void fsl_msi_cascade(unsigned int irq, struct irq_desc *desc)
> > +{
> > (...)
> > +	if ((msi_data->feature &  FSL_PIC_IP_MASK) == FSL_PIC_IP_IPIC) {
> > +		if (desc->chip->mask_ack)
> > +			desc->chip->mask_ack(irq);
> > +		else {
> > +			desc->chip->mask(irq);
> > +			desc->chip->ack(irq);
> > +		}
> > +	} else
> > +		desc->chip->eoi(irq);
> > +
> > (...)
> > +
> > +	desc->chip->unmask(irq);
> 
> Do we have to unmask here if we just eoi'ed on entry?
> 

eoi is actually mask+eoi in the mpic case.

-- 
Philippe.

Re: [Xenomai-help] Stuck MSI in normal Linux driver

[Richard Cochran]

On Mon, Mar 07, 2011 at 07:33:48PM +0100, Richard Cochran wrote:
> The only strange thing is that many more interrupts appear in
> /proc/interrupts than expected,

Actually, I lied.

I forgot how about a second interrupt source on this card. Every event
might raise two interrupts, but if they occur close together in time,
Linux sees only one in /proc/interrupts.

So, Philippe, it is working perfectly.

Thanks again,

Richard