[Xenomai-help] network stall

[Xenomai-help] network stall

[roderik.wildenburg]

On one of our platforms (PPC 5200B Kernel 2.4, Xenomai 2.4.8, PCI-bus) we face rare network stalls (once a day).
In this situation network communication seems to be completly dead, but obviously only the receive direction is affected as receive-interrupts in /proc/interrupts aren´t incremented while transmit interrupts are incremented.
Wireshark-protocoll (taken via port mirroring of the switch) says that packets are sent to the target, but tcpdump on the target does not show these packets (which is obvious, if receive-interrupt isn´t operational).
In the send direction only arp-requests are monitored (on the target and on the switch).
Unfortunatelly this situation can´t be reproduced in an laboratory environment

On our other platform, which is very similar (PPC 5200B Kernel 2.4, Xenomai 2.4.8) but without PCI we don´t have this problem. Therefore I am thinking whether Ipipe (in combination with PCI) could be responsible for the interrupt lock?

Does anybody have an idea/suggestion how we can track down the reason for interrupt blockade?
Are there some helpful /proc entries (/proc/ipipe wasn´t very useful for me)?
Unfortunatelly, as far as I know, Ipipe-tacer is only available for Kernel 2.6. Am I right?


I would appreciate any suggestion very much!

Roderik

--------------------------------------------------------
manroland AG
Vorsitzender des Aufsichtsrates: Hanno C. Fiedler
Vorstand: Gerd Finkbeiner (Vorsitzender), Dr. Ingo Koch, Dr. Markus Rall, Paul Steidle   
Sitz der Gesellschaft: Offenbach am Main, Registergericht: Amtsgericht Offenbach HRB-Nr. 42592
USt-Ident-Nr. DE 250200933

Re: [Xenomai-help] network stall

[Gilles Chanteperdrix]

roderik.wildenburg@domain.hid wrote:
> On one of our platforms (PPC 5200B Kernel 2.4, Xenomai 2.4.8,
> PCI-bus) we face rare network stalls (once a day). In this situation
> network communication seems to be completly dead, but obviously only
> the receive direction is affected as receive-interrupts in
> /proc/interrupts aren´t incremented while transmit interrupts are
> incremented. Wireshark-protocoll (taken via port mirroring of the
> switch) says that packets are sent to the target, but tcpdump on the
> target does not show these packets (which is obvious, if
> receive-interrupt isn´t operational). In the send direction only
> arp-requests are monitored (on the target and on the switch). 
> Unfortunatelly this situation can´t be reproduced in an laboratory
> environment
> 
> On our other platform, which is very similar (PPC 5200B Kernel 2.4,
> Xenomai 2.4.8) but without PCI we don´t have this problem. Therefore
> I am thinking whether Ipipe (in combination with PCI) could be
> responsible for the interrupt lock?
> 
> Does anybody have an idea/suggestion how we can track down the reason
> for interrupt blockade? Are there some helpful /proc entries
> (/proc/ipipe wasn´t very useful for me)? Unfortunatelly, as far as I
> know, Ipipe-tacer is only available for Kernel 2.6. Am I right?
> 
> 
> I would appreciate any suggestion very much!

Hi,

there used to be a bug in the Adeos patches for Linux 2.6, which made
the kernel forget to run the softirqs sometimes. If this happens at the
right time with an ethernet driver using the NAPI, this could result in
a network stall with the RX interrupt no longer increasing. That is
because with NAPI, the network driver sometimes stops the RX interrupts
and does polling in the softirqs.

Well, that is just an hypothesis, it could be something different (such
as the I-pipe suddenly loosing interrupts from one source). But to
investigate, you should try and find whether when this happens, the
interrupt generation is disabled at ethernet card level, at interrupt
controller level, or not disabled at all. This would in narrowing down
the issue.

Regards.

-- 
                                          Gilles

Re: [Xenomai-help] network stall

[Wolfgang Grandegger]

Gilles Chanteperdrix wrote:
> roderik.wildenburg@domain.hid wrote:
>> On one of our platforms (PPC 5200B Kernel 2.4, Xenomai 2.4.8,
>> PCI-bus) we face rare network stalls (once a day). In this situation
>> network communication seems to be completly dead, but obviously only
>> the receive direction is affected as receive-interrupts in
>> /proc/interrupts aren´t incremented while transmit interrupts are
>> incremented. Wireshark-protocoll (taken via port mirroring of the
>> switch) says that packets are sent to the target, but tcpdump on the
>> target does not show these packets (which is obvious, if
>> receive-interrupt isn´t operational). In the send direction only
>> arp-requests are monitored (on the target and on the switch). 
>> Unfortunatelly this situation can´t be reproduced in an laboratory
>> environment

Are there any FEC related kernel messages? Maybe (likely) the problem is
caused by Ethernet packet bombardments (I suspect bursts of ARP
request). You could try to simulate such bombardments by using pktgen. I
can send you a script in case you are interested.

>> On our other platform, which is very similar (PPC 5200B Kernel 2.4,
>> Xenomai 2.4.8) but without PCI we don´t have this problem. Therefore
>> I am thinking whether Ipipe (in combination with PCI) could be
>> responsible for the interrupt lock?
>>
>> Does anybody have an idea/suggestion how we can track down the reason
>> for interrupt blockade? Are there some helpful /proc entries
>> (/proc/ipipe wasn´t very useful for me)? Unfortunatelly, as far as I
>> know, Ipipe-tacer is only available for Kernel 2.6. Am I right?
>>
>>
>> I would appreciate any suggestion very much!
> 
> Hi,
> 
> there used to be a bug in the Adeos patches for Linux 2.6, which made
> the kernel forget to run the softirqs sometimes. If this happens at the
> right time with an ethernet driver using the NAPI, this could result in
> a network stall with the RX interrupt no longer increasing. That is
> because with NAPI, the network driver sometimes stops the RX interrupts
> and does polling in the softirqs.

The standard FEC Ethernet driver does not use NAPI.

Wolfgang.

Re: [Xenomai-help] network stall

[roderik.wildenburg]

> 
> Are there any FEC related kernel messages? Maybe (likely) the 
> problem is

We will check this.


> caused by Ethernet packet bombardments (I suspect bursts of ARP
> request). You could try to simulate such bombardments by 
> using pktgen. I
> can send you a script in case you are interested.

Sure I am interested! Would you be so kind to send me the script?

Roderik

--------------------------------------------------------
manroland AG
Vorsitzender des Aufsichtsrates: Hanno C. Fiedler
Vorstand: Gerd Finkbeiner (Vorsitzender), Dr. Ingo Koch, Dr. Markus Rall, Paul Steidle   
Sitz der Gesellschaft: Offenbach am Main, Registergericht: Amtsgericht Offenbach HRB-Nr. 42592
USt-Ident-Nr. DE 250200933

Re: [Xenomai-help] network stall

[Wolfgang Grandegger]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

Hallo,

roderik.wildenburg@domain.hid wrote:
>> Are there any FEC related kernel messages? Maybe (likely) the 
>> problem is
> 
> We will check this.
> 
> 
>> caused by Ethernet packet bombardments (I suspect bursts of ARP
>> request). You could try to simulate such bombardments by 
>> using pktgen. I
>> can send you a script in case you are interested.
> 
> Sure I am interested! Would you be so kind to send me the script?

See attachment. Please adapt the MAC address of the target and you may
need a "modprobe pktgen" before running the script. For further info,
please have a look to "Documentation/networking/pktgen.txt"

Wolfgang.

[-- Attachment #2: paketgenerator.sh --]
[-- Type: application/x-shellscript, Size: 1205 bytes --]

Re: [Xenomai-help] network stall

[Philippe Gerum]

On Tue, 2009-11-17 at 10:42 +0100, roderik.wildenburg@domain.hid
wrote:
> On one of our platforms (PPC 5200B Kernel 2.4, Xenomai 2.4.8, PCI-bus) we face rare network stalls (once a day).
> In this situation network communication seems to be completly dead, but obviously only the receive direction is affected as receive-interrupts in /proc/interrupts aren´t incremented while transmit interrupts are incremented.
> Wireshark-protocoll (taken via port mirroring of the switch) says that packets are sent to the target, but tcpdump on the target does not show these packets (which is obvious, if receive-interrupt isn´t operational).
> In the send direction only arp-requests are monitored (on the target and on the switch).
> Unfortunatelly this situation can´t be reproduced in an laboratory environment
> 
> On our other platform, which is very similar (PPC 5200B Kernel 2.4, Xenomai 2.4.8) but without PCI we don´t have this problem. Therefore I am thinking whether Ipipe (in combination with PCI) could be responsible for the interrupt lock?
> 
> Does anybody have an idea/suggestion how we can track down the reason for interrupt blockade?
> Are there some helpful /proc entries (/proc/ipipe wasn´t very useful for me)?
> Unfortunatelly, as far as I know, Ipipe-tacer is only available for Kernel 2.6. Am I right?

Yes.

> 
> 
> I would appreciate any suggestion very much!
> 

Make sure to disable CONFIG_XENO_OPT_ISHIELD. This option was reported
to introduce interrupt propagation issues at least once, certainly adds
some overhead, and won't buy you anything with your configuration. It
has been removed from 2.5.

> Roderik
> 
> --------------------------------------------------------
> manroland AG
> Vorsitzender des Aufsichtsrates: Hanno C. Fiedler
> Vorstand: Gerd Finkbeiner (Vorsitzender), Dr. Ingo Koch, Dr. Markus Rall, Paul Steidle   
> Sitz der Gesellschaft: Offenbach am Main, Registergericht: Amtsgericht Offenbach HRB-Nr. 42592
> USt-Ident-Nr. DE 250200933
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help


-- 
Philippe.