FW: SKB corruption on heavy traffic

FW: SKB corruption on heavy traffic

[Franca, Jose (NSN - PT/Portugal - MiniMD)]

>From our latest debugs we found that the problem occurs mainly on skbuff =
code. After some variable time kfree or kalloc result in kernel oops.

-----Original Message-----
From: Franca, Jose (NSN - PT/Portugal - MiniMD)=20
Sent: quarta-feira, 30 de Abril de 2008 9:44
To: 'ext Scott Wood'
Cc: =09
Subject: RE: SKB corruption on heavy traffic

Hello!

	Thank you for replying!
	It't quite dificult to say if the problem exists without our changes, =
since the all software is dependent on this changes so to work with the =
hardware. I can't answer to that right now on that, but I forgot to add =
one thing: we have ring buffer full problems on our fcc_enet driver from =
time to time. So, I think the problem could be on linux configurations =
(related to hw) because there is a lot of posts on the web related to =
problems similar to this (none of them has really solved the bottom =
problem).=20

Regards,
Filipe=20

-----Original Message-----
From: ext Scott Wood [mailto:scottwood@freescale.com]=20
Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
To: Franca, Jose (NSN - PT/Portugal - MiniMD)
Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
Subject: Re: SKB corruption on heavy traffic

On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal - MiniMD) wrote:
> 	We are developing a MPC8247 based telecom board (512MB), using
> linux 2.4 with some proprietary changes on IP stack and we are facing
> some problems when we have heavy traffic on our Ethernet interfaces...

Do you see these problems without the proprietary changes, and with a
current kernel?

-Scott

Re: FW: SKB corruption on heavy traffic

[Gerhard Pircher]

Hi,

I think I have the same problem here with all versions of the 2.6.x kernel
series (tested with kernel v2.6.8/14/16/18/25 on a PPC7455 machine with
different PCI network cards by transferring a big file over NFS/SCP). Data
corruption occurs under high load, but I don't get any kernel oops.

regards,

Gerhard

-------- Original-Nachricht --------
> Datum: Wed, 30 Apr 2008 10:07:15 +0100
> Von: "Franca, Jose (NSN - PT/Portugal - MiniMD)" <jose.franca@nsn.com>
> An: linuxppc-dev@ozlabs.org, linuxppc-embedded@ozlabs.org
> Betreff: FW: SKB corruption on heavy traffic

> >From our latest debugs we found that the problem occurs mainly on skbuff
> code. After some variable time kfree or kalloc result in kernel oops.
> 
> -----Original Message-----
> From: Franca, Jose (NSN - PT/Portugal - MiniMD) 
> Sent: quarta-feira, 30 de Abril de 2008 9:44
> To: 'ext Scott Wood'
> Cc: 	
> Subject: RE: SKB corruption on heavy traffic
> 
> Hello!
> 
> 	Thank you for replying!
> 	It't quite dificult to say if the problem exists without our
> changes, since the all software is dependent on this changes so to work
> with the hardware. I can't answer to that right now on that, but I forgot 
> to add one thing: we have ring buffer full problems on our fcc_enet
> driver from time to time. So, I think the problem could be on linux
> configurations (related to hw) because there is a lot of posts on the web 
> related to problems similar to this (none of them has really solved the
> bottom problem). 
> 
> Regards,
> Filipe 
> 
> -----Original Message-----
> From: ext Scott Wood [mailto:scottwood@freescale.com] 
> Sent: terça-feira, 29 de Abril de 2008 20:15
> To: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
> Subject: Re: SKB corruption on heavy traffic
> 
> On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - PT/Portugal
> - MiniMD) wrote:
> > 	We are developing a MPC8247 based telecom board (512MB), using
> > linux 2.4 with some proprietary changes on IP stack and we are facing
> > some problems when we have heavy traffic on our Ethernet interfaces...
> 
> Do you see these problems without the proprietary changes, and with a
> current kernel?
> 
> -Scott


-- 
Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games! 
http://games.entertainment.gmx.net/de/entertainment/games/free

RE: FW: SKB corruption on heavy traffic

[Franca, Jose (NSN - PT/Portugal - MiniMD)]

Hi!

	There was a sugestion to change slab to slub alocation method... I =
don't know quite well yet what is necessary to do this, but it seems =
that the current implementation of slub is more commonly available on =
2.6 kernels, not in 2.4 that I use :(...
	Any guesses or hints on this?


Regards!
Filipe.

-----Original Message-----
From: ext Gerhard Pircher [mailto:gerhard_pircher@gmx.net]=20
Sent: quarta-feira, 30 de Abril de 2008 13:26
To: Franca, Jose (NSN - PT/Portugal - MiniMD); Scott Wood
Cc: linuxppc-embedded@ozlabs.org; linuxppc-dev@ozlabs.org
Subject: Re: FW: SKB corruption on heavy traffic

Hi,

I think I have the same problem here with all versions of the 2.6.x =
kernel
series (tested with kernel v2.6.8/14/16/18/25 on a PPC7455 machine with
different PCI network cards by transferring a big file over NFS/SCP). =
Data
corruption occurs under high load, but I don't get any kernel oops.

regards,

Gerhard

-------- Original-Nachricht --------
> Datum: Wed, 30 Apr 2008 10:07:15 +0100
> Von: "Franca, Jose (NSN - PT/Portugal - MiniMD)" <jose.franca@nsn.com>
> An: linuxppc-dev@ozlabs.org, linuxppc-embedded@ozlabs.org
> Betreff: FW: SKB corruption on heavy traffic

> >From our latest debugs we found that the problem occurs mainly on =
skbuff
> code. After some variable time kfree or kalloc result in kernel oops.
>=20
> -----Original Message-----
> From: Franca, Jose (NSN - PT/Portugal - MiniMD)=20
> Sent: quarta-feira, 30 de Abril de 2008 9:44
> To: 'ext Scott Wood'
> Cc: =09
> Subject: RE: SKB corruption on heavy traffic
>=20
> Hello!
>=20
> 	Thank you for replying!
> 	It't quite dificult to say if the problem exists without our
> changes, since the all software is dependent on this changes so to =
work
> with the hardware. I can't answer to that right now on that, but I =
forgot=20
> to add one thing: we have ring buffer full problems on our fcc_enet
> driver from time to time. So, I think the problem could be on linux
> configurations (related to hw) because there is a lot of posts on the =
web=20
> related to problems similar to this (none of them has really solved =
the
> bottom problem).=20
>=20
> Regards,
> Filipe=20
>=20
> -----Original Message-----
> From: ext Scott Wood [mailto:scottwood@freescale.com]=20
> Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
> To: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
> Subject: Re: SKB corruption on heavy traffic
>=20
> On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal
> - MiniMD) wrote:
> > 	We are developing a MPC8247 based telecom board (512MB), using
> > linux 2.4 with some proprietary changes on IP stack and we are =
facing
> > some problems when we have heavy traffic on our Ethernet =
interfaces...
>=20
> Do you see these problems without the proprietary changes, and with a
> current kernel?
>=20
> -Scott


--=20
Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games! =

http://games.entertainment.gmx.net/de/entertainment/games/free

Re: RE: FW: SKB corruption on heavy traffic

[Gerhard Pircher]

-------- Original-Nachricht --------
> Datum: Wed, 30 Apr 2008 14:03:02 +0100
> Von: "Franca, Jose (NSN - PT/Portugal - MiniMD)" <jose.franca@nsn.com>
> An: "ext Gerhard Pircher" <gerhard_pircher@gmx.net>, "Scott Wood" <scottwood@freescale.com>
> CC: linuxppc-embedded@ozlabs.org, linuxppc-dev@ozlabs.org
> Betreff: RE: FW: SKB corruption on heavy traffic

> Hi!
> 
> 	There was a sugestion to change slab to slub alocation method... I > don't know quite well yet what is necessary to do this, but it seems that
> the current implementation of slub is more commonly available on 2.6
> kernels, not in 2.4 that I use :(...
> 	Any guesses or hints on this?
I'm using SLUB with kernel v2.6.25 and I still get data corruption on high
load. On the other side it is a good idea to enable SLAB debugging, as
Scott suggested. Maybe that sheds some light on this issue (even if most
of the network drivers make use of DMA).

regards,

Gerhard

> 
> Regards!
> Filipe.
> 
> -----Original Message-----
> From: ext Gerhard Pircher [mailto:gerhard_pircher@gmx.net] 
> Sent: quarta-feira, 30 de Abril de 2008 13:26
> To: Franca, Jose (NSN - PT/Portugal - MiniMD); Scott Wood
> Cc: linuxppc-embedded@ozlabs.org; linuxppc-dev@ozlabs.org
> Subject: Re: FW: SKB corruption on heavy traffic
> 
> Hi,
> 
> I think I have the same problem here with all versions of the 2.6.x
> kernel series (tested with kernel v2.6.8/14/16/18/25 on a PPC7455 machine 
> with different PCI network cards by transferring a big file over
> NFS/SCP). Data corruption occurs under high load, but I don't get any
> kernel oops.
> 
> regards,
> 
> Gerhard

-- 
Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten 
Browser-Versionen downloaden: http://www.gmx.net/de/go/browser

RE: SKB corruption on heavy traffic

[Myron.Dixon]

We have experience a very similar problem using a 2.4.18 kernel on an =
8260 ppc processor.
We have a telecomunication product that for some time only used the fec =
for TCP/IP ethernet=20
traffic only and worked just fine.

After we upgraded our product to implement TDM data over IP we started =
to notice an occasional
kernel oops.  We began to evaluate all of our products and determined =
that only some of the units
exhibted this behaviour at various rates of occurrence.  Further =
evaluation revealed that the pointers
located in dpram pointing to the fec's buffer descriptors were some how =
getting corrupted.
Note that the 8260 has 4 internal scc/fccs and we use all four for =
various aspects of our application
and each shares dpram for pointers to buffer descriptors that reside in =
sdram.  However, only the
fec that is used for IP experiences this buffer descriptor corruption =
and, then, apparently, only under=20
heavy traffic load.  We spent about six months evaluating this problem =
including contacting freescale,=20
but never found a solution.  We finally, decided to use an external =
ethernet chip on a daughter card=20
for our IP channel.

It is, however, our belief that our problem relates to a possible bug in =
the 8260 CPM, but have yet to
absolutely prove this.

If we are experiencing the same problem (and potentially others) and =
there is a solution we would be
very interested, as, we are not happy about the daughter card solution.  =



Myron L. Dixon
Sr. Software Engineer
L-3 Communications, GNS
1519 Grundy's Lane
Bristol, PA 19007
Phone:  215 957 3739
Fax:  215 957 3790
email:  Myron.Dixon@L-3Com.com

-----Original Message-----
From: linuxppc-dev-bounces+myron.dixon=3Dl-3com.com@ozlabs.org =
[mailto:linuxppc-dev-bounces+myron.dixon=3Dl-3com.com@ozlabs.org] On =
Behalf Of Franca, Jose (NSN - PT/Portugal - MiniMD)
Sent: Wednesday, April 30, 2008 5:07 AM
To: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
Subject: FW: SKB corruption on heavy traffic

>From our latest debugs we found that the problem occurs mainly on skbuff =
code. After some variable time kfree or kalloc result in kernel oops.

-----Original Message-----
From: Franca, Jose (NSN - PT/Portugal - MiniMD)
Sent: quarta-feira, 30 de Abril de 2008 9:44
To: 'ext Scott Wood'
Cc: =09
Subject: RE: SKB corruption on heavy traffic

Hello!

	Thank you for replying!
	It't quite dificult to say if the problem exists without our changes, =
since the all software is dependent on this changes so to work with the =
hardware. I can't answer to that right now on that, but I forgot to add =
one thing: we have ring buffer full problems on our fcc_enet driver from =
time to time. So, I think the problem could be on linux configurations =
(related to hw) because there is a lot of posts on the web related to =
problems similar to this (none of them has really solved the bottom =
problem).=20

Regards,
Filipe=20

-----Original Message-----
From: ext Scott Wood [mailto:scottwood@freescale.com]
Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
To: Franca, Jose (NSN - PT/Portugal - MiniMD)
Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
Subject: Re: SKB corruption on heavy traffic

On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal - MiniMD) wrote:
> 	We are developing a MPC8247 based telecom board (512MB), using linux=20
> 2.4 with some proprietary changes on IP stack and we are facing some=20
> problems when we have heavy traffic on our Ethernet interfaces...

Do you see these problems without the proprietary changes, and with a =
current kernel?

-Scott
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-dev

Re: SKB corruption on heavy traffic

[Matvejchikov Ilya]

[-- Attachment #1: Type: text/plain, Size: 4998 bytes --]

Hi all!

The same problem! I've tried 2.4.xx and 2.6.xx kernels. Nothing changed!
BUT. After many days of fucking with fs_enet driver I've found a
stable (as I see) solution. The bugs I've had:
 - kernel oopses
 - SKB data corruption
 - BDs status corruption
 - SKB ring full message
 - too many RX errors
 - may be something else :)

For now I have a 2.4.35 fs_enet driver that works on heavy load
24/7... I don't know what happens with my 8260 board, but with this
code it be very stable. I supposed that there are some errors in 8260
CPM core, but errata don't know about it :)

I've append an attachment with my 2.4.35 kernel patch. Sorry for a big
file and not for only fs_enet file. Moreover I've used CPM111 errata
microcode and NAPI in fs_enet driver.

If you have any questions I'm glad to hear it.

2008/5/2  <Myron.Dixon@l-3com.com>:
> We have experience a very similar problem using a 2.4.18 kernel on an 8260 ppc processor.
>  We have a telecomunication product that for some time only used the fec for TCP/IP ethernet
>  traffic only and worked just fine.
>
>  After we upgraded our product to implement TDM data over IP we started to notice an occasional
>  kernel oops.  We began to evaluate all of our products and determined that only some of the units
>  exhibted this behaviour at various rates of occurrence.  Further evaluation revealed that the pointers
>  located in dpram pointing to the fec's buffer descriptors were some how getting corrupted.
>  Note that the 8260 has 4 internal scc/fccs and we use all four for various aspects of our application
>  and each shares dpram for pointers to buffer descriptors that reside in sdram.  However, only the
>  fec that is used for IP experiences this buffer descriptor corruption and, then, apparently, only under
>  heavy traffic load.  We spent about six months evaluating this problem including contacting freescale,
>  but never found a solution.  We finally, decided to use an external ethernet chip on a daughter card
>  for our IP channel.
>
>  It is, however, our belief that our problem relates to a possible bug in the 8260 CPM, but have yet to
>  absolutely prove this.
>
>  If we are experiencing the same problem (and potentially others) and there is a solution we would be
>  very interested, as, we are not happy about the daughter card solution.
>
>
>  Myron L. Dixon
>  Sr. Software Engineer
>  L-3 Communications, GNS
>  1519 Grundy's Lane
>  Bristol, PA 19007
>  Phone:  215 957 3739
>  Fax:  215 957 3790
>  email:  Myron.Dixon@L-3Com.com
>
>  -----Original Message-----
>  From: linuxppc-dev-bounces+myron.dixon=l-3com.com@ozlabs.org [mailto:linuxppc-dev-bounces+myron.dixon=l-3com.com@ozlabs.org] On Behalf Of Franca, Jose (NSN - PT/Portugal - MiniMD)
>  Sent: Wednesday, April 30, 2008 5:07 AM
>  To: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
>  Subject: FW: SKB corruption on heavy traffic
>
>  From our latest debugs we found that the problem occurs mainly on skbuff code. After some variable time kfree or kalloc result in kernel oops.
>
>  -----Original Message-----
>  From: Franca, Jose (NSN - PT/Portugal - MiniMD)
>  Sent: quarta-feira, 30 de Abril de 2008 9:44
>  To: 'ext Scott Wood'
>  Cc:
>  Subject: RE: SKB corruption on heavy traffic
>
>  Hello!
>
>         Thank you for replying!
>         It't quite dificult to say if the problem exists without our changes, since the all software is dependent on this changes so to work with the hardware. I can't answer to that right now on that, but I forgot to add one thing: we have ring buffer full problems on our fcc_enet driver from time to time. So, I think the problem could be on linux configurations (related to hw) because there is a lot of posts on the web related to problems similar to this (none of them has really solved the bottom problem).
>
>  Regards,
>  Filipe
>
>  -----Original Message-----
>  From: ext Scott Wood [mailto:scottwood@freescale.com]
>  Sent: terça-feira, 29 de Abril de 2008 20:15
>  To: Franca, Jose (NSN - PT/Portugal - MiniMD)
>  Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
>  Subject: Re: SKB corruption on heavy traffic
>
>  On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - PT/Portugal - MiniMD) wrote:
>  >       We are developing a MPC8247 based telecom board (512MB), using linux
>  > 2.4 with some proprietary changes on IP stack and we are facing some
>  > problems when we have heavy traffic on our Ethernet interfaces...
>
>  Do you see these problems without the proprietary changes, and with a current kernel?
>
>  -Scott
>  _______________________________________________
>  Linuxppc-dev mailing list
>  Linuxppc-dev@ozlabs.org
>  https://ozlabs.org/mailman/listinfo/linuxppc-dev
>  _______________________________________________
>  Linuxppc-embedded mailing list
>  Linuxppc-embedded@ozlabs.org
>  https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>

[-- Attachment #2: linux-2.4.35-m82-2.patch.gz --]
[-- Type: application/x-gzip, Size: 21860 bytes --]

RE: SKB corruption on heavy traffic

[Franca, Jose (NSN - PT/Portugal - MiniMD)]

Hi!

  Thanks for your contribution! We will try it right away! :))
  As for patches, we also installed some patches provided by freescale =
and nothing so far... Still the same problems.
  I will post the results as soon as I can...

Thanks U ALL!
Filipe.

-----Original Message-----
From: ext Matvejchikov Ilya [mailto:matvejchikov@gmail.com]=20
Sent: s=E1bado, 3 de Maio de 2008 15:08
To: Myron.Dixon@l-3com.com
Cc: Franca, Jose (NSN - PT/Portugal - MiniMD); linuxppc-dev@ozlabs.org; =
linuxppc-embedded@ozlabs.org
Subject: Re: SKB corruption on heavy traffic

Hi all!

The same problem! I've tried 2.4.xx and 2.6.xx kernels. Nothing changed!
BUT. After many days of fucking with fs_enet driver I've found a
stable (as I see) solution. The bugs I've had:
 - kernel oopses
 - SKB data corruption
 - BDs status corruption
 - SKB ring full message
 - too many RX errors
 - may be something else :)

For now I have a 2.4.35 fs_enet driver that works on heavy load
24/7... I don't know what happens with my 8260 board, but with this
code it be very stable. I supposed that there are some errors in 8260
CPM core, but errata don't know about it :)

I've append an attachment with my 2.4.35 kernel patch. Sorry for a big
file and not for only fs_enet file. Moreover I've used CPM111 errata
microcode and NAPI in fs_enet driver.

If you have any questions I'm glad to hear it.

2008/5/2  <Myron.Dixon@l-3com.com>:
> We have experience a very similar problem using a 2.4.18 kernel on an =
8260 ppc processor.
>  We have a telecomunication product that for some time only used the =
fec for TCP/IP ethernet
>  traffic only and worked just fine.
>
>  After we upgraded our product to implement TDM data over IP we =
started to notice an occasional
>  kernel oops.  We began to evaluate all of our products and determined =
that only some of the units
>  exhibted this behaviour at various rates of occurrence.  Further =
evaluation revealed that the pointers
>  located in dpram pointing to the fec's buffer descriptors were some =
how getting corrupted.
>  Note that the 8260 has 4 internal scc/fccs and we use all four for =
various aspects of our application
>  and each shares dpram for pointers to buffer descriptors that reside =
in sdram.  However, only the
>  fec that is used for IP experiences this buffer descriptor corruption =
and, then, apparently, only under
>  heavy traffic load.  We spent about six months evaluating this =
problem including contacting freescale,
>  but never found a solution.  We finally, decided to use an external =
ethernet chip on a daughter card
>  for our IP channel.
>
>  It is, however, our belief that our problem relates to a possible bug =
in the 8260 CPM, but have yet to
>  absolutely prove this.
>
>  If we are experiencing the same problem (and potentially others) and =
there is a solution we would be
>  very interested, as, we are not happy about the daughter card =
solution.
>
>
>  Myron L. Dixon
>  Sr. Software Engineer
>  L-3 Communications, GNS
>  1519 Grundy's Lane
>  Bristol, PA 19007
>  Phone:  215 957 3739
>  Fax:  215 957 3790
>  email:  Myron.Dixon@L-3Com.com
>
>  -----Original Message-----
>  From: linuxppc-dev-bounces+myron.dixon=3Dl-3com.com@ozlabs.org =
[mailto:linuxppc-dev-bounces+myron.dixon=3Dl-3com.com@ozlabs.org] On =
Behalf Of Franca, Jose (NSN - PT/Portugal - MiniMD)
>  Sent: Wednesday, April 30, 2008 5:07 AM
>  To: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
>  Subject: FW: SKB corruption on heavy traffic
>
>  From our latest debugs we found that the problem occurs mainly on =
skbuff code. After some variable time kfree or kalloc result in kernel =
oops.
>
>  -----Original Message-----
>  From: Franca, Jose (NSN - PT/Portugal - MiniMD)
>  Sent: quarta-feira, 30 de Abril de 2008 9:44
>  To: 'ext Scott Wood'
>  Cc:
>  Subject: RE: SKB corruption on heavy traffic
>
>  Hello!
>
>         Thank you for replying!
>         It't quite dificult to say if the problem exists without our =
changes, since the all software is dependent on this changes so to work =
with the hardware. I can't answer to that right now on that, but I =
forgot to add one thing: we have ring buffer full problems on our =
fcc_enet driver from time to time. So, I think the problem could be on =
linux configurations (related to hw) because there is a lot of posts on =
the web related to problems similar to this (none of them has really =
solved the bottom problem).
>
>  Regards,
>  Filipe
>
>  -----Original Message-----
>  From: ext Scott Wood [mailto:scottwood@freescale.com]
>  Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
>  To: Franca, Jose (NSN - PT/Portugal - MiniMD)
>  Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
>  Subject: Re: SKB corruption on heavy traffic
>
>  On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal - MiniMD) wrote:
>  >       We are developing a MPC8247 based telecom board (512MB), =
using linux
>  > 2.4 with some proprietary changes on IP stack and we are facing =
some
>  > problems when we have heavy traffic on our Ethernet interfaces...
>
>  Do you see these problems without the proprietary changes, and with a =
current kernel?
>
>  -Scott
>  _______________________________________________
>  Linuxppc-dev mailing list
>  Linuxppc-dev@ozlabs.org
>  https://ozlabs.org/mailman/listinfo/linuxppc-dev
>  _______________________________________________
>  Linuxppc-embedded mailing list
>  Linuxppc-embedded@ozlabs.org
>  https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>