RE: 2.4.19 hang (was unlink perf pb !)

RE: 2.4.19 hang (was unlink perf pb !)

[Brasseur Valéry]

try a stock 2.4.19 without nfs patches : no hang !


seems that nfs patches broke something !

valery

> -----Original Message-----
> From: Lever, Charles [mailto:Charles.Lever@netapp.com]
> Sent: Tuesday, August 13, 2002 8:42 PM
> To: 'Brasseur Val=E9ry'
> Cc: nfs@lists.sourceforge.net
> Subject: RE: [NFS] 2.4.19 hang (was unlink perf pb !)
>=20
>=20
> if your client works OK with 2.4.18 but stops with your 2.4.19+nfs
> kernels, can you try hunting down exactly which kernel causes the
> problem?  does 2.4.19 without any nfs patches work, for example?
>=20
> > after some more investigation on my D state hang here is some=20
> > observations :
> >=20
> > * I couldn't get a clear trace of state : sysrq-t was not=20
> > dump on a file !
> > don't know why !
> >   so I hand wrote extract of what I see and let ksymsoops try !
> > =20
> >    got this : processes was in kmap or in flush_tbl !
> >=20
> > * now after a clean boot, lauchn my prog : after a few second=20
> > the LA grow,
> > blocked processes grow too ! (in my vmstat all the processes=20
> > are in block
> > queue !) and only 1 waiting process !
> >=20
> >   when in this state, I can't do  many things as all what I=20
> > can do seems to
> > hang too !
> >=20
> > =3D> I was not having this probleme in 2.4.18 !
> >=20
> > note : when this arrive I've got something like 10 to 20 Mbs=20
> > to nfs server.
> >=20
> >=20
> > valery
> >=20
> >=20
> >=20
> > -------------------------------------------------------
> > This sf.net email is sponsored by: Dice - The leading=20
> online job board
> > for high-tech professionals. Search and apply for tech jobs today!
> > http://seeker.dice.com/seeker.epl?rel_code=3D31
> > _______________________________________________
> > NFS maillist  -  NFS@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfs
> >=20
>=20


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 hang (was unlink perf pb !)

[Trond Myklebust]

>>>>> " " =3D=3D Brasseur Val=E9ry <Valery.Brasseur@atosorigin.com> writes:

     > try a stock 2.4.19 without nfs patches : no hang !  seems that
     > nfs patches broke something !

Can you get me a trace of the hang using the command

   echo "1" > /proc/sys/sunrpc/rpc_debug

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: 2.4.19 hang (was unlink perf pb !)

[Brasseur Valéry]

more on my problem ...

* it seems that when using tcp mount the hang appears more often and =
more
quickly !
* on stock 2.4.19 which I have is a limited perf but it works
* on 2.4.19 + nfs patches I have limited perf + hang !

I said I have limited perf because the NetApp filer I use as server is =
a 880
which is given for more than 15000 nfs/ops  and with my Linux box I =
can't do
more than 2/3000 ops when I am the only client !

how can I track this down ?

valery

> -----Original Message-----
> From: Brasseur Val=E9ry [mailto:Valery.Brasseur@atosorigin.com]
> Sent: Wednesday, August 14, 2002 6:22 PM
> To: 'Trond Myklebust'
> Cc: 'Lever, Charles'; nfs@lists.sourceforge.net
> Subject: RE: [NFS] 2.4.19 hang (was unlink perf pb !)
>=20
>=20
>=20
> well I try ! but for now :
> * all I have is a console message saying "RPC: waiting for=20
> request slot"
> * when I turn debug on : system slow to a crawl ! load=20
> average goes to more
> than 700 ! but it works
>          never crash !
> * when I turn debug of : crash with lot's of process blocked ! =
running
> process seems to be kswapd !
>=20
> I'v tried to but debug just when system near crash, but then=20
> it slow down,
> and get back to "life" slowly.
>=20
> more : I have some difficulties to reproduce what i was=20
> having yesterday !
>=20
> valery
>=20
> > -----Original Message-----
> > From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no]
> > Sent: Wednesday, August 14, 2002 2:53 PM
> > To: Brasseur Val=E9ry
> > Cc: 'Lever, Charles'; nfs@lists.sourceforge.net
> > Subject: Re: [NFS] 2.4.19 hang (was unlink perf pb !)
> >=20
> >=20
> > >>>>> " " =3D=3D Brasseur Val=E9ry=20
> <Valery.Brasseur@atosorigin.com> writes:
> >=20
> >      > try a stock 2.4.19 without nfs patches : no hang ! =20
> seems that
> >      > nfs patches broke something !
> >=20
> > Can you get me a trace of the hang using the command
> >=20
> >    echo "1" > /proc/sys/sunrpc/rpc_debug
> >=20
> > Cheers,
> >   Trond
> >=20
>=20
>=20
> -------------------------------------------------------
> This sf.net email is sponsored by: Dice - The leading online job =
board
> for high-tech professionals. Search and apply for tech jobs today!
> http://seeker.dice.com/seeker.epl?rel_code1
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
>=20


-------------------------------------------------------
This sf.net email is sponsored by: OSDN - Tired of that same old
cell phone?  Get a new here for FREE!
https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: 2.4.19 hang (was unlink perf pb !)

[Trond Myklebust]

>>>>> " " == Brasseur Valéry <Valery.Brasseur@atosorigin.com> writes:

     > more on my problem ...
     > * it seems that when using tcp mount the hang appears more
     >   often and more
     > quickly !
     > * on stock 2.4.19 which I have is a limited perf but it works
     > * on 2.4.19 + nfs patches I have limited perf + hang !

     > I said I have limited perf because the NetApp filer I use as
     > server is a 880 which is given for more than 15000 nfs/ops and
     > with my Linux box I can't do more than 2/3000 ops when I am the
     > only client !

     > how can I track this down ?

Unless you are using some form of NAT (a.k.a. masquerading) it sounds
rather like you have some serious networking problems. Certainly I
have never heard of anybody having problems with hangs on both UDP
*and* TCP mounts - the latter in particular was designed to work
around network congestion problems.

Have you therefore tried things like swapping out the networking card
for another (preferably of a different make and different Linux
driver), and ditto for any switches/HUBs?

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by: OSDN - Tired of that same old
cell phone?  Get a new here for FREE!
https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: 2.4.19 hang (was unlink perf pb !)

[Brasseur Valéry]

well I try ! but for now :
* all I have is a console message saying "RPC: waiting for request =
slot"
* when I turn debug on : system slow to a crawl ! load average goes to =
more
than 700 ! but it works
         never crash !
* when I turn debug of : crash with lot's of process blocked ! running
process seems to be kswapd !

I'v tried to but debug just when system near crash, but then it slow =
down,
and get back to "life" slowly.

more : I have some difficulties to reproduce what i was having =
yesterday !

valery

> -----Original Message-----
> From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no]
> Sent: Wednesday, August 14, 2002 2:53 PM
> To: Brasseur Val=E9ry
> Cc: 'Lever, Charles'; nfs@lists.sourceforge.net
> Subject: Re: [NFS] 2.4.19 hang (was unlink perf pb !)
>=20
>=20
> >>>>> " " =3D=3D Brasseur Val=E9ry <Valery.Brasseur@atosorigin.com> =
writes:
>=20
>      > try a stock 2.4.19 without nfs patches : no hang !  seems that
>      > nfs patches broke something !
>=20
> Can you get me a trace of the hang using the command
>=20
>    echo "1" > /proc/sys/sunrpc/rpc_debug
>=20
> Cheers,
>   Trond
>=20


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: 2.4.19 hang (was unlink perf pb !)

[Lever, Charles]

if your client works OK with 2.4.18 but stops with your 2.4.19+nfs
kernels, can you try hunting down exactly which kernel causes the
problem?  does 2.4.19 without any nfs patches work, for example?

> after some more investigation on my D state hang here is some 
> observations :
> 
> * I couldn't get a clear trace of state : sysrq-t was not 
> dump on a file !
> don't know why !
>   so I hand wrote extract of what I see and let ksymsoops try !
>  
>    got this : processes was in kmap or in flush_tbl !
> 
> * now after a clean boot, lauchn my prog : after a few second 
> the LA grow,
> blocked processes grow too ! (in my vmstat all the processes 
> are in block
> queue !) and only 1 waiting process !
> 
>   when in this state, I can't do  many things as all what I 
> can do seems to
> hang too !
> 
> => I was not having this probleme in 2.4.18 !
> 
> note : when this arrive I've got something like 10 to 20 Mbs 
> to nfs server.
> 
> 
> valery
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by: Dice - The leading online job board
> for high-tech professionals. Search and apply for tech jobs today!
> http://seeker.dice.com/seeker.epl?rel_code=31
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> 


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

2.4.19 hang (was unlink perf pb !)

[Brasseur Valéry]

after some more investigation on my D state hang here is some observations :

* I couldn't get a clear trace of state : sysrq-t was not dump on a file !
don't know why !
  so I hand wrote extract of what I see and let ksymsoops try !
 
   got this : processes was in kmap or in flush_tbl !

* now after a clean boot, lauchn my prog : after a few second the LA grow,
blocked processes grow too ! (in my vmstat all the processes are in block
queue !) and only 1 waiting process !

  when in this state, I can't do  many things as all what I can do seems to
hang too !

=> I was not having this probleme in 2.4.18 !

note : when this arrive I've got something like 10 to 20 Mbs to nfs server.


valery



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs