Kernel freezes on "exportfs -a"

Kernel freezes on "exportfs -a"

[Marian Eichholz]

Hello all, hi Neil, hi Olaf,

we recently found a strange bug in the recent 2.6 kernels
that freezes the whole kernel (down to even no response to
the Magic SysRq keys). And in fact sticks us to the 2.4
series we want to get rid of :)

This happens during "exportfs -a" when the following
sequence commences (tested by manual command execution,
with some seconds betweeen the commands):

	rpc.nfsd 128
	rpc.mountd
	rpc.statd
	exportfs -a

We cannot reproduce it on a "normal" machine.

The most recently tested kernel is a 2.6.9-rc1 (since each
test breaks the production service). exportfs is 1.0.6.

The particular server is the NFS file repository for a high
volume FTP cluster, thus beeing requested by the still
online 4 ftp backend servers *heavily*.

It has 2 Fast Ethernet ports in *bonding* mode (in peak hours they can
be filled with ease).

Probably it is this combinations that makes a problem.

There are only two directories being exported to 5 machines
each with really standard options.

The freeze can be reproduced even when the exported
directories are empty (no mount of the *some* GB to be
checked afterwards, nice for testing :)) and even with "strace
exportfs -a".

strace stops during the call of

	nfsservctl(0x1,...,0)

right after reading the still empty /proc/fs/nfs/exports.

I just hope that something helps.
If we can run a special test or provide suitable
information about the system, please let me know.

Thank You in advance, probably it's something easily fixed.

Yours sincerely,
- Marian


-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Kernel freezes on "exportfs -a"

[Neil Brown]

On Tuesday November 16, marian.eichholz@freenet-ag.de wrote:
> Hello all, hi Neil, hi Olaf,
> 
> we recently found a strange bug in the recent 2.6 kernels
> that freezes the whole kernel (down to even no response to
> the Magic SysRq keys). And in fact sticks us to the 2.4
> series we want to get rid of :)
> 
> This happens during "exportfs -a" when the following
> sequence commences (tested by manual command execution,
> with some seconds betweeen the commands):
> 
> 	rpc.nfsd 128
> 	rpc.mountd
> 	rpc.statd
> 	exportfs -a

It doesn't address your core problem, but this order is wrong.
The "exportfs -a" should be done *before* rpc.nfsd, otherwise there is
a window where clients could be told that the filesystem they want
isn't exported.

> 
> We cannot reproduce it on a "normal" machine.

Just how abnormal is the machine where you *can* export it?

> 
> The most recently tested kernel is a 2.6.9-rc1 (since each
> test breaks the production service). exportfs is 1.0.6.
> 
> The particular server is the NFS file repository for a high
> volume FTP cluster, thus beeing requested by the still
> online 4 ftp backend servers *heavily*.
> 
> It has 2 Fast Ethernet ports in *bonding* mode (in peak hours they can
> be filled with ease).
> 
> Probably it is this combinations that makes a problem.

So are you saying that the clients are hammering on the server while
you are trying to bring up NFS service?  If this is the case, then
maybe just moving the "exportfs -a" to the top will help.

NeilBrown


-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Kernel freezes on "exportfs -a"

[Marian Eichholz]

On Wed, Nov 17, 2004 at 01:42:28PM +1100, Neil Brown wrote:

> > 	rpc.nfsd 128
> > 	rpc.mountd
> > 	rpc.statd
> > 	exportfs -a
> 
> It doesn't address your core problem, but this order is wrong.
> The "exportfs -a" should be done *before* rpc.nfsd, otherwise there is
> a window where clients could be told that the filesystem they want
> isn't exported.

This makes sense. Too-many-generations-of-this-script problem.

> > We cannot reproduce it on a "normal" machine.
> 
> Just how abnormal is the machine where you *can* export it?

Smaller file systems (far much smaller), single processor,
completely different hardware, no clients and so on.

Although the size of the exported FS is of no interest,
because exportfs froze even when I did not mount it (to
save time on reboot-fsck).

> > The most recently tested kernel is a 2.6.9-rc1 (since each
> > test breaks the production service). exportfs is 1.0.6.

Interesting result: 2.6.10-rc2 appears to work perfectly.
Some racing condition anywhere in the kernel
infrastructure?

> > The particular server is the NFS file repository for a high
> > volume FTP cluster, thus beeing requested by the still
> > online 4 ftp backend servers *heavily*.
> > 
> > It has 2 Fast Ethernet ports in *bonding* mode (in peak hours they can
> > be filled with ease).
> > 
> > Probably it is this combinations that makes a problem.
> 
> So are you saying that the clients are hammering on the server while
> you are trying to bring up NFS service?  If this is the case, then
> maybe just moving the "exportfs -a" to the top will

I had no chance to go deep into detail, like Olaf mentioned
with the backtrace-NMI suggestion, because I had to bring
the machine back into production ASAP.

> 
> NeilBrown
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: InterSystems CACHE
> FREE OODBMS DOWNLOAD - A multidimensional database that combines
> robust object and relational technologies, making it a perfect match
> for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs

-- 
Mit freundlichen Gruessen / Yours sincerely

Marian Eichholz
Postmaster
freenet.de AG          Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
Deelbögenkamp 4c       Vorstand: Eckhard Spoerr (Vors.), Axel Krieger,
                                 Stephan Esch, Eric Berger
22297 Hamburg          Amtsgericht Hamburg, HRB 74048


-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Kernel freezes on "exportfs -a"

[Olaf Kirch]

On Tue, Nov 16, 2004 at 12:22:37PM +0100, Marian Eichholz wrote:
> Hello all, hi Neil, hi Olaf,
> 
> we recently found a strange bug in the recent 2.6 kernels
> that freezes the whole kernel (down to even no response to
> the Magic SysRq keys). And in fact sticks us to the 2.4
> series we want to get rid of :)

Did you try enabling the NMI watchdog so you get a backtrace
of where it hangs?

Olaf
-- 
Olaf Kirch     | Things that make Monday morning interesting, #1:
okir@suse.de   |        "I want to use NFS over AX25, can you help me?"
---------------+ 


-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs