PATCH [NFSd] NFSv3/TCP

PATCH [NFSd] NFSv3/TCP

[Oliver Tennert]

Hi Neil (and others),

is there any reason why current 2.4 kernels do not allow for
NFSSVC_MAXBLKSIZE to become as large as 32k?

The problem is when I use NFSv3/TCP with a 2.4.25, say, on the server
side, as well as on the client side, I am experiencing lockups when
copying medium-sized or large files from the NFS fs to a local fs.

But, after applying the following trivial patch, everything is well again:

--- linux-2.4.26-1/include/linux/nfsd/const.h.ORG       Mon May  3
16:20:18 2004
+++ linux-2.4.26-1/include/linux/nfsd/const.h   Mon May  3 16:20:32 2004
@@ -19,9 +19,9 @@
 #define NFSSVC_MAXVERS         3

 /*
- * Maximum blocksize supported by daemon currently at 8K
+ * Maximum blocksize supported by daemon currently at 32K
  */
-#define NFSSVC_MAXBLKSIZE      (8*1024)
+#define NFSSVC_MAXBLKSIZE      (32*1024)

 #ifdef __KERNEL__

32k is the maximum block size allowed by the NFS3 standard, as far as I
understand.

I realize that, before 2.4.19, this patch was part of the NFSv3/TCP server
patches, before this functionality got into the vanilla kernel. But with
the merge, this patch got lost! Is there any reason?

Best regards

Oliver

__
________________________________________creating IT solutions

Dr. Oliver Tennert			science + computing ag
phone   +49(0)7071 9457-598		Hagellocher Weg 71-75
fax     +49(0)7071 9457-411		D-72070 Tuebingen, Germany
O.Tennert@science-computing.de		www.science-computing.de

Re: PATCH [NFSd] NFSv3/TCP

[Neil Brown]

On Friday May 7, tennert@science-computing.de wrote:
> Hi Neil (and others),
> 
> is there any reason why current 2.4 kernels do not allow for
> NFSSVC_MAXBLKSIZE to become as large as 32k?

Yes.
At server thread creation, you need to be able to
   kmalloc(NFSSVC_MAXBLKSIZE+1024)
(or close to that) once per thread.  On most architectures this is a
high-order alloc_pages and can often fail.
Also, on every UDP write request, the server needs to 'kmalloc' a buffer
to hold the whole request (actually on every request that is
fragmented, but write is most common).  On most architectures, this
kmalloc will again require allocative several contiguous pages, which
can fail.

So this patch is only safe if you have a large-patch arch or only use
NFS over TCP.

There was once a patch floating around which allowed a larger
NFSSVC_MAXBLKSIZE on architectures with large page sizes, but it never
got properly submitted I think.


> 
> The problem is when I use NFSv3/TCP with a 2.4.25, say, on the server
> side, as well as on the client side, I am experiencing lockups when
> copying medium-sized or large files from the NFS fs to a local fs.

There must be some other cause.  Increasing the NFSSVC_MAXBLKSIZE is
just hiding a real problem I suspect.

NeilBrown

Re: PATCH [NFSd] NFSv3/TCP

[Oliver Tennert]

On Fri, 7 May 2004, Neil Brown wrote:

> On Friday May 7, tennert@science-computing.de wrote:
> > Hi Neil (and others),
> >
> > is there any reason why current 2.4 kernels do not allow for
> > NFSSVC_MAXBLKSIZE to become as large as 32k?
>
> Yes.
> At server thread creation, you need to be able to
>    kmalloc(NFSSVC_MAXBLKSIZE+1024)
> (or close to that) once per thread.  On most architectures this is a
> high-order alloc_pages and can often fail.
> Also, on every UDP write request, the server needs to 'kmalloc' a buffer
> to hold the whole request (actually on every request that is
> fragmented, but write is most common).  On most architectures, this
> kmalloc will again require allocative several contiguous pages, which
> can fail.
>
> So this patch is only safe if you have a large-patch arch or only use
> NFS over TCP.

So isn't it unsafe either with the NFSSVC_MAXBLKSIZE set to the original
8k? Or is it just less unsafe, then?

Anyway, I assure you the lockups occurred exactly when
server-side NFSv3/TCP got into the vanilla kernel at 2.4.19, and the only
significant difference I could see was this. I always used NFSv3/TCP
before then, and had no problems whatsoever. After that, I had to switch
to UDP.


>
> There was once a patch floating around which allowed a larger
> NFSSVC_MAXBLKSIZE on architectures with large page sizes, but it never
> got properly submitted I think.
>
>
> >
> > The problem is when I use NFSv3/TCP with a 2.4.25, say, on the server
> > side, as well as on the client side, I am experiencing lockups when
> > copying medium-sized or large files from the NFS fs to a local fs.
>
> There must be some other cause.  Increasing the NFSSVC_MAXBLKSIZE is
> just hiding a real problem I suspect.
>

If only the coincidence was not so clear.

Best regards and thanks

Oliver

__
________________________________________creating IT solutions

Dr. Oliver Tennert			science + computing ag
phone   +49(0)7071 9457-598		Hagellocher Weg 71-75
fax     +49(0)7071 9457-411		D-72070 Tuebingen, Germany
O.Tennert@science-computing.de		www.science-computing.de

Re: PATCH [NFSd] NFSv3/TCP

[Greg Banks]

Neil Brown wrote:
> 
> There was once a patch floating around which allowed a larger
> NFSSVC_MAXBLKSIZE on architectures with large page sizes, but it never
> got properly submitted I think.

Then please consider this a resend.  I'll appreciate any guidance
about proper submission.

This patch has been in SGI's ProPack kernel for 6 months and resulted
in a significant improvement in NFS throughput at a number of customer
sites.

--- /usr/tmp/TmpDir.16250-0/linux/linux/include/linux/nfsd/const.h_1.5	Fri May  7
17:20:22 2004
+++ /usr/tmp/TmpDir.16250-0/linux/linux/include/linux/nfsd/const.h	Fri May  7
17:20:22 2004
@@ -12,6 +12,7 @@
 #include <linux/nfs.h>
 #include <linux/nfs2.h>
 #include <linux/nfs3.h>
+#include <asm/page.h>
 
 /*
  * Maximum protocol version supported by knfsd
@@ -19,9 +20,16 @@
 #define NFSSVC_MAXVERS		3
 
 /*
- * Maximum blocksize supported by daemon currently at 8K
+ * Maximum blocksize supported by daemon.  We want the largest
+ * value which 1) fits in a UDP datagram less some headers
+ * 2) is a multiple of page size 3) can be successfully kmalloc()ed
+ * by each nfsd.   
  */
-#define NFSSVC_MAXBLKSIZE	(8*1024)
+#if PAGE_SIZE > (16*1024)
+#define NFSSVC_MAXBLKSIZE	(32*1024)
+#else
+#define NFSSVC_MAXBLKSIZE	(2*PAGE_SIZE)
+#endif
 
 #ifdef __KERNEL__
 


Greg.
-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.

Re: PATCH [NFSd] NFSv3/TCP

[Oliver Tennert]

As it does not any changes for say a i386 architecture, I cannot see why
after that my lockups should go away.

Are lockups no known problem at all? Am I the only one experiencing them?
They _definitely_ went away for me with NFSSVC_MAXBLKSIZE equal 32k, even
under high IO pressure.

Regards

Oliver


On Fri, 7 May 2004, Greg Banks wrote:

>
> Then please consider this a resend.  I'll appreciate any guidance
> about proper submission.
>
> This patch has been in SGI's ProPack kernel for 6 months and resulted
> in a significant improvement in NFS throughput at a number of customer
> sites.
>
> --- /usr/tmp/TmpDir.16250-0/linux/linux/include/linux/nfsd/const.h_1.5	Fri May  7
> 17:20:22 2004
> +++ /usr/tmp/TmpDir.16250-0/linux/linux/include/linux/nfsd/const.h	Fri May  7
> 17:20:22 2004
> @@ -12,6 +12,7 @@
>  #include <linux/nfs.h>
>  #include <linux/nfs2.h>
>  #include <linux/nfs3.h>
> +#include <asm/page.h>
>
>  /*
>   * Maximum protocol version supported by knfsd
> @@ -19,9 +20,16 @@
>  #define NFSSVC_MAXVERS		3
>
>  /*
> - * Maximum blocksize supported by daemon currently at 8K
> + * Maximum blocksize supported by daemon.  We want the largest
> + * value which 1) fits in a UDP datagram less some headers
> + * 2) is a multiple of page size 3) can be successfully kmalloc()ed
> + * by each nfsd.
>   */
> -#define NFSSVC_MAXBLKSIZE	(8*1024)
> +#if PAGE_SIZE > (16*1024)
> +#define NFSSVC_MAXBLKSIZE	(32*1024)
> +#else
> +#define NFSSVC_MAXBLKSIZE	(2*PAGE_SIZE)
> +#endif
>
>  #ifdef __KERNEL__
>
>
>
> Greg.
> --
> Greg Banks, R&D Software Engineer, SGI Australian Software Group.
> I don't speak for SGI.
>

__
________________________________________creating IT solutions

Dr. Oliver Tennert			science + computing ag
phone   +49(0)7071 9457-598		Hagellocher Weg 71-75
fax     +49(0)7071 9457-411		D-72070 Tuebingen, Germany
O.Tennert@science-computing.de		www.science-computing.de

Re: PATCH [NFSd] NFSv3/TCP

[Greg Banks]

Oliver Tennert wrote:
> 
> As it does not any changes for say a i386 architecture, 

Yes, by design.

> I cannot see why
> after that my lockups should go away.

I don't claim any such thing,  I was just resending a patch (which is of no
use to you) that Neil mentioned had been lost in the shuffle.

> Are lockups no known problem at all? Am I the only one experiencing them?
> They _definitely_ went away for me with NFSSVC_MAXBLKSIZE equal 32k, even
> under high IO pressure.

Sure, I believe you, I just have no idea what your problem is.

As a general statement of no particular import, I note that going to 32K
has a number of other side effects other than the obvious.  For streaming
reads and writes the call rate goes down by a factor of 4 so you may be
not exercising some race condition.  Also there may be different code paths
through READDIR and READDIR+ code.

Now if you had some kind of kernel debugger and could post some more
information, like process list and kernel stack traces from the hang,
someone (not me) may be able to figure out the real problem that you've
hidden by going to 32K.

Greg.
-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.

Strange Linux behaviour!?

[Oliver Pitzeier]

Hi folks!

Strange things are happinging some times, and here we have a new. :-)

We have a machine with five partitions mounted. One of those partitions
is /usr. We can created files on /usr, but we cannot created
directories. mkdir says, that there is no space left on device, but
there actually IS space as you can see and files can be created, so why
NO directories? Is it the kernel, is it the filesystem, is it the full
moon high in the sky? :-) I have no clue, but maybe you have... Any
help/idea is welcome!

Please have a look at this log:

uname -a
Linux apache2.dev.xxx.at 2.4.18-3 #1 Thu Apr 18 07:37:53 EDT 2002 i686
unknown

mount -l
/dev/sda7 on / type ext3 (rw) [/]
none on /proc type proc (rw)
/dev/sda1 on /boot type ext3 (rw) [/boot]
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
/dev/sda6 on /tmp type ext3 (rw) [/tmp]
/dev/sda2 on /usr type ext3 (rw) [/usr]
/dev/sda5 on /var/log type ext3 (rw) [/var/log]

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda7              10G  1.9G  8.0G  19% /
/dev/sda1              38M   15M   21M  40% /boot
none                  504M     0  503M   0% /dev/shm
/dev/sda6             243M  103M  128M  45% /tmp
/dev/sda2             4.8G  703M  3.8G  16% /usr
/dev/sda5             648M   19M  597M   3% /var/log
/usr

cd /usr
mkdir test
mkdir: cannot create directory `test': No space left on device

cd /tmp
mkdir test
ls
drwxr-xr-x    2 root     root         1024 May  7 09:08 test

mkdir --version
mkdir (fileutils) 4.1
Written by David MacKenzie.

Copyright (C) 2001 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is
NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.

Best regards,
 Oliver

PS: Please CC answers to dh@uptime.at

Re: Strange Linux behaviour!?

[Christoph Hellwig]

On Fri, May 07, 2004 at 10:33:02AM +0200, Oliver Pitzeier wrote:
> cd /usr
> mkdir test
> mkdir: cannot create directory `test': No space left on device

?Have you checked whether you're out of inodes?

Re: Strange Linux behaviour!?

[Keith Owens]

On Fri, 7 May 2004 10:33:02 +0200, 
"Oliver Pitzeier" <oliver@linux-kernel.at> wrote:
>We have a machine with five partitions mounted. One of those partitions
>is /usr. We can created files on /usr, but we cannot created
>directories. mkdir says, that there is no space left on device, but
>there actually IS space as you can see and files can be created, so why
>NO directories?

df -i, you have run out of inodes.

Re: Strange Linux behaviour!?

[Dick Streefland]

[Posted and mailed]

"Oliver Pitzeier" <oliver@linux-kernel.at> wrote:
| We have a machine with five partitions mounted. One of those partitions
| is /usr. We can created files on /usr, but we cannot created
| directories. mkdir says, that there is no space left on device, but
| there actually IS space as you can see and files can be created, so why
| NO directories? Is it the kernel, is it the filesystem, is it the full
| moon high in the sky? :-) I have no clue, but maybe you have... Any
| help/idea is welcome!

You probably ran out of inodes. Check "df -i".

-- 
Dick Streefland                      ////                      Altium BV
dick.streefland@altium.nl           (@ @)          http://www.altium.com
--------------------------------oOO--(_)--OOo---------------------------

RE: Strange Linux behaviour!?

[Oliver Pitzeier]

> On Fri, May 07, 2004 at 10:33:02AM +0200, Oliver Pitzeier wrote:
> > cd /usr
> > mkdir test
> > mkdir: cannot create directory `test': No space left on device
> 
> ?Have you checked whether you're out of inodes?

No we havn't. After rebooting the machine with 2.6 everything works...
So inodes are not our problem...

Best,
 Oliver

Re: Strange Linux behaviour!?

[DervishD]

    Hi Keith :)

 * Keith Owens <kaos@ocs.com.au> dixit:
> On Fri, 7 May 2004 10:33:02 +0200, 
> "Oliver Pitzeier" <oliver@linux-kernel.at> wrote:
> >We have a machine with five partitions mounted. One of those partitions
> >is /usr. We can created files on /usr, but we cannot created
> >directories. mkdir says, that there is no space left on device, but
> >there actually IS space as you can see and files can be created, so why
> >NO directories?
> df -i, you have run out of inodes.

    But if you have run out of inodes, you cannot create files
neither, am I wrong? Olives states clearly in his message that files
can be created, but no directories :?

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

Re: Strange Linux behaviour!?

[Timothy Miller]

Christoph Hellwig wrote:
> On Fri, May 07, 2004 at 10:33:02AM +0200, Oliver Pitzeier wrote:
> 
>>cd /usr
>>mkdir test
>>mkdir: cannot create directory `test': No space left on device
> 
> 
> ?Have you checked whether you're out of inodes?
> 


What happens when Linux runs out of inodes?  Why would it?  Doesn't it 
create more?

Re: Strange Linux behaviour!?

[DervishD]

    Hi Timothy :)

 * Timothy Miller <miller@techsource.com> dixit:
> >?Have you checked whether you're out of inodes?
> What happens when Linux runs out of inodes?

    It returns ENOSPC on write operations on the filesystem.

> Why would it?

    Because you created lots of dirs and files ;)

> Doesn't it create more?

    EXT2 and EXT3 doesn't, the number of inodes is specified when doing
mke2fs and it's fixed. Don't know what happens under other
filesystems, but for me doesn't make much sense to create more
inodes: inodes themselves occupy disk space, and if you've run out of
inodes, you probable are near to run out of disk space too. Moreover,
disk structures are a bit complex and adding inodes is not an easy
task in most filesystems :?

    I've been seeing this problem lately on myself. I have a disk to
store temporarily backups and large files in general, so I formatted
it with ext2 using one inode per megabyte of data. This filesystem
usually have 10-50 files, no more, and even with 1/1MB inode ratio,
there were more than 10000 inodes. But when I accidentally
uncompressed one of the backups in the disk, I run out of inodes
*FAST*. I mean, the disk was 80% empty and I didn't have free inodes,
but this is not the common case, since usually you will have an inode
per 4kB of data, so if you don't have free inodes it will usually
mean that your disk space will exhaust soon, too. This is the common
case, I think, so it doesn't worth the effort of adding a few more
inodes just for making the agony longer ;)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

Re: Strange Linux behaviour!?

[Pascal Schmidt]

On Fri, 07 May 2004 19:30:08 +0200, you wrote in linux.kernel:

> What happens when Linux runs out of inodes?  Why would it?  Doesn't it 
> create more?

For many filesystems, the number of inodes *on* *disk* is set at
mkfs time.

-- 
Ciao,
Pascal

Re: Strange Linux behaviour!?

[Richard B. Johnson]

On Fri, 7 May 2004, Pascal Schmidt wrote:

> On Fri, 07 May 2004 19:30:08 +0200, you wrote in linux.kernel:
>
> > What happens when Linux runs out of inodes?  Why would it?  Doesn't it
> > create more?
>
> For many filesystems, the number of inodes *on* *disk* is set at
> mkfs time.
>
> --
> Ciao,
> Pascal
> -

You are being suckered in. "Out of inodes", roughly translated,
means "disk or disk partition is full". You can use `df -k` to
see.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5557.45 BogoMips).
            Note 96.31% of all statistics are fiction.