Re: Strange delays on NFS server

[Ian Thurlbeck <ian@stams.strath.ac.uk>]

Greg Banks wrote:
> On Wed, Aug 25, 2004 at 09:40:36AM +0100, Ian Thurlbeck wrote:
> 
>>Server stats since reboot Monday morning.  It is lightly
>>loaded mostly:
>>
>>Server nfs v3:
>>null       getattr    setattr    lookup     access     readlink
>>85912   2% 1626277 46% 12375   0% 135833  3% 1295417 36% 773     0%
>>read       write      create     mkdir      symlink    mknod
>>136099  3% 151569  4% 7899    0% 64      0% 194     0% 0       0%
>>remove     rmdir      rename     link       readdir    readdirplus
>>6270    0% 37      0% 2931    0% 969     0% 266     0% 4893    0%
>>fsstat     fsinfo     pathconf   commit
>>1508    0% 890     0% 0       0% 41205   1%
> 
> 
> Yes, it is lightly loaded.  You said your block size is 8K, so
> the server has seen about 1.18 GB of writes in 3 days, which is
> just over twice RAM size.  How many of these slowdown events
> did you see in that time?

Well, at least 10 per day, probably more.

> Can you run "vmstat 1" and log the result please?

Not sure what you mean. Quick snapshot follows, but I'll leave it
running and watch what happens during an "event".

procs                      memory      swap          io     system 
    cpu
  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us 
sy wa id
  0  0  12248  43816  25148  70164    1    0   141   203  234   102  2 
2  0 96
  0  0  12248  43804  25148  70168    0    0     0     0  114    82  0 
2  0 98
  0  0  12248  43772  25180  70168    0    0     0   212  154   125  0 
1  0 99
  0  0  12248  43772  25180  70168    0    0     0     0  111    82  0 
1  0 99


> You said you have a 3ware RAID controller.  What is RAID
> configuration, especially RAID level and stripe width?  What options
> did you use to build the ext3 filesystem?

Capacity:	247.04 GB (482505216 blocks)
Write Cache:	In Use
Configuration:	Striped with Parity 64K (RAID 5)
Stripe size:	64K (65536 bytes)

Ext3 options: anyone know what the default is for Fedora Core 1?

Here is output from tune2fs -l:

tune2fs 1.34 (25-Jul-2003)
Filesystem volume name:   /spare
Last mounted on:          <not available>
Filesystem UUID:          5070c833-cdfe-408b-ae12-ee4587e7ffc4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery 
sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              30162944
Block count:              60312018
Reserved block count:     3015600
Free blocks:              34369240
Free inodes:              29320077
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Mon May 19 03:28:12 2003
Last mount time:          Mon Aug 23 08:45:09 2004
Last write time:          Mon Aug 23 08:45:09 2004
Mount count:              31
Maximum mount count:      -1
Last checked:             Mon May 19 03:28:12 2003
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      e37b261f-2740-4f6c-a970-d4aade2b54be

Ian

> 
>>So that's about 4 writes for every commit.
> 
> 
> Ok, so your clients are doing enough COMMITs.
> 
> 
>>File system export:
>>
>>/export/raid50  @unix(rw,sync,no_wdelay,hide,nocrossmnt,secure,root_squash,
>>no_all_squash,subtree_check,secure_locks,mapping=identity,
>>anonuid=-2,anongid=-2)
> 
> 
> Hmm, no "async". So much for that idea.
> 
> Greg.


-- 
Ian Thurlbeck                http://www.stams.strath.ac.uk/
Statistics and Modelling Science, University of Strathclyde
Livingstone Tower, 26 Richmond Street, Glasgow, UK,  G1 1XH
Tel: +44 (0)141 548 3667           Fax: +44 (0)141 552 2079



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs