rpc.mountd reads /etc/mtab 17028 times, 100% CPU.

rpc.mountd reads /etc/mtab 17028 times, 100% CPU.

[Sander Smeenk]

Hi,

I'm running a server that has 1500+ mounted local filesystems.
All of these local filesystems are exported through NFS by this server.

When an NFS-client tries to access one of these exported filesystems,
for example doing a simple 'ls' in a mounted NFS, rpc.mountd on the
server takes 100% CPU and starts to spin over reading /etc/mtab which is
linked to /proc/mounts and totals to about 200KB in size. The client
stalls all the while rpc.mountd is busy reading /etc/mtab for every
filesystem mounted, and then some more.

I made a strace of rpc.mountd during such a run to gather some stats:

# cut -d'(' -f1 straceout.25103 | sort | uniq -c | sort -n
      3 fadvise64
      6 ioctl
     11 lseek
     33 readlink
     43 select
     53 write
    809 statfs
    814 lstat
   1282 stat
  17043 mmap
  17043 munmap
  17088 fstat
  17895 open
  21728 openat
  37295 close
  40352 getdents
 458616 newfstatat
3763233 read

# grep -c "open.*mtab" straceout.25103 
17028
 
This entire process of reading /etc/mtab 17028 times takes a lot of time
during which the client stalls, but in the end 'it just works fine'.
It just takes ages when you try to tab-complete on a client.

What would be needed to debug and optimise this?
Could someone point me to the code that is involved in doing this?
I'm not a good C-coder but perhaps i can debug more...

Any insights appreciated!

With regards,
-Sander.
-- 
| Happiness isn't enough for me. I demand EUPHORIA!
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2

Re: rpc.mountd reads /etc/mtab 17028 times, 100% CPU.

[Steve Dickson]

On 03/25/2015 05:37 AM, Sander Smeenk wrote:
> I'm running a server that has 1500+ mounted local filesystems.
> All of these local filesystems are exported through NFS by this server.
> 
> When an NFS-client tries to access one of these exported filesystems,
> for example doing a simple 'ls' in a mounted NFS, rpc.mountd on the
> server takes 100% CPU and starts to spin over reading /etc/mtab which is
> linked to /proc/mounts and totals to about 200KB in size. The client
> stalls all the while rpc.mountd is busy reading /etc/mtab for every
> filesystem mounted, and then some more.
> 
> This entire process of reading /etc/mtab 17028 times takes a lot of time
> during which the client stalls, but in the end 'it just works fine'.
> It just takes ages when you try to tab-complete on a client.
Taking a quick look... it appears the only way rpc.mountd
will read /etc/mtab is if the 'crossmnt' export flag is 
set. Is this the case?

> 
> What would be needed to debug and optimise this?
Maybe reorder you exports so the 'hot' file systems are on the
top of the list...

> Could someone point me to the code that is involved in doing this?
utils/mountd/cache.c:nfsd_fh():path = next_mnt(&mnt, exp->m_export.e_path);

steved.

Re: rpc.mountd reads /etc/mtab 17028 times, 100% CPU.

[Sander Smeenk]

Quoting Steve Dickson (SteveD@redhat.com):

> > This entire process of reading /etc/mtab 17028 times takes a lot of
> > time during which the client stalls, but in the end 'it just works
> > fine'. It just takes ages when you try to tab-complete on a client.
> Taking a quick look... it appears the only way rpc.mountd will read
> /etc/mtab is if the 'crossmnt' export flag is set. Is this the case?

Yes.

IIRC this was necessary for this situation:
Server has filesytems mounted on /mnt/pt1 and /mnt/pt1/foo.
Both /mnt/pt1 and /mnt/pt1/foo are exported with 'crossmnt'.
Clients that mount just /mnt/pt1 can also reach /mnt/pt1/foo.


> > Could someone point me to the code that is involved in doing this?
> utils/mountd/cache.c:nfsd_fh():path = next_mnt(&mnt, exp->m_export.e_path);

Thanks. I'll dig around some more and see what i can find.


-Sndr.
-- 
| What are the little plastic endings on your shoelaces called?
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2