* Re: rpc.mountd high cpu usage [not found] ` <c50ec8735833f8c05a351928ae48b6df@roedie.nl> @ 2013-12-19 17:09 ` J. Bruce Fields 2013-12-23 11:01 ` Sander Klein 0 siblings, 1 reply; 2+ messages in thread From: J. Bruce Fields @ 2013-12-19 17:09 UTC (permalink / raw) To: Sander Klein; +Cc: linux-nfs, linux-nfs-owner, util-linux On Fri, Dec 13, 2013 at 08:32:56PM +0100, Sander Klein wrote: > On , J. Bruce Fields wrote: > >>>I think what happens is that exportfs flushes the kernel's export cache > >>>at which point every use of an uncached export triggers an upcall to > >>>mountd. That upcall is probably visible in the strace as a read of a > >>>file descriptor associated with /proc/net/sunrpc/nfsd.fh/content. > >>> > >>>That upcall is handled by nfs-utils/utils/mountd/cache.c:nfsd_fh(), > >>>which is given a filehandle fragment identifying the filesystem in > >>>question and has to match it to an export. > >>> > >>>That's done by match_fsid(). Which does do a stat of the export path, > >>>but not of all the devices.... That's probably happening in one of the > >>>libblkid calls in uuid_by_path()? I wonder if there's something wrong > >>>with libblkid configuration or with the way we're using it? > >> > >>Is there any way I can help getting this fixed? My coding skills are > >>limited but I am very willing to help in any way I can. > > > >I wonder if ltrace could help determine if libblkid is where most of > >those stat's are coming from (and if so, which calls)? > > > >May also be worth reading up on libblkid (man libblkid, etc.) and > >checking your configuration to make sure there's nothing obvious broken > >there. (If so, maybe the libblkid commandline tools would exhibit the > >same problem?) > > I didn't have any config in /etc/blkid.conf and the manpage says it > will use EVALUATE=udev,scan which in fact scans through all the > /dev/disk/by-* dirs and parses the /proc/partitions. I put > 'EVALUATE=scan' in /etc/blkid.conf but that doesn't help. I > restarted anything NFS related... > > I also attached ltrace to the rpc.mountd process. The output of the > trace is at http://pastebin.com/ika1Vetq . I'm not sure what I'm > looking for. The ltrace is done on a server with only 10 harddisks > but the strace showed the same behavour of newfstatat-ing every disk > in a lot of ways. Hm. I *think* that those SYS_262's are the newfstatat's. And it looks like they happen as part of the implementation of blkid_devno_to_devname (just because you see that call start above the SYS_262's and return only after). That is indeed called from utils/mountd/cache.c:get_uuid_blkdev(). So libblkid is mapping device numbers to paths by stat'ing lots of stuff under /dev. I'm not sure what to do about that.... Cc'ing util-linux in case they can help. --b. ^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: rpc.mountd high cpu usage 2013-12-19 17:09 ` rpc.mountd high cpu usage J. Bruce Fields @ 2013-12-23 11:01 ` Sander Klein 0 siblings, 0 replies; 2+ messages in thread From: Sander Klein @ 2013-12-23 11:01 UTC (permalink / raw) To: J. Bruce Fields; +Cc: linux-nfs, linux-nfs-owner, util-linux On , J. Bruce Fields wrote: > On Fri, Dec 13, 2013 at 08:32:56PM +0100, Sander Klein wrote: >> On , J. Bruce Fields wrote: >> >>>I think what happens is that exportfs flushes the kernel's export cache >> >>>at which point every use of an uncached export triggers an upcall to >> >>>mountd. That upcall is probably visible in the strace as a read of a >> >>>file descriptor associated with /proc/net/sunrpc/nfsd.fh/content. >> >>> >> >>>That upcall is handled by nfs-utils/utils/mountd/cache.c:nfsd_fh(), >> >>>which is given a filehandle fragment identifying the filesystem in >> >>>question and has to match it to an export. >> >>> >> >>>That's done by match_fsid(). Which does do a stat of the export path, >> >>>but not of all the devices.... That's probably happening in one of the >> >>>libblkid calls in uuid_by_path()? I wonder if there's something wrong >> >>>with libblkid configuration or with the way we're using it? >> >> >> >>Is there any way I can help getting this fixed? My coding skills are >> >>limited but I am very willing to help in any way I can. >> > >> >I wonder if ltrace could help determine if libblkid is where most of >> >those stat's are coming from (and if so, which calls)? >> > >> >May also be worth reading up on libblkid (man libblkid, etc.) and >> >checking your configuration to make sure there's nothing obvious broken >> >there. (If so, maybe the libblkid commandline tools would exhibit the >> >same problem?) >> >> I didn't have any config in /etc/blkid.conf and the manpage says it >> will use EVALUATE=udev,scan which in fact scans through all the >> /dev/disk/by-* dirs and parses the /proc/partitions. I put >> 'EVALUATE=scan' in /etc/blkid.conf but that doesn't help. I >> restarted anything NFS related... >> >> I also attached ltrace to the rpc.mountd process. The output of the >> trace is at http://pastebin.com/ika1Vetq . I'm not sure what I'm >> looking for. The ltrace is done on a server with only 10 harddisks >> but the strace showed the same behavour of newfstatat-ing every disk >> in a lot of ways. > > Hm. I *think* that those SYS_262's are the newfstatat's. And it looks > like they happen as part of the implementation of > blkid_devno_to_devname > (just because you see that call start above the SYS_262's and return > only after). > > That is indeed called from utils/mountd/cache.c:get_uuid_blkdev(). > > So libblkid is mapping device numbers to paths by stat'ing lots of > stuff > under /dev. > > I'm not sure what to do about that.... Cc'ing util-linux in case they > can help. > > --b. Anything else I can do to help a bit getting this fixed? Regards, Sander ^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2013-12-23 11:07 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <889b64df295ba04d47f941762ebe0bac@roedie.nl>
[not found] ` <20131212154642.GA11521@fieldses.org>
[not found] ` <3e92cb841fa81d6a16f3d471cdd8bb20@roedie.nl>
[not found] ` <20131212212213.GC13467@fieldses.org>
[not found] ` <c50ec8735833f8c05a351928ae48b6df@roedie.nl>
2013-12-19 17:09 ` rpc.mountd high cpu usage J. Bruce Fields
2013-12-23 11:01 ` Sander Klein
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox