mountd prevents spindown of non-exported disk

mountd prevents spindown of non-exported disk

[Timo Reimann]

Hi all,

I have two disks in my server, one of them (hda) being used for backups
solely. To reduce noise level and power consumption, I have been trying
to keep it running in standby mode (as opposed to active) most of the time.

Although there should be nothing accessing the disk except my custom
backup cron job initiating at 5am daily, something was constantly
bringing it back into active state after a rough 20-25 minutes. With the
help of blktrace, I monitored every single I/O access to the disk and
found a single process only that would cause the wake-up:


$ sudo blkparse -i hda.blktrace.0
Input file hda.blktrace.0 added
[...]
  3,0    0        6    88.950000000  6806  Q   R 447 + 8 [rpc.mountd]
[...]


So for some reason, rpc.mountd issues this disk request in regular
intervals although nothing on the disk is being NFS-exported according
to /etc/exports.

mountd in debug mode did not yield anything further, so I did another
run with mountd hooked up to strace, and found out that the requests
happen when I NFS-mount my other drive on the server (sda), or if it's
already mounted, access files on sda (read: not hda). I assume the
spinup event can be pinpointed down to calls to stat64 or open on hda:

  open("/dev/mapper/backup-backup1", O_RDONLY) = 12

I should mention at this point that both disks have their partitions set
up facilitating LVM2, and additionally use encryption layers through
dm-crypt/LUKS on top of that. However, both sda and hda are treated in
completely separate physical volumes, respectively, not interweaved in
any way. So any access to sda should not touch hda filesystem-wise.

I'd glad if anyone could help me out by explaining why NFS operations
influence disks that are not participating in a particular operation.
Preferably with a solution how I can have both NFS running on sda and
keep hda asleep unintermittedly.

By the way, I'm using NFSv3.


Cheers,

--Timo

Re: mountd prevents spindown of non-exported disk

[Neil Brown]

On Wednesday February 20, mailinglist-d4LLFNs4DFRA7UZ8SB9NFg@public.gmane.org wrote:
> Hi all,
> 
> I have two disks in my server, one of them (hda) being used for backups
> solely. To reduce noise level and power consumption, I have been trying
> to keep it running in standby mode (as opposed to active) most of the time.
> 
> Although there should be nothing accessing the disk except my custom
> backup cron job initiating at 5am daily, something was constantly
> bringing it back into active state after a rough 20-25 minutes. With the
> help of blktrace, I monitored every single I/O access to the disk and
> found a single process only that would cause the wake-up:
> 
> 
> $ sudo blkparse -i hda.blktrace.0
> Input file hda.blktrace.0 added
> [...]
>   3,0    0        6    88.950000000  6806  Q   R 447 + 8 [rpc.mountd]
> [...]
> 
> 
> So for some reason, rpc.mountd issues this disk request in regular
> intervals although nothing on the disk is being NFS-exported according
> to /etc/exports.

This is doubtlessly something related to libblkid.

We use libblkid to get a unique UID for each filesystem and use that
as an identified in the filehandle.

We only ever ask it for the UID of specific devices that have been
exported.  However it is quite possible that it touches other devices
as well...

I'm using libblkid in a way that it wasn't originally designed to be
used.  It was (as far as I can tell) designed to find a device given a
UUID or similar.  In that case you would expect it to touch every
device.

I'm doing the reverse and it probably isn't doing things the ideal
way.

I guess someone should figure out how libblkid works and make it work
better for us......

You could always build your own nfs-utils and configure with
--without-uuid.

Not an ideal solution...

NeilBrown

Re: mountd prevents spindown of non-exported disk

[Timo Reimann]

Hi Neil,

first of all, thanks for replying.


Neil Brown wrote:
> On Wednesday February 20, mailinglist-d4LLFNs4DFRA7UZ8SB9NFg@public.gmane.org wrote:
>> Although there should be nothing accessing the disk except my custom
>> backup cron job initiating at 5am daily, something was constantly
>> bringing it back into active state after a rough 20-25 minutes. With the
>> help of blktrace, I monitored every single I/O access to the disk and
>> found a single process only that would cause the wake-up:
>>
>>
>> So for some reason, rpc.mountd issues this disk request in regular
>> intervals although nothing on the disk is being NFS-exported according
>> to /etc/exports.
> 
> This is doubtlessly something related to libblkid.
> 
> We use libblkid to get a unique UID for each filesystem and use that
> as an identified in the filehandle.
> 
> We only ever ask it for the UID of specific devices that have been
> exported.  However it is quite possible that it touches other devices
> as well...
> 
> I'm using libblkid in a way that it wasn't originally designed to be
> used.  It was (as far as I can tell) designed to find a device given a
> UUID or similar.  In that case you would expect it to touch every
> device.
> 
> You could always build your own nfs-utils and configure with
> --without-uuid.
> 
> Not an ideal solution...

What would be the downside of that solution? I suppose NFS would keep on
working even if I disabled that switch, but you gave the impression
there was some (major) drawback.


Cheers,

--Timo

Re: mountd prevents spindown of non-exported disk

[Neil Brown]

On Thursday February 21, mailinglist-d4LLFNs4DFRA7UZ8SB9NFg@public.gmane.org wrote:
> > 
> > You could always build your own nfs-utils and configure with
> > --without-uuid.
> > 
> > Not an ideal solution...
> 
> What would be the downside of that solution? I suppose NFS would keep on
> working even if I disabled that switch, but you gave the impression
> there was some (major) drawback.
> 

The fact that you have to compile your own tools rather than just use
what the distro provides is always a bit of a bore.

Also, if you change the server to stop using UUIDs, then any clients
which have the filesystem mounted will need to remount as the old
mount will become stale.

So it isn't a major problem, but nor is it an ideal solution.

NeilBrown

Re: mountd prevents spindown of non-exported disk

[Timo Reimann]

Neil Brown wrote:
> On Thursday February 21, mailinglist-d4LLFNs4DFRA7UZ8SB9NFg@public.gmane.org wrote:
>>> You could always build your own nfs-utils and configure with
>>> --without-uuid.
>>>
>>> Not an ideal solution...
>> What would be the downside of that solution? I suppose NFS would keep on
>> working even if I disabled that switch, but you gave the impression
>> there was some (major) drawback.
>>
> 
> The fact that you have to compile your own tools rather than just use
> what the distro provides is always a bit of a bore.
> 
> Also, if you change the server to stop using UUIDs, then any clients
> which have the filesystem mounted will need to remount as the old
> mount will become stale.
> 
> So it isn't a major problem, but nor is it an ideal solution.

Is there a proper place to file a bug report for this issue? I'm lacking
the expertise to know whose responsibility it is, but I remember you
mentioned libblkid before.


--Timo

Re: mountd prevents spindown of non-exported disk

[Timo Reimann]

Neil Brown wrote:
> On Wednesday February 20, mailinglist-d4LLFNs4DFRA7UZ8SB9NFg@public.gmane.org wrote:
>> Although there should be nothing accessing the disk except my custom
>> backup cron job initiating at 5am daily, something was constantly
>> bringing it back into active state after a rough 20-25 minutes. With the
>> help of blktrace, I monitored every single I/O access to the disk and
>> found a single process only that would cause the wake-up:
>>
>>
>> $ sudo blkparse -i hda.blktrace.0
>> Input file hda.blktrace.0 added
>> [...]
>>   3,0    0        6    88.950000000  6806  Q   R 447 + 8 [rpc.mountd]
>> [...]
>>
>>
>> So for some reason, rpc.mountd issues this disk request in regular
>> intervals although nothing on the disk is being NFS-exported according
>> to /etc/exports.
> 
> This is doubtlessly something related to libblkid.
> 
> We only ever ask it for the UID of specific devices that have been
> exported.  However it is quite possible that it touches other devices
> as well...
> 
> I'm using libblkid in a way that it wasn't originally designed to be
> used.  It was (as far as I can tell) designed to find a device given a
> UUID or similar.  In that case you would expect it to touch every
> device.
> 
> You could always build your own nfs-utils and configure with
> --without-uuid.

I did so and re-compiled the nfs-utils package, resulting in new
packages of nfs-common and nfs-kernel-server under Ubuntu 7.10.

However, another re-run of blktrace yielded that the problem did not go
away. There are still lots of disk accesses from mountd, and the strace
output continuously contains open calls to the unmounted backup filesystem.


Is there anything else I can do in attempt to resolve this issue? I
suppose pasting the complete output of strace might help, but I wouldn't
do so without explicit confirmation of yours as the log-file is 3.2 MB
large.


--Timo

Re: mountd prevents spindown of non-exported disk

[Benjamin Coddington]

Timo Reimann wrote:
> Is there anything else I can do in attempt to resolve this issue? I
> suppose pasting the complete output of strace might help, but I wouldn't
> do so without explicit confirmation of yours as the log-file is 3.2 MB
> large.

Maybe this is too simplistic, but have you tried exportfs -f?  Old entry 
in the kernel table?

Whats in /var/lib/nfs/xtab?

~Ben

Re: mountd prevents spindown of non-exported disk

[Timo Reimann]

Benjamin Coddington wrote:
> Timo Reimann wrote:
>> Is there anything else I can do in attempt to resolve this issue? I
>> suppose pasting the complete output of strace might help, but I wouldn't
>> do so without explicit confirmation of yours as the log-file is 3.2 MB
>> large.
> 
> Maybe this is too simplistic, but have you tried exportfs -f?  Old entry
> in the kernel table?

I had to reboot the machine a couple of times, so I suppose that is
similar to calling exportfs -f.

Just to make sure, I tried the command too, but to no avail.


> Whats in /var/lib/nfs/xtab?

It's empty. Is that reason to worry?


--Timo

Re: mountd prevents spindown of non-exported disk

[J. Bruce Fields]

On Mon, Feb 25, 2008 at 03:24:49PM +0100, Timo Reimann wrote:
> Neil Brown wrote:
> > On Wednesday February 20, mailinglist-d4LLFNs4DFRA7UZ8SB9NFg@public.gmane.org wrote:
> >> Although there should be nothing accessing the disk except my custom
> >> backup cron job initiating at 5am daily, something was constantly
> >> bringing it back into active state after a rough 20-25 minutes. With the
> >> help of blktrace, I monitored every single I/O access to the disk and
> >> found a single process only that would cause the wake-up:
> >>
> >>
> >> $ sudo blkparse -i hda.blktrace.0
> >> Input file hda.blktrace.0 added
> >> [...]
> >>   3,0    0        6    88.950000000  6806  Q   R 447 + 8 [rpc.mountd]
> >> [...]
> >>
> >>
> >> So for some reason, rpc.mountd issues this disk request in regular
> >> intervals although nothing on the disk is being NFS-exported according
> >> to /etc/exports.
> > 
> > This is doubtlessly something related to libblkid.
> > 
> > We only ever ask it for the UID of specific devices that have been
> > exported.  However it is quite possible that it touches other devices
> > as well...
> > 
> > I'm using libblkid in a way that it wasn't originally designed to be
> > used.  It was (as far as I can tell) designed to find a device given a
> > UUID or similar.  In that case you would expect it to touch every
> > device.
> > 
> > You could always build your own nfs-utils and configure with
> > --without-uuid.
> 
> I did so and re-compiled the nfs-utils package, resulting in new
> packages of nfs-common and nfs-kernel-server under Ubuntu 7.10.
> 
> However, another re-run of blktrace yielded that the problem did not go
> away. There are still lots of disk accesses from mountd, and the strace
> output continuously contains open calls to the unmounted backup filesystem.
> 
> 
> Is there anything else I can do in attempt to resolve this issue? I
> suppose pasting the complete output of strace might help, but I wouldn't
> do so without explicit confirmation of yours as the log-file is 3.2 MB
> large.

Probably the most likely to read such an strace are me, Neil, and Steve
(<steved@redhat.com>), so you might just send it privately to the three
of us.

--b.