Some way of telling which block devices are in use (and how)

Some way of telling which block devices are in use (and how)

[Theodore Ts'o]

It would be useful if there was a way to be able to determine,
conclusively, whether a partciular block is in device, and how (i.e.,
whether some process has the block open, or it is mounted, or it is
being used as part of a device mapper, or md setup, etc.).

That way when users complain that trying to open a particular device in
exclusive mode returns EBUSY, there's an easy way to figure out why this
might be the case.  The lsof program works for the first method, but it
doesn't help for the other cases; and using /proc/mounts doesn't help if
the block device is mounted in some other namespace; you could try to
look at /proc/*/mounts for every single process, but that's exquisitely
painful, and only works for block devices used by mounted file systems.

So the question is, what's the best way of doing this?  Adding a new
flags field after the block device name in /proc/partitions would be the
simplest, but that might break some userspace programs that aren't
expecting it.  There's the holders directory in sysfs, i.e.,
/sys/dev/block/8:5/holders/*, but that only seems to only record usage
by device mapper.

Should we extend the holders directory in the block device directory in
sysfs?  Add a new directory to record when a file system might be
mounted?  Add a new /proc file?

What do folks think makes the most amount of sense?  Al, Greg, do you
have any opinions or suggestions?

Thanks,

						- Ted

Re: Some way of telling which block devices are in use (and how)

[Greg KH]

On Sun, Apr 29, 2012 at 02:57:11PM -0400, Theodore Ts'o wrote:
> 
> It would be useful if there was a way to be able to determine,
> conclusively, whether a partciular block is in device, and how (i.e.,
> whether some process has the block open, or it is mounted, or it is
> being used as part of a device mapper, or md setup, etc.).
> 
> That way when users complain that trying to open a particular device in
> exclusive mode returns EBUSY, there's an easy way to figure out why this
> might be the case.  The lsof program works for the first method, but it
> doesn't help for the other cases; and using /proc/mounts doesn't help if
> the block device is mounted in some other namespace; you could try to
> look at /proc/*/mounts for every single process, but that's exquisitely
> painful, and only works for block devices used by mounted file systems.
> 
> So the question is, what's the best way of doing this?  Adding a new
> flags field after the block device name in /proc/partitions would be the
> simplest, but that might break some userspace programs that aren't
> expecting it.  There's the holders directory in sysfs, i.e.,
> /sys/dev/block/8:5/holders/*, but that only seems to only record usage
> by device mapper.
> 
> Should we extend the holders directory in the block device directory in
> sysfs?  Add a new directory to record when a file system might be
> mounted?  Add a new /proc file?

Don't we already have a way to do all of this through sysfs today?  How
does gnome/kde handle this when you try to eject a device with an active
mount that is busy?  It pops up a dialog telling you why this can't be
ejected, and which device it is.  I think everything you want is already
there, if not, perhaps dm just needs to add a few more sysfs links to
tie it all together.

And no, no more /proc files for filesystem stuff like this, that way
lies madness.

thanks,

greg k-h

Re: Some way of telling which block devices are in use (and how)

[Ted Ts'o]

On Sun, Apr 29, 2012 at 01:18:30PM -0700, Greg KH wrote:
> Don't we already have a way to do all of this through sysfs today?  How
> does gnome/kde handle this when you try to eject a device with an active
> mount that is busy?  It pops up a dialog telling you why this can't be
> ejected, and which device it is.  I think everything you want is already
> there, if not, perhaps dm just needs to add a few more sysfs links to
> tie it all together.

I'm pretty sure GNOME/KDE is just checking /proc/mounts.  Which won't
help if someone is playing games with mount namespaces; the file
system might not be mounted in the namespace where the sysadmin is
trying to do the umount, so the sysadmin will be clueless as to why
they can't unmount the block device.  (And even if they know that it
has been mounted, there's no way to enumerate or name the mount
namespaces, so all the sysadmin can do is to check /proc/$pid/mounts
for every single pid currently running on the system, which is both
painful and something you can't do in a race-free manner.  Fun, fun,
fun....)

And then there are also all of the other ways which the block device
might be busy.  What GNOME/KDE do may often good enough such that it
works most of the time, but there are cases such as

	https://bugs.launchpad.net/bugs/711799

where it's not so obvious why the block device is busy, and it would
be nice if there's an easy way to definitively tell *why* the block
device is busy.

					- Ted

Re: Some way of telling which block devices are in use (and how)

[Ted Ts'o]

I've been looking more closely at this, and I think we can make things
better by having the file system create a symlink from partition's
holders directory to the file system's kobject (assuming the file
system has a kobject, which most do not have).  i.e., I'd add to
ext4_fill_super() something like this:

	ret = add_symlink(bdev->bd_part->holder_dir, &ext4_kset->kobj);

Currently, the holders directory will show links for md and
devicemapper users of the block device.  This would add a link if ext4
has the block device mounted.

It's not a complete solution, but I think it would help, and as far as
I can tell, it's in the spirit of how the holders directory was
(partially) implemented.

					- Ted

Re: Some way of telling which block devices are in use (and how)

[Greg KH]

On Sun, Apr 29, 2012 at 06:05:12PM -0400, Ted Ts'o wrote:
> I've been looking more closely at this, and I think we can make things
> better by having the file system create a symlink from partition's
> holders directory to the file system's kobject (assuming the file
> system has a kobject, which most do not have).  i.e., I'd add to
> ext4_fill_super() something like this:
> 
> 	ret = add_symlink(bdev->bd_part->holder_dir, &ext4_kset->kobj);

But, as you point out, not all filesystems have /sys/fs/ entries.  In
fact, the large majority do not.

And why would you be doing this at a fs-specific level?  If you want to
know what type of filesystem is mounted on each block device, yes, that
would matter, but you don't.  You want to know what is "busy", right?

And "busy" means different things, including the fact that the whole
block device underneath can disappear at any moment no matter how much
it isn't nice that this happens.

So a combination of 'lsof' and other things might just be the best that
we can do, like GNOME and KDE are doing today.  As you point out the
mount namespace issue, it gets really tricky to try to figure it all
out, so maybe we really don't want to?

thanks,

greg k-h

Re: Some way of telling which block devices are in use (and how)

[Ted Ts'o]

On Mon, Apr 30, 2012 at 01:10:33AM -0400, Greg KH wrote:
> 
> And why would you be doing this at a fs-specific level?  If you want to
> know what type of filesystem is mounted on each block device, yes, that
> would matter, but you don't.  You want to know what is "busy", right?

Well, I'd much rather do something at the VFS layer.  But my
experience is that getting consensus across all of the various FS
maintainers is sometimes, well, hard.

And the meantime, I'd like it to be easier to debug various problems.
The complete problem I'd like to solve is to be able to answer, in a
debugging situation, why a particular block device is "busy".  If I
can't get that, in the worst case, I'd like to be able to answer the
question, is this block device being used by ext4?  And if that's
something I can solve by myself, where there's resistance to solving
the whole problem, at least I can make my patch of the world a little
easier to debug.

> And "busy" means different things, including the fact that the whole
> block device underneath can disappear at any moment no matter how much
> it isn't nice that this happens.

Sure, at which point it's not my problem.  :-)

> So a combination of 'lsof' and other things might just be the best that
> we can do, like GNOME and KDE are doing today.  As you point out the
> mount namespace issue, it gets really tricky to try to figure it all
> out, so maybe we really don't want to?

The mount namespace issue is one of the ones that has always worried
me, because it's very hard to debug.  And as it gets more and more
use, it would be nice if there was an answer better than, "just
iterate over /proc/$pid/mounts".  Think of the issue from the point of
view of a someone at a RHEL or SLES help desk, trying to debug a
problem where they don't have access to the remote system, and want to
tell the user to issue a command which gathers as much information as
possible.  Do we really think the best thing to do is to gather up
information on a per-pid basis?

As to your last point, we're kernel developers.  Since when has
something been hard been a reason not to do the right thing?   :-)

	       	    	   	      	 - Ted

Re: Some way of telling which block devices are in use (and how)

[Greg KH]

On Mon, Apr 30, 2012 at 05:11:19AM -0400, Ted Ts'o wrote:
> On Mon, Apr 30, 2012 at 01:10:33AM -0400, Greg KH wrote:
> > 
> > And why would you be doing this at a fs-specific level?  If you want to
> > know what type of filesystem is mounted on each block device, yes, that
> > would matter, but you don't.  You want to know what is "busy", right?
> 
> Well, I'd much rather do something at the VFS layer.  But my
> experience is that getting consensus across all of the various FS
> maintainers is sometimes, well, hard.

I agree, I think it would be better to do this at the VFS layer, but at
that point, we don't really "know" what filesystem is mounted here, do
we?  As the dm/lvm/md/ecryptfs/union/etc. stacking starts to add up,
where should we even start to point at as well?

> And the meantime, I'd like it to be easier to debug various problems.
> The complete problem I'd like to solve is to be able to answer, in a
> debugging situation, why a particular block device is "busy".

Well, what specifically can cause a block device to be marked "busy"
today?  Is this "just" a reference count being held on the superblock?
Or something else?

> If I can't get that, in the worst case, I'd like to be able to answer
> the question, is this block device being used by ext4?

I think you know that today with /sys/fs/ext4/ right?  (Which, btw, I
hadn't noticed before, very nice, but why not have a symlink back to the
/sys/devices/ tree to the "real" disk device that ext4 is mounted on?  A
"device" symlink would be nice to create, don't you think?

> And if that's something I can solve by myself, where there's
> resistance to solving the whole problem, at least I can make my patch
> of the world a little easier to debug.

If part of this is just to provide a common "core" set of code that any
fs can use to create the /sys/fs/FSNAME/ block links, that would be
great to have and make it easier to add to each of the filesystems.
That in itself would be worthy.

> > And "busy" means different things, including the fact that the whole
> > block device underneath can disappear at any moment no matter how much
> > it isn't nice that this happens.
> 
> Sure, at which point it's not my problem.  :-)

My dmesg seems to differ at times, given the number of nasty warnings
that get spit out when this happens :)

> > So a combination of 'lsof' and other things might just be the best that
> > we can do, like GNOME and KDE are doing today.  As you point out the
> > mount namespace issue, it gets really tricky to try to figure it all
> > out, so maybe we really don't want to?
> 
> The mount namespace issue is one of the ones that has always worried
> me, because it's very hard to debug.  And as it gets more and more
> use, it would be nice if there was an answer better than, "just
> iterate over /proc/$pid/mounts".  Think of the issue from the point of
> view of a someone at a RHEL or SLES help desk, trying to debug a
> problem where they don't have access to the remote system, and want to
> tell the user to issue a command which gathers as much information as
> possible.  Do we really think the best thing to do is to gather up
> information on a per-pid basis?

I think /proc/$pid/mounts needs to be where the namespace is expressed,
and have /sys/* be "namespace neutral" if at all possible.  That being
said, sysfs does have namespace controls and I know the networking layer
takes advantage of this, so it is possible to do this in sysfs as well.

thanks,

greg k-h