[linux-lvm] Creation of snapshots failing with `in use' errors

[linux-lvm] Creation of snapshots failing with `in use' errors

[Nix]

(This is with kernel 2.6.17.11, device-mapper 1.02.09, and lvm 2.02.09.)

I'm trying to create a snapshot-the-whole-visible-filesystem-tree script
to use in backup scripts, which makes readonly[1] mounts of network
filesystems and non-LVMed block devices, remounts pseudo-filesystems
(other than /dev which gets bind-mounted), re-binds bind mounts, and
creates and mounts snapshots of LVMed filesystems, before chrooting
into the resulting tree, running some command in there, and then
reversing all its work.

But the apparent asychronousness of LVM is throwing me off.

Specifically, snapshot creation randomly fails if you've recently done
another LVM command. The command might be another snapshot creation
(since they seem to be asynchronous): or it might be much simpler.  By
its very nature this script creates (or removes) about a dozen snapshots
in quick succession, first using lvs(8) to determine the size of the
original filesystem, since LVM maintains a snapshot percentage counter
but doesn't appear to expose it in the commands (at least if it does
it's not documented) and I want these snapshots to have a fixed
percentage of their source available for expansion.

Here's an example:

loki:~# lvs; lvcreate --size 600M --name snap-packages.bin-oKHOHQzu --snapshot /dev/raid/packages.bin
  LV           VG    Attr   LSize   Origin Snap%  Move Log Copy%
  music        disks -wi-ao  17.50G
  news         disks -wn-ao 912.00M
  news-bofh    disks -wn-ao   1.00G
  spam         disks -wi-ao   1.00G
  swap0        disks -wc-ao 592.00M
  swap1        disks -wc-ao 608.00M
  swap2        disks -wc-ao 608.00M
  doc          raid  -wi-ao   1.00G
  home         raid  -wi-ao   6.00G
  mirror       raid  -wi-ao   1.00G
  non-free     raid  -wi-ao   1.00G
  packages     raid  -wi-ao  20.00G
  packages.bin raid  -wi-ao   6.00G
  root         raid  -wi-ao 200.00M
  usr          raid  -wi-ao   6.00G
  var          raid  -wi-ao   1.00G
  LV raid/snap-packages.bin-oKHOHQzu in use: not deactivating
  Couldn't deactivate new snapshot.

This has not only not worked for no good reason (it's not in use! it's
brand new!) but has then proceed to leave a sort of garbage LV behind,
not marked as a snapshot but no good as anything else either:

loki:~# lvs
  LV                         VG    Attr   LSize   Origin Snap%  Move Log Copy%
  music                      disks -wi-ao  17.50G
  news                       disks -wn-ao 912.00M
  news-bofh                  disks -wn-ao   1.00G
  spam                       disks -wi-ao   1.00G
  swap0                      disks -wc-ao 592.00M
  swap1                      disks -wc-ao 608.00M
  swap2                      disks -wc-ao 608.00M
  doc                        raid  -wi-ao   1.00G
  home                       raid  -wi-ao   6.00G
  mirror                     raid  -wi-ao   1.00G
  non-free                   raid  -wi-ao   1.00G
  packages                   raid  -wi-ao  20.00G
  packages.bin               raid  -wi-ao   6.00G
  root                       raid  -wi-ao 200.00M
  snap-packages.bin-oKHOHQzu raid  -wi-a- 600.00M
  usr                        raid  -wi-ao   6.00G
  var                        raid  -wi-ao   1.00G

Is there any way I can arrange for lvcreate and lvremove to actually
*work* consistently such that I can usefully use them in scripts like
this? I'd have no objection to lvcreate/lvremove blocking until whatever
it is that's locking the VG is completed, but dying and leaving debris
behind seems a bit off.


[1] when 2.6.18 comes out so readonly mounts work

-- 
`In typical emacs fashion, it is both absurdly ornate and
 still not really what one wanted.' --- jdev

Re: [linux-lvm] Creation of snapshots failing with `in use' errors

[Alasdair G Kergon]

On Sun, Aug 27, 2006 at 06:57:37PM +0100, Nix wrote:
> Specifically, snapshot creation randomly fails if you've recently done
> another LVM command. 

See if it's the usual udev problem: completely disable udev for your
lvm devices.  [Common problem is udev causing the device to be opened to
look for a label - then lvm command fails because it needs exclusive
access.]

> original filesystem, since LVM maintains a snapshot percentage counter

lvm2 doesn't maintain it, but while a device is active, kernel dm tells
userspace lvm2 the percentage full.

> This has not only not worked for no good reason (it's not in use! it's
> brand new!) 

udev is probably using it.
Old udev packages used to work OK, but something got changed - we never
discovered what (ref. an old debian bug) - and this started to break.

Some people at suse are working on a proper fix for this problem (so kernel
will inform udev when device is usable, instead of udev running too early as
it can now; udev will inform lvm2 when it has finished whatever pointless
things it wants to do and lvm2 will wait for that message).

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] Creation of snapshots failing with `in use' errors

[Nix]

On Sun, 27 Aug 2006, Alasdair G. Kergon uttered the following:
> On Sun, Aug 27, 2006 at 06:57:37PM +0100, Nix wrote:
>> Specifically, snapshot creation randomly fails if you've recently done
>> another LVM command. 
> 
> See if it's the usual udev problem: completely disable udev for your
> lvm devices.  [Common problem is udev causing the device to be opened to
> look for a label - then lvm command fails because it needs exclusive
> access.]

I knew about this but did I remember about it? I did not. `Oh, look at
the idiots not disabling udev' I said as I lurked, and then I set up
a new system, and, whoops...

I need a holiday. (Oh, look, a bank holiday. How convenient.)


Of course, it'd be nice to keep udev active for my non-snapshot devices:
the /dev/disk/* persistent naming really *is* useful (pretty much
critically so on one of my systems, with hot-pluggable devices with an
LVM VG on them). I just don't need it for snapshots.

Which leads me to another problem. Creation of a snapshot creates at least
one temporary block device, but it gets removed so fast that I can't tell
anything about its characteristics to get udev to ignore it:

UEVENT[1156762986.267714] remove@/block/dm-16
UEVENT[1156762990.264106] add@/block/dm-16
UEVENT[1156762990.418076] remove@/block/dm-16
UEVENT[1156762990.592966] add@/block/dm-16
UEVENT[1156762990.606790] add@/block/dm-17
UEVENT[1156762990.609193] add@/block/dm-18

However, that's not all that problematic as there's nothing about dm-16
to dm-18 that tells me they're part of a snapshot either. It looks like
the latter dm-16 is a -real device, while -18 is a -cow, but again
there's absolutely nothing about the -16 that I can sensibly key off: it
doesn't even have all-zero SYSFS{stat} like the snapshot devices do.

It'd be nice if we had a sysfs key that we could use to determine `this
is a snapshot', or an event which was fired to indicate that the device
can be safely opened because lvcreate is finished with it. Right now we
can't even do a dmsetup ls or an lvs from a udev rule to attempt to
determine if the new device is a snapshot, because there's no obvious
way to get from that -cow device, say, to the LV name, and because the
LV isn't even *marked* as a snapshot until a short time after creation.

... hah! Got one, although it's a vile hack. If `dmsetup deps' indicates
a dependency on other device-mapper devices (assuming `major 16' is
enough to tell) then this is a snapshot or mirror some other class of
derived device, and udev should either hands-off it or go to sleep for a
few seconds to let lvcreate or whatever do its job. (That's assuming
that the dependencies are set up at creation time. If they aren't, udev
will see no dependencies, and I'm right back where I started. Let's
see...)

>> original filesystem, since LVM maintains a snapshot percentage counter
> 
> lvm2 doesn't maintain it, but while a device is active, kernel dm tells
> userspace lvm2 the percentage full.

Ah. That explains the absence of setters :)

>> This has not only not worked for no good reason (it's not in use! it's
>> brand new!) 
> 
> udev is probably using it.
> Old udev packages used to work OK, but something got changed - we never
> discovered what (ref. an old debian bug) - and this started to break.

Well, I'm using handwritten udev rules, so it's perfectly likely that
this is the problem.

> Some people at suse are working on a proper fix for this problem (so kernel
> will inform udev when device is usable, instead of udev running too early as
> it can now; udev will inform lvm2 when it has finished whatever pointless
> things it wants to do and lvm2 will wait for that message).

Yep, that's what we're looking for :)

-- 
`In typical emacs fashion, it is both absurdly ornate and
 still not really what one wanted.' --- jdev