[linux-lvm] snapshots and nfs

[linux-lvm] snapshots and nfs

[Kristina Clair]

Hello,

I'm trying to create snapshots on a volume whose filesystem is mounted
over nfs on multiple webservers.  lvcreate is hanging trying to
suspend the volume, and I suspect that the problem is that it is being
used by the webservers.  Does this sound like a reasonable theory?  If
so, is there a way to have lvcreate not suspend the volume?  I'm using
xfs, so I should be able to freeze the filesystem.

Thanks for any advice,
Kristina

Re: [linux-lvm] snapshots and nfs

[Alasdair G Kergon]

On Tue, Jan 18, 2005 at 12:36:06PM -0500, Kristina Clair wrote:
> so, is there a way to have lvcreate not suspend the volume?  I'm using
> xfs, so I should be able to freeze the filesystem.
 
Quick question: you're not running xfs_freeze are you?
If so, try without, as device-mapper does the equivalent of that
internally and xfs can't handle things if it gets run twice.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] snapshots and nfs

[Kristina Clair]

On Tue, 18 Jan 2005 19:44:49 +0000, Alasdair G Kergon <agk@redhat.com> wrote:
> On Tue, Jan 18, 2005 at 12:36:06PM -0500, Kristina Clair wrote:
> > so, is there a way to have lvcreate not suspend the volume?  I'm using
> > xfs, so I should be able to freeze the filesystem.
>  
> Quick question: you're not running xfs_freeze are you?
> If so, try without, as device-mapper does the equivalent of that
> internally and xfs can't handle things if it gets run twice.

No, if I run xfs_freeze first, lvcreate hangs when it tries to suspend
the volume, so I stopped doing that.
Now, if I run lvcreate shortly after the machine is booted, it runs
successfully.  However, if I try to run the same command the next day,
after it's been online and the webservers have had the filesystem
mounted, it hangs when trying to suspend the volume.

So I'm wondering if the problem is that the webservers have the
filesystem mounted and in use (apache is serving pages from it),  I'm
trying to find a way to keep the filesystem mounted over nfs but still
be able to create snapshots, and I was wondering if I could use
xfs_freeze and have lvcreate not try to suspend the volume.

Thanks,
Kristina

Re: [linux-lvm] snapshots and nfs

[Alasdair G Kergon]

On Tue, Jan 18, 2005 at 02:54:41PM -0500, Kristina Clair wrote:
> Now, if I run lvcreate shortly after the machine is booted, it runs
> successfully.  However, if I try to run the same command the next day,
> after it's been online and the webservers have had the filesystem
> mounted, it hangs when trying to suspend the volume.
 
How does it hang?  Do you get kernel error messages?
Try watching kernel memory usage just before the failures:
It could be one of the out-of-memory problems with snapshots 
that we're working on fixing.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] snapshots and nfs

[Randall A. Jones]

Kristina Clair wrote:

>On Tue, 18 Jan 2005 19:44:49 +0000, Alasdair G Kergon <agk@redhat.com> wrote:
>  
>
>>On Tue, Jan 18, 2005 at 12:36:06PM -0500, Kristina Clair wrote:
>>    
>>
>>>so, is there a way to have lvcreate not suspend the volume?  I'm using
>>>xfs, so I should be able to freeze the filesystem.
>>>      
>>>
>> 
>>Quick question: you're not running xfs_freeze are you?
>>If so, try without, as device-mapper does the equivalent of that
>>internally and xfs can't handle things if it gets run twice.
>>    
>>
>
>No, if I run xfs_freeze first, lvcreate hangs when it tries to suspend
>the volume, so I stopped doing that.
>Now, if I run lvcreate shortly after the machine is booted, it runs
>successfully.  However, if I try to run the same command the next day,
>after it's been online and the webservers have had the filesystem
>mounted, it hangs when trying to suspend the volume.
>
>  
>
Is it possible that mounting the NFS mounts on the webservers read-only 
as a temporary measure while creating the snapshot will help?

>So I'm wondering if the problem is that the webservers have the
>filesystem mounted and in use (apache is serving pages from it),  I'm
>trying to find a way to keep the filesystem mounted over nfs but still
>be able to create snapshots, and I was wondering if I could use
>xfs_freeze and have lvcreate not try to suspend the volume.
>
>Thanks,
>Kristina
>
>  
>
-- 
..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
rajones@svs.gsfc.nasa.gov      Code 610.3      301-286-2239

Re: [linux-lvm] snapshots and nfs

[Kristina Clair]

On Tue, 18 Jan 2005 20:01:28 +0000, Alasdair G Kergon <agk@redhat.com> wrote:
> On Tue, Jan 18, 2005 at 02:54:41PM -0500, Kristina Clair wrote:
> > Now, if I run lvcreate shortly after the machine is booted, it runs
> > successfully.  However, if I try to run the same command the next day,
> > after it's been online and the webservers have had the filesystem
> > mounted, it hangs when trying to suspend the volume.
>  
> How does it hang?  Do you get kernel error messages?
> Try watching kernel memory usage just before the failures:
> It could be one of the out-of-memory problems with snapshots
> that we're working on fixing.

I'm not sure about kernel memory usage - actually I don't even know
how to watch kernel memory usage :-)
But by "hang", I just mean that the output hangs, and the command is
still running but is "sleeping".  When it runs successfully, it
doesn't pause at all after it suspends the volume.
One weird thing is that when it is hanging, if I strace the process,
the lvcreate command gets killed.

here is the output of lvcreate and then strace (i apologize for the
backslashes):

\ \ \ \ Creating\ volume\ group\ backup\ \"/etc/lvm/backup/nfsvg\"\
\ \ \ \ \ \ Finding\ volume\ group\ for\ uuid\
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBvh0UnmPyjHyWMwgczC9WXDQFZtF4Ykie\
\ \ \ \ \ \ /dev/sdb:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdc:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdb:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdc:\ lvm2\ label\ detected\
\ \ \ \ Found\ volume\ group\ \"nfsvg\"\
\ \ \ \ \ \ Setting\ activation/missing_stripe_filler\ to\ /dev/ioerror\
\ \ \ \ \ \ Locking\ memory\
\ \ \ \ Loading\ nfsvg-20050118\
\ \ \ \ \ \ Activated\ nfsvg-20050118\
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBvh0UnmPyjHyWMwgczC9WXDQFZtF4Ykie\
253:008\
\ \ \ \ \ \ Unlocking\ memory\
\ \ \ \ \ \ Linking\ /dev/nfsvg/20050118\ -\>\
/dev/mapper/nfsvg-20050118\
\ \ \ \ Zeroing\ start\ of\ logical\ volume\ \"20050118\"\
\ \ \ \ \ \ Finding\ volume\ group\ for\ uuid\
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBvh0UnmPyjHyWMwgczC9WXDQFZtF4Ykie\
\ \ \ \ \ \ /dev/sdb:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdc:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdb:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdc:\ lvm2\ label\ detected\
\ \ \ \ Found\ volume\ group\ \"nfsvg\"\
\ \ \ \ \ \ Locking\ memory\
\ \ \ \ Removing\ nfsvg-20050118\
\ \ \ \ \ \ Unlocking\ memory\
\ \ \ \ \ \ Removing\ link\ /dev/nfsvg/20050118\
\ \ \ \ \ \ Finding\ volume\ group\ for\ uuid\
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBvh0UnmPyjHyWMwgczC9WXDQFZtF4Ykie\
\ \ \ \ \ \ /dev/sdb:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdc:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdb:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdc:\ lvm2\ label\ detected\
\ \ \ \ Found\ volume\ group\ \"nfsvg\"\
\ \ \ \ \ \ Finding\ volume\ group\ for\ uuid\
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBz048ZiCMdqNZhJV1ImaBs77WsnFRlsLD\
\ \ \ \ \ \ /dev/sdb:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdc:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdb:\ lvm2\ label\ detected\
\ \ \ \ \ \ /dev/sdc:\ lvm2\ label\ detected\
\ \ \ \ Found\ volume\ group\ \"nfsvg\"\
\ \ \ \ \ \ Locking\ memory\
\ \ \ \ \ \ Suspending\ nfsvg-lvol0\

[ at this point i run strace and it kills lvcreate ]

\ \ device-mapper\ ioctl\ cmd\ 6\ failed:\ Interrupted\ system\ call\
\ \ Couldn\'t\ suspend\ device\ \'nfsvg-lvol0\'\
\ \ \ \ \ \ Unlocking\ memory\
\ \ Failed\ to\ suspend\ origin\ lvol0\
\ \ \ \ \ \ Unlocking\ /var/lock/lvm/V_nfsvg\

% strace
write(2, "  ", 2)                       = 2
write(2, "    ", 4)                     = 4
write(2, "Unlocking memory", 16)        = 16
write(2, "\n", 1)                       = 1
brk(0x810b000)                          = 0x810b000
setpriority(PRIO_PROCESS, 0, 0)         = 0
close(3)                                = 0
write(2, "  ", 2)                       = 2
write(2, "Failed to suspend origin lvol0", 30) = 30
write(2, "\n", 1)                       = 1
time([1106062664])                      = 1106062664
rt_sigaction(SIGPIPE, {0xb7f0e860, [], 0}, {SIG_DFL}, 8) = 0
socket(PF_FILE, SOCK_DGRAM, 0)          = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
connect(3, {sa_family=AF_FILE, path="/dev/log"}, 16) = 0
send(3, "<11>Jan 18 08:37:44 lvm[5829]: F"..., 61, 0) = 61
rt_sigaction(SIGPIPE, {SIG_DFL}, NULL, 8) = 0
stat64("/proc/lvm/VGs/nfsvg", 0xbffff530) = -1 ENOENT (No such file or
directory)
write(2, "  ", 2)                       = 2
write(2, "    ", 4)                     = 4
write(2, "Unlocking /var/lock/lvm/V_nfsvg", 31) = 31
write(2, "\n", 1)                       = 1
flock(4, LOCK_NB|LOCK_UN)               = 0
flock(4, LOCK_EX|LOCK_NB)               = 0
stat64("/var/lock/lvm/V_nfsvg", {st_mode=S_IFREG|0700, st_size=0, ...})
= 0
fstat64(4, {st_mode=S_IFREG|0700, st_size=0, ...}) = 0
unlink("/var/lock/lvm/V_nfsvg")         = 0
close(4)                                = 0
close(5)                                = 0
close(6)                                = 0
close(7)                                = 0
close(8)                                = 0
close(9)                                = 0
close(10)                               = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
open("/etc/lvm/.cache", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 4
fstat64(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7feb000
write(4, "# This file is automatically mai"..., 184) = 184
close(4)                                = 0
munmap(0xb7feb000, 4096)                = 0
brk(0x80f7000)                          = 0x80f7000
brk(0x80e7000)                          = 0x80e7000
close(3)                                = 0
exit_group(5)                           = ?

Kristina

Re: [linux-lvm] snapshots and nfs

[Kristina Clair]

On Tue, 18 Jan 2005 15:07:04 -0500, Randall A. Jones
<rajones@svs.gsfc.nasa.gov> wrote:
> Kristina Clair wrote:
> 
> >No, if I run xfs_freeze first, lvcreate hangs when it tries to suspend
> >the volume, so I stopped doing that.
> >Now, if I run lvcreate shortly after the machine is booted, it runs
> >successfully.  However, if I try to run the same command the next day,
> >after it's been online and the webservers have had the filesystem
> >mounted, it hangs when trying to suspend the volume.
> >
> >
> Is it possible that mounting the NFS mounts on the webservers read-only
> as a temporary measure while creating the snapshot will help?

Actually, I just tried running lvcreate while the fliesystem is not
mounted on any of the webservers, and I'm having the same problem.
If there are any logs or command output that would be especially
helpful to see, let me know.  There is nothing in either
/var/log/messages or dmesg...

Kristina

[linux-lvm] kernel bug of snapshot

[Chung Hsien Hu]

Hi,
For testing snapshot of LVM2, I am trying to make snapshot over 3 kinds of
filesystem, xfs, ext3 and reiserfs. A crontab is made like this:
*/1 * * * * root snap_rotate xfs_volume
*/2 * * * * root snap_rotate ext_volume
*/3 * * * * root snap_rotate reiserfs_volume

What snap_rotate is a script to remove the old snapshot, and then make 
an new one
over the same logical volume after that. And the crontab do these on 
xfs, ext3 and
reiserfs volumes by every 1, 2 and 3 minutes seperately.

This crontab job lasted for about 10hours (equal to 1000 times of 
rotate), utill
it reported kernel bug, and I found lvm program hang in "D" state.

Any clues? Seems to be some memory allocation problem?

------------[ cut here ]------------
kernel BUG at drivers/md/kcopyd.c:145!
invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in: bonding lm85 i2c_sensor i2c_i801 usb_storage i2c_dev 
i2c_core dm_snapshot uhci_hcd usbcore appletalk ipx p8022 psnap llc nfsd 
exportfs lockd sunrpc quota_v2 nls_iso8859_1 nls_cp437 vfat fat xfs 
reiserfs ext3 jbd st sg loop iscsi_trgt 3w_9xxx tg3 dm_mod sd_mod 
scsi_mod 8250 serial_core unix
CPU:    0
EIP:    0060:[<f886b84b>]    Not tainted VLI
EFLAGS: 00010283   (2.6.10-ac10)
EIP is at client_free_pages+0x3b/0x50 [dm_mod]
eax: 00000100   ebx: f658f3e0   ecx: f6d60b00   edx: 00000000
esi: f8a7e080   edi: 00000000   ebp: 00000000   esp: f3353f10
ds: 007b   es: 007b   ss: 0068
Process lvm (pid: 27317, threadinfo=f3352000 task=f7241a20)
Stack: f658f3e0 f658f3e0 f886c34e f658f3e0 f5a35780 f895da04 f658f3e0 
f702fa20
       f8a7e080 f6d5ec80 f88660ea f8a7e080 f5183e80 f8992000 00000004 
080c3b48
       f8868492 f6d5ec80 00000000 f8872ce0 f8868d21 f5183e80 f3352000 
00000000
Call Trace:
 [<f886c34e>] kcopyd_client_destroy+0x1e/0x3a [dm_mod]
 [<f895da04>] snapshot_dtr+0x74/0x90 [dm_snapshot]
 [<f88660ea>] table_destroy+0x8a/0xa0 [dm_mod]
 [<f8868492>] __hash_remove+0x62/0x90 [dm_mod]
 [<f8868d21>] dev_remove+0x51/0xf0 [dm_mod]
 [<f886a411>] ctl_ioctl+0xf1/0x150 [dm_mod]
 [<f8868cd0>] dev_remove+0x0/0xf0 [dm_mod]
 [<c016b5aa>] sys_ioctl+0xca/0x230
 [<c01028af>] syscall_call+0x7/0xb
Code: 14 75 28 8b 43 0c 89 04 24 e8 52 ff ff ff c7 43 0c 00 00 00 00 c7 
43 10 00 00 00 00 c7 43 14 00 00 00 00 8b 5c 24 04 83 c4 08 c3 <0f> 0b 
91 00 d1 c6 86 f8 eb ce 8d 74 26 00 8d bc 27 00 00 00 00
 <1>Unable to handle kernel paging request at virtual address f8bcc070
 printing eip:
f895efbf
*pde = 00000000
Oops: 0002 [#2]
PREEMPT SMP
Modules linked in: bonding lm85 i2c_sensor i2c_i801 usb_storage i2c_dev 
i2c_core dm_snapshot uhci_hcd usbcore appletalk ipx p8022 psnap llc nfsd 
exportfs lockd sunrpc quota_v2 nls_iso8859_1 nls_cp437 vfat fat xfs 
reiserfs ext3 jbd st sg loop iscsi_trgt 3w_9xxx tg3 dm_mod sd_mod 
scsi_mod 8250 serial_core unix
CPU:    0
EIP:    0060:[<f895efbf>]    Not tainted VLI
EFLAGS: 00010282   (2.6.10-ac10)
EIP is at write_exception+0x2f/0x50 [dm_snapshot]
eax: 0019e872   ebx: f7389ed0   ecx: f8bcc070   edx: 00000000
esi: 00000000   edi: f732eedc   ebp: f895dd40   esp: f7389eb4
ds: 007b   es: 007b   ss: 0068
Process kcopyd (pid: 1330, threadinfo=f7388000 task=f71b8020)
Stack: f6968940 00000007 f6968940 f895f377 f6968940 00000007 f7389ed0 
0019e872
       00000000 00000009 00000000 f6c95468 00000000 f732eedc f895dd85 
f5a357dc
       f732eedc f895dd30 f732eedc f886b9c3 00000000 00000000 f732eedc 
00000000
Call Trace:
 [<f895f377>] persistent_commit+0x47/0x110 [dm_snapshot]
 [<f895dd85>] copy_callback+0x45/0x50 [dm_snapshot]
 [<f895dd30>] commit_callback+0x0/0x10 [dm_snapshot]
 [<f886b9c3>] run_complete_job+0x63/0x80 [dm_mod]
 [<f886bc25>] process_jobs+0x55/0xd0 [dm_mod]
 [<f886bcb8>] do_work+0x18/0x50 [dm_mod]
 [<f886b960>] run_complete_job+0x0/0x80 [dm_mod]
 [<c012ab09>] worker_thread+0x1b9/0x260
 [<f886bca0>] do_work+0x0/0x50 [dm_mod]
 [<c01163a0>] default_wake_function+0x0/0x20
 [<c01163a0>] default_wake_function+0x0/0x20
 [<c012a950>] worker_thread+0x0/0x260
 [<c012ef4a>] kthread+0xba/0xc0
 [<c012ee90>] kthread+0x0/0xc0
 [<c01009b5>] kernel_thread_helper+0x5/0x10
Code: 44 24 14 89 5c 24 08 8b 5c 24 18 89 44 24 04 8b 44 24 10 89 04 24 
e8 71 ff ff ff 89 c1 85 c9 b8 ea ff ff ff 74 18 8b 03 8b 53 04 <89> 01 
8b 43 08 89 51 04 8b 53 0c 89 41 08 31 c0 89 51 0c 8b 5c

-- 
Mozo Hu. mozo@coventive
R&D Assistant Manager, Coventive Advance Technologies

Re: [linux-lvm] lvcreate snapshot problems

[Kristina Clair]

On Tue, 18 Jan 2005 20:01:28 +0000, Alasdair G Kergon <agk@redhat.com> wrote:
> On Tue, Jan 18, 2005 at 02:54:41PM -0500, Kristina Clair wrote:
> > Now, if I run lvcreate shortly after the machine is booted, it runs
> > successfully.  However, if I try to run the same command the next day,
> > after it's been online and the webservers have had the filesystem
> > mounted, it hangs when trying to suspend the volume.
>  
> How does it hang?  Do you get kernel error messages?
> Try watching kernel memory usage just before the failures:
> It could be one of the out-of-memory problems with snapshots
> that we're working on fixing.

The total memory usage goes up slightly when lvcreate starts running,
and then it gradually goes down as the process is hanging.  But the
difference is only about 10M.   Is there something else I can look at
to determine if this is a kernel problem or not?  We are using the
2.6.10 kernel, with DM_SNAPSHOT enabled as a module.

I think when I sent the output of lvcreate last time, it wasn't the
full output.  Here is the full output:

% lvcreate -vvv -L100G -s -n 20050119 /dev/nfsvg/lvol0
        Processing: lvcreate -vvv -L100G -s -n 20050119 /dev/nfsvg/lvol0
        O_DIRECT will be used
      Setting global/locking_type to 1
      Setting global/locking_dir to /var/lock/lvm
      File-based locking enabled.
    Setting chunksize to 16 sectors.
      Getting target version for snapshot
        dm version    
        dm versions    
      Getting target version for snapshot-origin
        dm versions    
      Locking /var/lock/lvm/V_nfsvg WB
    Finding volume group "nfsvg"
        Opened /dev/sda1 RW
        /dev/sda1: block size is 1024 bytes
      /dev/sda1: No label detected
        Opened /dev/sda2 RW
        /dev/sda2: block size is 4096 bytes
      /dev/sda2: No label detected
        Opened /dev/sda3 RW
        /dev/sda3: block size is 4096 bytes
      /dev/sda3: No label detected
        Opened /dev/sda4 RW
        /dev/sda4: block size is 4096 bytes
      /dev/sda4: No label detected
        Opened /dev/sdb RW
        /dev/sdb: block size is 4096 bytes
      /dev/sdb: lvm2 label detected
        Opened /dev/sdc RW
        /dev/sdc: block size is 4096 bytes
      /dev/sdc: lvm2 label detected
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (123) from /dev/sdb at 110080 size 2439
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (123) from /dev/sdc at 110080 size 2439
    Creating logical volume 20050119
        Allowing allocation on /dev/sdb start PE 182272 length 25600
        Allowing allocation on /dev/sdb start PE 233472 length 124139
        Allowing allocation on /dev/sdc start PE 131072 length 226539
    Archiving volume group "nfsvg" metadata.
        Writing nfsvg metadata to /dev/sdb at 112640 len 2680
        Writing nfsvg metadata to /dev/sdc at 112640 len 2680
    Creating volume group backup "/etc/lvm/backup/nfsvg"
        Writing nfsvg metadata to /etc/lvm/backup
.lvm_nfs.phx.serve.com_13289_966521642
        Committing nfsvg metadata (124)
        Renaming /etc/lvm/backup/nfsvg.tmp to /etc/lvm/backup/nfsvg
        Committing nfsvg metadata (124) to /dev/sdb header at 2048
        Committing nfsvg metadata (124) to /dev/sdc header at 2048
        Locking LV
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX (R)
      Finding volume group for uuid
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (124) from /dev/sdb at 112640 size 2680
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (124) from /dev/sdc at 112640 size 2680
    Found volume group "nfsvg"
      Setting activation/missing_stripe_filler to /dev/ioerror
        Getting device info for nfsvg-20050119
        dm info 
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
        dm info nfsvg-20050119   
      Locking memory
        memlock_count inc to 1
        dm names    
        Found existing layer 'nfsvg-lvol0-real'
        Getting device info for nfsvg-lvol0-real
        dm info nfsvg-lvol0-real   
        Found existing layer 'nfsvg-20050111.weekly-cow'
        Getting device info for nfsvg-20050111.weekly-cow
        dm info nfsvg-20050111.weekly-cow   
        Found existing layer 'nfsvg-20050111.weekly'
        Getting device info for nfsvg-20050111.weekly
        dm info nfsvg-20050111.weekly   
        Found existing layer 'nfsvg-20050113-cow'
        Getting device info for nfsvg-20050113-cow
        dm info nfsvg-20050113-cow   
        Found existing layer 'nfsvg-lvol0'
        Getting device info for nfsvg-lvol0
        dm info nfsvg-lvol0   
        Found existing layer 'nfsvg-20050114'
        Getting device info for nfsvg-20050114
        dm info nfsvg-20050114   
        Found existing layer 'nfsvg-20050113'
        Getting device info for nfsvg-20050113
        dm info nfsvg-20050113   
        Found existing layer 'nfsvg-20050114-cow'
        Getting device info for nfsvg-20050114-cow
        dm info nfsvg-20050114-cow   
        Found active lv lvol0
        Found active lv 20050111.weekly
        Found active lv 20050113
        Found active lv 20050114
        Getting device info for nfsvg-20050119
        dm info 
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
        dm info nfsvg-20050119   
    Loading nfsvg-20050119
        Adding target: 0 209715200 linear /dev/sdb 1493172608
        dm create nfsvg-20050119
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
        dm reload nfsvg-20050119   
        dm resume nfsvg-20050119   
      Activated nfsvg-20050119
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX 253:0
08
      Unlocking memory
        memlock_count dec to 0
      Linking /dev/nfsvg/20050119 -> /dev/mapper/nfsvg-20050119
    Zeroing start of logical volume "20050119"
        /dev/nfsvg/20050119: Added to device cache
        Opened /dev/nfsvg/20050119 RW
        Wiping /dev/nfsvg/20050119 at sector 0 length 8 sectors
        /dev/nfsvg/20050119: block size is 4096 bytes
        Closed /dev/nfsvg/20050119
        Locking LV
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX (NL)
      Finding volume group for uuid
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (124) from /dev/sdb at 112640 size 2680
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (124) from /dev/sdc at 112640 size 2680
    Found volume group "nfsvg"
        Getting device info for nfsvg-20050119
        dm info 
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
      Locking memory
        memlock_count inc to 1
        dm names    
        Found existing layer 'nfsvg-lvol0-real'
        Getting device info for nfsvg-lvol0-real
        dm info nfsvg-lvol0-real   
        Found existing layer 'nfsvg-20050111.weekly-cow'
        Getting device info for nfsvg-20050111.weekly-cow
        dm info nfsvg-20050111.weekly-cow   
        Found existing layer 'nfsvg-20050119'
        Getting device info for nfsvg-20050119
        dm info nfsvg-20050119   
        Found existing layer 'nfsvg-20050111.weekly'
        Getting device info for nfsvg-20050111.weekly
        dm info nfsvg-20050111.weekly   
        Found existing layer 'nfsvg-20050113-cow'
        Getting device info for nfsvg-20050113-cow
        dm info nfsvg-20050113-cow   
        Found existing layer 'nfsvg-lvol0'
        Getting device info for nfsvg-lvol0
        dm info nfsvg-lvol0   
        Found existing layer 'nfsvg-20050114'
        Getting device info for nfsvg-20050114
        dm info nfsvg-20050114   
        Found existing layer 'nfsvg-20050113'
        Getting device info for nfsvg-20050113
        dm info nfsvg-20050113   
        Found existing layer 'nfsvg-20050114-cow'
        Getting device info for nfsvg-20050114-cow
        dm info nfsvg-20050114-cow   
        Found active lv lvol0
        Found active lv 20050111.weekly
        Found active lv 20050113
        Found active lv 20050114
        Found active lv 20050119
    Removing nfsvg-20050119
      Unlocking memory
        memlock_count dec to 0
      Removing link /dev/nfsvg/20050119
        Unlocking LV
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
      Finding volume group for uuid
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (124) from /dev/sdb at 112640 size 2680
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (124) from /dev/sdc at 112640 size 2680
    Found volume group "nfsvg"
        Getting device info for nfsvg-20050119
        dm info 
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
        dm info nfsvg-20050119   
        Locking LV
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBz048ZiCMdqNZhJV1ImaBs77WsnFRlsLD (W)
      Finding volume group for uuid
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBz048ZiCMdqNZhJV1ImaBs77WsnFRlsLD
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (124) from /dev/sdb at 112640 size 2680
      /dev/sdb: lvm2 label detected
      /dev/sdc: lvm2 label detected
        Read nfsvg metadata (124) from /dev/sdc at 112640 size 2680
    Found volume group "nfsvg"
        Getting device info for nfsvg-lvol0
        dm info 
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBz048ZiCMdqNZhJV1ImaBs77WsnFRlsLD
      Locking memory
        memlock_count inc to 1
        dm names    
        Found existing layer 'nfsvg-lvol0-real'
        Getting device info for nfsvg-lvol0-real
        dm info nfsvg-lvol0-real   
        Found existing layer 'nfsvg-20050111.weekly-cow'
        Getting device info for nfsvg-20050111.weekly-cow
        dm info nfsvg-20050111.weekly-cow   
        Found existing layer 'nfsvg-20050111.weekly'
        Getting device info for nfsvg-20050111.weekly
        dm info nfsvg-20050111.weekly   
        Found existing layer 'nfsvg-20050113-cow'
        Getting device info for nfsvg-20050113-cow
        dm info nfsvg-20050113-cow   
        Found existing layer 'nfsvg-lvol0'
        Getting device info for nfsvg-lvol0
        dm info nfsvg-lvol0   
        Found existing layer 'nfsvg-20050114'
        Getting device info for nfsvg-20050114
        dm info nfsvg-20050114   
        Found existing layer 'nfsvg-20050113'
        Getting device info for nfsvg-20050113
        dm info nfsvg-20050113   
        Found existing layer 'nfsvg-20050114-cow'
        Getting device info for nfsvg-20050114-cow
        dm info nfsvg-20050114-cow   
        Found active lv lvol0
        Found active lv 20050111.weekly
        Found active lv 20050113
        Found active lv 20050114
        Getting device info for nfsvg-20050119
        dm info 
fo4GChqZZR0sMnwSN0BD0jqqQtROe4MBCmnoDgnlzi1fQbzciXC2S2GT9byJmFqX
        dm info nfsvg-20050119   
      Suspending nfsvg-lvol0
        dm suspend nfsvg-lvol0