LVM snapshot and log record CRC mismatch

LVM snapshot and log record CRC mismatch

[Gionatan Danti]

Hi list,
I have a question about LVM snapshot and XFS.

Some preliminary information: CentOS Linux release 7.0.1406 (Core), with 
kernel version 3.10.0-123.20.1.el7.x86_64

Basically, each time I mount a LVM snapshot (with the "nouuid" option), 
I have the following dmesg entries:

[824501.391058] XFS (dm-7): Mounting Filesystem
[824502.085459] XFS (dm-7): Starting recovery (logdev: internal)
[824502.120435] XFS (dm-7): log record CRC mismatch: found 0x324293f8, 
expected 0x43a1b3b5.
[824502.121035] ffffc90018271000: 00 00 00 10 00 00 00 00 69 01 00 00 6d 
a0 9d a7  ........i...m...
[824502.121649] ffffc90018271010: 00 00 00 10 69 00 00 00 4e 41 52 54 2a 
00 00 00  ....i...NART*...
[824502.122403] XFS (dm-7): log record CRC mismatch: found 0x873f6b2, 
expected 0x49045c16.
[824502.122952] ffffc90018271000: 00 00 00 10 00 00 00 08 aa 20 00 00 6e 
55 00 00  ......... ..nU..
[824502.123508] ffffc90018271010: 00 00 00 00 69 00 00 00 4e 41 52 54 2a 
00 00 00  ....i...NART*...
[824502.124239] XFS (dm-7): log record CRC mismatch: found 0x2f6e47af, 
expected 0x7efb222.
[824502.124792] ffffc90018271000: 00 00 00 10 00 00 00 00 69 01 00 00 a7 
66 d7 2e  ........i....f..
[824502.125359] ffffc90018271010: 00 00 00 10 69 00 00 00 4e 41 52 54 2a 
00 00 00  ....i...NART*...
[824502.126173] XFS (dm-7): log record CRC mismatch: found 0x324293f8, 
expected 0x43a1b3b5.
[824502.126751] ffffc900182b2000: 00 00 00 10 00 00 00 00 69 01 00 00 6d 
a0 9d a7  ........i...m...
[824502.127332] ffffc900182b2010: 00 00 00 10 69 00 00 00 4e 41 52 54 2a 
00 00 00  ....i...NART*...
[824502.153510] XFS (dm-7): log record CRC mismatch: found 0x873f6b2, 
expected 0x49045c16.
[824502.154158] ffffc900182b2000: 00 00 00 10 00 00 00 08 aa 20 00 00 6e 
55 00 00  ......... ..nU..
[824502.154818] ffffc900182b2010: 00 00 00 00 69 00 00 00 4e 41 52 54 2a 
00 00 00  ....i...NART*...
[824502.155654] XFS (dm-7): log record CRC mismatch: found 0x2f6e47af, 
expected 0x7efb222.
[824502.156258] ffffc900182b2000: 00 00 00 10 00 00 00 00 69 01 00 00 a7 
66 d7 2e  ........i....f..
[824502.156869] ffffc900182b2010: 00 00 00 10 69 00 00 00 4e 41 52 54 2a 
00 00 00  ....i...NART*...
[824502.906702] XFS (dm-7): Ending recovery (logdev: internal)
[824502.906714] SELinux: initialized (dev dm-7, type xfs), uses xattr

 From my understanding, this should be more or less the expected 
behavior: the snapshot is a "crash consistent" backup point, and 
mounting the snapshotted filesystem can led to mismatched CRC. Anyway, I 
can read from the snapshot without problems.

However, issuing a xfs_info on the original volume shows that crc 
options is 0 (crc=0), so I am somewhat puzzled about the warning above. 
Moreover, on a more recent CentOS 7.2 server (kernel version 
3.10.0-327.13.1.el7.x86_64), I can not reproduce this warning.

So I ask:
1) it really is the expected behavior, or should I be worried by the 
dmesg entries?
2) this is kernel-related, should I update the kernel to newer version?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LVM snapshot and log record CRC mismatch

[Dave Chinner]

On Tue, May 24, 2016 at 02:58:51PM +0200, Gionatan Danti wrote:
> Hi list,
> I have a question about LVM snapshot and XFS.
> 
> Some preliminary information: CentOS Linux release 7.0.1406 (Core),
> with kernel version 3.10.0-123.20.1.el7.x86_64
> 
> Basically, each time I mount a LVM snapshot (with the "nouuid"
> option), I have the following dmesg entries:
> 
> [824501.391058] XFS (dm-7): Mounting Filesystem
> [824502.085459] XFS (dm-7): Starting recovery (logdev: internal)
> [824502.120435] XFS (dm-7): log record CRC mismatch: found
> 0x324293f8, expected 0x43a1b3b5.
> [824502.121035] ffffc90018271000: 00 00 00 10 00 00 00 00 69 01 00
> 00 6d a0 9d a7  ........i...m...
> [824502.121649] ffffc90018271010: 00 00 00 10 69 00 00 00 4e 41 52
> 54 2a 00 00 00  ....i...NART*...
> [824502.122403] XFS (dm-7): log record CRC mismatch: found
> 0x873f6b2, expected 0x49045c16.
....
> 
> From my understanding, this should be more or less the expected
> behavior: the snapshot is a "crash consistent" backup point, and
> mounting the snapshotted filesystem can led to mismatched CRC.

No, a snapshot should not have mismatched CRC errors in the log.
However, log recovery should be seeing only an unmount record during
recovery, so seeing multiple CRC errors indicates that the
filesystem was not quiesced correctly when the snapshot was taken.

How did you take the snapshot? (full command lines, please)

> Anyway, I can read from the snapshot without problems.

Well, until you hit whatever was not replayed out of the log....

> However, issuing a xfs_info on the original volume shows that crc
> options is 0 (crc=0), so I am somewhat puzzled about the warning
> above.

CRCs are now always enabled on the journal. The difference is that
for crc=1 filesystems this is a fatal error and the fs won't mount,
while for crc=0 it is simply a canary for developers when bugs are
reported that something might have gone wrong in log recovery....

> Moreover, on a more recent CentOS 7.2 server (kernel version
> 3.10.0-327.13.1.el7.x86_64), I can not reproduce this warning.
> 
> So I ask:
> 1) it really is the expected behavior, or should I be worried by the
> dmesg entries?

Not expected. indicates a problem with the snapshot image.

> 2) this is kernel-related

Maybe.

> should I update the kernel to newer version?

Depends if it is kernel related...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LVM snapshot and log record CRC mismatch

[Gionatan Danti]

On 25/05/2016 00:12, Dave Chinner wrote:
>
> How did you take the snapshot? (full command lines, please)
>

The snapshot is accomplished using the following commands:

lvcreate -s hypervisor/KVM_Storage --name Backup_SNAP
lvchange -ay -K hypervisor/Backup_SNAP
mount -o nouuid /dev/hypervisor/Backup_SNAP /mnt/lvm/

As additional information, this is my logical volume setup (I am using 
thin provisioning):

[root@hypervisor ~]# lvs
   LV           VG         Attr       LSize  Pool         Origin Data% 
Move Log Cpy%Sync Convert
   000-ThinPool hypervisor twi-a-t---  1.75t                      42.42 

   KVM_Storage  hypervisor Vwi-aot---  1.70t 000-ThinPool         43.61 

   root         hypervisor -wi-ao---- 64.00g 

   swap         hypervisor -wi-ao----  7.73g


Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LVM snapshot and log record CRC mismatch

[Emmanuel Florac]

Le Wed, 25 May 2016 09:37:41 +0200
Gionatan Danti <g.danti@assyoma.it> écrivait:

> The snapshot is accomplished using the following commands:
> 
> lvcreate -s hypervisor/KVM_Storage --name Backup_SNAP
> lvchange -ay -K hypervisor/Backup_SNAP
> mount -o nouuid /dev/hypervisor/Backup_SNAP /mnt/lvm/
> 

You need to use xfs_freeze to make the filesystem coherent before
snapshot:

xfs_freeze /dev/hypervisor/KVM_Storage
lvcreate -s hypervisor/KVM_Storage --name Backup_SNAP
xfs_freeze -u /dev/hypervisor/KVM_Storage
lvchange -ay -K hypervisor/Backup_SNAP

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LVM snapshot and log record CRC mismatch

[Gionatan Danti]

On 26/05/2016 12:23, Emmanuel Florac wrote:
> Le Wed, 25 May 2016 09:37:41 +0200
> Gionatan Danti <g.danti@assyoma.it> écrivait:
>
>> The snapshot is accomplished using the following commands:
>>
>> lvcreate -s hypervisor/KVM_Storage --name Backup_SNAP
>> lvchange -ay -K hypervisor/Backup_SNAP
>> mount -o nouuid /dev/hypervisor/Backup_SNAP /mnt/lvm/
>>
>
> You need to use xfs_freeze to make the filesystem coherent before
> snapshot:
>
> xfs_freeze /dev/hypervisor/KVM_Storage
> lvcreate -s hypervisor/KVM_Storage --name Backup_SNAP
> xfs_freeze -u /dev/hypervisor/KVM_Storage
> lvchange -ay -K hypervisor/Backup_SNAP
>

It was my understanding that filesystem freezing was automatically 
managed by LVM, and integrated into VFS long ago.

I am missing something?

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LVM snapshot and log record CRC mismatch

[Emmanuel Florac]

Le Thu, 26 May 2016 13:17:04 +0200
Gionatan Danti <g.danti@assyoma.it> écrivait:

> >
> > You need to use xfs_freeze to make the filesystem coherent before
> > snapshot:
> >
> > xfs_freeze /dev/hypervisor/KVM_Storage
> > lvcreate -s hypervisor/KVM_Storage --name Backup_SNAP
> > xfs_freeze -u /dev/hypervisor/KVM_Storage
> > lvchange -ay -K hypervisor/Backup_SNAP
> >  
> 
> It was my understanding that filesystem freezing was automatically 
> managed by LVM, and integrated into VFS long ago.
> 
> I am missing something?

Yes, I thought that too, however it doesn't seem to be working in your
case, and freezing twice can't do any harm anyway :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LVM snapshot and log record CRC mismatch

[Eric Sandeen]

On 5/26/16 6:17 AM, Gionatan Danti wrote:
> On 26/05/2016 12:23, Emmanuel Florac wrote:
>> Le Wed, 25 May 2016 09:37:41 +0200
>> Gionatan Danti <g.danti@assyoma.it> écrivait:
>>
>>> The snapshot is accomplished using the following commands:
>>>
>>> lvcreate -s hypervisor/KVM_Storage --name Backup_SNAP
>>> lvchange -ay -K hypervisor/Backup_SNAP
>>> mount -o nouuid /dev/hypervisor/Backup_SNAP /mnt/lvm/
>>>
>>
>> You need to use xfs_freeze to make the filesystem coherent before
>> snapshot:
>>
>> xfs_freeze /dev/hypervisor/KVM_Storage
>> lvcreate -s hypervisor/KVM_Storage --name Backup_SNAP
>> xfs_freeze -u /dev/hypervisor/KVM_Storage
>> lvchange -ay -K hypervisor/Backup_SNAP
>>
> 
> It was my understanding that filesystem freezing was automatically managed by LVM, and integrated into VFS long ago.
> 
> I am missing something?

Nope, you are correct.  lvm snapshots handle fs quiesce.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LVM snapshot and log record CRC mismatch

[Gionatan Danti]

Il 26-05-2016 13:34 Emmanuel Florac ha scritto:
> 
> Yes, I thought that too, however it doesn't seem to be working in your
> case, and freezing twice can't do any harm anyway :)

Hi all,
after upgrading to CentOS 7.2, kernel version 
3.10.0-327.18.2.el7.x86_64, the problem was solved: mounting a snapshot 
shows no more CRC error (ie: the filesystem is quiesced before taking 
the snapshot).

I am not sure of what component upgrade solved the problem, if kernel or 
lvm tools. I are more inclined to the latter, anyway.

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs