corrupt xfs log

corrupt xfs log

[Ingard -]

After a server crash we've encountered a corrupt xfs filesystem. When
trying to mount said filesystem normally the system hangs.
This was initially on a ubuntu trusty server with 3.13 kernel with
xfsprogs 3.1.9

We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
4.12.0 from source. We're still not able to mount the filesystem (and
replay the log) normally.
We are able to mount it -o ro,norecovery, but we're reluctant to do
xfs_repair -L without trying everything we can first. The filesystem
is browsable albeit a few paths which gives an error : "Structure
needs cleaning"

Does anyone have any advice as to how we might recover/repair the
corrupt log so we can replay it? Or is xfs_repair -L the only way
forward?


Excerpt from kern.log:
2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
(sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
inconsistent.

2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
(sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
[xfs], xfs_inode block 0x81c9c210
2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
(sdd1): Unmount and run xfs_repair
2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
(sdd1): First 64 bytes of corrupted metadata buffer:
2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
?.3T[U..|....QGA
2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
....\..z...].r..
2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
..:v.. ...5..6..
2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
..Bu.P...,-..-..

kind regards
ingard

Re: corrupt xfs log

[Bill O'Donnell]

On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> After a server crash we've encountered a corrupt xfs filesystem. When
> trying to mount said filesystem normally the system hangs.
> This was initially on a ubuntu trusty server with 3.13 kernel with
> xfsprogs 3.1.9
> 
> We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> 4.12.0 from source. We're still not able to mount the filesystem (and
> replay the log) normally.
> We are able to mount it -o ro,norecovery, but we're reluctant to do
> xfs_repair -L without trying everything we can first. The filesystem
> is browsable albeit a few paths which gives an error : "Structure
> needs cleaning"
> 
> Does anyone have any advice as to how we might recover/repair the
> corrupt log so we can replay it? Or is xfs_repair -L the only way
> forward?

Can you try xfs_repair -n (only scans the fs and reports what repairs
would be made)?

Thanks-
Bill


> 
> 
> Excerpt from kern.log:
> 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> inconsistent.
> 
> 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> [xfs], xfs_inode block 0x81c9c210
> 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> (sdd1): Unmount and run xfs_repair
> 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> (sdd1): First 64 bytes of corrupted metadata buffer:
> 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> ?.3T[U..|....QGA
> 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> ....\..z...].r..
> 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> ..:v.. ...5..6..
> 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> ..Bu.P...,-..-..
> 
> kind regards
> ingard
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Brian Foster]

On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> > After a server crash we've encountered a corrupt xfs filesystem. When
> > trying to mount said filesystem normally the system hangs.
> > This was initially on a ubuntu trusty server with 3.13 kernel with
> > xfsprogs 3.1.9
> > 
> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> > 4.12.0 from source. We're still not able to mount the filesystem (and
> > replay the log) normally.
> > We are able to mount it -o ro,norecovery, but we're reluctant to do
> > xfs_repair -L without trying everything we can first. The filesystem
> > is browsable albeit a few paths which gives an error : "Structure
> > needs cleaning"
> > 
> > Does anyone have any advice as to how we might recover/repair the
> > corrupt log so we can replay it? Or is xfs_repair -L the only way
> > forward?
> 
> Can you try xfs_repair -n (only scans the fs and reports what repairs
> would be made)?
> 

An xfs_metadump of the fs might be useful as well. Then we can see if we
can reproduce the mount hang on latest kernels and if so, potentially
try and root cause it.

Brian

> Thanks-
> Bill
> 
> 
> > 
> > 
> > Excerpt from kern.log:
> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> > inconsistent.
> > 
> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> > [xfs], xfs_inode block 0x81c9c210
> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> > (sdd1): Unmount and run xfs_repair
> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> > (sdd1): First 64 bytes of corrupted metadata buffer:
> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> > ?.3T[U..|....QGA
> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> > ....\..z...].r..
> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> > ..:v.. ...5..6..
> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> > ..Bu.P...,-..-..
> > 
> > kind regards
> > ingard
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Ingard -]

On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
> On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
>> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
>> > After a server crash we've encountered a corrupt xfs filesystem. When
>> > trying to mount said filesystem normally the system hangs.
>> > This was initially on a ubuntu trusty server with 3.13 kernel with
>> > xfsprogs 3.1.9
>> >
>> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
>> > 4.12.0 from source. We're still not able to mount the filesystem (and
>> > replay the log) normally.
>> > We are able to mount it -o ro,norecovery, but we're reluctant to do
>> > xfs_repair -L without trying everything we can first. The filesystem
>> > is browsable albeit a few paths which gives an error : "Structure
>> > needs cleaning"
>> >
>> > Does anyone have any advice as to how we might recover/repair the
>> > corrupt log so we can replay it? Or is xfs_repair -L the only way
>> > forward?
>>
>> Can you try xfs_repair -n (only scans the fs and reports what repairs
>> would be made)?
>>
>
> An xfs_metadump of the fs might be useful as well. Then we can see if we
> can reproduce the mount hang on latest kernels and if so, potentially
> try and root cause it.
>
> Brian

Here is a link for the metadump :
https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
And the repair -n output :
https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d

kind regards
ingard

>
>> Thanks-
>> Bill
>>
>>
>> >
>> >
>> > Excerpt from kern.log:
>> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
>> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
>> > inconsistent.
>> >
>> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
>> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
>> > [xfs], xfs_inode block 0x81c9c210
>> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
>> > (sdd1): Unmount and run xfs_repair
>> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
>> > (sdd1): First 64 bytes of corrupted metadata buffer:
>> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
>> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
>> > ?.3T[U..|....QGA
>> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
>> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
>> > ....\..z...].r..
>> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
>> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
>> > ..:v.. ...5..6..
>> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
>> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
>> > ..Bu.P...,-..-..
>> >
>> > kind regards
>> > ingard
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Brian Foster]

On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> >> > After a server crash we've encountered a corrupt xfs filesystem. When
> >> > trying to mount said filesystem normally the system hangs.
> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
> >> > xfsprogs 3.1.9
> >> >
> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
> >> > replay the log) normally.
> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
> >> > xfs_repair -L without trying everything we can first. The filesystem
> >> > is browsable albeit a few paths which gives an error : "Structure
> >> > needs cleaning"
> >> >
> >> > Does anyone have any advice as to how we might recover/repair the
> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
> >> > forward?
> >>
> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
> >> would be made)?
> >>
> >
> > An xfs_metadump of the fs might be useful as well. Then we can see if we
> > can reproduce the mount hang on latest kernels and if so, potentially
> > try and root cause it.
> >
> > Brian
> 
> Here is a link for the metadump :
> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff

This points to a 29GB image file, apparently uncompressed..? Could you
upload a compressed file? Thanks.

Brian

> And the repair -n output :
> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
> 
> kind regards
> ingard
> 
> >
> >> Thanks-
> >> Bill
> >>
> >>
> >> >
> >> >
> >> > Excerpt from kern.log:
> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> >> > inconsistent.
> >> >
> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> >> > [xfs], xfs_inode block 0x81c9c210
> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> >> > (sdd1): Unmount and run xfs_repair
> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> >> > ?.3T[U..|....QGA
> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> >> > ....\..z...].r..
> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> >> > ..:v.. ...5..6..
> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> >> > ..Bu.P...,-..-..
> >> >
> >> > kind regards
> >> > ingard
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Ingard -]

On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
> On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
>> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
>> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
>> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
>> >> > After a server crash we've encountered a corrupt xfs filesystem. When
>> >> > trying to mount said filesystem normally the system hangs.
>> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
>> >> > xfsprogs 3.1.9
>> >> >
>> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
>> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
>> >> > replay the log) normally.
>> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
>> >> > xfs_repair -L without trying everything we can first. The filesystem
>> >> > is browsable albeit a few paths which gives an error : "Structure
>> >> > needs cleaning"
>> >> >
>> >> > Does anyone have any advice as to how we might recover/repair the
>> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
>> >> > forward?
>> >>
>> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
>> >> would be made)?
>> >>
>> >
>> > An xfs_metadump of the fs might be useful as well. Then we can see if we
>> > can reproduce the mount hang on latest kernels and if so, potentially
>> > try and root cause it.
>> >
>> > Brian
>>
>> Here is a link for the metadump :
>> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
>
> This points to a 29GB image file, apparently uncompressed..? Could you
> upload a compressed file? Thanks.

Hi. Sorry about that. Didnt realize the output would be compressable.
Here is a link to the compressed tgz (6G)
https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae

>
> Brian
>
>> And the repair -n output :
>> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
>>
>> kind regards
>> ingard
>>
>> >
>> >> Thanks-
>> >> Bill
>> >>
>> >>
>> >> >
>> >> >
>> >> > Excerpt from kern.log:
>> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
>> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
>> >> > inconsistent.
>> >> >
>> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
>> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
>> >> > [xfs], xfs_inode block 0x81c9c210
>> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
>> >> > (sdd1): Unmount and run xfs_repair
>> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
>> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
>> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
>> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
>> >> > ?.3T[U..|....QGA
>> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
>> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
>> >> > ....\..z...].r..
>> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
>> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
>> >> > ..:v.. ...5..6..
>> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
>> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
>> >> > ..Bu.P...,-..-..
>> >> >
>> >> > kind regards
>> >> > ingard
>> >> > --
>> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> > the body of a message to majordomo@vger.kernel.org
>> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Ingard -]

On Mon, Aug 21, 2017 at 10:24 PM, Ingard - <ingard1@gmail.com> wrote:
> On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
>> On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
>>> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
>>> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
>>> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
>>> >> > After a server crash we've encountered a corrupt xfs filesystem. When
>>> >> > trying to mount said filesystem normally the system hangs.
>>> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
>>> >> > xfsprogs 3.1.9
>>> >> >
>>> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
>>> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
>>> >> > replay the log) normally.
>>> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
>>> >> > xfs_repair -L without trying everything we can first. The filesystem
>>> >> > is browsable albeit a few paths which gives an error : "Structure
>>> >> > needs cleaning"
>>> >> >
>>> >> > Does anyone have any advice as to how we might recover/repair the
>>> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
>>> >> > forward?
>>> >>
>>> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
>>> >> would be made)?
>>> >>
>>> >
>>> > An xfs_metadump of the fs might be useful as well. Then we can see if we
>>> > can reproduce the mount hang on latest kernels and if so, potentially
>>> > try and root cause it.
>>> >
>>> > Brian
>>>
>>> Here is a link for the metadump :
>>> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
>>
>> This points to a 29GB image file, apparently uncompressed..? Could you
>> upload a compressed file? Thanks.
>
> Hi. Sorry about that. Didnt realize the output would be compressable.
> Here is a link to the compressed tgz (6G)
> https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae

Hi
Did you have a chance to look at the log/metadump ?

ingard
>
>>
>> Brian
>>
>>> And the repair -n output :
>>> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
>>>
>>> kind regards
>>> ingard
>>>
>>> >
>>> >> Thanks-
>>> >> Bill
>>> >>
>>> >>
>>> >> >
>>> >> >
>>> >> > Excerpt from kern.log:
>>> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
>>> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
>>> >> > inconsistent.
>>> >> >
>>> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
>>> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
>>> >> > [xfs], xfs_inode block 0x81c9c210
>>> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
>>> >> > (sdd1): Unmount and run xfs_repair
>>> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
>>> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
>>> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
>>> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
>>> >> > ?.3T[U..|....QGA
>>> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
>>> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
>>> >> > ....\..z...].r..
>>> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
>>> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
>>> >> > ..:v.. ...5..6..
>>> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
>>> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
>>> >> > ..Bu.P...,-..-..
>>> >> >
>>> >> > kind regards
>>> >> > ingard
>>> >> > --
>>> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>>> >> > the body of a message to majordomo@vger.kernel.org
>>> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> >> --
>>> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>>> >> the body of a message to majordomo@vger.kernel.org
>>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Brian Foster]

On Mon, Aug 28, 2017 at 10:56:13AM +0200, Ingard - wrote:
> On Mon, Aug 21, 2017 at 10:24 PM, Ingard - <ingard1@gmail.com> wrote:
> > On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
> >> On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
> >>> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
> >>> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
> >>> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> >>> >> > After a server crash we've encountered a corrupt xfs filesystem. When
> >>> >> > trying to mount said filesystem normally the system hangs.
> >>> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
> >>> >> > xfsprogs 3.1.9
> >>> >> >
> >>> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> >>> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
> >>> >> > replay the log) normally.
> >>> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
> >>> >> > xfs_repair -L without trying everything we can first. The filesystem
> >>> >> > is browsable albeit a few paths which gives an error : "Structure
> >>> >> > needs cleaning"
> >>> >> >
> >>> >> > Does anyone have any advice as to how we might recover/repair the
> >>> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
> >>> >> > forward?
> >>> >>
> >>> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
> >>> >> would be made)?
> >>> >>
> >>> >
> >>> > An xfs_metadump of the fs might be useful as well. Then we can see if we
> >>> > can reproduce the mount hang on latest kernels and if so, potentially
> >>> > try and root cause it.
> >>> >
> >>> > Brian
> >>>
> >>> Here is a link for the metadump :
> >>> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
> >>
> >> This points to a 29GB image file, apparently uncompressed..? Could you
> >> upload a compressed file? Thanks.
> >
> > Hi. Sorry about that. Didnt realize the output would be compressable.
> > Here is a link to the compressed tgz (6G)
> > https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae
> 
> Hi
> Did you have a chance to look at the log/metadump ?
> 

Not yet. I have it downloaded somewhere but I haven't got to it yet.
It's on the todo list, hopefully soon.

Brian

> ingard
> >
> >>
> >> Brian
> >>
> >>> And the repair -n output :
> >>> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
> >>>
> >>> kind regards
> >>> ingard
> >>>
> >>> >
> >>> >> Thanks-
> >>> >> Bill
> >>> >>
> >>> >>
> >>> >> >
> >>> >> >
> >>> >> > Excerpt from kern.log:
> >>> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> >>> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> >>> >> > inconsistent.
> >>> >> >
> >>> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> >>> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> >>> >> > [xfs], xfs_inode block 0x81c9c210
> >>> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> >>> >> > (sdd1): Unmount and run xfs_repair
> >>> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> >>> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
> >>> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> >>> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> >>> >> > ?.3T[U..|....QGA
> >>> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> >>> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> >>> >> > ....\..z...].r..
> >>> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> >>> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> >>> >> > ..:v.. ...5..6..
> >>> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> >>> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> >>> >> > ..Bu.P...,-..-..
> >>> >> >
> >>> >> > kind regards
> >>> >> > ingard
> >>> >> > --
> >>> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >>> >> > the body of a message to majordomo@vger.kernel.org
> >>> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>> >> --
> >>> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >>> >> the body of a message to majordomo@vger.kernel.org
> >>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Brian Foster]

[-- Attachment #1: Type: text/plain, Size: 5803 bytes --]

On Mon, Aug 21, 2017 at 10:24:32PM +0200, Ingard - wrote:
> On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
> > On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
> >> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
> >> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
> >> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> >> >> > After a server crash we've encountered a corrupt xfs filesystem. When
> >> >> > trying to mount said filesystem normally the system hangs.
> >> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
> >> >> > xfsprogs 3.1.9
> >> >> >
> >> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> >> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
> >> >> > replay the log) normally.
> >> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
> >> >> > xfs_repair -L without trying everything we can first. The filesystem
> >> >> > is browsable albeit a few paths which gives an error : "Structure
> >> >> > needs cleaning"
> >> >> >
> >> >> > Does anyone have any advice as to how we might recover/repair the
> >> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
> >> >> > forward?
> >> >>
> >> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
> >> >> would be made)?
> >> >>
> >> >
> >> > An xfs_metadump of the fs might be useful as well. Then we can see if we
> >> > can reproduce the mount hang on latest kernels and if so, potentially
> >> > try and root cause it.
> >> >
> >> > Brian
> >>
> >> Here is a link for the metadump :
> >> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
> >
> > This points to a 29GB image file, apparently uncompressed..? Could you
> > upload a compressed file? Thanks.
> 
> Hi. Sorry about that. Didnt realize the output would be compressable.
> Here is a link to the compressed tgz (6G)
> https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae
> 

I finally played around with this image a bit. Note that mount does not
hang on latest kernels. Instead, log recovery emits a torn write message
due to a bad crc at the head of the log and then ultimately fails due to
a bad crc at the tail of the log. I ran a couple experiments to skip the
bad crc records and/or to completely ignore all bad crc's and both still
either fail to mount (due to other corruption) or continue to show
corruption in the recovered fs. 

It's not clear to me what would have caused this corruption or log
state. Have you encountered any corruption before? If not, is this kind
of crash or unclean shutdown of the server an uncommon event?

That aside, I think the best course of action is to run 'xfs_repair -L'
on the fs. I ran a v4.12 version against the metadump image and it
successfully repaired the fs. I've attached the repair output for
reference, but I would recommend to first restore your metadump to a
temporary location, attempt to repair that and examine the results
before repairing the original fs. Note that the metadump will not have
any file content, but will represent which files might be cleared, moved
to lost+found, etc.

Brian

> >
> > Brian
> >
> >> And the repair -n output :
> >> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
> >>
> >> kind regards
> >> ingard
> >>
> >> >
> >> >> Thanks-
> >> >> Bill
> >> >>
> >> >>
> >> >> >
> >> >> >
> >> >> > Excerpt from kern.log:
> >> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> >> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> >> >> > inconsistent.
> >> >> >
> >> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> >> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> >> >> > [xfs], xfs_inode block 0x81c9c210
> >> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> >> >> > (sdd1): Unmount and run xfs_repair
> >> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> >> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
> >> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> >> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> >> >> > ?.3T[U..|....QGA
> >> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> >> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> >> >> > ....\..z...].r..
> >> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> >> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> >> >> > ..:v.. ...5..6..
> >> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> >> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> >> >> > ..Bu.P...,-..-..
> >> >> >
> >> >> > kind regards
> >> >> > ingard
> >> >> > --
> >> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> > the body of a message to majordomo@vger.kernel.org
> >> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: xfs_repair.out.gz --]
[-- Type: application/gzip, Size: 4457 bytes --]

Re: corrupt xfs log

[Ingard -]

On Wed, Aug 30, 2017 at 4:58 PM, Brian Foster <bfoster@redhat.com> wrote:
> On Mon, Aug 21, 2017 at 10:24:32PM +0200, Ingard - wrote:
>> On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
>> > On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
>> >> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
>> >> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
>> >> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
>> >> >> > After a server crash we've encountered a corrupt xfs filesystem. When
>> >> >> > trying to mount said filesystem normally the system hangs.
>> >> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
>> >> >> > xfsprogs 3.1.9
>> >> >> >
>> >> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
>> >> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
>> >> >> > replay the log) normally.
>> >> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
>> >> >> > xfs_repair -L without trying everything we can first. The filesystem
>> >> >> > is browsable albeit a few paths which gives an error : "Structure
>> >> >> > needs cleaning"
>> >> >> >
>> >> >> > Does anyone have any advice as to how we might recover/repair the
>> >> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
>> >> >> > forward?
>> >> >>
>> >> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
>> >> >> would be made)?
>> >> >>
>> >> >
>> >> > An xfs_metadump of the fs might be useful as well. Then we can see if we
>> >> > can reproduce the mount hang on latest kernels and if so, potentially
>> >> > try and root cause it.
>> >> >
>> >> > Brian
>> >>
>> >> Here is a link for the metadump :
>> >> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
>> >
>> > This points to a 29GB image file, apparently uncompressed..? Could you
>> > upload a compressed file? Thanks.
>>
>> Hi. Sorry about that. Didnt realize the output would be compressable.
>> Here is a link to the compressed tgz (6G)
>> https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae
>>
>
> I finally played around with this image a bit. Note that mount does not
> hang on latest kernels. Instead, log recovery emits a torn write message
> due to a bad crc at the head of the log and then ultimately fails due to
> a bad crc at the tail of the log. I ran a couple experiments to skip the
> bad crc records and/or to completely ignore all bad crc's and both still
> either fail to mount (due to other corruption) or continue to show
> corruption in the recovered fs.
>
> It's not clear to me what would have caused this corruption or log
> state. Have you encountered any corruption before? If not, is this kind
> of crash or unclean shutdown of the server an uncommon event?
We failed to notice the log messages of corrupt fs at first. After a
few days of these messages the filesystem got shut down due to
excessive? corruption.
At that point we tried to reboot normally, but ended up with having to
do a hard reset of the server.
It is not clear to us either why the corruption happened in the first
place either. The underlying raid has been in optimal state the whole
time

>
> That aside, I think the best course of action is to run 'xfs_repair -L'
> on the fs. I ran a v4.12 version against the metadump image and it
> successfully repaired the fs. I've attached the repair output for
> reference, but I would recommend to first restore your metadump to a
> temporary location, attempt to repair that and examine the results
> before repairing the original fs. Note that the metadump will not have
> any file content, but will represent which files might be cleared, moved
> to lost+found, etc.
Ok. Thanks for looking into it. We'll proceed with the suggested
course of action.

ingard
>
> Brian
>
>> >
>> > Brian
>> >
>> >> And the repair -n output :
>> >> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
>> >>
>> >> kind regards
>> >> ingard
>> >>
>> >> >
>> >> >> Thanks-
>> >> >> Bill
>> >> >>
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> > Excerpt from kern.log:
>> >> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
>> >> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
>> >> >> > inconsistent.
>> >> >> >
>> >> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
>> >> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
>> >> >> > [xfs], xfs_inode block 0x81c9c210
>> >> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
>> >> >> > (sdd1): Unmount and run xfs_repair
>> >> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
>> >> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
>> >> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
>> >> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
>> >> >> > ?.3T[U..|....QGA
>> >> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
>> >> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
>> >> >> > ....\..z...].r..
>> >> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
>> >> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
>> >> >> > ..:v.. ...5..6..
>> >> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
>> >> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
>> >> >> > ..Bu.P...,-..-..
>> >> >> >
>> >> >> > kind regards
>> >> >> > ingard
>> >> >> > --
>> >> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> >> > the body of a message to majordomo@vger.kernel.org
>> >> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> >> --
>> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> >> the body of a message to majordomo@vger.kernel.org
>> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Brian Foster]

On Thu, Aug 31, 2017 at 09:27:52AM +0200, Ingard - wrote:
> On Wed, Aug 30, 2017 at 4:58 PM, Brian Foster <bfoster@redhat.com> wrote:
> > On Mon, Aug 21, 2017 at 10:24:32PM +0200, Ingard - wrote:
> >> On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
> >> > On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
> >> >> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
> >> >> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
> >> >> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> >> >> >> > After a server crash we've encountered a corrupt xfs filesystem. When
> >> >> >> > trying to mount said filesystem normally the system hangs.
> >> >> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
> >> >> >> > xfsprogs 3.1.9
> >> >> >> >
> >> >> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> >> >> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
> >> >> >> > replay the log) normally.
> >> >> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
> >> >> >> > xfs_repair -L without trying everything we can first. The filesystem
> >> >> >> > is browsable albeit a few paths which gives an error : "Structure
> >> >> >> > needs cleaning"
> >> >> >> >
> >> >> >> > Does anyone have any advice as to how we might recover/repair the
> >> >> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
> >> >> >> > forward?
> >> >> >>
> >> >> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
> >> >> >> would be made)?
> >> >> >>
> >> >> >
> >> >> > An xfs_metadump of the fs might be useful as well. Then we can see if we
> >> >> > can reproduce the mount hang on latest kernels and if so, potentially
> >> >> > try and root cause it.
> >> >> >
> >> >> > Brian
> >> >>
> >> >> Here is a link for the metadump :
> >> >> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
> >> >
> >> > This points to a 29GB image file, apparently uncompressed..? Could you
> >> > upload a compressed file? Thanks.
> >>
> >> Hi. Sorry about that. Didnt realize the output would be compressable.
> >> Here is a link to the compressed tgz (6G)
> >> https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae
> >>
> >
> > I finally played around with this image a bit. Note that mount does not
> > hang on latest kernels. Instead, log recovery emits a torn write message
> > due to a bad crc at the head of the log and then ultimately fails due to
> > a bad crc at the tail of the log. I ran a couple experiments to skip the
> > bad crc records and/or to completely ignore all bad crc's and both still
> > either fail to mount (due to other corruption) or continue to show
> > corruption in the recovered fs.
> >
> > It's not clear to me what would have caused this corruption or log
> > state. Have you encountered any corruption before? If not, is this kind
> > of crash or unclean shutdown of the server an uncommon event?
> We failed to notice the log messages of corrupt fs at first. After a
> few days of these messages the filesystem got shut down due to
> excessive? corruption.
> At that point we tried to reboot normally, but ended up with having to
> do a hard reset of the server.
> It is not clear to us either why the corruption happened in the first
> place either. The underlying raid has been in optimal state the whole
> time
> 

Ok, so corruption was the first problem. If the filesystem shutdown with
something other than a log I/O error, chances are the log was flushed at
that time. It is strange that log records end up corrupted, though not
terribly out of the ordinary for the mount to ultimately fail if
recovery stumbled over existing on-disk corruption, for instance.
An xfs_repair was probably a foregone conclusion given the corruption
started on disk, anyways.

Brian

> >
> > That aside, I think the best course of action is to run 'xfs_repair -L'
> > on the fs. I ran a v4.12 version against the metadump image and it
> > successfully repaired the fs. I've attached the repair output for
> > reference, but I would recommend to first restore your metadump to a
> > temporary location, attempt to repair that and examine the results
> > before repairing the original fs. Note that the metadump will not have
> > any file content, but will represent which files might be cleared, moved
> > to lost+found, etc.
> Ok. Thanks for looking into it. We'll proceed with the suggested
> course of action.
> 
> ingard
> >
> > Brian
> >
> >> >
> >> > Brian
> >> >
> >> >> And the repair -n output :
> >> >> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
> >> >>
> >> >> kind regards
> >> >> ingard
> >> >>
> >> >> >
> >> >> >> Thanks-
> >> >> >> Bill
> >> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> > Excerpt from kern.log:
> >> >> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> >> >> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> >> >> >> > inconsistent.
> >> >> >> >
> >> >> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> >> >> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> >> >> >> > [xfs], xfs_inode block 0x81c9c210
> >> >> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> >> >> >> > (sdd1): Unmount and run xfs_repair
> >> >> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> >> >> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
> >> >> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> >> >> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> >> >> >> > ?.3T[U..|....QGA
> >> >> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> >> >> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> >> >> >> > ....\..z...].r..
> >> >> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> >> >> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> >> >> >> > ..:v.. ...5..6..
> >> >> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> >> >> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> >> >> >> > ..Bu.P...,-..-..
> >> >> >> >
> >> >> >> > kind regards
> >> >> >> > ingard
> >> >> >> > --
> >> >> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> >> > the body of a message to majordomo@vger.kernel.org
> >> >> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >> >> --
> >> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Ingard -]

On Thu, Aug 31, 2017 at 12:20 PM, Brian Foster <bfoster@redhat.com> wrote:
> On Thu, Aug 31, 2017 at 09:27:52AM +0200, Ingard - wrote:
>> On Wed, Aug 30, 2017 at 4:58 PM, Brian Foster <bfoster@redhat.com> wrote:
>> > On Mon, Aug 21, 2017 at 10:24:32PM +0200, Ingard - wrote:
>> >> On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
>> >> > On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
>> >> >> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
>> >> >> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
>> >> >> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
>> >> >> >> > After a server crash we've encountered a corrupt xfs filesystem. When
>> >> >> >> > trying to mount said filesystem normally the system hangs.
>> >> >> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
>> >> >> >> > xfsprogs 3.1.9
>> >> >> >> >
>> >> >> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
>> >> >> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
>> >> >> >> > replay the log) normally.
>> >> >> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
>> >> >> >> > xfs_repair -L without trying everything we can first. The filesystem
>> >> >> >> > is browsable albeit a few paths which gives an error : "Structure
>> >> >> >> > needs cleaning"
>> >> >> >> >
>> >> >> >> > Does anyone have any advice as to how we might recover/repair the
>> >> >> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
>> >> >> >> > forward?
>> >> >> >>
>> >> >> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
>> >> >> >> would be made)?
>> >> >> >>
>> >> >> >
>> >> >> > An xfs_metadump of the fs might be useful as well. Then we can see if we
>> >> >> > can reproduce the mount hang on latest kernels and if so, potentially
>> >> >> > try and root cause it.
>> >> >> >
>> >> >> > Brian
>> >> >>
>> >> >> Here is a link for the metadump :
>> >> >> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
>> >> >
>> >> > This points to a 29GB image file, apparently uncompressed..? Could you
>> >> > upload a compressed file? Thanks.
>> >>
>> >> Hi. Sorry about that. Didnt realize the output would be compressable.
>> >> Here is a link to the compressed tgz (6G)
>> >> https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae
>> >>
>> >
>> > I finally played around with this image a bit. Note that mount does not
>> > hang on latest kernels. Instead, log recovery emits a torn write message
>> > due to a bad crc at the head of the log and then ultimately fails due to
>> > a bad crc at the tail of the log. I ran a couple experiments to skip the
>> > bad crc records and/or to completely ignore all bad crc's and both still
>> > either fail to mount (due to other corruption) or continue to show
>> > corruption in the recovered fs.
>> >
>> > It's not clear to me what would have caused this corruption or log
>> > state. Have you encountered any corruption before? If not, is this kind
>> > of crash or unclean shutdown of the server an uncommon event?
>> We failed to notice the log messages of corrupt fs at first. After a
>> few days of these messages the filesystem got shut down due to
>> excessive? corruption.
>> At that point we tried to reboot normally, but ended up with having to
>> do a hard reset of the server.
>> It is not clear to us either why the corruption happened in the first
>> place either. The underlying raid has been in optimal state the whole
>> time
>>
>
> Ok, so corruption was the first problem. If the filesystem shutdown with
> something other than a log I/O error, chances are the log was flushed at
> that time. It is strange that log records end up corrupted, though not
> terribly out of the ordinary for the mount to ultimately fail if
> recovery stumbled over existing on-disk corruption, for instance.
> An xfs_repair was probably a foregone conclusion given the corruption
> started on disk, anyways.

Out of curiosity, how long did the xfs_mdrestore command run ? I'm
pushing 20ish hours now and noticed the following in kern.log :
2017-09-01T08:47:23.414139+02:00 dn-238 kernel: [1278740.983304] XFS:
xfs_mdrestore(5176) possible memory allocation deadlock size 37136 in
kmem_alloc (mode:0x2400240)

ingard

>
> Brian
>
>> >
>> > That aside, I think the best course of action is to run 'xfs_repair -L'
>> > on the fs. I ran a v4.12 version against the metadump image and it
>> > successfully repaired the fs. I've attached the repair output for
>> > reference, but I would recommend to first restore your metadump to a
>> > temporary location, attempt to repair that and examine the results
>> > before repairing the original fs. Note that the metadump will not have
>> > any file content, but will represent which files might be cleared, moved
>> > to lost+found, etc.
>> Ok. Thanks for looking into it. We'll proceed with the suggested
>> course of action.
>>
>> ingard
>> >
>> > Brian
>> >
>> >> >
>> >> > Brian
>> >> >
>> >> >> And the repair -n output :
>> >> >> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
>> >> >>
>> >> >> kind regards
>> >> >> ingard
>> >> >>
>> >> >> >
>> >> >> >> Thanks-
>> >> >> >> Bill
>> >> >> >>
>> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Excerpt from kern.log:
>> >> >> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
>> >> >> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
>> >> >> >> > inconsistent.
>> >> >> >> >
>> >> >> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
>> >> >> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
>> >> >> >> > [xfs], xfs_inode block 0x81c9c210
>> >> >> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
>> >> >> >> > (sdd1): Unmount and run xfs_repair
>> >> >> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
>> >> >> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
>> >> >> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
>> >> >> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
>> >> >> >> > ?.3T[U..|....QGA
>> >> >> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
>> >> >> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
>> >> >> >> > ....\..z...].r..
>> >> >> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
>> >> >> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
>> >> >> >> > ..:v.. ...5..6..
>> >> >> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
>> >> >> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
>> >> >> >> > ..Bu.P...,-..-..
>> >> >> >> >
>> >> >> >> > kind regards
>> >> >> >> > ingard
>> >> >> >> > --
>> >> >> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> >> >> > the body of a message to majordomo@vger.kernel.org
>> >> >> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> >> >> --
>> >> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> >> >> the body of a message to majordomo@vger.kernel.org
>> >> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> >> --
>> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> >> the body of a message to majordomo@vger.kernel.org
>> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Brian Foster]

On Fri, Sep 01, 2017 at 08:48:03AM +0200, Ingard - wrote:
> On Thu, Aug 31, 2017 at 12:20 PM, Brian Foster <bfoster@redhat.com> wrote:
> > On Thu, Aug 31, 2017 at 09:27:52AM +0200, Ingard - wrote:
> >> On Wed, Aug 30, 2017 at 4:58 PM, Brian Foster <bfoster@redhat.com> wrote:
> >> > On Mon, Aug 21, 2017 at 10:24:32PM +0200, Ingard - wrote:
> >> >> On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
> >> >> > On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
> >> >> >> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
> >> >> >> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
> >> >> >> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> >> >> >> >> > After a server crash we've encountered a corrupt xfs filesystem. When
> >> >> >> >> > trying to mount said filesystem normally the system hangs.
> >> >> >> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
> >> >> >> >> > xfsprogs 3.1.9
> >> >> >> >> >
> >> >> >> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> >> >> >> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
> >> >> >> >> > replay the log) normally.
> >> >> >> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
> >> >> >> >> > xfs_repair -L without trying everything we can first. The filesystem
> >> >> >> >> > is browsable albeit a few paths which gives an error : "Structure
> >> >> >> >> > needs cleaning"
> >> >> >> >> >
> >> >> >> >> > Does anyone have any advice as to how we might recover/repair the
> >> >> >> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
> >> >> >> >> > forward?
> >> >> >> >>
> >> >> >> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
> >> >> >> >> would be made)?
> >> >> >> >>
> >> >> >> >
> >> >> >> > An xfs_metadump of the fs might be useful as well. Then we can see if we
> >> >> >> > can reproduce the mount hang on latest kernels and if so, potentially
> >> >> >> > try and root cause it.
> >> >> >> >
> >> >> >> > Brian
> >> >> >>
> >> >> >> Here is a link for the metadump :
> >> >> >> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
> >> >> >
> >> >> > This points to a 29GB image file, apparently uncompressed..? Could you
> >> >> > upload a compressed file? Thanks.
> >> >>
> >> >> Hi. Sorry about that. Didnt realize the output would be compressable.
> >> >> Here is a link to the compressed tgz (6G)
> >> >> https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae
> >> >>
> >> >
> >> > I finally played around with this image a bit. Note that mount does not
> >> > hang on latest kernels. Instead, log recovery emits a torn write message
> >> > due to a bad crc at the head of the log and then ultimately fails due to
> >> > a bad crc at the tail of the log. I ran a couple experiments to skip the
> >> > bad crc records and/or to completely ignore all bad crc's and both still
> >> > either fail to mount (due to other corruption) or continue to show
> >> > corruption in the recovered fs.
> >> >
> >> > It's not clear to me what would have caused this corruption or log
> >> > state. Have you encountered any corruption before? If not, is this kind
> >> > of crash or unclean shutdown of the server an uncommon event?
> >> We failed to notice the log messages of corrupt fs at first. After a
> >> few days of these messages the filesystem got shut down due to
> >> excessive? corruption.
> >> At that point we tried to reboot normally, but ended up with having to
> >> do a hard reset of the server.
> >> It is not clear to us either why the corruption happened in the first
> >> place either. The underlying raid has been in optimal state the whole
> >> time
> >>
> >
> > Ok, so corruption was the first problem. If the filesystem shutdown with
> > something other than a log I/O error, chances are the log was flushed at
> > that time. It is strange that log records end up corrupted, though not
> > terribly out of the ordinary for the mount to ultimately fail if
> > recovery stumbled over existing on-disk corruption, for instance.
> > An xfs_repair was probably a foregone conclusion given the corruption
> > started on disk, anyways.
> 
> Out of curiosity, how long did the xfs_mdrestore command run ? I'm
> pushing 20ish hours now and noticed the following in kern.log :
> 2017-09-01T08:47:23.414139+02:00 dn-238 kernel: [1278740.983304] XFS:
> xfs_mdrestore(5176) possible memory allocation deadlock size 37136 in
> kmem_alloc (mode:0x2400240)
> 

Heh. It certainly wasn't quick since it had to restore ~30GB or so of
metadata, but it didn't take that long. If I had to guess, I'd say it
restored within an hour.

It seems like you're running into the in-core extent list problem, which
causes pain for highly sparse or fragmented files because we store the
entire extent list in memory. An fiemap of the restored image I have
lying around shows over 1.5m extents. :/ You may need a box with more
RAM (I had 32GB) or otherwise find another large enough block device to
use the metadump. :/ If you had to bypass that step, you could at least
run 'xfs_repair -n' on the original fs to see whether repair runs to
completion in your environment.

Brian

> ingard
> 
> >
> > Brian
> >
> >> >
> >> > That aside, I think the best course of action is to run 'xfs_repair -L'
> >> > on the fs. I ran a v4.12 version against the metadump image and it
> >> > successfully repaired the fs. I've attached the repair output for
> >> > reference, but I would recommend to first restore your metadump to a
> >> > temporary location, attempt to repair that and examine the results
> >> > before repairing the original fs. Note that the metadump will not have
> >> > any file content, but will represent which files might be cleared, moved
> >> > to lost+found, etc.
> >> Ok. Thanks for looking into it. We'll proceed with the suggested
> >> course of action.
> >>
> >> ingard
> >> >
> >> > Brian
> >> >
> >> >> >
> >> >> > Brian
> >> >> >
> >> >> >> And the repair -n output :
> >> >> >> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
> >> >> >>
> >> >> >> kind regards
> >> >> >> ingard
> >> >> >>
> >> >> >> >
> >> >> >> >> Thanks-
> >> >> >> >> Bill
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Excerpt from kern.log:
> >> >> >> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> >> >> >> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> >> >> >> >> > inconsistent.
> >> >> >> >> >
> >> >> >> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> >> >> >> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> >> >> >> >> > [xfs], xfs_inode block 0x81c9c210
> >> >> >> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> >> >> >> >> > (sdd1): Unmount and run xfs_repair
> >> >> >> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> >> >> >> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
> >> >> >> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> >> >> >> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> >> >> >> >> > ?.3T[U..|....QGA
> >> >> >> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> >> >> >> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> >> >> >> >> > ....\..z...].r..
> >> >> >> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> >> >> >> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> >> >> >> >> > ..:v.. ...5..6..
> >> >> >> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> >> >> >> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> >> >> >> >> > ..Bu.P...,-..-..
> >> >> >> >> >
> >> >> >> >> > kind regards
> >> >> >> >> > ingard
> >> >> >> >> > --
> >> >> >> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> >> >> > the body of a message to majordomo@vger.kernel.org
> >> >> >> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >> >> >> --
> >> >> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >> >> --
> >> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Darrick J. Wong]

On Fri, Sep 01, 2017 at 07:33:07AM -0400, Brian Foster wrote:
> On Fri, Sep 01, 2017 at 08:48:03AM +0200, Ingard - wrote:
> > On Thu, Aug 31, 2017 at 12:20 PM, Brian Foster <bfoster@redhat.com> wrote:
> > > On Thu, Aug 31, 2017 at 09:27:52AM +0200, Ingard - wrote:
> > >> On Wed, Aug 30, 2017 at 4:58 PM, Brian Foster <bfoster@redhat.com> wrote:
> > >> > On Mon, Aug 21, 2017 at 10:24:32PM +0200, Ingard - wrote:
> > >> >> On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
> > >> >> > On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
> > >> >> >> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
> > >> >> >> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
> > >> >> >> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> > >> >> >> >> > After a server crash we've encountered a corrupt xfs filesystem. When
> > >> >> >> >> > trying to mount said filesystem normally the system hangs.
> > >> >> >> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
> > >> >> >> >> > xfsprogs 3.1.9
> > >> >> >> >> >
> > >> >> >> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> > >> >> >> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
> > >> >> >> >> > replay the log) normally.
> > >> >> >> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
> > >> >> >> >> > xfs_repair -L without trying everything we can first. The filesystem
> > >> >> >> >> > is browsable albeit a few paths which gives an error : "Structure
> > >> >> >> >> > needs cleaning"
> > >> >> >> >> >
> > >> >> >> >> > Does anyone have any advice as to how we might recover/repair the
> > >> >> >> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
> > >> >> >> >> > forward?
> > >> >> >> >>
> > >> >> >> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
> > >> >> >> >> would be made)?
> > >> >> >> >>
> > >> >> >> >
> > >> >> >> > An xfs_metadump of the fs might be useful as well. Then we can see if we
> > >> >> >> > can reproduce the mount hang on latest kernels and if so, potentially
> > >> >> >> > try and root cause it.
> > >> >> >> >
> > >> >> >> > Brian
> > >> >> >>
> > >> >> >> Here is a link for the metadump :
> > >> >> >> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
> > >> >> >
> > >> >> > This points to a 29GB image file, apparently uncompressed..? Could you
> > >> >> > upload a compressed file? Thanks.
> > >> >>
> > >> >> Hi. Sorry about that. Didnt realize the output would be compressable.
> > >> >> Here is a link to the compressed tgz (6G)
> > >> >> https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae
> > >> >>
> > >> >
> > >> > I finally played around with this image a bit. Note that mount does not
> > >> > hang on latest kernels. Instead, log recovery emits a torn write message
> > >> > due to a bad crc at the head of the log and then ultimately fails due to
> > >> > a bad crc at the tail of the log. I ran a couple experiments to skip the
> > >> > bad crc records and/or to completely ignore all bad crc's and both still
> > >> > either fail to mount (due to other corruption) or continue to show
> > >> > corruption in the recovered fs.
> > >> >
> > >> > It's not clear to me what would have caused this corruption or log
> > >> > state. Have you encountered any corruption before? If not, is this kind
> > >> > of crash or unclean shutdown of the server an uncommon event?
> > >> We failed to notice the log messages of corrupt fs at first. After a
> > >> few days of these messages the filesystem got shut down due to
> > >> excessive? corruption.
> > >> At that point we tried to reboot normally, but ended up with having to
> > >> do a hard reset of the server.
> > >> It is not clear to us either why the corruption happened in the first
> > >> place either. The underlying raid has been in optimal state the whole
> > >> time
> > >>
> > >
> > > Ok, so corruption was the first problem. If the filesystem shutdown with
> > > something other than a log I/O error, chances are the log was flushed at
> > > that time. It is strange that log records end up corrupted, though not
> > > terribly out of the ordinary for the mount to ultimately fail if
> > > recovery stumbled over existing on-disk corruption, for instance.
> > > An xfs_repair was probably a foregone conclusion given the corruption
> > > started on disk, anyways.
> > 
> > Out of curiosity, how long did the xfs_mdrestore command run ? I'm
> > pushing 20ish hours now and noticed the following in kern.log :
> > 2017-09-01T08:47:23.414139+02:00 dn-238 kernel: [1278740.983304] XFS:
> > xfs_mdrestore(5176) possible memory allocation deadlock size 37136 in
> > kmem_alloc (mode:0x2400240)
> > 
> 
> Heh. It certainly wasn't quick since it had to restore ~30GB or so of
> metadata, but it didn't take that long. If I had to guess, I'd say it
> restored within an hour.
> 
> It seems like you're running into the in-core extent list problem, which
> causes pain for highly sparse or fragmented files because we store the
> entire extent list in memory. An fiemap of the restored image I have
> lying around shows over 1.5m extents. :/ You may need a box with more
> RAM (I had 32GB) or otherwise find another large enough block device to
> use the metadump. :/ If you had to bypass that step, you could at least
> run 'xfs_repair -n' on the original fs to see whether repair runs to
> completion in your environment.

/me wonders if it'd help if mdrestore had a command line arg to
fallocate the target file beforehand (lots of wasted space but fewer
extents) or set an extent size hint (only useful if metadata isn't
fragmented) but yes we should fix the incore extent cache memory usage
problem.

--D

> 
> Brian
> 
> > ingard
> > 
> > >
> > > Brian
> > >
> > >> >
> > >> > That aside, I think the best course of action is to run 'xfs_repair -L'
> > >> > on the fs. I ran a v4.12 version against the metadump image and it
> > >> > successfully repaired the fs. I've attached the repair output for
> > >> > reference, but I would recommend to first restore your metadump to a
> > >> > temporary location, attempt to repair that and examine the results
> > >> > before repairing the original fs. Note that the metadump will not have
> > >> > any file content, but will represent which files might be cleared, moved
> > >> > to lost+found, etc.
> > >> Ok. Thanks for looking into it. We'll proceed with the suggested
> > >> course of action.
> > >>
> > >> ingard
> > >> >
> > >> > Brian
> > >> >
> > >> >> >
> > >> >> > Brian
> > >> >> >
> > >> >> >> And the repair -n output :
> > >> >> >> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
> > >> >> >>
> > >> >> >> kind regards
> > >> >> >> ingard
> > >> >> >>
> > >> >> >> >
> > >> >> >> >> Thanks-
> > >> >> >> >> Bill
> > >> >> >> >>
> > >> >> >> >>
> > >> >> >> >> >
> > >> >> >> >> >
> > >> >> >> >> > Excerpt from kern.log:
> > >> >> >> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> > >> >> >> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> > >> >> >> >> > inconsistent.
> > >> >> >> >> >
> > >> >> >> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> > >> >> >> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> > >> >> >> >> > [xfs], xfs_inode block 0x81c9c210
> > >> >> >> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> > >> >> >> >> > (sdd1): Unmount and run xfs_repair
> > >> >> >> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> > >> >> >> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
> > >> >> >> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> > >> >> >> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> > >> >> >> >> > ?.3T[U..|....QGA
> > >> >> >> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> > >> >> >> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> > >> >> >> >> > ....\..z...].r..
> > >> >> >> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> > >> >> >> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> > >> >> >> >> > ..:v.. ...5..6..
> > >> >> >> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> > >> >> >> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> > >> >> >> >> > ..Bu.P...,-..-..
> > >> >> >> >> >
> > >> >> >> >> > kind regards
> > >> >> >> >> > ingard
> > >> >> >> >> > --
> > >> >> >> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > >> >> >> >> > the body of a message to majordomo@vger.kernel.org
> > >> >> >> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >> >> >> >> --
> > >> >> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > >> >> >> >> the body of a message to majordomo@vger.kernel.org
> > >> >> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >> >> >> --
> > >> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > >> >> >> the body of a message to majordomo@vger.kernel.org
> > >> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >> >> --
> > >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > >> >> the body of a message to majordomo@vger.kernel.org
> > >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > >> the body of a message to majordomo@vger.kernel.org
> > >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Brian Foster]

On Fri, Sep 01, 2017 at 08:11:19AM -0700, Darrick J. Wong wrote:
> On Fri, Sep 01, 2017 at 07:33:07AM -0400, Brian Foster wrote:
> > On Fri, Sep 01, 2017 at 08:48:03AM +0200, Ingard - wrote:
> > > On Thu, Aug 31, 2017 at 12:20 PM, Brian Foster <bfoster@redhat.com> wrote:
> > > > On Thu, Aug 31, 2017 at 09:27:52AM +0200, Ingard - wrote:
> > > >> On Wed, Aug 30, 2017 at 4:58 PM, Brian Foster <bfoster@redhat.com> wrote:
> > > >> > On Mon, Aug 21, 2017 at 10:24:32PM +0200, Ingard - wrote:
> > > >> >> On Mon, Aug 21, 2017 at 5:51 PM, Brian Foster <bfoster@redhat.com> wrote:
> > > >> >> > On Mon, Aug 21, 2017 at 02:08:43PM +0200, Ingard - wrote:
> > > >> >> >> On Fri, Aug 18, 2017 at 2:17 PM, Brian Foster <bfoster@redhat.com> wrote:
> > > >> >> >> > On Fri, Aug 18, 2017 at 07:02:24AM -0500, Bill O'Donnell wrote:
> > > >> >> >> >> On Fri, Aug 18, 2017 at 01:56:31PM +0200, Ingard - wrote:
> > > >> >> >> >> > After a server crash we've encountered a corrupt xfs filesystem. When
> > > >> >> >> >> > trying to mount said filesystem normally the system hangs.
> > > >> >> >> >> > This was initially on a ubuntu trusty server with 3.13 kernel with
> > > >> >> >> >> > xfsprogs 3.1.9
> > > >> >> >> >> >
> > > >> >> >> >> > We've installed a newer kernel (4.4.0-92) and compiled xfsprogs v
> > > >> >> >> >> > 4.12.0 from source. We're still not able to mount the filesystem (and
> > > >> >> >> >> > replay the log) normally.
> > > >> >> >> >> > We are able to mount it -o ro,norecovery, but we're reluctant to do
> > > >> >> >> >> > xfs_repair -L without trying everything we can first. The filesystem
> > > >> >> >> >> > is browsable albeit a few paths which gives an error : "Structure
> > > >> >> >> >> > needs cleaning"
> > > >> >> >> >> >
> > > >> >> >> >> > Does anyone have any advice as to how we might recover/repair the
> > > >> >> >> >> > corrupt log so we can replay it? Or is xfs_repair -L the only way
> > > >> >> >> >> > forward?
> > > >> >> >> >>
> > > >> >> >> >> Can you try xfs_repair -n (only scans the fs and reports what repairs
> > > >> >> >> >> would be made)?
> > > >> >> >> >>
> > > >> >> >> >
> > > >> >> >> > An xfs_metadump of the fs might be useful as well. Then we can see if we
> > > >> >> >> > can reproduce the mount hang on latest kernels and if so, potentially
> > > >> >> >> > try and root cause it.
> > > >> >> >> >
> > > >> >> >> > Brian
> > > >> >> >>
> > > >> >> >> Here is a link for the metadump :
> > > >> >> >> https://www.jottacloud.com/p/ingardme/95ec2e45ba80431d962345981d38bdff
> > > >> >> >
> > > >> >> > This points to a 29GB image file, apparently uncompressed..? Could you
> > > >> >> > upload a compressed file? Thanks.
> > > >> >>
> > > >> >> Hi. Sorry about that. Didnt realize the output would be compressable.
> > > >> >> Here is a link to the compressed tgz (6G)
> > > >> >> https://www.jottacloud.com/p/ingardme/cac6939649e14b98b928647f5222a2ae
> > > >> >>
> > > >> >
> > > >> > I finally played around with this image a bit. Note that mount does not
> > > >> > hang on latest kernels. Instead, log recovery emits a torn write message
> > > >> > due to a bad crc at the head of the log and then ultimately fails due to
> > > >> > a bad crc at the tail of the log. I ran a couple experiments to skip the
> > > >> > bad crc records and/or to completely ignore all bad crc's and both still
> > > >> > either fail to mount (due to other corruption) or continue to show
> > > >> > corruption in the recovered fs.
> > > >> >
> > > >> > It's not clear to me what would have caused this corruption or log
> > > >> > state. Have you encountered any corruption before? If not, is this kind
> > > >> > of crash or unclean shutdown of the server an uncommon event?
> > > >> We failed to notice the log messages of corrupt fs at first. After a
> > > >> few days of these messages the filesystem got shut down due to
> > > >> excessive? corruption.
> > > >> At that point we tried to reboot normally, but ended up with having to
> > > >> do a hard reset of the server.
> > > >> It is not clear to us either why the corruption happened in the first
> > > >> place either. The underlying raid has been in optimal state the whole
> > > >> time
> > > >>
> > > >
> > > > Ok, so corruption was the first problem. If the filesystem shutdown with
> > > > something other than a log I/O error, chances are the log was flushed at
> > > > that time. It is strange that log records end up corrupted, though not
> > > > terribly out of the ordinary for the mount to ultimately fail if
> > > > recovery stumbled over existing on-disk corruption, for instance.
> > > > An xfs_repair was probably a foregone conclusion given the corruption
> > > > started on disk, anyways.
> > > 
> > > Out of curiosity, how long did the xfs_mdrestore command run ? I'm
> > > pushing 20ish hours now and noticed the following in kern.log :
> > > 2017-09-01T08:47:23.414139+02:00 dn-238 kernel: [1278740.983304] XFS:
> > > xfs_mdrestore(5176) possible memory allocation deadlock size 37136 in
> > > kmem_alloc (mode:0x2400240)
> > > 
> > 
> > Heh. It certainly wasn't quick since it had to restore ~30GB or so of
> > metadata, but it didn't take that long. If I had to guess, I'd say it
> > restored within an hour.
> > 
> > It seems like you're running into the in-core extent list problem, which
> > causes pain for highly sparse or fragmented files because we store the
> > entire extent list in memory. An fiemap of the restored image I have
> > lying around shows over 1.5m extents. :/ You may need a box with more
> > RAM (I had 32GB) or otherwise find another large enough block device to
> > use the metadump. :/ If you had to bypass that step, you could at least
> > run 'xfs_repair -n' on the original fs to see whether repair runs to
> > completion in your environment.
> 
> /me wonders if it'd help if mdrestore had a command line arg to
> fallocate the target file beforehand (lots of wasted space but fewer
> extents) or set an extent size hint (only useful if metadata isn't
> fragmented) but yes we should fix the incore extent cache memory usage
> problem.
> 

That's an interesting idea. I suppose one could set a really large
extent size hint on the directory to try and achieve a similar outcome
as an initial fallocate.

Eh, though looking at the restored image, the original fs is 88TB (with
only 4TB free space), so that might not be so useful in this particular
case. Heh, I have another metadump lying around of a significantly
smaller fs with what looks like about double the amount of metadata
based on the metadump size and extent count. :P There may be a middle
ground here where an fallocate could be useful.

Brian

> --D
> 
> > 
> > Brian
> > 
> > > ingard
> > > 
> > > >
> > > > Brian
> > > >
> > > >> >
> > > >> > That aside, I think the best course of action is to run 'xfs_repair -L'
> > > >> > on the fs. I ran a v4.12 version against the metadump image and it
> > > >> > successfully repaired the fs. I've attached the repair output for
> > > >> > reference, but I would recommend to first restore your metadump to a
> > > >> > temporary location, attempt to repair that and examine the results
> > > >> > before repairing the original fs. Note that the metadump will not have
> > > >> > any file content, but will represent which files might be cleared, moved
> > > >> > to lost+found, etc.
> > > >> Ok. Thanks for looking into it. We'll proceed with the suggested
> > > >> course of action.
> > > >>
> > > >> ingard
> > > >> >
> > > >> > Brian
> > > >> >
> > > >> >> >
> > > >> >> > Brian
> > > >> >> >
> > > >> >> >> And the repair -n output :
> > > >> >> >> https://www.jottacloud.com/p/ingardme/0205c6ca6f7e495ebcda5f255b96f63d
> > > >> >> >>
> > > >> >> >> kind regards
> > > >> >> >> ingard
> > > >> >> >>
> > > >> >> >> >
> > > >> >> >> >> Thanks-
> > > >> >> >> >> Bill
> > > >> >> >> >>
> > > >> >> >> >>
> > > >> >> >> >> >
> > > >> >> >> >> >
> > > >> >> >> >> > Excerpt from kern.log:
> > > >> >> >> >> > 2017-08-17T13:40:41.122121+02:00 dn-238 kernel: [  294.300347] XFS
> > > >> >> >> >> > (sdd1): Mounting V4 filesystem in no-recovery mode. Filesystem will be
> > > >> >> >> >> > inconsistent.
> > > >> >> >> >> >
> > > >> >> >> >> > 2017-08-17T17:04:54.794194+02:00 dn-238 kernel: [12548.400260] XFS
> > > >> >> >> >> > (sdd1): Metadata corruption detected at xfs_inode_buf_verify+0x6f/0xd0
> > > >> >> >> >> > [xfs], xfs_inode block 0x81c9c210
> > > >> >> >> >> > 2017-08-17T17:04:54.794216+02:00 dn-238 kernel: [12548.400342] XFS
> > > >> >> >> >> > (sdd1): Unmount and run xfs_repair
> > > >> >> >> >> > 2017-08-17T17:04:54.794218+02:00 dn-238 kernel: [12548.400374] XFS
> > > >> >> >> >> > (sdd1): First 64 bytes of corrupted metadata buffer:
> > > >> >> >> >> > 2017-08-17T17:04:54.794220+02:00 dn-238 kernel: [12548.400418]
> > > >> >> >> >> > ffff880171fff000: 3f 1a 33 54 5b 55 85 0b 7c f5 c6 d5 cf 51 47 41
> > > >> >> >> >> > ?.3T[U..|....QGA
> > > >> >> >> >> > 2017-08-17T17:04:54.794222+02:00 dn-238 kernel: [12548.400473]
> > > >> >> >> >> > ffff880171fff010: 97 ba ba 03 5c e4 02 7a e6 bc fb 5d f1 72 db c1
> > > >> >> >> >> > ....\..z...].r..
> > > >> >> >> >> > 2017-08-17T17:04:54.794223+02:00 dn-238 kernel: [12548.400527]
> > > >> >> >> >> > ffff880171fff020: c8 ad 3a 76 c7 e4 20 92 88 a2 35 0c 1f 36 cf b5
> > > >> >> >> >> > ..:v.. ...5..6..
> > > >> >> >> >> > 2017-08-17T17:04:54.794226+02:00 dn-238 kernel: [12548.400581]
> > > >> >> >> >> > ffff880171fff030: 8a bc 42 75 86 50 a0 a2 be 2c 2d 99 96 2d e1 ee
> > > >> >> >> >> > ..Bu.P...,-..-..
> > > >> >> >> >> >
> > > >> >> >> >> > kind regards
> > > >> >> >> >> > ingard
> > > >> >> >> >> > --
> > > >> >> >> >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > > >> >> >> >> > the body of a message to majordomo@vger.kernel.org
> > > >> >> >> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >> >> >> >> --
> > > >> >> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > > >> >> >> >> the body of a message to majordomo@vger.kernel.org
> > > >> >> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >> >> >> --
> > > >> >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > > >> >> >> the body of a message to majordomo@vger.kernel.org
> > > >> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >> >> --
> > > >> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > > >> >> the body of a message to majordomo@vger.kernel.org
> > > >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >> --
> > > >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > > >> the body of a message to majordomo@vger.kernel.org
> > > >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt xfs log

[Ingard -]

[-- Attachment #1: Type: text/plain, Size: 35 bytes --]

output from xfs_repair -n attached

[-- Attachment #2: xfs_repair.txt --]
[-- Type: text/plain, Size: 44810 bytes --]

$ xfs_repair -n /dev/sdd1
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
ignored because the -n option was used.  Expect spurious inconsistencies
which may be resolved by first mounting the filesystem to replay the log.
        - scan filesystem freespace and inode maps...
Metadata corruption detected at xfs_allocbt block 0xb000cbff8/0x1000
btree block 22/105855 is suspect, error -117
bad magic # 0x530939a7 in btbno block 22/105855
agf_btreeblks 390, counted 389 in ag 22
agi unlinked bucket 60 is 298941500 in ag 14 (inode=60428483644)
Metadata corruption detected at xfs_allocbt block 0x29000059e0/0x1000
btree block 82/8124 is suspect, error -117
bad magic # 0xd36c598e in btbno block 82/8124
agf_btreeblks 403, counted 402 in ag 82
sb_icount 64, counted 79473472
sb_ifree 61, counted 2941
sb_fdblocks 23439345295, counted 1093782972
        - 14:09:48: scanning filesystem freespace - 88 of 88 allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - 14:09:48: scanning agi unlinked lists - 88 of 88 allocation groups done
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 15
        - agno = 60
        - agno = 45
        - agno = 75
        - agno = 30
^C
14:09:54 [root@dn-238[DP]:/mnt] $
14:09:54 [root@dn-238[DP]:/mnt] $
14:09:54 [root@dn-238[DP]:/mnt] $
14:09:54 [root@dn-238[DP]:/mnt] $
14:09:55 [root@dn-238[DP]:/mnt] $
14:09:55 [root@dn-238[DP]:/mnt] $ xfs_repair -n /dev/sdd1 |tee xfsrepair.txt
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
ignored because the -n option was used.  Expect spurious inconsistencies
which may be resolved by first mounting the filesystem to replay the log.
        - scan filesystem freespace and inode maps...
Metadata corruption detected at xfs_allocbt block 0xb000cbff8/0x1000
btree block 22/105855 is suspect, error -117
bad magic # 0x530939a7 in btbno block 22/105855
agf_btreeblks 390, counted 389 in ag 22
agi unlinked bucket 60 is 298941500 in ag 14 (inode=60428483644)
Metadata corruption detected at xfs_allocbt block 0x29000059e0/0x1000
btree block 82/8124 is suspect, error -117
bad magic # 0xd36c598e in btbno block 82/8124
agf_btreeblks 403, counted 402 in ag 82
sb_icount 64, counted 79473472
sb_ifree 61, counted 2941
sb_fdblocks 23439345295, counted 1093782972
        - 14:10:30: scanning filesystem freespace - 88 of 88 allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - 14:10:30: scanning agi unlinked lists - 88 of 88 allocation groups done
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 15
        - agno = 30
        - agno = 45
        - agno = 60
        - agno = 75
        - agno = 31
        - agno = 46
        - agno = 61
        - agno = 16
        - agno = 76
        - agno = 1
        - agno = 47
        - agno = 32
        - agno = 17
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
        - agno = 62
        - agno = 77
bad magic number 0x3f1a on inode 4354967584
bad version number 0x5b on inode 4354967584
bad (negative) size -4743366169855139346 on inode 4354967584
bad magic number 0xf749 on inode 4354967585
bad version number 0x19 on inode 4354967585
bad (negative) size -4925401835952624351 on inode 4354967585
bad magic number 0xefc6 on inode 4354967586
bad version number 0xffffffc0 on inode 4354967586
bad (negative) size -6663211212969851433 on inode 4354967586
bad magic number 0x93db on inode 4354967587
bad version number 0x5f on inode 4354967587
bad inode format in inode 4354967587
bad magic number 0x54da on inode 4354967588
bad version number 0xffffff86 on inode 4354967588
bad inode format in inode 4354967588
bad magic number 0x5893 on inode 4354967589
bad version number 0x0 on inode 4354967589
bad inode format in inode 4354967589
bad magic number 0x309f on inode 4354967590
bad version number 0xffffffc2 on inode 4354967590
bad (negative) size -7620939567173939239 on inode 4354967590
bad magic number 0x9fce on inode 4354967591
bad version number 0x38 on inode 4354967591
bad inode format in inode 4354967591
bad magic number 0xb59f on inode 4354967592
bad version number 0x4 on inode 4354967592
bad inode format in inode 4354967592
bad magic number 0x53b4 on inode 4354967593
bad version number 0xfffffff4 on inode 4354967593
bad (negative) size -6181984635646857242 on inode 4354967593
bad magic number 0x7c77 on inode 4354967594
bad version number 0x3e on inode 4354967594
bad (negative) size -5208493334119488045 on inode 4354967594
bad magic number 0x6698 on inode 4354967595
bad version number 0x51 on inode 4354967595
bad inode format in inode 4354967595
bad magic number 0xe872 on inode 4354967596
bad version number 0xffffffa1 on inode 4354967596
bad (negative) size -4217771995784516902 on inode 4354967596
bad magic number 0x2c98 on inode 4354967597
bad version number 0xffffff8e on inode 4354967597
bad (negative) size -1829068493719510754 on inode 4354967597
bad magic number 0x5ead on inode 4354967598
bad version number 0xffffffa3 on inode 4354967598
bad (negative) size -3586320840011193807 on inode 4354967598
bad magic number 0xf5ac on inode 4354967599
bad version number 0x4d on inode 4354967599
bad (negative) size -7822733889952692968 on inode 4354967599
bad magic number 0x5c7e on inode 4354967600
bad version number 0x4e on inode 4354967600
bad (negative) size -8116057573661653832 on inode 4354967600
bad magic number 0x57c5 on inode 4354967601
bad version number 0xffffffe5 on inode 4354967601
bad inode format in inode 4354967601
bad magic number 0xcc0e on inode 4354967602
bad version number 0x74 on inode 4354967602
bad (negative) size -7175657235803173389 on inode 4354967602
bad magic number 0x17a3 on inode 4354967603
bad version number 0x3b on inode 4354967603
bad inode format in inode 4354967603
bad magic number 0xf085 on inode 4354967604
bad version number 0xffffffd3 on inode 4354967604
bad (negative) size -4829046689272463617 on inode 4354967604
bad magic number 0xd47a on inode 4354967605
bad version number 0xe on inode 4354967605
bad inode format in inode 4354967605
bad magic number 0xcb77 on inode 4354967606
bad version number 0xfffffff6 on inode 4354967606
bad (negative) size -4813752938159903980 on inode 4354967606
bad magic number 0xfb56 on inode 4354967607
bad version number 0x0 on inode 4354967607
bad (negative) size -5002825908862883310 on inode 4354967607
bad magic number 0xf74f on inode 4354967608
bad version number 0x31 on inode 4354967608
bad (negative) size -2804153168172737527 on inode 4354967608
bad magic number 0xa92e on inode 4354967609
bad version number 0xffffffad on inode 4354967609
bad (negative) size -3138139371176466993 on inode 4354967609
bad magic number 0x75e5 on inode 4354967610
bad version number 0x66 on inode 4354967610
bad inode format in inode 4354967610
bad magic number 0x6b8d on inode 4354967611
bad version number 0x26 on inode 4354967611
bad inode format in inode 4354967611
bad magic number 0xf803 on inode 4354967612
bad version number 0x7f on inode 4354967612
bad inode format in inode 4354967612
bad magic number 0xaa2f on inode 4354967613
bad version number 0x31 on inode 4354967613
bad (negative) size -8402135599746543486 on inode 4354967613
bad magic number 0xbd27 on inode 4354967614
bad version number 0xffffffec on inode 4354967614
bad inode format in inode 4354967614
bad magic number 0x27bd on inode 4354967615
bad version number 0x1f on inode 4354967615
bad inode format in inode 4354967615
bad magic number 0x3f1a on inode 4354967584, would reset magic number
bad version number 0x5b on inode 4354967584, would reset version number
bad (negative) size -4743366169855139346 on inode 4354967584
would have cleared inode 4354967584
bad magic number 0xf749 on inode 4354967585, would reset magic number
bad version number 0x19 on inode 4354967585, would reset version number
bad (negative) size -4925401835952624351 on inode 4354967585
would have cleared inode 4354967585
bad magic number 0xefc6 on inode 4354967586, would reset magic number
bad version number 0xffffffc0 on inode 4354967586, would reset version number
bad (negative) size -6663211212969851433 on inode 4354967586
would have cleared inode 4354967586
bad magic number 0x93db on inode 4354967587, would reset magic number
bad version number 0x5f on inode 4354967587, would reset version number
bad inode format in inode 4354967587
would have cleared inode 4354967587
bad magic number 0x54da on inode 4354967588, would reset magic number
bad version number 0xffffff86 on inode 4354967588, would reset version number
bad inode format in inode 4354967588
would have cleared inode 4354967588
bad magic number 0x5893 on inode 4354967589, would reset magic number
bad version number 0x0 on inode 4354967589, would reset version number
bad inode format in inode 4354967589
would have cleared inode 4354967589
bad magic number 0x309f on inode 4354967590, would reset magic number
bad version number 0xffffffc2 on inode 4354967590, would reset version number
bad (negative) size -7620939567173939239 on inode 4354967590
would have cleared inode 4354967590
bad magic number 0x9fce on inode 4354967591, would reset magic number
bad version number 0x38 on inode 4354967591, would reset version number
bad inode format in inode 4354967591
would have cleared inode 4354967591
bad magic number 0xb59f on inode 4354967592, would reset magic number
bad version number 0x4 on inode 4354967592, would reset version number
bad inode format in inode 4354967592
would have cleared inode 4354967592
bad magic number 0x53b4 on inode 4354967593, would reset magic number
bad version number 0xfffffff4 on inode 4354967593, would reset version number
bad (negative) size -6181984635646857242 on inode 4354967593
would have cleared inode 4354967593
bad magic number 0x7c77 on inode 4354967594, would reset magic number
bad version number 0x3e on inode 4354967594, would reset version number
bad (negative) size -5208493334119488045 on inode 4354967594
would have cleared inode 4354967594
bad magic number 0x6698 on inode 4354967595, would reset magic number
bad version number 0x51 on inode 4354967595, would reset version number
bad inode format in inode 4354967595
would have cleared inode 4354967595
bad magic number 0xe872 on inode 4354967596, would reset magic number
bad version number 0xffffffa1 on inode 4354967596, would reset version number
bad (negative) size -4217771995784516902 on inode 4354967596
would have cleared inode 4354967596
bad magic number 0x2c98 on inode 4354967597, would reset magic number
bad version number 0xffffff8e on inode 4354967597, would reset version number
bad (negative) size -1829068493719510754 on inode 4354967597
would have cleared inode 4354967597
bad magic number 0x5ead on inode 4354967598, would reset magic number
bad version number 0xffffffa3 on inode 4354967598, would reset version number
bad (negative) size -3586320840011193807 on inode 4354967598
would have cleared inode 4354967598
bad magic number 0xf5ac on inode 4354967599, would reset magic number
bad version number 0x4d on inode 4354967599, would reset version number
bad (negative) size -7822733889952692968 on inode 4354967599
would have cleared inode 4354967599
bad magic number 0x5c7e on inode 4354967600, would reset magic number
bad version number 0x4e on inode 4354967600, would reset version number
bad (negative) size -8116057573661653832 on inode 4354967600
would have cleared inode 4354967600
bad magic number 0x57c5 on inode 4354967601, would reset magic number
bad version number 0xffffffe5 on inode 4354967601, would reset version number
bad inode format in inode 4354967601
would have cleared inode 4354967601
bad magic number 0xcc0e on inode 4354967602, would reset magic number
bad version number 0x74 on inode 4354967602, would reset version number
bad (negative) size -7175657235803173389 on inode 4354967602
would have cleared inode 4354967602
bad magic number 0x17a3 on inode 4354967603, would reset magic number
bad version number 0x3b on inode 4354967603, would reset version number
bad inode format in inode 4354967603
would have cleared inode 4354967603
bad magic number 0xf085 on inode 4354967604, would reset magic number
bad version number 0xffffffd3 on inode 4354967604, would reset version number
bad (negative) size -4829046689272463617 on inode 4354967604
would have cleared inode 4354967604
bad magic number 0xd47a on inode 4354967605, would reset magic number
bad version number 0xe on inode 4354967605, would reset version number
bad inode format in inode 4354967605
would have cleared inode 4354967605
bad magic number 0xcb77 on inode 4354967606, would reset magic number
bad version number 0xfffffff6 on inode 4354967606, would reset version number
bad (negative) size -4813752938159903980 on inode 4354967606
would have cleared inode 4354967606
bad magic number 0xfb56 on inode 4354967607, would reset magic number
bad version number 0x0 on inode 4354967607, would reset version number
bad (negative) size -5002825908862883310 on inode 4354967607
would have cleared inode 4354967607
bad magic number 0xf74f on inode 4354967608, would reset magic number
bad version number 0x31 on inode 4354967608, would reset version number
bad (negative) size -2804153168172737527 on inode 4354967608
would have cleared inode 4354967608
bad magic number 0xa92e on inode 4354967609, would reset magic number
bad version number 0xffffffad on inode 4354967609, would reset version number
bad (negative) size -3138139371176466993 on inode 4354967609
would have cleared inode 4354967609
bad magic number 0x75e5 on inode 4354967610, would reset magic number
bad version number 0x66 on inode 4354967610, would reset version number
bad inode format in inode 4354967610
would have cleared inode 4354967610
bad magic number 0x6b8d on inode 4354967611, would reset magic number
bad version number 0x26 on inode 4354967611, would reset version number
bad inode format in inode 4354967611
would have cleared inode 4354967611
bad magic number 0xf803 on inode 4354967612, would reset magic number
bad version number 0x7f on inode 4354967612, would reset version number
bad inode format in inode 4354967612
would have cleared inode 4354967612
bad magic number 0xaa2f on inode 4354967613, would reset magic number
bad version number 0x31 on inode 4354967613, would reset version number
bad (negative) size -8402135599746543486 on inode 4354967613
would have cleared inode 4354967613
bad magic number 0xbd27 on inode 4354967614, would reset magic number
bad version number 0xffffffec on inode 4354967614, would reset version number
bad inode format in inode 4354967614
would have cleared inode 4354967614
bad magic number 0x27bd on inode 4354967615, would reset magic number
bad version number 0x1f on inode 4354967615, would reset version number
bad inode format in inode 4354967615
would have cleared inode 4354967615
        - agno = 2
        - agno = 33
        - agno = 48
        - agno = 63
        - agno = 18
        - agno = 78
        - agno = 49
        - agno = 34
        - agno = 3
        - agno = 19
        - agno = 64
        - agno = 35
        - agno = 50
        - agno = 79
        - agno = 20
        - agno = 65
        - agno = 4
        - agno = 36
        - agno = 51
        - agno = 21
        - agno = 66
        - agno = 80
        - agno = 5
        - agno = 37
        - agno = 52
        - agno = 22
        - agno = 67
        - agno = 38
        - agno = 53
        - agno = 81
        - agno = 6
        - agno = 23
        - agno = 68
        - agno = 39
        - agno = 54
        - agno = 82
        - agno = 7
        - agno = 24
        - agno = 69
        - agno = 40
        - agno = 55
        - agno = 25
        - agno = 8
        - agno = 83
        - agno = 41
        - agno = 70
        - agno = 56
        - agno = 26
        - agno = 42
        - agno = 57
        - agno = 71
        - agno = 9
        - agno = 84
        - agno = 27
        - agno = 43
        - agno = 58
        - agno = 72
        - agno = 10
        - agno = 85
        - agno = 44
        - agno = 28
        - agno = 59
        - agno = 73
        - agno = 11
        - agno = 86
        - agno = 29
        - agno = 74
        - agno = 12
        - agno = 87
        - agno = 13
        - agno = 14
        - 14:32:06: process known inodes and inode discovery - 79473472 of 64 inodes done
        - process newly discovered inodes...
        - 14:32:06: process newly discovered inodes - 88 of 88 allocation groups done
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - 14:32:09: setting up duplicate extent list - 88 of 88 allocation groups done
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 30
        - agno = 60
        - agno = 15
        - agno = 75
        - agno = 45
entry "102" at block 3 offset 160 in directory inode 128854679580 references free inode 4354967595
        would clear inode number in entry at offset 160...
        - agno = 31
        - agno = 46
entry "123" at block 0 offset 1088 in directory inode 257750135845 references free inode 4354967607
        would clear inode number in entry at offset 1088...
        - agno = 16
        - agno = 61
        - agno = 76
        - agno = 1
entry "755" at block 0 offset 256 in directory inode 133160166402 references free inode 4354967608
        would clear inode number in entry at offset 256...
entry "335" at block 0 offset 800 in directory inode 133160166402 references free inode 4354967612
        would clear inode number in entry at offset 800...
        - agno = 32
        - agno = 47
entry "553" at block 0 offset 2112 in directory inode 68723563568 references free inode 4354967601
        would clear inode number in entry at offset 2112...
entry "478" at block 3 offset 704 in directory inode 262041560107 references free inode 4354967588
        would clear inode number in entry at offset 704...
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
        - agno = 17
        - agno = 62
entry "868747126f598190569cc83b2028d3c3" in shortform directory 4295693332 references free inode 4354967584
would have junked entry "868747126f598190569cc83b2028d3c3" in directory inode 4295693332
would have corrected i8 count in directory 4295693332 from 3 to 2
entry "88e7499849585ec20b4b95d014632d62" at block 0 offset 144 in directory inode 4307964959 references free inode 4354967606
        would clear inode number in entry at offset 144...
entry "a22e4ffdf3bcf47d316a7f20c4b65927" in shortform directory 4310804523 references free inode 4354967597
would have junked entry "a22e4ffdf3bcf47d316a7f20c4b65927" in directory inode 4310804523
would have corrected i8 count in directory 4310804523 from 2 to 1
entry "aff7afd9850b2b80fbaa45e4e261150c" in shortform directory 4318433342 references free inode 4354967585
would have junked entry "aff7afd9850b2b80fbaa45e4e261150c" in directory inode 4318433342
would have corrected i8 count in directory 4318433342 from 2 to 1
entry "8d7e5ad25683e71cd1dfe4949a754bc5" in shortform directory 4318560261 references free inode 4354967615
would have junked entry "8d7e5ad25683e71cd1dfe4949a754bc5" in directory inode 4318560261
would have corrected i8 count in directory 4318560261 from 2 to 1
        - agno = 77
entry "acfbd6d039cc61f057ab7fd49d59770f" at block 0 offset 48 in directory inode 4344747031 references free inode 4354967594
        would clear inode number in entry at offset 48...
entry "b872391b03d655726a8ceb9e6a3b3b14" at block 0 offset 48 in directory inode 4354966589 references free inode 4354967592
        would clear inode number in entry at offset 48...
entry "6d5dffea5cd56f445fb0b425ee93691b" in shortform directory 4354967571 references free inode 4354967604
would have junked entry "6d5dffea5cd56f445fb0b425ee93691b" in directory inode 4354967571
would have corrected i8 count in directory 4354967571 from 2 to 1
bad magic number 0x3f1a on inode 4354967584, would reset magic number
bad version number 0x5b on inode 4354967584, would reset version number
bad (negative) size -4743366169855139346 on inode 4354967584
would have cleared inode 4354967584
bad magic number 0xf749 on inode 4354967585, would reset magic number
bad version number 0x19 on inode 4354967585, would reset version number
bad (negative) size -4925401835952624351 on inode 4354967585
would have cleared inode 4354967585
bad magic number 0xefc6 on inode 4354967586, would reset magic number
bad version number 0xffffffc0 on inode 4354967586, would reset version number
bad (negative) size -6663211212969851433 on inode 4354967586
would have cleared inode 4354967586
bad magic number 0x93db on inode 4354967587, would reset magic number
bad version number 0x5f on inode 4354967587, would reset version number
bad inode format in inode 4354967587
would have cleared inode 4354967587
bad magic number 0x54da on inode 4354967588, would reset magic number
bad version number 0xffffff86 on inode 4354967588, would reset version number
bad inode format in inode 4354967588
would have cleared inode 4354967588
bad magic number 0x5893 on inode 4354967589, would reset magic number
bad version number 0x0 on inode 4354967589, would reset version number
bad inode format in inode 4354967589
would have cleared inode 4354967589
bad magic number 0x309f on inode 4354967590, would reset magic number
bad version number 0xffffffc2 on inode 4354967590, would reset version number
bad (negative) size -7620939567173939239 on inode 4354967590
would have cleared inode 4354967590
bad magic number 0x9fce on inode 4354967591, would reset magic number
bad version number 0x38 on inode 4354967591, would reset version number
bad inode format in inode 4354967591
would have cleared inode 4354967591
bad magic number 0xb59f on inode 4354967592, would reset magic number
bad version number 0x4 on inode 4354967592, would reset version number
bad inode format in inode 4354967592
would have cleared inode 4354967592
bad magic number 0x53b4 on inode 4354967593, would reset magic number
bad version number 0xfffffff4 on inode 4354967593, would reset version number
bad (negative) size -6181984635646857242 on inode 4354967593
would have cleared inode 4354967593
bad magic number 0x7c77 on inode 4354967594, would reset magic number
bad version number 0x3e on inode 4354967594, would reset version number
bad (negative) size -5208493334119488045 on inode 4354967594
would have cleared inode 4354967594
bad magic number 0x6698 on inode 4354967595, would reset magic number
bad version number 0x51 on inode 4354967595, would reset version number
bad inode format in inode 4354967595
would have cleared inode 4354967595
bad magic number 0xe872 on inode 4354967596, would reset magic number
bad version number 0xffffffa1 on inode 4354967596, would reset version number
bad (negative) size -4217771995784516902 on inode 4354967596
would have cleared inode 4354967596
bad magic number 0x2c98 on inode 4354967597, would reset magic number
bad version number 0xffffff8e on inode 4354967597, would reset version number
bad (negative) size -1829068493719510754 on inode 4354967597
would have cleared inode 4354967597
bad magic number 0x5ead on inode 4354967598, would reset magic number
bad version number 0xffffffa3 on inode 4354967598, would reset version number
bad (negative) size -3586320840011193807 on inode 4354967598
would have cleared inode 4354967598
bad magic number 0xf5ac on inode 4354967599, would reset magic number
bad version number 0x4d on inode 4354967599, would reset version number
bad (negative) size -7822733889952692968 on inode 4354967599
would have cleared inode 4354967599
bad magic number 0x5c7e on inode 4354967600, would reset magic number
bad version number 0x4e on inode 4354967600, would reset version number
bad (negative) size -8116057573661653832 on inode 4354967600
would have cleared inode 4354967600
bad magic number 0x57c5 on inode 4354967601, would reset magic number
bad version number 0xffffffe5 on inode 4354967601, would reset version number
bad inode format in inode 4354967601
would have cleared inode 4354967601
bad magic number 0xcc0e on inode 4354967602, would reset magic number
bad version number 0x74 on inode 4354967602, would reset version number
bad (negative) size -7175657235803173389 on inode 4354967602
would have cleared inode 4354967602
bad magic number 0x17a3 on inode 4354967603, would reset magic number
bad version number 0x3b on inode 4354967603, would reset version number
bad inode format in inode 4354967603
would have cleared inode 4354967603
bad magic number 0xf085 on inode 4354967604, would reset magic number
bad version number 0xffffffd3 on inode 4354967604, would reset version number
bad (negative) size -4829046689272463617 on inode 4354967604
would have cleared inode 4354967604
bad magic number 0xd47a on inode 4354967605, would reset magic number
bad version number 0xe on inode 4354967605, would reset version number
bad inode format in inode 4354967605
would have cleared inode 4354967605
bad magic number 0xcb77 on inode 4354967606, would reset magic number
bad version number 0xfffffff6 on inode 4354967606, would reset version number
bad (negative) size -4813752938159903980 on inode 4354967606
would have cleared inode 4354967606
bad magic number 0xfb56 on inode 4354967607, would reset magic number
bad version number 0x0 on inode 4354967607, would reset version number
bad (negative) size -5002825908862883310 on inode 4354967607
would have cleared inode 4354967607
bad magic number 0xf74f on inode 4354967608, would reset magic number
bad version number 0x31 on inode 4354967608, would reset version number
bad (negative) size -2804153168172737527 on inode 4354967608
would have cleared inode 4354967608
bad magic number 0xa92e on inode 4354967609, would reset magic number
bad version number 0xffffffad on inode 4354967609, would reset version number
bad (negative) size -3138139371176466993 on inode 4354967609
would have cleared inode 4354967609
bad magic number 0x75e5 on inode 4354967610, would reset magic number
bad version number 0x66 on inode 4354967610, would reset version number
bad inode format in inode 4354967610
would have cleared inode 4354967610
bad magic number 0x6b8d on inode 4354967611, would reset magic number
bad version number 0x26 on inode 4354967611, would reset version number
bad inode format in inode 4354967611
would have cleared inode 4354967611
bad magic number 0xf803 on inode 4354967612, would reset magic number
bad version number 0x7f on inode 4354967612, would reset version number
bad inode format in inode 4354967612
would have cleared inode 4354967612
bad magic number 0xaa2f on inode 4354967613, would reset magic number
bad version number 0x31 on inode 4354967613, would reset version number
bad (negative) size -8402135599746543486 on inode 4354967613
would have cleared inode 4354967613
bad magic number 0xbd27 on inode 4354967614, would reset magic number
bad version number 0xffffffec on inode 4354967614, would reset version number
bad inode format in inode 4354967614
would have cleared inode 4354967614
bad magic number 0x27bd on inode 4354967615, would reset magic number
bad version number 0x1f on inode 4354967615, would reset version number
bad inode format in inode 4354967615
would have cleared inode 4354967615
        - agno = 2
entry "831" at block 2 offset 1680 in directory inode 201865261100 references free inode 4354967605
        would clear inode number in entry at offset 1680...
        - agno = 33
        - agno = 48

        - agno = 18
        - agno = 63
entry "503" at block 0 offset 3584 in directory inode 330718645273 references free inode 4354967613
        would clear inode number in entry at offset 3584...
        - agno = 78
        - agno = 34
        - agno = 49
        - agno = 3
        - agno = 19
entry "125" at block 0 offset 2160 in directory inode 270592950284 references free inode 4354967596
        would clear inode number in entry at offset 2160...
entry "746" at block 0 offset 2448 in directory inode 270649053207 references free inode 4354967611
        would clear inode number in entry at offset 2448...
        - agno = 64
        - agno = 35
        - agno = 50
        - agno = 79
        - agno = 20
        - agno = 65
        - agno = 4
        - agno = 36
        - agno = 51
        - agno = 21
        - agno = 66
        - agno = 80
        - agno = 5
entry "937" at block 0 offset 944 in directory inode 154623186993 references free inode 4354967610
        would clear inode number in entry at offset 944...
        - agno = 37
        - agno = 52
        - agno = 22
        - agno = 67
        - agno = 38
        - agno = 53
        - agno = 81
        - agno = 6
        - agno = 23
        - agno = 68
        - agno = 39
entry "612" at block 1 offset 1840 in directory inode 227634536482 references free inode 4354967599
        would clear inode number in entry at offset 1840...
        - agno = 54
        - agno = 7
        - agno = 82
        - agno = 24
entry "361" at block 1 offset 272 in directory inode 231928297598 references free inode 4354967591
        would clear inode number in entry at offset 272...
entry "875" at block 0 offset 3840 in directory inode 231928298526 references free inode 4354967593
        would clear inode number in entry at offset 3840...
entry "592" at block 0 offset 3728 in directory inode 292065894452 references free inode 4354967600
        would clear inode number in entry at offset 3728...
        - agno = 69
        - agno = 40
        - agno = 55
entry "922" at block 0 offset 3216 in directory inode 103079291917 references free inode 4354967589
        would clear inode number in entry at offset 3216...
entry "561" at block 0 offset 944 in directory inode 103079473192 references free inode 4354967609
        would clear inode number in entry at offset 944...
entry "033" at block 0 offset 544 in directory inode 103136944171 references free inode 4354967590
        would clear inode number in entry at offset 544...
        - agno = 8
        - agno = 25
entry "702" at block 0 offset 960 in directory inode 352256730114 references free inode 4354967586
        would clear inode number in entry at offset 960...
entry "965" at block 0 offset 1168 in directory inode 352256730114 references free inode 4354967587
        would clear inode number in entry at offset 1168...
        - agno = 83
        - agno = 41
        - agno = 70
entry "510" at block 1 offset 336 in directory inode 236351051817 references free inode 4354967602
        would clear inode number in entry at offset 336...
        - agno = 56
        - agno = 26
        - agno = 42
        - agno = 57
        - agno = 9
        - agno = 71
        - agno = 84
        - agno = 27
        - agno = 43
        - agno = 58
        - agno = 72
        - agno = 10
        - agno = 85
        - agno = 44
        - agno = 28
        - agno = 59
        - agno = 73
        - agno = 11
        - agno = 86
        - agno = 29
        - agno = 74
        - agno = 12
entry "990" at block 1 offset 224 in directory inode 369367197707 references free inode 4354967598
        would clear inode number in entry at offset 224...
        - agno = 87
        - agno = 13
        - agno = 14
entry "479" at block 3 offset 144 in directory inode 60193580050 references free inode 4354967603
        would clear inode number in entry at offset 144...
        - 14:53:41: check for inodes claiming duplicate blocks - 79473472 of 64 inodes done
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
Metadata corruption detected at xfs_inode block 0x81c9c210/0x2000
entry "868747126f598190569cc83b2028d3c3" in shortform directory inode 4295693332 points to free inode 4354967584
would junk entry
would fix i8count in inode 4295693332
entry "88e7499849585ec20b4b95d014632d62" in directory inode 4307964959 points to free inode 4354967606, would junk entry
bad hash table for directory inode 4307964959 (no data entry): would rebuild
entry "a22e4ffdf3bcf47d316a7f20c4b65927" in shortform directory inode 4310804523 points to free inode 4354967597
would junk entry
would fix i8count in inode 4310804523
entry "aff7afd9850b2b80fbaa45e4e261150c" in shortform directory inode 4318433342 points to free inode 4354967585
would junk entry
would fix i8count in inode 4318433342
entry "8d7e5ad25683e71cd1dfe4949a754bc5" in shortform directory inode 4318560261 points to free inode 4354967615
would junk entry
would fix i8count in inode 4318560261
entry "acfbd6d039cc61f057ab7fd49d59770f" in directory inode 4344747031 points to free inode 4354967594, would junk entry
bad hash table for directory inode 4344747031 (no data entry): would rebuild
entry "b872391b03d655726a8ceb9e6a3b3b14" in directory inode 4354966589 points to free inode 4354967592, would junk entry
bad hash table for directory inode 4354966589 (no data entry): would rebuild
entry "6d5dffea5cd56f445fb0b425ee93691b" in shortform directory inode 4354967571 points to free inode 4354967604
would junk entry
would fix i8count in inode 4354967571
entry "479" in directory inode 60193580050 points to free inode 4354967603, would junk entry
entry "553" in directory inode 68723563568 points to free inode 4354967601, would junk entry
bad hash table for directory inode 68723563568 (no data entry): would rebuild
entry "922" in directory inode 103079291917 points to free inode 4354967589, would junk entry
bad hash table for directory inode 103079291917 (no data entry): would rebuild
entry "561" in directory inode 103079473192 points to free inode 4354967609, would junk entry
bad hash table for directory inode 103079473192 (no data entry): would rebuild
entry "033" in directory inode 103136944171 points to free inode 4354967590, would junk entry
bad hash table for directory inode 103136944171 (no data entry): would rebuild
entry "102" in directory inode 128854679580 points to free inode 4354967595, would junk entry
entry "755" in directory inode 133160166402 points to free inode 4354967608, would junk entry
entry "335" in directory inode 133160166402 points to free inode 4354967612, would junk entry
entry "937" in directory inode 154623186993 points to free inode 4354967610, would junk entry
bad hash table for directory inode 154623186993 (no data entry): would rebuild
entry "831" in directory inode 201865261100 points to free inode 4354967605, would junk entry

entry "612" in directory inode 227634536482 points to free inode 4354967599, would junk entry
entry "361" in directory inode 231928297598 points to free inode 4354967591, would junk entry
bad hash table for directory inode 231928297598 (no data entry): would rebuild
entry "875" in directory inode 231928298526 points to free inode 4354967593, would junk entry
entry "510" in directory inode 236351051817 points to free inode 4354967602, would junk entry
entry "123" in directory inode 257750135845 points to free inode 4354967607, would junk entry
entry "478" in directory inode 262041560107 points to free inode 4354967588, would junk entry
entry "125" in directory inode 270592950284 points to free inode 4354967596, would junk entry
entry "746" in directory inode 270649053207 points to free inode 4354967611, would junk entry
entry "592" in directory inode 292065894452 points to free inode 4354967600, would junk entry
bad hash table for directory inode 292065894452 (no data entry): would rebuild
entry "503" in directory inode 330718645273 points to free inode 4354967613, would junk entry
entry "702" in directory inode 352256730114 points to free inode 4354967586, would junk entry
entry "965" in directory inode 352256730114 points to free inode 4354967587, would junk entry
entry "990" in directory inode 369367197707 points to free inode 4354967598, would junk entry
bad hash table for directory inode 369367197707 (no data entry): would rebuild
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 4354971706, would move to lost+found
disconnected inode 4355534875, would move to lost+found
disconnected inode 4356321307, would move to lost+found
disconnected inode 4356453437, would move to lost+found
disconnected inode 4365067280, would move to lost+found
disconnected inode 4367228968, would move to lost+found
disconnected inode 60428483644, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 103079291917 nlinks from 406 to 405
would have reset inode 68723563568 nlinks from 314 to 313
would have reset inode 154623186993 nlinks from 146 to 145
would have reset inode 60193580050 nlinks from 982 to 981
would have reset inode 128854679580 nlinks from 971 to 970
would have reset inode 103079473192 nlinks from 127 to 126
would have reset inode 60428483644 nlinks from 0 to 1
would have reset inode 133160166402 nlinks from 619 to 617
would have reset inode 231928297598 nlinks from 377 to 376
would have reset inode 227634536482 nlinks from 979 to 978
would have reset inode 262041560107 nlinks from 954 to 953
would have reset inode 270592950284 nlinks from 792 to 791
would have reset inode 292065894452 nlinks from 353 to 352
would have reset inode 231928298526 nlinks from 916 to 915
would have reset inode 330718645273 nlinks from 556 to 555
would have reset inode 369367197707 nlinks from 443 to 442
would have reset inode 270649053207 nlinks from 951 to 950
would have reset inode 201865261100 nlinks from 715 to 714
would have reset inode 257750135845 nlinks from 1001 to 1000
would have reset inode 103136944171 nlinks from 56 to 55
would have reset inode 236351051817 nlinks from 662 to 661
would have reset inode 352256730114 nlinks from 1001 to 999
        - 15:22:58: verify and correct link counts - 88 of 88 allocation groups done
No modify flag set, skipping filesystem flush and exiting.
15:23:00 [root@dn-238[DP]:/mnt] $
15:23:00 [root@dn-238[DP]:/mnt] $
15:23:00 [root@dn-238[DP]:/mnt] $
15:30:30 [root@dn-238[DP]:/mnt] $
15:32:26 [root@dn-238[DP]:/mnt] $
15:32:27 [root@dn-238[DP]:/mnt] $
15:32:27 [root@dn-238[DP]:/mnt] $
15:32:30 [root@dn-238[DP]:/mnt] $
15:32:30 [root@dn-238[DP]:/mnt] $