Re: XFS breakage in 2.6.18-rc1

Re: XFS breakage in 2.6.18-rc1

[Nathan Scott]

On Wed, Jul 19, 2006 at 09:59:33AM -0500, Jeffrey E. Hundstad wrote:
> I did try the xfs_repair 2.8.4 for a volume running on 2.6.17.4 and it 
> annihilated the volume.  This volume was not showing signs of crashing.  
> So... I guess I would certainly not run xfs_repair unless there is good 
> reason.

Erm, wha..?  Can you expand on "annihilated" a bit?  (please send
me the full xfs_repair output if you still have it).

thanks.

-- 
Nathan

Re: XFS breakage in 2.6.18-rc1

[Jeffrey Hundstad]

Nathan Scott wrote:
> On Wed, Jul 19, 2006 at 09:59:33AM -0500, Jeffrey E. Hundstad wrote:
>   
>> I did try the xfs_repair 2.8.4 for a volume running on 2.6.17.4 and it 
>> annihilated the volume.  This volume was not showing signs of crashing.  
>> So... I guess I would certainly not run xfs_repair unless there is good 
>> reason.
>>     
>
> Erm, wha..?  Can you expand on "annihilated" a bit?  (please send
> me the full xfs_repair output if you still have it).
>   

Nathan Scott,

I'm very sorry; I don't have the output anymore.  By annihilated I mean 
that there were several directories trees that /didn't work/.  If you 
tried to cd into the directory or take a directory listing... or used a 
file that you knew was in these certain directories then you'd get pages 
of debug message to the console; and no usable data.  I re-ran 
xfs_repair and retried several times but the condition never seemed to 
improve or get worse for that matter.

I /incorrectly/ figured it was a known issue or I'd have saved the 
output.  Sorry again.

-- 
Jeffrey Hundstad

Re: XFS breakage in 2.6.18-rc1

[Nathan Scott]

On Wed, Jul 19, 2006 at 12:29:41AM +0200, Torsten Landschoff wrote:
> Hi friends, 

Hi Torsten,

> I upgraded to 2.6.18-rc1 on sunday, with the following results (taken
> from my /var/log/kern.log), which ultimately led me to reinstall my 
> system:
> 
> Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216
> Jul 17 07:33:53 pulsar kernel: dir: inode 54526538

I suspect you had some residual directory corruption from using the
2.6.17 XFS (which is known to have a lurking dir2 corruption issue,
fixed in the latest -stable point release).

> of programs fail in mysterious ways. I tried to recover using xfs_repair
> but I feel that my partition is thorougly borked. Of course no data was 
> lost due to backups but still I'd like this bug to be fixed ;-)

2.6.18-rc1 should be fine (contains the corruption fix).  Did you
mkfs and restore?  Or at least get a full repair run?  If you did,
and you still see issues in .18-rc1, please let me know asap.

thanks.

-- 
Nathan

Re: XFS breakage in 2.6.18-rc1

[Alistair John Strachan]

On Tuesday 18 July 2006 23:57, Nathan Scott wrote:
[snip]
> > of programs fail in mysterious ways. I tried to recover using xfs_repair
> > but I feel that my partition is thorougly borked. Of course no data was
> > lost due to backups but still I'd like this bug to be fixed ;-)
>
> 2.6.18-rc1 should be fine (contains the corruption fix).  Did you
> mkfs and restore?  Or at least get a full repair run?  If you did,
> and you still see issues in .18-rc1, please let me know asap.

Just out of interest, I've got a few XFS volumes that were created 24 months 
ago on a machine that I upgraded to 2.6.17 about a month ago. I haven't seen 
any crashes so far.

Assuming I get the newest XFS repair tools on there, what's the disadvantage 
of repairing versus creating a new filesystem? What special circumstances are 
required to cause a crash?

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: XFS breakage in 2.6.18-rc1

[Nathan Scott]

On Wed, Jul 19, 2006 at 09:08:30AM +0100, Alistair John Strachan wrote:
> On Tuesday 18 July 2006 23:57, Nathan Scott wrote:
> [snip]
> > > of programs fail in mysterious ways. I tried to recover using xfs_repair
> > > but I feel that my partition is thorougly borked. Of course no data was
> > > lost due to backups but still I'd like this bug to be fixed ;-)
> >
> > 2.6.18-rc1 should be fine (contains the corruption fix).  Did you
> > mkfs and restore?  Or at least get a full repair run?  If you did,
> > and you still see issues in .18-rc1, please let me know asap.
> 
> Just out of interest, I've got a few XFS volumes that were created 24 months 
> ago on a machine that I upgraded to 2.6.17 about a month ago. I haven't seen 
> any crashes so far.
> 
> Assuming I get the newest XFS repair tools on there, what's the disadvantage 
> of repairing versus creating a new filesystem? What special circumstances are 
> required to cause a crash?

There should be no disadvantage to repairing.  I will update the FAQ
shortly to describe all the details of the problem, recommendations
on how to address it, which kernel version is affected, etc.

cheers.

-- 
Nathan

Re: XFS breakage in 2.6.18-rc1

[Kasper Sandberg]

On Thu, 2006-07-20 at 08:56 +1000, Nathan Scott wrote:
> On Wed, Jul 19, 2006 at 09:08:30AM +0100, Alistair John Strachan wrote:
> > On Tuesday 18 July 2006 23:57, Nathan Scott wrote:
> > [snip]
> > > > of programs fail in mysterious ways. I tried to recover using xfs_repair
> > > > but I feel that my partition is thorougly borked. Of course no data was
> > > > lost due to backups but still I'd like this bug to be fixed ;-)
> > >
> > > 2.6.18-rc1 should be fine (contains the corruption fix).  Did you
> > > mkfs and restore?  Or at least get a full repair run?  If you did,
> > > and you still see issues in .18-rc1, please let me know asap.
> > 
> > Just out of interest, I've got a few XFS volumes that were created 24 months 
> > ago on a machine that I upgraded to 2.6.17 about a month ago. I haven't seen 
> > any crashes so far.
> > 
> > Assuming I get the newest XFS repair tools on there, what's the disadvantage 
> > of repairing versus creating a new filesystem? What special circumstances are 
> > required to cause a crash?
> 
> There should be no disadvantage to repairing.  I will update the FAQ
> shortly to describe all the details of the problem, recommendations
> on how to address it, which kernel version is affected, etc.
this FAQ, is it this: http://oss.sgi.com/projects/xfs/faq.html#dir2 ?
(btw, it seems that while only in the TOC once, you have the same about
2.6.17 twice..)..

which version of xfsprogs should i use while doing the xfs_check ?

> 
> cheers.
> 

Re: XFS breakage in 2.6.18-rc1

[Kasper Sandberg]

On Wed, 2006-07-19 at 08:57 +1000, Nathan Scott wrote:
> On Wed, Jul 19, 2006 at 12:29:41AM +0200, Torsten Landschoff wrote:
> > Hi friends, 
> 
> Hi Torsten,
> 
> > I upgraded to 2.6.18-rc1 on sunday, with the following results (taken
> > from my /var/log/kern.log), which ultimately led me to reinstall my 
> > system:
> > 
> > Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216
> > Jul 17 07:33:53 pulsar kernel: dir: inode 54526538
> 
> I suspect you had some residual directory corruption from using the
> 2.6.17 XFS (which is known to have a lurking dir2 corruption issue,
> fixed in the latest -stable point release).
This has me very worried.

i just upgraded to .18-rc1-git5 when it came out, i used .17-rc3 before.
does this mean my .17-rc3 may have corrupted my filesystem?

what action do you suggest i do now?

> 
> > of programs fail in mysterious ways. I tried to recover using xfs_repair
> > but I feel that my partition is thorougly borked. Of course no data was 
> > lost due to backups but still I'd like this bug to be fixed ;-)
> 
> 2.6.18-rc1 should be fine (contains the corruption fix).  Did you
> mkfs and restore?  Or at least get a full repair run?  If you did,
> and you still see issues in .18-rc1, please let me know asap.
> 
> thanks.
> 

Re: XFS breakage in 2.6.18-rc1

[Alistair John Strachan]

On Wednesday 19 July 2006 11:21, Kasper Sandberg wrote:
> On Wed, 2006-07-19 at 08:57 +1000, Nathan Scott wrote:
> > On Wed, Jul 19, 2006 at 12:29:41AM +0200, Torsten Landschoff wrote:
> > > Hi friends,
> >
> > Hi Torsten,
> >
> > > I upgraded to 2.6.18-rc1 on sunday, with the following results (taken
> > > from my /var/log/kern.log), which ultimately led me to reinstall my
> > > system:
> > >
> > > Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216
> > > Jul 17 07:33:53 pulsar kernel: dir: inode 54526538
> >
> > I suspect you had some residual directory corruption from using the
> > 2.6.17 XFS (which is known to have a lurking dir2 corruption issue,
> > fixed in the latest -stable point release).
>
> This has me very worried.
>
> i just upgraded to .18-rc1-git5 when it came out, i used .17-rc3 before.
> does this mean my .17-rc3 may have corrupted my filesystem?
>
> what action do you suggest i do now?
>
> > > of programs fail in mysterious ways. I tried to recover using
> > > xfs_repair but I feel that my partition is thorougly borked. Of course
> > > no data was lost due to backups but still I'd like this bug to be fixed
> > > ;-)
> >
> > 2.6.18-rc1 should be fine (contains the corruption fix).  Did you
> > mkfs and restore?  Or at least get a full repair run?  If you did,
> > and you still see issues in .18-rc1, please let me know asap.
> >
> > thanks.

According to another thread Nathan just responded to, it sounds like we need 
to wait for a new version of the xfsprogs package, and then run xfs_repair on 
the affected filesystems. I wouldn't worry about it too much if you've not 
had any crashes. The damage can be repaired, just not right now.

I'm still waiting for a crash on a machine that has been under heavy load for 
28 days, so it's obviously not _that_ easy to trigger.

-- 
Cheers,
Alistair.

Third year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: XFS breakage in 2.6.18-rc1

[Kasper Sandberg]

On Wed, 2006-07-19 at 13:43 +0100, Alistair John Strachan wrote:
> On Wednesday 19 July 2006 11:21, Kasper Sandberg wrote:
> > On Wed, 2006-07-19 at 08:57 +1000, Nathan Scott wrote:
> > > On Wed, Jul 19, 2006 at 12:29:41AM +0200, Torsten Landschoff wrote:
> > > > Hi friends,
> > >
> > > Hi Torsten,
> > >
> > > > I upgraded to 2.6.18-rc1 on sunday, with the following results (taken
> > > > from my /var/log/kern.log), which ultimately led me to reinstall my
> > > > system:
> > > >
> > > > Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216
> > > > Jul 17 07:33:53 pulsar kernel: dir: inode 54526538
> > >
> > > I suspect you had some residual directory corruption from using the
> > > 2.6.17 XFS (which is known to have a lurking dir2 corruption issue,
> > > fixed in the latest -stable point release).
> >
> > This has me very worried.
> >
> > i just upgraded to .18-rc1-git5 when it came out, i used .17-rc3 before.
> > does this mean my .17-rc3 may have corrupted my filesystem?
> >
> > what action do you suggest i do now?
> >
> > > > of programs fail in mysterious ways. I tried to recover using
> > > > xfs_repair but I feel that my partition is thorougly borked. Of course
> > > > no data was lost due to backups but still I'd like this bug to be fixed
> > > > ;-)
> > >
> > > 2.6.18-rc1 should be fine (contains the corruption fix).  Did you
> > > mkfs and restore?  Or at least get a full repair run?  If you did,
> > > and you still see issues in .18-rc1, please let me know asap.
> > >
> > > thanks.
> 
> According to another thread Nathan just responded to, it sounds like we need 
> to wait for a new version of the xfsprogs package, and then run xfs_repair on 
> the affected filesystems. I wouldn't worry about it too much if you've not 
> had any crashes. The damage can be repaired, just not right now.
without ANY loss? because even though it would be abit painful for me to
do, i do have the option of smashing in a new drive, copy everything,
and reinitialize my filesystem.
> 
> I'm still waiting for a crash on a machine that has been under heavy load for 
> 28 days, so it's obviously not _that_ easy to trigger.
so basically if i upgrade to a safe kernel before i do get these errors,
im good?


> 

Re: XFS breakage in 2.6.18-rc1

[Nathan Scott]

On Wed, Jul 19, 2006 at 12:21:08PM +0200, Kasper Sandberg wrote:
> On Wed, 2006-07-19 at 08:57 +1000, Nathan Scott wrote:
> > On Wed, Jul 19, 2006 at 12:29:41AM +0200, Torsten Landschoff wrote:
> > > Hi friends, 
> > 
> > Hi Torsten,
> > 
> > > I upgraded to 2.6.18-rc1 on sunday, with the following results (taken
> > > from my /var/log/kern.log), which ultimately led me to reinstall my 
> > > system:
> > > 
> > > Jul 17 07:33:53 pulsar kernel: xfs_da_do_buf: bno 16777216
> > > Jul 17 07:33:53 pulsar kernel: dir: inode 54526538
> > 
> > I suspect you had some residual directory corruption from using the
> > 2.6.17 XFS (which is known to have a lurking dir2 corruption issue,
> > fixed in the latest -stable point release).
> This has me very worried.
> 
> i just upgraded to .18-rc1-git5 when it came out, i used .17-rc3 before.
> does this mean my .17-rc3 may have corrupted my filesystem?
> 
> what action do you suggest i do now?

The odds are decent that you're unaffected.  You can check your filesystem
using xfs_check or xfs_repair -n and these will give you a good indication
as to whether further action is required.

cheers.

-- 
Nathan

Re: XFS breakage in 2.6.18-rc1

[Torsten Landschoff]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

Hi Nathan, 

On Wed, Jul 19, 2006 at 08:57:31AM +1000, Nathan Scott wrote:
 
> I suspect you had some residual directory corruption from using the
> 2.6.17 XFS (which is known to have a lurking dir2 corruption issue,
> fixed in the latest -stable point release).

That probably the cause of my problem. Thanks for the info!

BTW: I think there was nothing important on the broken filesystems, but
I'd like to keep what's still there anyway just in case... How would you
suggest should I copy that data? I fear, just mounting and using cp 
might break and shutdown the FS again, would xfsdump be more
appropriate?

Thanks for XFS, I am using it for years in production servers!

Greetings

	Torsten

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: XFS breakage in 2.6.18-rc1

[Nathan Scott]

On Wed, Jul 19, 2006 at 11:14:02PM +0200, Torsten Landschoff wrote:
> On Wed, Jul 19, 2006 at 08:57:31AM +1000, Nathan Scott wrote:
> > I suspect you had some residual directory corruption from using the
> > 2.6.17 XFS (which is known to have a lurking dir2 corruption issue,
> > fixed in the latest -stable point release).
> 
> That probably the cause of my problem. Thanks for the info!
> 
> BTW: I think there was nothing important on the broken filesystems, but
> I'd like to keep what's still there anyway just in case... How would you
> suggest should I copy that data? I fear, just mounting and using cp 
> might break and shutdown the FS again, would xfsdump be more
> appropriate?

Yeah, xfsdumps not a bad idea, the interfaces it uses may well
be able to avoid the cases that trigger shutdown.  Otherwise it
is a case of identifying the problem directory inode (the inum
is reported in the shutdown trace) and avoiding that path when
cp'ing - you can match inum to path via xfs_ncheck.

> Thanks for XFS, I am using it for years in production servers!

Thanks for the kind words, they're much appreciated at times
like these. :-]

cheers.

-- 
Nathan

Re: XFS breakage in 2.6.18-rc1

[Christian Kujau]

Hi folks,

On Wed, 19 Jul 2006, Nathan Scott wrote:
> 2.6.18-rc1 should be fine (contains the corruption fix).  Did you
> mkfs and restore?  Or at least get a full repair run?  If you did,
> and you still see issues in .18-rc1, please let me know asap.

well, at least for me, corruption/errors *started* with 2.6.18-rc1:

http://oss.sgi.com/archives/xfs/2006-07/msg00151.html

I downgraded to 2.6.17.5 and the errors stopped. Now I've upgraded to 
2.6.18-rc2 and see the same errors:

xfs_da_do_buf: bno 16777216
dir: inode 24472381
Filesystem "md0": XFS internal error xfs_da_do_buf(1) at line 1992 of file fs/xfs/xfs_da_btree.c.  Caller 0xc0219230
Filesystem "md0": XFS internal error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c.  Caller 0xc024d717

Please see the whole error/.config/logs here:

http://nerdbynature.de/bits/2.6.18-rc2/

Thanks,
Christian.
-- 
BOFH excuse #38:

secretary plugged hairdryer into UPS

Re: XFS breakage in 2.6.18-rc1

[Nathan Scott]

On Sat, Jul 22, 2006 at 05:27:24PM +0100, Christian Kujau wrote:
> On Wed, 19 Jul 2006, Nathan Scott wrote:
> > 2.6.18-rc1 should be fine (contains the corruption fix).  Did you
> > mkfs and restore?  Or at least get a full repair run?  If you did,
> > and you still see issues in .18-rc1, please let me know asap.
> 
> well, at least for me, corruption/errors *started* with 2.6.18-rc1:
> ...
> I downgraded to 2.6.17.5 and the errors stopped. Now I've upgraded to 
> 2.6.18-rc2 and see the same errors:
> 
> xfs_da_do_buf: bno 16777216
> dir: inode 24472381

This is an ondisk corruption - downgrading the kernel will not
resolve it.  The problem must be triggered by a combination of
operations on a directory; I'm certain that if you access inode
24472381 on your filesystem on 2.6.17, that it'll shutdown your
filesystem too.  See the FAQ entry for a description on how to
translate inums to paths, and also the repair -n step to detect
any corruption ondisk.

cheers.

-- 
Nathan

Re: XFS breakage in 2.6.18-rc1

[Jan Engelhardt]

Now that 2.6.17.7 is finally there, I am going to eyeball my XFS 
filesystems. Does xfsprogs 2.7.11 suffice?
Please update the XFS FAQ about this problem (and the solution, for 
others that did not read LKML), thanks!


Jan Engelhardt
-- 

Re: XFS breakage in 2.6.18-rc1

[Nathan Scott]

On Tue, Jul 25, 2006 at 10:16:10PM +0200, Jan Engelhardt wrote:
> Now that 2.6.17.7 is finally there, I am going to eyeball my XFS 
> filesystems. Does xfsprogs 2.7.11 suffice?

Sure, except for the repair issue which Barry is working on
(see the FAQ).

> Please update the XFS FAQ about this problem (and the solution, for 
> others that did not read LKML), thanks!

That was done last week.  You shoulda read it before sending
this, and saved me some time replying. ;)

cheers.

-- 
Nathan

Re: XFS breakage in 2.6.18-rc1

[Christian Kujau]

Hello again,

On Mon, 24 Jul 2006, Nathan Scott wrote:
> filesystem too.  See the FAQ entry for a description on how to
> translate inums to paths, and also the repair -n step to detect
> any corruption ondisk.

I had two xfs filesystems and I first noticed that /data/Scratch was 
befallen from this bug. I did not care much about this (hence the
name :)) and I wanted to postpone the xfs_db surgery.

Unfortunately I forgot that "/" was also an XFS and it crashed 
yesterday. remounting ro helped a bit (so no process attempted to write 
on it. however, cp'ing from the ro-mounted xfs sometimes hung, 
unkillable), I setup a mini-root somewhere else and followed the
instructions in the FAQ. It did not go too well, lots of 
stuff was moved to lost+found, but every subsequent xfs_repair run 
found more and more errors. I decided to mkfs the partition and make use 
of my backups. my other "scratch" partition is still XFS but mounted ro 
and I'll try the xfsprogs fixes Nathan published on this one.

Oh, and I dd'ed the corrupt xfs-filesystem to a file, so I can play 
around with this one as well.

If anyone is interested, here are the typescripts from the horrible 
xfs_repair runs: http://nerdbynature.de/bits/2.6.18-rc2/log/

cheers,
Christian.
-- 
BOFH excuse #21:

POSIX compliance problem

Re: XFS breakage in 2.6.18-rc1

[Nathan Scott]

On Fri, Jul 28, 2006 at 05:01:24PM +0000, Christian Kujau wrote:
> I had two xfs filesystems and I first noticed that /data/Scratch was 
> befallen from this bug. I did not care much about this (hence the
> name :)) and I wanted to postpone the xfs_db surgery.
> ...
> found more and more errors. I decided to mkfs the partition and make use 
> of my backups. my other "scratch" partition is still XFS but mounted ro 
> and I'll try the xfsprogs fixes Nathan published on this one.

Barry sent an xfs_repair patch to resolve this issue to the xfs@oss.sgi.com
list yesterday; please give that a go and let us know how it fares.

cheers.

-- 
Nathan

Re: XFS breakage in 2.6.18-rc1

[Ralf Hildebrandt]

* Nathan Scott <nathans@sgi.com>:

> Barry sent an xfs_repair patch to resolve this issue to the xfs@oss.sgi.com
> list yesterday; please give that a go and let us know how it fares.

Just to let you know, I did a cvs checkout of xfs-cmds
as described on http://oss.sgi.com/projects/xfs/source.html

Then I saved the patch from
http://oss.sgi.com/archives/xfs/2006-07/msg00374.html using the
"Original" link on hat page.

I build a xfs_Repair binary using that, transferred it onto an old
KLAX boot cd I had and repaired the XFS root on my laptop.

I got 5000 files in lost and found, mostly the whole manpages from my
system. Had to reinstall a few packages to restore lost binaries, but
that's all.

When will that horrible bug be fixed in 2.6.x? 

-- 
Ralf Hildebrandt (i.A. des IT-Zentrums)         Ralf.Hildebrandt@charite.de
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF                 send no mail to spamtrap@charite.de

Re: XFS breakage in 2.6.18-rc1

[David Chatterton]

Ralf Hildebrandt wrote:
> * Nathan Scott <nathans@sgi.com>:
> 
>> Barry sent an xfs_repair patch to resolve this issue to the xfs@oss.sgi.com
>> list yesterday; please give that a go and let us know how it fares.
> 
> Just to let you know, I did a cvs checkout of xfs-cmds
> as described on http://oss.sgi.com/projects/xfs/source.html
> 
> Then I saved the patch from
> http://oss.sgi.com/archives/xfs/2006-07/msg00374.html using the
> "Original" link on hat page.
> 
> I build a xfs_Repair binary using that, transferred it onto an old
> KLAX boot cd I had and repaired the XFS root on my laptop.
> 
> I got 5000 files in lost and found, mostly the whole manpages from my
> system. Had to reinstall a few packages to restore lost binaries, but
> that's all.
> 
> When will that horrible bug be fixed in 2.6.x? 
> 

The bug is fixed in 2.6.17.7.

David