* FS corruption and repair problem
@ 2006-10-16 8:25 Jason White
2006-10-16 9:52 ` martin f krafft
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Jason White @ 2006-10-16 8:25 UTC (permalink / raw)
To: xfs
[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]
This is my desktop system, running kernel 2.6.17.11. Kernel 2.6.17 was
previously installed, but I upgraded several weeks ago after I found out about
the corruption bug in 2.6.17, which may or may not be related to the
following.
The file system was shut down during a Debian package upgrade while trying to
access apt-related files. I booted into an old 2.4 kernel on a separate
partition and ran an old version of xfs_repair. The output is attached.
Xfs_repair terminated abnormally (see the end of the output).
I can copy a newer version of xfs_repair to the backup partition if that would
help, though I'm not sure which libraries to copy along with it.
Also, if there is any indication of what might have caused this I would be
interested, as this hasn't happened before and I'de especially like to find
out whether it is a hardware problem, or related to the current kernel, or
indeed a manifestation of the 2.6.17 bug. I've successfully mounted the
corrupted file system and a lot of it is still relatively intact - plenty of
files in lost+found, especially from the /usr/share/man3 directory for obvious
reasons.
It's a straightforward desktop machine with an IDE drive. I haven't updated
Debian packages on here for a while, so it isn't clear when the corruption
happened.
[-- Attachment #2: repair --]
[-- Type: text/plain, Size: 13777 bytes --]
jw:/home/jason# xfs_repair /dev/hda5
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
corrupt block 0 in directory inode 75501223
will junk block
no . entry for directory 75501223
no .. entry for directory 75501223
bad magic number 0x1362 on inode 75528256
bad version number 0x33 on inode 75528256
bad inode format in inode 75528256
bad magic number 0x1362 on inode 75528256, resetting magic number
bad version number 0x33 on inode 75528256, resetting version number
bad inode format in inode 75528256
cleared inode 75528256
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- clear lost+found (if it exists) ...
- clearing existing "lost+found" inode
- marking entry "lost+found" to be deleted
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
corrupt block 0 in directory inode 75501223
will junk block
no . entry for directory 75501223
no .. entry for directory 75501223
entry "cabinfo" in shortform directory 75528250 references free inode 75528256
junking entry "cabinfo" in directory inode 75528250
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- ensuring existence of lost+found directory
- traversing filesystem starting at / ...
rebuilding directory inode 128
free block 16777216 for directory inode 75555671 bad nused
rebuilding directory inode 75555671
free block 16777216 for directory inode 46172285 bad nused
rebuilding directory inode 46172285
free block 16777216 for directory inode 29690737 bad nused
rebuilding directory inode 29690737
free block 16777216 for directory inode 96485213 bad nused
rebuilding directory inode 96485213
free block 16777216 for directory inode 50930667 bad nused
rebuilding directory inode 50930667
free block 16777216 for directory inode 67109183 bad nused
rebuilding directory inode 67109183
free block 16777216 for directory inode 46458573 bad nused
rebuilding directory inode 46458573
entry "man3" in directory inode 62936853 not consistent with .. value (18446744073709551615) in ino 75501223,
will clear entry "man3"
rebuilding directory inode 62936853
free block 16777216 for directory inode 71308695 bad nused
rebuilding directory inode 71308695
free block 16777216 for directory inode 67134600 bad nused
rebuilding directory inode 67134600
free block 16777216 for directory inode 113257373 bad nused
rebuilding directory inode 113257373
free block 16777216 for directory inode 34419700 bad nused
rebuilding directory inode 34419700
free block 16777216 for directory inode 96469270 bad nused
rebuilding directory inode 96469270
free block 16777216 for directory inode 83886208 bad nused
rebuilding directory inode 83886208
- traversal finished ...
- traversing all unattached subtrees ...
corrupt block 0 in directory inode 75501223: junking block
rebuilding directory inode 75501223
creating missing "." entry in dir ino 75501223
- traversals finished ...
- moving disconnected inodes to lost+found ...
disconnected inode 12589404, moving to lost+found
disconnected inode 12591780, moving to lost+found
disconnected inode 12629480, moving to lost+found
disconnected inode 12646097, moving to lost+found
disconnected inode 12654206, moving to lost+found
disconnected inode 12654359, moving to lost+found
disconnected inode 12894519, moving to lost+found
disconnected inode 12894520, moving to lost+found
disconnected inode 12895231, moving to lost+found
disconnected inode 16777353, moving to lost+found
disconnected inode 16778367, moving to lost+found
disconnected inode 33608565, moving to lost+found
disconnected inode 37761455, moving to lost+found
disconnected inode 37776070, moving to lost+found
disconnected inode 46277904, moving to lost+found
disconnected inode 75497741, moving to lost+found
disconnected dir inode 75501223, moving to lost+found
disconnected inode 75501249, moving to lost+found
disconnected inode 75501510, moving to lost+found
disconnected inode 75510902, moving to lost+found
disconnected inode 75520655, moving to lost+found
disconnected inode 75520656, moving to lost+found
disconnected inode 75521710, moving to lost+found
disconnected inode 75528257, moving to lost+found
disconnected inode 75528268, moving to lost+found
disconnected inode 75533032, moving to lost+found
disconnected inode 75533049, moving to lost+found
disconnected inode 75533053, moving to lost+found
disconnected inode 75534873, moving to lost+found
disconnected inode 75541216, moving to lost+found
disconnected inode 75549879, moving to lost+found
disconnected inode 75560335, moving to lost+found
disconnected inode 75616906, moving to lost+found
disconnected inode 75616909, moving to lost+found
disconnected inode 75616910, moving to lost+found
disconnected inode 75616911, moving to lost+found
disconnected inode 75616912, moving to lost+found
disconnected inode 75616914, moving to lost+found
disconnected inode 75616917, moving to lost+found
disconnected inode 75616918, moving to lost+found
disconnected inode 75616919, moving to lost+found
disconnected inode 75616920, moving to lost+found
disconnected inode 75616921, moving to lost+found
disconnected inode 75616922, moving to lost+found
disconnected inode 75616923, moving to lost+found
disconnected inode 75616924, moving to lost+found
disconnected inode 75616927, moving to lost+found
disconnected inode 75619840, moving to lost+found
disconnected inode 75619854, moving to lost+found
disconnected inode 75619856, moving to lost+found
disconnected inode 75619858, moving to lost+found
disconnected inode 75619860, moving to lost+found
disconnected inode 75619862, moving to lost+found
disconnected inode 75619874, moving to lost+found
disconnected inode 75619876, moving to lost+found
disconnected inode 75619880, moving to lost+found
disconnected inode 75619882, moving to lost+found
disconnected inode 75619884, moving to lost+found
disconnected inode 75619892, moving to lost+found
disconnected inode 75619896, moving to lost+found
disconnected inode 75619898, moving to lost+found
disconnected inode 75619902, moving to lost+found
disconnected inode 75620194, moving to lost+found
disconnected inode 75620206, moving to lost+found
disconnected inode 75620208, moving to lost+found
disconnected inode 75633584, moving to lost+found
disconnected inode 75633587, moving to lost+found
disconnected inode 75633597, moving to lost+found
disconnected inode 75633601, moving to lost+found
disconnected inode 75633602, moving to lost+found
disconnected inode 75633608, moving to lost+found
disconnected inode 75633616, moving to lost+found
disconnected inode 75633621, moving to lost+found
disconnected inode 75633625, moving to lost+found
disconnected inode 75633627, moving to lost+found
disconnected inode 75633629, moving to lost+found
disconnected inode 75636070, moving to lost+found
disconnected inode 75636091, moving to lost+found
disconnected inode 75636251, moving to lost+found
disconnected inode 75636253, moving to lost+found
disconnected inode 75636254, moving to lost+found
disconnected inode 75636255, moving to lost+found
disconnected inode 75637216, moving to lost+found
disconnected inode 75638721, moving to lost+found
disconnected inode 75638722, moving to lost+found
disconnected inode 75638724, moving to lost+found
disconnected inode 75638725, moving to lost+found
disconnected inode 75638726, moving to lost+found
disconnected inode 75638727, moving to lost+found
disconnected inode 75638729, moving to lost+found
disconnected inode 75638730, moving to lost+found
disconnected inode 75638777, moving to lost+found
disconnected inode 75638780, moving to lost+found
disconnected inode 75638782, moving to lost+found
disconnected inode 75638783, moving to lost+found
disconnected inode 75641638, moving to lost+found
disconnected inode 75641639, moving to lost+found
disconnected inode 75641640, moving to lost+found
disconnected inode 75641641, moving to lost+found
disconnected inode 75641646, moving to lost+found
disconnected inode 75641647, moving to lost+found
disconnected inode 75641648, moving to lost+found
disconnected inode 75641651, moving to lost+found
disconnected inode 75641652, moving to lost+found
disconnected inode 75641653, moving to lost+found
disconnected inode 75641655, moving to lost+found
disconnected inode 75641657, moving to lost+found
disconnected inode 75641659, moving to lost+found
disconnected inode 75641661, moving to lost+found
disconnected inode 75641665, moving to lost+found
disconnected inode 75641667, moving to lost+found
disconnected inode 75641681, moving to lost+found
disconnected inode 75641685, moving to lost+found
disconnected inode 75641693, moving to lost+found
disconnected inode 75647265, moving to lost+found
disconnected inode 75647283, moving to lost+found
disconnected inode 75647327, moving to lost+found
disconnected inode 75648228, moving to lost+found
disconnected inode 75648232, moving to lost+found
disconnected inode 75648233, moving to lost+found
disconnected inode 75649508, moving to lost+found
disconnected inode 75649511, moving to lost+found
disconnected inode 75649513, moving to lost+found
disconnected inode 75649515, moving to lost+found
disconnected inode 75649888, moving to lost+found
disconnected inode 75655519, moving to lost+found
disconnected inode 75658594, moving to lost+found
disconnected inode 75658595, moving to lost+found
disconnected inode 75658596, moving to lost+found
disconnected inode 75658598, moving to lost+found
disconnected inode 75658599, moving to lost+found
disconnected inode 75658608, moving to lost+found
disconnected inode 75658611, moving to lost+found
disconnected inode 75658613, moving to lost+found
disconnected inode 75658614, moving to lost+found
disconnected inode 75658620, moving to lost+found
disconnected inode 75658622, moving to lost+found
disconnected inode 75658625, moving to lost+found
disconnected inode 75658626, moving to lost+found
disconnected inode 75658629, moving to lost+found
disconnected inode 75658630, moving to lost+found
disconnected inode 75658632, moving to lost+found
disconnected inode 75658633, moving to lost+found
disconnected inode 75658634, moving to lost+found
disconnected inode 75658637, moving to lost+found
disconnected inode 75658640, moving to lost+found
disconnected inode 75658648, moving to lost+found
disconnected inode 75659115, moving to lost+found
disconnected inode 75659116, moving to lost+found
disconnected inode 75659117, moving to lost+found
disconnected inode 75659119, moving to lost+found
disconnected inode 75683988, moving to lost+found
disconnected inode 75683989, moving to lost+found
disconnected inode 75683997, moving to lost+found
disconnected inode 75684027, moving to lost+found
disconnected inode 75689664, moving to lost+found
disconnected inode 75689666, moving to lost+found
disconnected inode 75689673, moving to lost+found
disconnected inode 75689674, moving to lost+found
disconnected inode 75689675, moving to lost+found
disconnected inode 75689676, moving to lost+found
disconnected inode 75689679, moving to lost+found
disconnected inode 75689682, moving to lost+found
disconnected inode 75689695, moving to lost+found
disconnected inode 75689696, moving to lost+found
disconnected inode 75689703, moving to lost+found
disconnected inode 75689704, moving to lost+found
disconnected inode 75689711, moving to lost+found
disconnected inode 75689719, moving to lost+found
disconnected inode 75689752, moving to lost+found
disconnected inode 75689753, moving to lost+found
disconnected inode 75689758, moving to lost+found
disconnected inode 75689762, moving to lost+found
disconnected inode 75689764, moving to lost+found
disconnected inode 75689770, moving to lost+found
disconnected inode 75689773, moving to lost+found
disconnected inode 75689782, moving to lost+found
disconnected inode 75689786, moving to lost+found
disconnected inode 75689788, moving to lost+found
disconnected inode 75689874, moving to lost+found
disconnected inode 92336260, moving to lost+found
Phase 7 - verify and correct link counts...
Terminated
jw:/home/jason# echo $?
143
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: FS corruption and repair problem
2006-10-16 8:25 FS corruption and repair problem Jason White
@ 2006-10-16 9:52 ` martin f krafft
2006-10-16 10:10 ` Justin Piszcz
2006-10-16 13:58 ` Eric Sandeen
2 siblings, 0 replies; 7+ messages in thread
From: martin f krafft @ 2006-10-16 9:52 UTC (permalink / raw)
To: xfs
[-- Attachment #1: Type: text/plain, Size: 893 bytes --]
also sprach Jason White <jasonjgw@internode.on.net> [2006.10.16.1025 +0200]:
> The file system was shut down during a Debian package upgrade while trying to
> access apt-related files. I booted into an old 2.4 kernel on a separate
> partition and ran an old version of xfs_repair. The output is attached.
> Xfs_repair terminated abnormally (see the end of the output).
FYI:
http://blog.madduck.net/geek/2006.08.09-through-with-xfs
http://blog.madduck.net/geek/2006.08.11-xfs-zeroes
http://blog.madduck.net/geek/2006.08.30-lvm-for-filesystem-recovery
You may be able to rescue the filesystem with xfs_dump and
xfs_restore via a separate storage medium.
--
martin; (greetings from the heart of the sun.)
\____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
spamtraps: madduck.bogus@madduck.net
http://lavender.cime.net/~ricky/badgers.txt
[-- Attachment #2: Digital signature (GPG/PGP) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: FS corruption and repair problem
2006-10-16 8:25 FS corruption and repair problem Jason White
2006-10-16 9:52 ` martin f krafft
@ 2006-10-16 10:10 ` Justin Piszcz
2006-10-16 13:58 ` Eric Sandeen
2 siblings, 0 replies; 7+ messages in thread
From: Justin Piszcz @ 2006-10-16 10:10 UTC (permalink / raw)
To: Jason White; +Cc: xfs
On Mon, 16 Oct 2006, Jason White wrote:
> This is my desktop system, running kernel 2.6.17.11. Kernel 2.6.17 was
> previously installed, but I upgraded several weeks ago after I found out about
> the corruption bug in 2.6.17, which may or may not be related to the
> following.
>
> The file system was shut down during a Debian package upgrade while trying to
> access apt-related files. I booted into an old 2.4 kernel on a separate
> partition and ran an old version of xfs_repair. The output is attached.
> Xfs_repair terminated abnormally (see the end of the output).
>
> I can copy a newer version of xfs_repair to the backup partition if that would
> help, though I'm not sure which libraries to copy along with it.
>
> Also, if there is any indication of what might have caused this I would be
> interested, as this hasn't happened before and I'de especially like to find
> out whether it is a hardware problem, or related to the current kernel, or
> indeed a manifestation of the 2.6.17 bug. I've successfully mounted the
> corrupted file system and a lot of it is still relatively intact - plenty of
> files in lost+found, especially from the /usr/share/man3 directory for obvious
> reasons.
>
> It's a straightforward desktop machine with an IDE drive. I haven't updated
> Debian packages on here for a while, so it isn't clear when the corruption
> happened.
>
>
Most likely if you booted and mounted the partition r/w with 2.6.17(.1-.6)
then it is most likely a result of that.
Either follow the instructions on the XFS FAQ on how to fix the problem or
re-install and bypass 2.6.17-2.6.17.6.
Backup your data first if possible before you do anything!
Justin.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: FS corruption and repair problem
2006-10-16 8:25 FS corruption and repair problem Jason White
2006-10-16 9:52 ` martin f krafft
2006-10-16 10:10 ` Justin Piszcz
@ 2006-10-16 13:58 ` Eric Sandeen
2006-10-17 5:35 ` Jason White
2 siblings, 1 reply; 7+ messages in thread
From: Eric Sandeen @ 2006-10-16 13:58 UTC (permalink / raw)
To: Jason White; +Cc: xfs
Jason White wrote:
> This is my desktop system, running kernel 2.6.17.11. Kernel 2.6.17 was
> previously installed, but I upgraded several weeks ago after I found out about
> the corruption bug in 2.6.17, which may or may not be related to the
> following.
>
> The file system was shut down during a Debian package upgrade while trying to
> access apt-related files. I booted into an old 2.4 kernel on a separate
> partition and ran an old version of xfs_repair. The output is attached.
> Xfs_repair terminated abnormally (see the end of the output).
The original reason for the shutdown would probably be interesting here, that's
missing information about the first problem you encountered.
But, from the repair output, it looks like corrupted directory data on disk,
hard to say when/why it occurred. Newer repair is always a good idea, but if
the directory is badly corrupted then there's not a lot of magic to be done.
-Eric
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: FS corruption and repair problem
2006-10-16 13:58 ` Eric Sandeen
@ 2006-10-17 5:35 ` Jason White
2006-10-17 7:37 ` Martin Steigerwald
0 siblings, 1 reply; 7+ messages in thread
From: Jason White @ 2006-10-17 5:35 UTC (permalink / raw)
To: xfs
On Mon, Oct 16, 2006 at 08:58:02AM -0500, Eric Sandeen wrote:
> The original reason for the shutdown would probably be interesting here, that's
> missing information about the first problem you encountered.
Unfortunately, yes. The first I knew about it came in the form of i/o errors
and processes terminating with error 990.
>
> But, from the repair output, it looks like corrupted directory data on disk,
> hard to say when/why it occurred. Newer repair is always a good idea, but if
> the directory is badly corrupted then there's not a lot of magic to be done.
The newer repair ran to completion and corrected more errors.
If the corruption recurs, the next step will be to reinstall everything.
Fortunately, I have multiple backups of important files, though one of them
was to an XFS file system created under kernel 2.6.17, which had minor
directory corruption that was fixed by xfs_repair. After mounting that drive
under 2.6.18, I created an entirely new backup and compared it with the old;
the only files which had changed were those which I knew to have been updated
since the previous backup.
After unmounting the backup fs I ran xfs_check, which reported no problems.
Thus I suspect that in the case of the backup drive, the FS corruption was due
to the 2.6.17 bug, since copying gigabytes of files to it failed to reproduce
the problem, and all seems fine under 2.6.18.
I am still pleased with XFS as I have been using it since 2001, and the only
problems have been due to bugs that were quickly fixed by the developers, and
most of these happened prior to the integration of XFS into the mainline.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: FS corruption and repair problem
2006-10-17 5:35 ` Jason White
@ 2006-10-17 7:37 ` Martin Steigerwald
2006-10-17 13:18 ` Martin Steigerwald
0 siblings, 1 reply; 7+ messages in thread
From: Martin Steigerwald @ 2006-10-17 7:37 UTC (permalink / raw)
To: xfs
Am Dienstag 17 Oktober 2006 07:35 schrieb Jason White:
> Unfortunately, yes. The first I knew about it came in the form of i/o
> errors and processes terminating with error 990.
Hello Jason,
this is the error code XFS returns when it detects corruption. AFAIK it
has been changed to something more meaningful lately.
> After unmounting the backup fs I ran xfs_check, which reported no
> problems. Thus I suspect that in the case of the backup drive, the FS
> corruption was due to the 2.6.17 bug, since copying gigabytes of files
> to it failed to reproduce the problem, and all seems fine under 2.6.18.
>
> I am still pleased with XFS as I have been using it since 2001, and the
> only problems have been due to bugs that were quickly fixed by the
> developers, and most of these happened prior to the integration of XFS
> into the mainline.
Since 2.6.17.7 I had no fs corruption with XFS anymore. Neither on my
laptop nor on my workstation. On my laptop I had quite some issues with
2.6.16 which I omitted on my Workstation except 2.6.16rc1.
I am currently running 2.6.18 and am pretty pleased.
Make sure that write barriers are enabled (is default since 2.6.17, was
not the default 2.6.16 which IMHO at least partly contributed to my
problems with it). Umount an XFS partition, do tail -f /var/log/syslog or
tail -f /var/log/messages and mount it again. If XFS can't do write
barriers on your machine, it should put an error message into log. You
can find more about this is in the approbiate log entry.
Regards,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2006-10-17 14:24 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-10-16 8:25 FS corruption and repair problem Jason White
2006-10-16 9:52 ` martin f krafft
2006-10-16 10:10 ` Justin Piszcz
2006-10-16 13:58 ` Eric Sandeen
2006-10-17 5:35 ` Jason White
2006-10-17 7:37 ` Martin Steigerwald
2006-10-17 13:18 ` Martin Steigerwald
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox