invalid directory entry - bad magic number on inode

invalid directory entry - bad magic number on inode

[Martin Steigerwald]

Hello,

today, I got a strange problem with XFS that seemed to affect only one 
file. I stumbled upon it as I tried to remove the package serendipity on 
my Debian mostly Etch/some Sid/probably some Experimental:

-----------------------------------------------------------------------
deepdance:~#130> aptitude purge serendipity
Paketlisten werden gelesen... Fertig
[...]
Entferne serendipity ...
dpkg: Fehler beim Bearbeiten von serendipity (--purge):
 
Kann »/usr/share/serendipity/www/plugins/serendipity_event_livesearch/lang_pt_PT.inc.php« 
nicht entfernen: Das Argument ist ungültig
Fehler traten auf beim Bearbeiten von:
 serendipity
[...]
-----------------------------------------------------------------------

That means that the package could not removed cause that PHP file could 
not be removed

"Das Argument ist ungültig" means in english => "The argument is invalid"

-----------------------------------------------------------------------
deepdance:~#2> 
ls -l /usr/share/serendipity/www/plugins/serendipity_event_livesearch/lang_pt_PT.inc.php
ls: /usr/share/serendipity/www/plugins/serendipity_event_livesearch/lang_pt_PT.inc.php: 
Das Argument ist ungültig
deepdance:~#2> 
ls -l /usr/share/serendipity/www/plugins/serendipity_event_livesearch/
insgesamt 0
?--------- ? ? ? ?                ? /usr/share/serendipity/www/plugins/serendipity_event_livesearch/lang_pt_PT.inc.php
-----------------------------------------------------------------------

Now that looks strange. I thought I better boot into SUSE 10.1 and check 
my Debian root partition.

-----------------------------------------------------------------------
deepdance:~ # xfs_check /dev/hda5
agi unlinked bucket 14 is 4110 in ag 0 (inode=4110)
agi unlinked bucket 25 is 12633 in ag 0 (inode=12633)
agi unlinked bucket 33 is 929 in ag 0 (inode=929)
[... more of that kinda usual stuff ...]
agi unlinked bucket 60 is 233532 in ag 9 (inode=37982268)
agi unlinked bucket 61 is 233533 in ag 9 (inode=37982269)
bad magic number 0 for inode 43113648
agi unlinked bucket 0 is 74624 in ag 11 (inode=46211968)
agi unlinked bucket 1 is 74625 in ag 11 (inode=46211969)
[...]
agi unlinked bucket 57 is 47865 in ag 15 (inode=62962425)
agi unlinked bucket 58 is 164986 in ag 15 (inode=63079546)
block 10/73160 type unknown not expected
allocated inode 4110 has 0 link count
allocated inode 929 has 0 link count
[... more of that ...]
allocated inode 37982268 has 0 link count
allocated inode 37982269 has 0 link count
link count mismatch for inode 43113648 (name ?), nlink 0, counted 1
allocated inode 46140958 has 0 link count
allocated inode 46137510 has 0 link count
[... more of that ...]
-----------------------------------------------------------------------

Hmmm, seems 43113648 is corrupted.

As I need the laptop tomorrow for work I ran xfs_repair without much 
further diagnostics. It seems to have fixed it the problem properly:

-----------------------------------------------------------------------
deepdance:~ # xfs_repair /dev/hda5
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
error following ag 1 unlinked list
error following ag 2 unlinked list
error following ag 3 unlinked list
error following ag 8 unlinked list
error following ag 11 unlinked list
error following ag 12 unlinked list
error following ag 15 unlinked list

-----------------------------------------------------------------------

Are those above harmless?

-----------------------------------------------------------------------

Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
bad magic number 0x0 on inode 43113648
bad version number 0x2d on inode 43113648
bad inode format in inode 43113648
bad magic number 0x0 on inode 43113648, resetting magic number
bad version number 0x2d on inode 43113648, resetting version number
bad inode format in inode 43113648
cleared inode 43113648
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - clear lost+found (if it exists) ...
        - clearing existing "lost+found" inode
        - marking entry "lost+found" to be deleted
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
entry "lang_pt_PT.inc.php" in shortform directory 43058698 references free 
inode 43113648
junking entry "lang_pt_PT.inc.php" in directory inode 43058698
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - ensuring existence of lost+found directory
        - traversing filesystem starting at / ...
rebuilding directory inode 128
        - traversal finished ...
        - traversing all unattached subtrees ...
        - traversals finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 929, moving to lost+found
disconnected inode 3366, moving to lost+found
[...more of those...]
-----------------------------------------------------------------------

Another run of xfs_check remains completely silent. All seems well again.

Anyone any idea what that might have been?

Unfortunately I cannot provide much further info. I do not even now when 
this problem might have occured first.

There are no traces of XFS problems in /var/log/syslog.

I will do a backup now and then run a SMART offline check just to be 
sure...

Any hints on what I could try if it ever happens again? Suppose I can save 
out the bad inode contents before I let xfs_repair fix it...

Aside that so far no problems with XFS since 2.6.17.7 whatsoever ;-).

This is on IBM ThinkPad T23 with 768 MB (it swapped out nevertheless... I 
had two KDE sessions running and whatnot...) and this kernel:

deepdance:~> cat /proc/version
Linux version 2.6.18.1-ck1-tp23-sws2-2.2.8 (root@deepdance) (gcc-Version 
4.1.2 20060901 (prerelease) (Debian 4.1.1-13)) #1 PREEMPT Wed Oct 25 
11:55:21 CEST 2006

It contains suspend2 patches by Nigel Cunningham and responsive ness 
patches by Con Koliva. Apart from that it is vanilla.

Regards,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: invalid directory entry - bad magic number on inode

[Martin Steigerwald]

Am Montag 11 Dezember 2006 15:05 schrieb Lachlan McIlroy:

> > Any hints on what I could try if it ever happens again? Suppose I can
> > save out the bad inode contents before I let xfs_repair fix it...
>
> If you see it again could you run xfs_check in verbose mode (ie
> xfs_check -v /dev/hda5 and xfs_check -v -i <inode number> /dev/hda5)?

Hello Lachlan,

thanks for the hints. So far it didn't occur again (at least not that I am 
aware of) - well I don't expect it to happen often and am not really sure 
whether it will happen again at all.

> Thanks for reporting this problem.

You are welcome. Without further info reporting it to the bugtracker 
doesn't make much sense to me. I will keep an eye on it. If it happens 
again, I  will try the hints you gave me. I run xfs_check anyway and thus 
can easily give it a "-v >xfs_check.txt". I thought I would have to use 
xfs_db stuff to get further info.

Regards,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: invalid directory entry - bad magic number on inode

[Lachlan McIlroy]

Martin Steigerwald wrote:
> Am Montag 11 Dezember 2006 15:05 schrieb Lachlan McIlroy:
> 
> 
>>>Any hints on what I could try if it ever happens again? Suppose I can
>>>save out the bad inode contents before I let xfs_repair fix it...
>>
>>If you see it again could you run xfs_check in verbose mode (ie
>>xfs_check -v /dev/hda5 and xfs_check -v -i <inode number> /dev/hda5)?
> 
> 
> Hello Lachlan,
> 
> thanks for the hints. So far it didn't occur again (at least not that I am 
> aware of) - well I don't expect it to happen often and am not really sure 
> whether it will happen again at all.
> 
> 
>>Thanks for reporting this problem.
> 
> 
> You are welcome. Without further info reporting it to the bugtracker 
> doesn't make much sense to me. I will keep an eye on it. If it happens 
> again, I  will try the hints you gave me. I run xfs_check anyway and thus 
> can easily give it a "-v >xfs_check.txt". I thought I would have to use 
> xfs_db stuff to get further info.

Now that you mention it, printing out the inode in xfs_db might be useful.

> 
> Regards,

hints on how to help debugging as FAQ entry (Re: invalid directory entry - bad magic number on inode)

[Martin Steigerwald]

Am Dienstag 12 Dezember 2006 01:40 schrieb Lachlan McIlroy:

> > You are welcome. Without further info reporting it to the bugtracker
> > doesn't make much sense to me. I will keep an eye on it. If it
> > happens again, I  will try the hints you gave me. I run xfs_check
> > anyway and thus can easily give it a "-v >xfs_check.txt". I thought I
> > would have to use xfs_db stuff to get further info.
>
> Now that you mention it, printing out the inode in xfs_db might be
> useful.

Hello Lachlan,

well I can do that too... my problem is just: As I use the notebook for 
daily work I have to fix it up quickly when problems arise. So usually I 
can not afford to report first and await instructions on what to do. I 
also currently often have no storage space left to store the complete 
partition onto. 

So ideally I have some hints on how to help debugging before an incident 
happens. I think this would make a nice FAQ entry, too. 

It could contain:

1) xfs_check -v <device>
2) xfs_check -v -i <inode> <device>
3) xfs_db stuff
4) probably some hints to determine a useful range for dd / ddrescue from 
xfs_check output so that people with either very large partitions or low 
storage space can just copy a part of the defect partition for 
inspection. Well if thats useful. A complete partition would still be 
better cause it is possible to use the XFS tools on it
5) probably some hints on how to store a partition in a file with 
compression... somewhere along the lines of piping dd into bzip2 and 
bzip2 into a file. maybe "cat /dev/hda1 | bzip2 >mypartition"

What do you think?

Those hints could help XFS developers to get useful bug reports...

I am willing to collect the hints and writing a FAQ entry about it that 
you can include in your FAQ.

Well that would probably basically be an extension to:

" Q: What information should I include when reporting a problem?"

Regards,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: hints on how to help debugging as FAQ entry (Re: invalid directory entry - bad magic number on inode)

[Lachlan McIlroy]

Martin Steigerwald wrote:
> Am Dienstag 12 Dezember 2006 01:40 schrieb Lachlan McIlroy:
> 
> 
>>>You are welcome. Without further info reporting it to the bugtracker
>>>doesn't make much sense to me. I will keep an eye on it. If it
>>>happens again, I  will try the hints you gave me. I run xfs_check
>>>anyway and thus can easily give it a "-v >xfs_check.txt". I thought I
>>>would have to use xfs_db stuff to get further info.
>>
>>Now that you mention it, printing out the inode in xfs_db might be
>>useful.
> 
> 
> Hello Lachlan,
> 
> well I can do that too... my problem is just: As I use the notebook for 
> daily work I have to fix it up quickly when problems arise. So usually I 
> can not afford to report first and await instructions on what to do. I 
> also currently often have no storage space left to store the complete 
> partition onto. 
> 
> So ideally I have some hints on how to help debugging before an incident 
> happens. I think this would make a nice FAQ entry, too. 
> 
> It could contain:
> 
> 1) xfs_check -v <device>
> 2) xfs_check -v -i <inode> <device>
> 3) xfs_db stuff
> 4) probably some hints to determine a useful range for dd / ddrescue from 
> xfs_check output so that people with either very large partitions or low 
> storage space can just copy a part of the defect partition for 
> inspection. Well if thats useful. A complete partition would still be 
> better cause it is possible to use the XFS tools on it

Since the tools wont operate on a fragment of a partition I don't think
this would be feasible.  It might be possible to debug a single allocation
group on it's own but not right now as far as I'm aware.

> 5) probably some hints on how to store a partition in a file with 
> compression... somewhere along the lines of piping dd into bzip2 and 
> bzip2 into a file. maybe "cat /dev/hda1 | bzip2 >mypartition"

Is this so that the problem can be debugged while the filesystem is still
in use or is repaired?  If the file system can be repaired then the output
from repair should be enough to tell us what the problem is (the cause is 
another matter but we're unlikely to get more information from a dump of
the partition).  If the partition cannot be repaired then I don't see a
point in dumping the partition to a file since the filesystem needs to be
fixed before it should be used again.

> 
> What do you think?

Sounds good.  Our FAQ is lacking in bug triage processes so any hints to
gather additional information would be a welcome addition.

> 
> Those hints could help XFS developers to get useful bug reports...
> 
> I am willing to collect the hints and writing a FAQ entry about it that 
> you can include in your FAQ.
> 
> Well that would probably basically be an extension to:
> 
> " Q: What information should I include when reporting a problem?"
> 
> Regards,