Filesystem corrupted: "Sorry, could not find valid secondary superblock"

Filesystem corrupted: "Sorry, could not find valid secondary superblock"

[Nicolas STRANSKY]

Hi,

I'm using Debian Lenny with xfsprogs 2.9.8-1lenny1 and a kernel
2.6.26-2-amd64. Recently I had some disk failures on a 1.5TB RAID 5
array, which is now running in a degraded mode but the xfs filesystem on
this array is inconsistent. I can mount it with -o ro,norecovery and
read some files, but I can't read most of them.
Using xfs_repair, I tried to repair the filesystem. It went for 12
hours, displaying these messages:
"found candidate secondary superblock...
unable to verify superblock, continuing..."
and finally:
"Sorry, could not find valid secondary superblock
Exiting now."

Is there anything else I could do to repair the filesystem? Upgrade to a
newer version of xfsprogs? Is it possible that there is no valid
secondary superblock on the whole disk or are they just not found?

Thanks a lot for any help,
NS.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Filesystem corrupted: "Sorry, could not find valid secondary superblock"

[Eric Sandeen]

Nicolas STRANSKY wrote:
> Hi,
> 
> I'm using Debian Lenny with xfsprogs 2.9.8-1lenny1 and a kernel
> 2.6.26-2-amd64. Recently I had some disk failures on a 1.5TB RAID 5
> array, which is now running in a degraded mode but the xfs filesystem on
> this array is inconsistent. I can mount it with -o ro,norecovery and
> read some files, but I can't read most of them.
> Using xfs_repair, I tried to repair the filesystem. It went for 12
> hours, displaying these messages:
> "found candidate secondary superblock...
> unable to verify superblock, continuing..."
> and finally:
> "Sorry, could not find valid secondary superblock
> Exiting now."
> 
> Is there anything else I could do to repair the filesystem? Upgrade to a
> newer version of xfsprogs? Is it possible that there is no valid
> secondary superblock on the whole disk or are they just not found?
> 
> Thanks a lot for any help,
> NS.

xfs_repair needs to be smarter about this; I at least wish it'd say
-why- the candidate was not valid.

Can you include the whole repair output?  Then maybe we can direct
you to some xfs_db jujitsu to examine the fs.

Trying newer xfsprogs would not be a -bad- idea...

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Filesystem corrupted: "Sorry, could not find valid secondary superblock"

[Nicolas Stransky]

On 1/29/10 10:38 AM, Eric Sandeen wrote:
> xfs_repair needs to be smarter about this; I at least wish it'd say 
> -why- the candidate was not valid.
> 
> Can you include the whole repair output?  Then maybe we can direct 
> you to some xfs_db jujitsu to examine the fs.
> 
> Trying newer xfsprogs would not be a -bad- idea...
> 
> -Eric

The output is as follows:

# xfs_repair /dev/sda
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
[The previous one repeats hundreds of times over many hours]
......................
Sorry, could not find valid secondary superblock
Exiting now.


I am trying right now xfsprogs 3.1.0 from Debian testing, will see what
it gives...

Thanks!
NS

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Filesystem corrupted: "Sorry, could not find valid secondary superblock"

[Eric Sandeen]

Nicolas Stransky wrote:
> On 1/29/10 10:38 AM, Eric Sandeen wrote:
>> xfs_repair needs to be smarter about this; I at least wish it'd say 
>> -why- the candidate was not valid.
>>
>> Can you include the whole repair output?  Then maybe we can direct 
>> you to some xfs_db jujitsu to examine the fs.
>>
>> Trying newer xfsprogs would not be a -bad- idea...
>>
>> -Eric
> 
> The output is as follows:
> 
> # xfs_repair /dev/sda
> Phase 1 - find and verify superblock...
> bad primary superblock - bad magic number !!!

xfs_db /dev/sda
xfs_db> sb 0
xfs_db> p

might be interesting to see what's there.

-Eric

> attempting to find secondary superblock...
> ......................
> found candidate secondary superblock...
> unable to verify superblock, continuing...
> ......................
> found candidate secondary superblock...
> unable to verify superblock, continuing...
> ......................
> [The previous one repeats hundreds of times over many hours]
> ......................
> Sorry, could not find valid secondary superblock
> Exiting now.
> 
> 
> I am trying right now xfsprogs 3.1.0 from Debian testing, will see what
> it gives...
> 
> Thanks!
> NS
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Filesystem corrupted: "Sorry, could not find valid secondary superblock"

[Nicolas Stransky]

OK here is a bunch of new stuff:

# xfs_repair /dev/sda1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs
to be replayed.  Mount the filesystem to replay the log, and unmount it
before re-running xfs_repair.  If you are unable to mount the
filesystem, then use the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a
mount of the filesystem before doing this.

# mount/ dev/sda1
mount: Structure needs cleaning


Is the use of -L the only way to go? I wonder, since it will cause data
loss...


Also here is the output of xfs_db:

# xfs_db /dev/sda1
xfs_db> sb 0
xfs_db> p
magicnum = 0x58465342
blocksize = 4096
dblocks = 366197650
rblocks = 0
rextents = 0
uuid = dab73302-1744-4fc0-89db-5c6670a02aa0
logstart = 134217732
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 16
agblocks = 6866155
agcount = 54
rbmblocks = 0
logblocks = 32768
versionnum = 0x3484
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 23
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 4115008
ifree = 954636
fdblocks = 85242778
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 1
features2 = 0
bad_features2 = 0

Thanks!

On 1/29/10 12:12 PM, Eric Sandeen wrote:
> Nicolas Stransky wrote:
>> On 1/29/10 10:38 AM, Eric Sandeen wrote:
>>> xfs_repair needs to be smarter about this; I at least wish it'd say 
>>> -why- the candidate was not valid.
>>>
>>> Can you include the whole repair output?  Then maybe we can direct 
>>> you to some xfs_db jujitsu to examine the fs.
>>>
>>> Trying newer xfsprogs would not be a -bad- idea...
>>>
>>> -Eric
>>
>> The output is as follows:
>>
>> # xfs_repair /dev/sda
>> Phase 1 - find and verify superblock...
>> bad primary superblock - bad magic number !!!
> 
> xfs_db /dev/sda
> xfs_db> sb 0
> xfs_db> p
> 
> might be interesting to see what's there.
> 
> -Eric
> 
>> attempting to find secondary superblock...
>> ......................
>> found candidate secondary superblock...
>> unable to verify superblock, continuing...
>> ......................
>> found candidate secondary superblock...
>> unable to verify superblock, continuing...
>> ......................
>> [The previous one repeats hundreds of times over many hours]
>> ......................
>> Sorry, could not find valid secondary superblock
>> Exiting now.
>>
>>
>> I am trying right now xfsprogs 3.1.0 from Debian testing, will see what
>> it gives...
>>
>> Thanks!
>> NS
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
>>
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Filesystem corrupted: "Sorry, could not find valid secondary superblock"

[Eric Sandeen]

Nicolas Stransky wrote:
> OK here is a bunch of new stuff:
> 
> # xfs_repair /dev/sda1
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
> ERROR: The filesystem has valuable metadata changes in a log which needs
> to be replayed.  Mount the filesystem to replay the log, and unmount it
> before re-running xfs_repair.  If you are unable to mount the
> filesystem, then use the -L option to destroy the log and attempt a repair.
> Note that destroying the log may cause corruption -- please attempt a
> mount of the filesystem before doing this.
> 
> # mount/ dev/sda1
> mount: Structure needs cleaning

dmesg at this point would be good

> 
> Is the use of -L the only way to go? I wonder, since it will cause data
> loss...

if you can't mount it, yeah

> 
> Also here is the output of xfs_db:
> 
> # xfs_db /dev/sda1
> xfs_db> sb 0
> xfs_db> p
> magicnum = 0x58465342

ok, odd; you got:

>> # xfs_repair /dev/sda
>> Phase 1 - find and verify superblock...
>> bad primary superblock - bad magic number !!!

but this magic number is fine ...

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Filesystem corrupted: "Sorry, could not find valid secondary superblock"

[Nicolas Stransky]

Hi Eric,

Thanks so much for your help,

On 1/29/10 1:16 PM, Eric Sandeen wrote:
>> # mount/ dev/sda1 mount: Structure needs cleaning
> 
> dmesg at this point would be good

This particular command does not produce any output in dmesg, but
previous commands did, especially when I mounted the filesystem with
no-recovery.

[205411.967412] XFS: log mount/recovery failed: error 117
[205411.972696] XFS: log mount failed
[205469.205586] Mounting filesystem "sda1" in no-recovery mode.
Filesystem will be inconsistent.
[205483.946417] XFS mounting filesystem sda1
[205484.136159] Starting XFS recovery on filesystem: sda1 (logdev: internal)
[205484.488176] Filesystem "sda1": xfs_inode_recover: Bad inode magic
number, dino ptr = 0xffff81004d109f00, dino bp = 0xffff81002e85be80, ino
= 2954558655
[205484.493476] Filesystem "sda1": XFS internal error
xlog_recover_do_inode_trans(1) at line 2326 of file
fs/xfs/xfs_log_recover.c.  Caller 0xffffffffa018553a
[205484.498909] Pid: 27800, comm: mount Not tainted 2.6.26-2-amd64 #1
[205484.506907]
[205484.506908] Call Trace:
[205484.516417]  [<ffffffffa018553a>] :xfs:xlog_recover_do_trans+0x64/0xfd
[205484.523162]  [<ffffffffa0184e33>]
:xfs:xlog_recover_do_inode_trans+0x23c/0x766
[205484.528729]  [<ffffffffa01966b0>] :xfs:xfs_buf_iostart+0x29/0x86
[205484.534304]  [<ffffffffa018553a>] :xfs:xlog_recover_do_trans+0x64/0xfd
[205484.539907]  [<ffffffffa0185605>]
:xfs:xlog_recover_commit_trans+0x32/0x4b
[205484.545489]  [<ffffffffa018576b>]
:xfs:xlog_recover_process_data+0x14d/0x1c9
[205484.551117]  [<ffffffffa0185a2f>] :xfs:xlog_do_recovery_pass+0x248/0x661
[205484.556736]  [<ffffffffa0185e9a>] :xfs:xlog_do_log_recovery+0x52/0x75
[205484.564000]  [<ffffffffa0185ec9>] :xfs:xlog_do_recover+0xc/0xf3
[205484.568026]  [<ffffffffa018602a>] :xfs:xlog_recover+0x7a/0x84
[205484.573631]  [<ffffffffa0180460>] :xfs:xfs_log_mount+0xb1/0x105
[205484.579270]  [<ffffffffa01880dc>] :xfs:xfs_mountfs+0x25a/0x5ac
[205484.584866]  [<ffffffffa0193311>] :xfs:kmem_alloc+0x60/0xc4
[205484.590483]  [<ffffffffa01933d1>] :xfs:kmem_zalloc+0x9/0x21
[205484.596066]  [<ffffffffa018deb4>] :xfs:xfs_mount+0x29b/0x347
[205484.601625]  [<ffffffffa019c3ea>] :xfs:xfs_fs_fill_super+0x0/0x1ee
[205484.607231]  [<ffffffffa019c49f>] :xfs:xfs_fs_fill_super+0xb5/0x1ee
[205484.612727]  [<ffffffff8029d290>] get_sb_bdev+0xf8/0x145
[205484.618174]  [<ffffffff8029ccb4>] vfs_kern_mount+0x93/0x11b
[205484.623745]  [<ffffffff8029cd8f>] do_kern_mount+0x43/0xe3
[205484.629183]  [<ffffffff802b1805>] do_new_mount+0x5b/0x95
[205484.634627]  [<ffffffff802b19fc>] do_mount+0x1bd/0x1e7
[205484.639990]  [<ffffffff80276909>] __alloc_pages_internal+0xd6/0x3bf
[205484.645219]  [<ffffffff802b1ab0>] sys_mount+0x8a/0xce
[205484.650275]  [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f

>> Is the use of -L the only way to go? I wonder, since it will cause
>> data loss...
> 
> if you can't mount it, yeah

I'm going ahead and doing that.
There is an awful lot of output but at least it's doing something.

>> Also here is the output of xfs_db:
>> 
>> # xfs_db /dev/sda1 xfs_db> sb 0 xfs_db> p magicnum = 0x58465342
> 
> ok, odd; you got:
> 
>>> # xfs_repair /dev/sda Phase 1 - find and verify superblock... bad
>>> primary superblock - bad magic number !!!
> 
> but this magic number is fine ...

Yes, I figured that I had used /dev/sda instead of /dev/sda1, my bad :(
So now the magic number is ok...

-- 
Nico

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Filesystem corrupted: "Sorry, could not find valid secondary superblock"

[Nicolas Stransky]

Hello,

Unfortunately I am still having big problems with this filesystem. I can
mount it OK but there are problems reading files:

du: cannot access `./DCIS 2 AB': Structure needs cleaning

The previous goes together with a lot of these in dmesg:

[  431.976517] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
[  431.988072] Filesystem "sda1": XFS internal error xfs_da_do_buf(2) at
line 2085 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffffa016749f
[  431.993185] Pid: 4246, comm: du Not tainted 2.6.26-2-amd64 #1
[  431.998308]
[  431.998309] Call Trace:
[  432.010348]  [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29
[  432.015530]  [<ffffffffa016734d>] :xfs:xfs_da_do_buf+0x54e/0x636
[  432.020696]  [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29
[  432.025860]  [<ffffffffa019580c>] :xfs:_xfs_buf_lookup_pages+0x298/0x2ca
[  432.031974]  [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29
[  432.037173]  [<ffffffffa016c66f>] :xfs:xfs_dir2_leaf_getdents+0x381/0x61a
[  432.042391]  [<ffffffffa016c66f>] :xfs:xfs_dir2_leaf_getdents+0x381/0x61a
[  432.047617]  [<ffffffffa0196f68>] :xfs:xfs_hack_filldir+0x0/0x5b
[  432.052673]  [<ffffffffa0196f68>] :xfs:xfs_hack_filldir+0x0/0x5b
[  432.057517]  [<ffffffffa01695d8>] :xfs:xfs_readdir+0xa6/0xb5
[  432.062197]  [<ffffffffa0196f1b>] :xfs:xfs_file_readdir+0xff/0x14c
[  432.066835]  [<ffffffff802a6bec>] filldir+0x0/0xb7
[  432.071987]  [<ffffffff802a6bec>] filldir+0x0/0xb7
[  432.076557]  [<ffffffff802a6d18>] vfs_readdir+0x75/0xa7
[  432.081137]  [<ffffffff802a6f68>] sys_getdents+0x75/0xbd
[  432.085716]  [<ffffffff802a65da>] sys_fcntl+0x2eb/0x2f7
[  432.090267]  [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f

The server has 2GB of RAM and 4GB of swap. I have tried several times to
xfs_repair but it always fails because of memory problems, even if I use
'xfs_repair -m1500 /dev/sda1' to limit the memory usage.
xfs_repair produces an awful lot of output. I could of course send it
but it's 130k lines long. Here is what it is currently printing, I have
started "xfs_repair -t15 -m500 /dev/sda1" again a few minutes ago. It is
currently in Phase 3.

bp=(bno 1211109312, len 8192 bytes) key=(bno 1211109312, len 4096 bytes)
429f7950: Badness in key lookup (length)
bp=(bno 1211109328, len 8192 bytes) key=(bno 1211109328, len 4096 bytes)
bad directory block magic # 0x2f49cdbc in block 2 for directory inode
2952790372
corrupt block 2 in directory inode 2952790372
        will junk block
bad directory block magic # 0 in block 5 for directory inode 2952790372
corrupt block 5 in directory inode 2952790372
        will junk block
no . entry for directory 2952790372
no .. entry for directory 2952790372
bad directory block magic # 0 in block 0 for directory inode 2952790373
corrupt block 0 in directory inode 2952790373
        will junk block
bad directory block magic # 0 in block 1 for directory inode 2952790373
corrupt block 1 in directory inode 2952790373
        will junk block
doubling cache size to 66768
doubling cache size to 133536
doubling cache size to 267072
doubling cache size to 534144
doubling cache size to 1068288


Any ideas on how I could repair this filesystem?

Thanks a lot!

-- 
Nico

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs