warning: is_tree_node: node level 56362 does not match to the expected one 1

warning: is_tree_node: node level 56362 does not match to the expected one 1

[Ralf Gross]

Hi,

I'm not sure if this is the right list, but I couldn't find a -user
list. So I'll try my luck here.

Last Friday a backup server running backuppc crashed and did not
respond to pings anymore. I logged in using the IPMI management card
and found this (sorry I couldn't scroll up or get the output from any
logfile):

http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console.png

Next I reset the system and booted. But I couldn't boot the system to
the login prompt. Many reiserfs warnings appeared.

To start reiserfsck I booted grml from CD. The first run with the
check option resulted in errors and I restarted reiserfsck with the
--rebuilt-tree option which complained after 40% about a possible
harware problems.

http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console-1.png

With the rescue CD I could also take a look at the system logfiles
from the time of the crash:


Jun  5 21:59:20 server -- MARK --
Jun  5 22:05:54 server kernel: ReiserFS: warning: is_tree_node: node level 56362 does not match to the expected one 1
Jun  5 22:05:54 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514510379. Fsck?
Jun  5 22:05:54 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:06:04 server kernel: ReiserFS: warning: is_tree_node: node level 26161 does not match to the expected one 1
Jun  5 22:06:04 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514464681. Fsck?
Jun  5 22:06:04 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:07:12 server kernel: ReiserFS: warning: is_tree_node: node level 2318 does not match to the expected one 1
Jun  5 22:07:12 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514480243. Fsck?
Jun  5 22:07:12 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:07:16 server kernel: ReiserFS: warning: is_tree_node: node level 26161 does not match to the expected one 1
Jun  5 22:07:16 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514464681. Fsck?
Jun  5 22:07:16 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:20 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:20 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:20 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:29 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:29 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:29 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:29 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:29 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:29 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:49 server kernel: ReiserFS: warning: is_tree_node: node level 26161 does not match to the expected one 1
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514464681. Fsck?
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:49 server kernel: ReiserFS: warning: is_tree_node: node level 26161 does not match to the expected one 1
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514464681. Fsck?
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:49 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:49 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:49 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:49 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:49 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:10:49 server kernel: ReiserFS: warning: is_tree_node: node level 25139 does not match to the expected one 1
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514472403. Fsck?
Jun  5 22:10:49 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
Jun  5 22:39:20 server -- MARK --
Jun  5 22:59:20 server -- MARK --
Jun  5 23:19:20 server -- MARK --


Here is the complete logfile:

http://pirx.askja.de/messages



The server is running Debian Etch with 2.6.18 kernel.

reiser4progs	1.0.5-2
reiserfsprogs   3.6.19-4    

The reiserfs is built on top of 3 raid volumes (Areca sata controller,
7 x 750 GB, raid 5, each volume 1,5 TB) and one large LVM vg.

  --- Volume group ---
  VG Name               vg-data
  System ID             
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               4.09 TB
  PE Size               4.00 MB
  Total PE              1072880
  Alloc PE / Size       1072880 / 4.09 TB
  Free  PE / Size       0 / 0   
  VG UUID               gvgzD2-0Xff-GEDR-1WlY-7OOD-Bwi4-5tig7


Hardware:

Supermicro Mainboard
C2D CPU
4 GB ECC RAM
Areca SATA RAID Controller

The system was running as backuppc server for 18 months without
problems. No power failure or other harware problems were detected
before the fail.

Since Friday I've been running memtest86+ (20 passes, app. 25 hours),
prime95 and badblocks. No problems so far.


What can I do next? Any chance to save the fs? Gladly I've a second
backup system, so not all backups are lost. But if I can get this fs
back, I'd be happy.

Ralf

Re: warning: is_tree_node: node level 56362 does not match to the expected one 1

[Edward Shishkin]

Ralf Gross wrote:
> Hi,
>   

Hello.

> I'm not sure if this is the right list, but I couldn't find a -user
> list. So I'll try my luck here.
>
> Last Friday a backup server running backuppc crashed and did not
> respond to pings anymore. I logged in using the IPMI management card
> and found this (sorry I couldn't scroll up or get the output from any
> logfile):
>
> http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console.png
>   

There is no useful info here..

I also don't have successful experience with such remote consoles:
It doesn't fit a stacktrace. Desperate attempts to parse packets with
ngrep didn't' t lead to happy end..

> Next I reset the system and booted. But I couldn't boot the system to
> the login prompt. Many reiserfs warnings appeared.
>
> To start reiserfsck I booted grml from CD. The first run with the
> check option resulted in errors and I restarted reiserfsck with the
> --rebuilt-tree option which complained after 40% about a possible
> harware problems.
>
> http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console-1.png
>
> With the rescue CD I could also take a look at the system logfiles
> from the time of the crash:
>
>   

olala... it definitely seems like hardware problems..

> Jun  5 21:59:20 server -- MARK --
> Jun  5 22:05:54 server kernel: ReiserFS: warning: is_tree_node: node level 56362 does not match to the expected one 1
> Jun  5 22:05:54 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514510379. Fsck?
> Jun  5 22:05:54 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
>   
[...]
>
> Here is the complete logfile:
>
> http://pirx.askja.de/messages
>
>
>
> The server is running Debian Etch with 2.6.18 kernel.
>
> reiser4progs	1.0.5-2
> reiserfsprogs   3.6.19-4    
>
> The reiserfs is built on top of 3 raid volumes (Areca sata controller,
> 7 x 750 GB, raid 5, each volume 1,5 TB) and one large LVM vg.
>
>   --- Volume group ---
>   VG Name               vg-data
>   System ID             
>   Format                lvm2
>   Metadata Areas        3
>   Metadata Sequence No  2
>   VG Access             read/write
>   VG Status             resizable
>   MAX LV                0
>   Cur LV                1
>   Open LV               0
>   Max PV                0
>   Cur PV                3
>   Act PV                3
>   VG Size               4.09 TB
>   PE Size               4.00 MB
>   Total PE              1072880
>   Alloc PE / Size       1072880 / 4.09 TB
>   Free  PE / Size       0 / 0   
>   VG UUID               gvgzD2-0Xff-GEDR-1WlY-7OOD-Bwi4-5tig7
>
>
> Hardware:
>
> Supermicro Mainboard
> C2D CPU
> 4 GB ECC RAM
> Areca SATA RAID Controller
>
> The system was running as backuppc server for 18 months without
> problems. No power failure or other harware problems were detected
> before the fail.
>
> Since Friday I've been running memtest86+ (20 passes, app. 25 hours),
> prime95 and badblocks. No problems so far.
>
>   

Did you check all your hard drives with badblocks after the crash?

If yes, then try to build and check the same configuration with the
same hard drives on another box. No more ideas..

Thanks,
Edward.

> What can I do next? Any chance to save the fs? Gladly I've a second
> backup system, so not all backups are lost. But if I can get this fs
> back, I'd be happy.
>
> Ralf
> --
> To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   

Re: warning: is_tree_node: node level 56362 does not match to the expected one 1

[Ralf Gross]

Edward Shishkin schrieb:
> > Last Friday a backup server running backuppc crashed and did not
> > respond to pings anymore. I logged in using the IPMI management card
> > and found this (sorry I couldn't scroll up or get the output from any
> > logfile):
> >
> > http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console.png
> >   
> 
> There is no useful info here..
> 
> I also don't have successful experience with such remote consoles:
> It doesn't fit a stacktrace. Desperate attempts to parse packets with
> ngrep didn't' t lead to happy end..

Maybe I should use netconsole on my servers, but as the server was not
responding to ping packets, I doubt it would have been a big help.
 
> > Next I reset the system and booted. But I couldn't boot the system to
> > the login prompt. Many reiserfs warnings appeared.
> >
> > To start reiserfsck I booted grml from CD. The first run with the
> > check option resulted in errors and I restarted reiserfsck with the
> > --rebuilt-tree option which complained after 40% about a possible
> > harware problems.
> >
> > http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console-1.png
> >
> > With the rescue CD I could also take a look at the system logfiles
> > from the time of the crash:
> >
> >   
> 
> olala... it definitely seems like hardware problems..

But I can't find any hardware problems for the last 2 days.
 
> > Jun  5 21:59:20 server -- MARK --
> > Jun  5 22:05:54 server kernel: ReiserFS: warning: is_tree_node: node level 56362 does not match to the expected one 1
> > Jun  5 22:05:54 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514510379. Fsck?
> > Jun  5 22:05:54 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
> >   
> [...]
> > Hardware:
> >
> > Supermicro Mainboard
> > C2D CPU
> > 4 GB ECC RAM
> > Areca SATA RAID Controller
> >
> > The system was running as backuppc server for 18 months without
> > problems. No power failure or other harware problems were detected
> > before the fail.
> >
> > Since Friday I've been running memtest86+ (20 passes, app. 25 hours),
> > prime95 and badblocks. No problems so far.
> >
> >   
> 
> Did you check all your hard drives with badblocks after the crash?

Not one by one, but the 3 raid volume sets (sdc, sdd and sde). No bad
blocks were found (non destructiv badblock test of course). I also
used the "Check Volume Set" option of the Arcea controller. 
 
> If yes, then try to build and check the same configuration with the
> same hard drives on another box. No more ideas.

Moving the disks to another server is not possible I fear.


Ralf

Re: warning: is_tree_node: node level 56362 does not match to the expected one 1

[Edward Shishkin]

Ralf Gross wrote:
> Edward Shishkin schrieb:
>   
>>> Last Friday a backup server running backuppc crashed and did not
>>> respond to pings anymore. I logged in using the IPMI management card
>>> and found this (sorry I couldn't scroll up or get the output from any
>>> logfile):
>>>
>>> http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console.png
>>>   
>>>       
>> There is no useful info here..
>>
>> I also don't have successful experience with such remote consoles:
>> It doesn't fit a stacktrace. Desperate attempts to parse packets with
>> ngrep didn't' t lead to happy end..
>>     
>
> Maybe I should use netconsole on my servers, but as the server was not
> responding to ping packets, I doubt it would have been a big help.
>  
>   
>>> Next I reset the system and booted. But I couldn't boot the system to
>>> the login prompt. Many reiserfs warnings appeared.
>>>
>>> To start reiserfsck I booted grml from CD. The first run with the
>>> check option resulted in errors and I restarted reiserfsck with the
>>> --rebuilt-tree option which complained after 40% about a possible
>>> harware problems.
>>>
>>> http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console-1.png
>>>
>>> With the rescue CD I could also take a look at the system logfiles
>>> from the time of the crash:
>>>
>>>   
>>>       
>> olala... it definitely seems like hardware problems..
>>     
>
> But I can't find any hardware problems for the last 2 days.
>  
>   

How did you test it?
Were relevant controllers, cables, etc. involved?

>>> Jun  5 21:59:20 server -- MARK --
>>> Jun  5 22:05:54 server kernel: ReiserFS: warning: is_tree_node: node level 56362 does not match to the expected one 1
>>> Jun  5 22:05:54 server kernel: ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in block 514510379. Fsck?
>>> Jun  5 22:05:54 server kernel: ReiserFS: dm-0: warning: zam-7001: io error in reiserfs_find_entry
>>>   
>>>       
>> [...]
>>     
>>> Hardware:
>>>
>>> Supermicro Mainboard
>>> C2D CPU
>>> 4 GB ECC RAM
>>> Areca SATA RAID Controller
>>>
>>> The system was running as backuppc server for 18 months without
>>> problems. No power failure or other harware problems were detected
>>> before the fail.
>>>
>>> Since Friday I've been running memtest86+ (20 passes, app. 25 hours),
>>> prime95 and badblocks. No problems so far.
>>>
>>>   
>>>       
>> Did you check all your hard drives with badblocks after the crash?
>>     
>
> Not one by one, but the 3 raid volume sets (sdc, sdd and sde). No bad
> blocks were found (non destructiv badblock test of course). I also
> used the "Check Volume Set" option of the Arcea controller. 
>  
>   
>> If yes, then try to build and check the same configuration with the
>> same hard drives on another box. No more ideas.
>>     
>
> Moving the disks to another server is not possible I fear.
>   

How much space is occupied on dm-0? (df -h)

Re: warning: is_tree_node: node level 56362 does not match to the expected one 1

[Ralf Gross]

Edward Shishkin schrieb:
> Ralf Gross wrote:
> > Edward Shishkin schrieb:
> >>>
> >>> http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console-1.png
> >>>
> >>> With the rescue CD I could also take a look at the system logfiles
> >>> from the time of the crash:
> >>>
> >>>   
> >>>       
> >> olala... it definitely seems like hardware problems..
> >>     
> >
> > But I can't find any hardware problems for the last 2 days.
> >  
> >   
> 
> How did you test it?
> Were relevant controllers, cables, etc. involved?

I didn't check the cables or opened the chassis. But neither memtest
nor badblocks found anything interesting. 

> [...]
> >> If yes, then try to build and check the same configuration with the
> >> same hard drives on another box. No more ideas.
> >>     
> >
> > Moving the disks to another server is not possible I fear.
> >   
> 
> How much space is occupied on dm-0? (df -h)

~ 1 TB.

Anyway, I couldn't wait any longer to get the system online again. So
I started from scratch and created one new raid volume (instead of 3)
and don't use lvm anymore. As the controller has the ability to expand
raid volumes, I won't need lvm.

Next I'll fill the filesystem and calculate the md5sums for a couple
of times. If this doesn't shows any problems, I *hope* that I can
trust the hardware.  

Any other ideas? Is there a tool that does the above thing (fill a fs
and check the md5sums x times)?

Ralf

Re: warning: is_tree_node: node level 56362 does not match to the expected one 1

[Edward Shishkin]

Ralf Gross wrote:
> Edward Shishkin schrieb:
>   
>> Ralf Gross wrote:
>>     
>>> Edward Shishkin schrieb:
>>>       
>>>>> http://pirx.askja.de/Supermicro_Daughter_Card_Remote_Console-1.png
>>>>>
>>>>> With the rescue CD I could also take a look at the system logfiles
>>>>> from the time of the crash:
>>>>>
>>>>>   
>>>>>       
>>>>>           
>>>> olala... it definitely seems like hardware problems..
>>>>     
>>>>         
>>> But I can't find any hardware problems for the last 2 days.
>>>  
>>>   
>>>       
>> How did you test it?
>> Were relevant controllers, cables, etc. involved?
>>     
>
> I didn't check the cables or opened the chassis. But neither memtest
> nor badblocks found anything interesting. 
>
>   
>> [...]
>>     
>>>> If yes, then try to build and check the same configuration with the
>>>> same hard drives on another box. No more ideas.
>>>>     
>>>>         
>>> Moving the disks to another server is not possible I fear.
>>>   
>>>       
>> How much space is occupied on dm-0? (df -h)
>>     
>
> ~ 1 TB.
>
> Anyway, I couldn't wait any longer to get the system online again. So
> I started from scratch and created one new raid volume (instead of 3)
> and don't use lvm anymore. As the controller has the ability to expand
> raid volumes, I won't need lvm.
>
> Next I'll fill the filesystem and calculate the md5sums for a couple
> of times. If this doesn't shows any problems, I *hope* that I can
> trust the hardware.  
>
> Any other ideas? Is there a tool that does the above thing (fill a fs
> and check the md5sums x times)?
>   

I find md5sums over fs rather useless.

If rebuild-tree stops at the same block, then it would make sense to export
the partition somehow to look whats up..