Reiserfs 1300G partition on lvm problem ...

Reiserfs 1300G partition on lvm problem ...

[Matthias Barremaecker]

Hi,

I have a huge problem.

setup : LVM with 8 disks

fs : reiserfs

All of the sudden it stoped working, I can't mount it anymore.

reiserfsck said it had 43 errors and I should do

    reiserfsck --rebuild-tree

what I did.

but that sais it is a fysical drive error

Point is : it is 1300 gig. and I can't mount it.

What can I do.

Thanx.

Re: Reiserfs 1300G partition on lvm problem ...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Sun, 29 May 2005 21:25:54 +0200, Matthias Barremaecker said:

> but that sais it is a fysical drive error

Physical drive errors.  Your hardware is broken.  Isn't much that Reiserfs
can do about it.

> What can I do.

1) Call whoever you get hardware support from.

2) Be ready to restore from backups.

3) If you didn't have RAID-5 (or similar) set up, or a good backup, consider
it a learning experience.

If your data is important enough that you'll care if you lose it, you should take
steps to make sure you won't lose it... It's that simple.

(Just for the record, if we have important info, it gets at least RAID5, a
backup to tape or other device, *and* a *second* backup off-site.  And my shop
is far from the most paranoid about such things.)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Reiserfs 1300G partition on lvm problem ...

[Matthias Barremaecker]

Hi,

Thanx for your reply.

The data is not THAT importend, all our importend data is backuped 4 
times, inc. original (well, 3 times now, since the 1300gig machine is 
broke).

I did a bit furtur reasearch and maybe this is something to think about 
if you use reiserfs :

I did a bad block check and I have 10 bad blocks of 4096bytes on 1300Gig 
and ... that is the reason reiserfs will not work anymore.

I guess this sux. I rather have that the data on the bad blocks is just 
corupted but the rest is accesseble.

I'm doing a --rebuild-tree with the bad block list. Hopes this works.


Aren't there any tools to substract data from a broken reiserfs partition ?


kind regardes,

Matthias.



Valdis.Kletnieks@vt.edu wrote:
> On Sun, 29 May 2005 21:25:54 +0200, Matthias Barremaecker said:
> 
> 
>>but that sais it is a fysical drive error
> 
> 
> Physical drive errors.  Your hardware is broken.  Isn't much that Reiserfs
> can do about it.
> 
> 
>>What can I do.
> 
> 
> 1) Call whoever you get hardware support from.
> 
> 2) Be ready to restore from backups.
> 
> 3) If you didn't have RAID-5 (or similar) set up, or a good backup, consider
> it a learning experience.
> 
> If your data is important enough that you'll care if you lose it, you should take
> steps to make sure you won't lose it... It's that simple.
> 
> (Just for the record, if we have important info, it gets at least RAID5, a
> backup to tape or other device, *and* a *second* backup off-site.  And my shop
> is far from the most paranoid about such things.)
> 

-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

Re: Reiserfs 1300G partition on lvm problem ...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

On Mon, 30 May 2005 08:17:00 +0200, Matthias Barremaecker said:

> I did a bad block check and I have 10 bad blocks of 4096bytes on 1300Gig 
> and ... that is the reason reiserfs will not work anymore.

> I guess this sux. I rather have that the data on the bad blocks is just 
> corupted but the rest is accesseble.

It all depends on which 10 blocks go bad.  If it's a block that's allocated to
a file, you lose the 4K or whatever that's in that block.

If it's a block that an inode lives in, you're probably going to have the
entire file evaporate.

If it's a block that contains something even more important, you're going to
have large sections of the file system evaporate.

It's a tradeoff issue - how many times do you replicate metadata on the
filesystem, against how well the file system deals with errors.  The problem is
that if you just say "let's have 2 copies of everything, just in case", it
takes a lot more disk space to *store* 2 copies of the metadata.  Also, your
disk performance falls through the floor - most journalled filesystems have
enough trouble making sure that *one* copy of things like the free list is on
disk and consistent with the journal.  Making 2 copies is going to probably
triple your disk I/O and complicate matters a *lot* for fsck (if you crash and
the two copies aren't consistent, which one do you believe?)

That's why almost all filesystems designers just punt and assume that the media
actually works, and suggest if your media might not be 100% reliable, that you
use RAID or similar solutions....


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Minor note FYI

[Artem B. Bityuckiy]

Hello,

Look at the create_child_common() function at fs/reiser4/plugin/dir/dir.c:

There are _7_ places with the code like this in it:

result = blah();
if (result) {
	warning();
         DQUOT_FREE_INODE(object);
         object->i_flags |= S_NOQUOTA;
         return result;
}

isn't it cuter to use the classical way:

result = blah();
if (result) {
	warning();
	goto error;
}

...

error:
         DQUOT_FREE_INODE(object);
         object->i_flags |= S_NOQUOTA;
         return result;

?

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: Reiserfs 1300G partition on lvm problem ...

[Dan Oglesby]

Matthias Barremaecker wrote:
> Hi,
> 
> Thanx for your reply.
> 
> The data is not THAT importend, all our importend data is backuped 4 
> times, inc. original (well, 3 times now, since the 1300gig machine is 
> broke).
> 
> I did a bit furtur reasearch and maybe this is something to think about 
> if you use reiserfs :
> 
> I did a bad block check and I have 10 bad blocks of 4096bytes on 1300Gig 
> and ... that is the reason reiserfs will not work anymore.
> 
> I guess this sux. I rather have that the data on the bad blocks is just 
> corupted but the rest is accesseble.
> 
> I'm doing a --rebuild-tree with the bad block list. Hopes this works.
> 
> 
> Aren't there any tools to substract data from a broken reiserfs partition ?
> 
> 
> kind regardes,
> 
> Matthias.
> 
> 
> 


I'm currently in a similar situation:  1TB (RAID-5) ReiserFS filesystem 
running on RedHat 7.2 with a Promise SX6000 controller.  Everything was 
running stock version/firmware/BIOS, and it suddently developed 39 bad 
blocks after a power outage.

reiserfsck --rebuild-tree (version 3.6.4, the latest for RH 7.2) failed 
to complete after the controller kept hard locking/crashing.

So, I've compiled a plain 2.4.30 kernel (from kernel.org, not RedHat), 
updated the BIOS/Firmware on the SX6000, and compiled and installed the 
latest reiserfsprogs (3.6.19, I believe?), and have been running 
badblocks (non-destructive) for the last four days.  Nothing has turned 
up so far.

I'm hoping the next time I run a --rebuild-tree, it will be able to 
complete.  I really do need to get the data off of this filesystem.

Good luck with your recovery.

--Dan




> Valdis.Kletnieks@vt.edu wrote:
> 
>> On Sun, 29 May 2005 21:25:54 +0200, Matthias Barremaecker said:
>>
>>
>>> but that sais it is a fysical drive error
>>
>>
>>
>> Physical drive errors.  Your hardware is broken.  Isn't much that 
>> Reiserfs
>> can do about it.
>>
>>
>>> What can I do.
>>
>>
>>
>> 1) Call whoever you get hardware support from.
>>
>> 2) Be ready to restore from backups.
>>
>> 3) If you didn't have RAID-5 (or similar) set up, or a good backup, 
>> consider
>> it a learning experience.
>>
>> If your data is important enough that you'll care if you lose it, you 
>> should take
>> steps to make sure you won't lose it... It's that simple.
>>
>> (Just for the record, if we have important info, it gets at least 
>> RAID5, a
>> backup to tape or other device, *and* a *second* backup off-site.  And 
>> my shop
>> is far from the most paranoid about such things.)
>>
> 

Re: Reiserfs 1300G partition on lvm problem ...

[Matthias Barremaecker]

Hi Dan,

My recovery goes rather well...

Does the badblocks count or anything ? With me it counted.

You have to write the bad block no's to a file and feed that to the 
reiserfschk.

I didn't completed a full badblock check coz I knew the badblocks could 
only be at the beginning of the lvm 'array', but I excpect it to take as 
long as a reiserfsck --tree-rebuild.

If you don't see anything counting -- start worrying.

Good luck to you to.

I have still 18 hours to go... I realy hope I can mount the dam thing 
after that and then I can start looking for the bad disk.

And I'm not using any type of raid for that data.

kind regardes,

Matthias.

Dan Oglesby wrote:
> Matthias Barremaecker wrote:
> 
>> Hi,
>>
>> Thanx for your reply.
>>
>> The data is not THAT importend, all our importend data is backuped 4 
>> times, inc. original (well, 3 times now, since the 1300gig machine is 
>> broke).
>>
>> I did a bit furtur reasearch and maybe this is something to think 
>> about if you use reiserfs :
>>
>> I did a bad block check and I have 10 bad blocks of 4096bytes on 
>> 1300Gig and ... that is the reason reiserfs will not work anymore.
>>
>> I guess this sux. I rather have that the data on the bad blocks is 
>> just corupted but the rest is accesseble.
>>
>> I'm doing a --rebuild-tree with the bad block list. Hopes this works.
>>
>>
>> Aren't there any tools to substract data from a broken reiserfs 
>> partition ?
>>
>>
>> kind regardes,
>>
>> Matthias.
>>
>>
>>
> 
> 
> I'm currently in a similar situation:  1TB (RAID-5) ReiserFS filesystem 
> running on RedHat 7.2 with a Promise SX6000 controller.  Everything was 
> running stock version/firmware/BIOS, and it suddently developed 39 bad 
> blocks after a power outage.
> 
> reiserfsck --rebuild-tree (version 3.6.4, the latest for RH 7.2) failed 
> to complete after the controller kept hard locking/crashing.
> 
> So, I've compiled a plain 2.4.30 kernel (from kernel.org, not RedHat), 
> updated the BIOS/Firmware on the SX6000, and compiled and installed the 
> latest reiserfsprogs (3.6.19, I believe?), and have been running 
> badblocks (non-destructive) for the last four days.  Nothing has turned 
> up so far.
> 
> I'm hoping the next time I run a --rebuild-tree, it will be able to 
> complete.  I really do need to get the data off of this filesystem.
> 
> Good luck with your recovery.
> 
> --Dan
> 
> 
> 
> 
>> Valdis.Kletnieks@vt.edu wrote:
>>
>>> On Sun, 29 May 2005 21:25:54 +0200, Matthias Barremaecker said:
>>>
>>>
>>>> but that sais it is a fysical drive error
>>>
>>>
>>>
>>>
>>> Physical drive errors.  Your hardware is broken.  Isn't much that 
>>> Reiserfs
>>> can do about it.
>>>
>>>
>>>> What can I do.
>>>
>>>
>>>
>>>
>>> 1) Call whoever you get hardware support from.
>>>
>>> 2) Be ready to restore from backups.
>>>
>>> 3) If you didn't have RAID-5 (or similar) set up, or a good backup, 
>>> consider
>>> it a learning experience.
>>>
>>> If your data is important enough that you'll care if you lose it, you 
>>> should take
>>> steps to make sure you won't lose it... It's that simple.
>>>
>>> (Just for the record, if we have important info, it gets at least 
>>> RAID5, a
>>> backup to tape or other device, *and* a *second* backup off-site.  
>>> And my shop
>>> is far from the most paranoid about such things.)
>>>
>>
> 
> 
> 

-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

Re: Reiserfs 1300G partition on lvm problem ...

[Dan Oglesby]

Matthias Barremaecker wrote:
> Hi Dan,
> 
> My recovery goes rather well...
> 
> Does the badblocks count or anything ? With me it counted.
> 

So far, I haven't seen any bad blocks written to my output file.  I'm 
not in front of the machine (remote location), so I can't see what's on 
the console, where I'm actually running the command.

> You have to write the bad block no's to a file and feed that to the 
> reiserfschk.
> 

Yeah, that's what I'm doing.

> I didn't completed a full badblock check coz I knew the badblocks could 
> only be at the beginning of the lvm 'array', but I excpect it to take as 
> long as a reiserfsck --tree-rebuild.
> 
> If you don't see anything counting -- start worrying.
> 

I have no idea if there are even real bad blocks on my array.  I think 
the controller's BIOS/firmware was so old, it didn't know how to deal 
with the power outage in a sane manner.  The last reiserfsck failed 
after a day of running, so I'm thinking if I have a problem, it's 
towards the end of my array.  Time will tell.

> Good luck to you to.
> 
> I have still 18 hours to go... I realy hope I can mount the dam thing 
> after that and then I can start looking for the bad disk.
> 

Thanks, and same here...  :-)

> And I'm not using any type of raid for that data.
> 

JBOD?

--Dan

> kind regardes,
> 
> Matthias.
> 
> Dan Oglesby wrote:
> 
>> Matthias Barremaecker wrote:
>>
>>> Hi,
>>>
>>> Thanx for your reply.
>>>
>>> The data is not THAT importend, all our importend data is backuped 4 
>>> times, inc. original (well, 3 times now, since the 1300gig machine is 
>>> broke).
>>>
>>> I did a bit furtur reasearch and maybe this is something to think 
>>> about if you use reiserfs :
>>>
>>> I did a bad block check and I have 10 bad blocks of 4096bytes on 
>>> 1300Gig and ... that is the reason reiserfs will not work anymore.
>>>
>>> I guess this sux. I rather have that the data on the bad blocks is 
>>> just corupted but the rest is accesseble.
>>>
>>> I'm doing a --rebuild-tree with the bad block list. Hopes this works.
>>>
>>>
>>> Aren't there any tools to substract data from a broken reiserfs 
>>> partition ?
>>>
>>>
>>> kind regardes,
>>>
>>> Matthias.
>>>
>>>
>>>
>>
>>
>> I'm currently in a similar situation:  1TB (RAID-5) ReiserFS 
>> filesystem running on RedHat 7.2 with a Promise SX6000 controller.  
>> Everything was running stock version/firmware/BIOS, and it suddently 
>> developed 39 bad blocks after a power outage.
>>
>> reiserfsck --rebuild-tree (version 3.6.4, the latest for RH 7.2) 
>> failed to complete after the controller kept hard locking/crashing.
>>
>> So, I've compiled a plain 2.4.30 kernel (from kernel.org, not RedHat), 
>> updated the BIOS/Firmware on the SX6000, and compiled and installed 
>> the latest reiserfsprogs (3.6.19, I believe?), and have been running 
>> badblocks (non-destructive) for the last four days.  Nothing has 
>> turned up so far.
>>
>> I'm hoping the next time I run a --rebuild-tree, it will be able to 
>> complete.  I really do need to get the data off of this filesystem.
>>
>> Good luck with your recovery.
>>
>> --Dan
>>
>>
>>
>>
>>> Valdis.Kletnieks@vt.edu wrote:
>>>
>>>> On Sun, 29 May 2005 21:25:54 +0200, Matthias Barremaecker said:
>>>>
>>>>
>>>>> but that sais it is a fysical drive error
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Physical drive errors.  Your hardware is broken.  Isn't much that 
>>>> Reiserfs
>>>> can do about it.
>>>>
>>>>
>>>>> What can I do.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 1) Call whoever you get hardware support from.
>>>>
>>>> 2) Be ready to restore from backups.
>>>>
>>>> 3) If you didn't have RAID-5 (or similar) set up, or a good backup, 
>>>> consider
>>>> it a learning experience.
>>>>
>>>> If your data is important enough that you'll care if you lose it, 
>>>> you should take
>>>> steps to make sure you won't lose it... It's that simple.
>>>>
>>>> (Just for the record, if we have important info, it gets at least 
>>>> RAID5, a
>>>> backup to tape or other device, *and* a *second* backup off-site.  
>>>> And my shop
>>>> is far from the most paranoid about such things.)
>>>>
>>>
>>
>>
>>
> 

Re: Reiserfs 1300G partition on lvm problem ...

[Christian]

On Tue, May 31, 2005 17:09, Dan Oglesby said:
>
> So far, I haven't seen any bad blocks written to my output file.  I'm
> not in front of the machine (remote location), so I can't see what's on
> the console, where I'm actually running the command.

use screendump(1) to see what's on the console. it comes in a package
called "console-tools" here.

> with the power outage in a sane manner.  The last reiserfsck failed
> after a day of running, so I'm thinking if I have a problem, it's
> towards the end of my array.  Time will tell.

sorry if i missed the rest of the thread: please make sure you're using
the latest version of reiserfsprogs.

...my 2 cents,
Christian.
-- 
make bzImage, not war

Reiserfsck 1300Gig failed with following error

[Matthias Barremaecker]

Hi,

using reiserfsprogs-3.6.19

Here is the error and messages : ...


2 directory entries were hashed with not set hash.
731305 directory entries were hashed with "r5" hash.
         "r5" hash is selected
Flushing..finished
         Read blocks (but not data blocks) 313507947
                 Leaves among those 439626
                         - corrected leaves 46
                         - leaves all contents of which could not be 
saved and deleted 56
                 pointers in indirect items to wrong area 3997 (zeroed)
                 Objectids found 731997

Pass 1 (will try to insert 439570 leaves):
####### Pass 1 #######
Looking for allocable blocks .. finished
0%....20%....40%pass1.c 212 balance_condition_2_fails
balance_condition_2_fails: block 1288402, pointer 168: The left 
delimiting key [1412453424 1412453424 0x10001000 ??? (15)] of the block 
(170046457) is wrong,the item cannot be found
Aborted


Anyone any suggestions ?

thanx!

kind regardes,


Matthias.


-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

Re: Reiserfsck 1300Gig failed with following error

[Vitaly Fertman]

On Wednesday 01 June 2005 10:47, Matthias Barremaecker wrote:
> Hi,
> 
> using reiserfsprogs-3.6.19
> 
> Here is the error and messages : ...
> 
> 
> 2 directory entries were hashed with not set hash.
> 731305 directory entries were hashed with "r5" hash.
>          "r5" hash is selected
> Flushing..finished
>          Read blocks (but not data blocks) 313507947
>                  Leaves among those 439626
>                          - corrected leaves 46
>                          - leaves all contents of which could not be 
> saved and deleted 56
>                  pointers in indirect items to wrong area 3997 (zeroed)
>                  Objectids found 731997
> 
> Pass 1 (will try to insert 439570 leaves):
> ####### Pass 1 #######
> Looking for allocable blocks .. finished
> 0%....20%....40%pass1.c 212 balance_condition_2_fails
> balance_condition_2_fails: block 1288402, pointer 168: The left 
> delimiting key [1412453424 1412453424 0x10001000 ??? (15)] of the block 
> (170046457) is wrong,the item cannot be found

the key (which is in the tree already and supposed to be correct) is 
completely corrupted, no one key component is correct. data could be 
corrupted in the memory or on read/write -- bad cable, controller, 
dying harddrive, etc. 

> 
> Anyone any suggestions ?

running fsck with the list of bad blocks is useful when you have the 
known fixed number of bad blocks, you know that the number will not 
change soon, in other words other blocks are relable. 

first of all you even have not complete badblocks run, so you cannot 
be sure you know the full bad block list. 

and the second, if your hardware is dying you need to fix it before 
running fsck on it.

-- 
Thanks,
Vitaly Fertman

Re: Reiserfsck 1300Gig failed with following error

[Matthias Barremaecker]

Hi,

> the key (which is in the tree already and supposed to be correct) is 
> completely corrupted, no one key component is correct. data could be 
> corrupted in the memory or on read/write -- bad cable, controller, 
> dying harddrive, etc. 
> 
> 
>>Anyone any suggestions ?
> 
> 
> running fsck with the list of bad blocks is useful when you have the 
> known fixed number of bad blocks, you know that the number will not 
> change soon, in other words other blocks are relable. 
> 
> first of all you even have not complete badblocks run, so you cannot 
> be sure you know the full bad block list. 
> 
> and the second, if your hardware is dying you need to fix it before 
> running fsck on it.

How do you sugest I swap a hd in a LVM raid with already data on it ?

thanx.

kind regardes,

Matthias.



-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

Re: Reiserfsck 1300Gig failed with following error

[Vitaly Fertman]

On Wednesday 01 June 2005 14:57, Matthias Barremaecker wrote:
> Hi,
> 
> > the key (which is in the tree already and supposed to be correct) is 
> > completely corrupted, no one key component is correct. data could be 
> > corrupted in the memory or on read/write -- bad cable, controller, 
> > dying harddrive, etc. 
> > 
> > 
> >>Anyone any suggestions ?
> > 
> > 
> > running fsck with the list of bad blocks is useful when you have the 
> > known fixed number of bad blocks, you know that the number will not 
> > change soon, in other words other blocks are relable. 
> > 
> > first of all you even have not complete badblocks run, so you cannot 
> > be sure you know the full bad block list. 
> > 
> > and the second, if your hardware is dying you need to fix it before 
> > running fsck on it.
> 
> How do you sugest I swap a hd in a LVM raid with already data on it ?

dd_rescue hd to a relable one & change lvm according to the change.
if you need our assistance please use our www.namesys.com/support.html 
service.

-- 
Thanks,
Vitaly Fertman

Re: Reiserfsck 1300Gig failed with following error

[Matthias Barremaecker]

>>How do you sugest I swap a hd in a LVM raid with already data on it ?
> 
> 
> dd_rescue hd to a relable one & change lvm according to the change.
> if you need our assistance please use our www.namesys.com/support.html 
> service.

I start to find that answer super funny.:)) I'm willing to pay if you 
fix my problem. Maybe you can start saying if the problem is fixable.

Situation :

LVM with reiserfs

2 onboard IDE controllers => 8 disks

I can't connect another disk fysicly

system boots from a scsi disk, that has nothing to do with the LVM 
except booting and mounting it .

What should I do before I pay you to work out a solution.

thanx.

kind regardes,


Matthias.



-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

Re: Reiserfsck 1300Gig failed with following error

[Vitaly Fertman]

On Wednesday 01 June 2005 16:28, Matthias Barremaecker wrote:
> >>How do you sugest I swap a hd in a LVM raid with already data on it ?
> > 
> > 
> > dd_rescue hd to a relable one & change lvm according to the change.
> > if you need our assistance please use our www.namesys.com/support.html 
> > service.
> 
> I start to find that answer super funny.:)) I'm willing to pay if you 
> fix my problem. Maybe you can start saying if the problem is fixable.
> 
> Situation :
> 
> LVM with reiserfs
> 
> 2 onboard IDE controllers => 8 disks
> 
> I can't connect another disk fysicly

perform dd_rescue'ing on another computer.

or alternatively you can pull out a working disk out, put a 
new one, dd_rescue the broken one to a new one, put the new 
one instead of the broken one and put the pulled out disk back 
to its own place.

> system boots from a scsi disk, that has nothing to do with the LVM 
> except booting and mounting it .
>
> What should I do before I pay you to work out a solution.
> 
> thanx.
> 
> kind regardes,
> 
> 
> Matthias.
> 
> 
> 

-- 
Thanks,
Vitaly Fertman

Re: Reiserfsck 1300Gig failed with following error

[Matthias Barremaecker]

>>Situation :
>>
>>LVM with reiserfs
>>
>>2 onboard IDE controllers => 8 disks
>>
>>I can't connect another disk fysicly
> 
> 
> perform dd_rescue'ing on another computer.
> 
> or alternatively you can pull out a working disk out, put a 
> new one, dd_rescue the broken one to a new one, put the new 
> one instead of the broken one and put the pulled out disk back 
> to its own place.

Omg omg! How I didn't think of that :) thanx.

> 
> 
>>system boots from a scsi disk, that has nothing to do with the LVM 
>>except booting and mounting it .
>>
>>What should I do before I pay you to work out a solution.
>>
>>thanx.
>>
>>kind regardes,
>>
>>
>>Matthias.
>>
>>
>>
> 
> 

-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

13000Gig partition badblock check is the same -- do a reiserfsck again ?

[Matthias Barremaecker]

Hi,

The first time, I did a bad block check, and feeded that list to the 
reiserfsck --rebuild-tree.

The reiserfsck FAILED in phase 2.

Now I have done a bad block check again and the list is the same, so no 
new bad blocks have occoured.

Is it sane to do the reiserfsck --rebuild-tree again, or will it fail again?

thanx.

kind regardes,

Matthias.
-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

Re: 13000Gig partition badblock check is the same -- do a reiserfsck again ?

[Dan Oglesby]

Matthias Barremaecker wrote:

> Hi,
>
> The first time, I did a bad block check, and feeded that list to the 
> reiserfsck --rebuild-tree.
>
> The reiserfsck FAILED in phase 2.
>
> Now I have done a bad block check again and the list is the same, so 
> no new bad blocks have occoured.
>
> Is it sane to do the reiserfsck --rebuild-tree again, or will it fail 
> again?
>
> thanx.
>
> kind regardes,
>
> Matthias.


What error did you get?  Same as before?

FWIW, I was able to successfully recover my corrupted array!  Had to 
update all hardware to latest BIOS/Firmware, all software and drivers to 
latest versions.  Took two days to run, but it completed, and I ended up 
only losing 2 files out of over 1.1 million files on a 1TB RAID-5 
array.  That's not too bad, considering how many times the machine went 
up and down due to bad power in the building.

--Dan

Re: 13000Gig partition badblock check is the same -- do a reiserfsck again ?

[Matthias Barremaecker]

Hi Dan,

I'm glad you have your data back and just lost 2 files. It gives me a 
bit of ... hope :))

The error I've got was :
--------------------------------------------
2 directory entries were hashed with not set hash.
  731305 directory entries were hashed with "r5" hash.
           "r5" hash is selected
  Flushing..finished
           Read blocks (but not data blocks) 313507947
                   Leaves among those 439626
                           - corrected leaves 46
                           - leaves all contents of which could not be
  saved and deleted 56
                   pointers in indirect items to wrong area 3997 (zeroed)
                   Objectids found 731997

  Pass 1 (will try to insert 439570 leaves):
  ####### Pass 1 #######
  Looking for allocable blocks .. finished
  0%....20%....40%pass1.c 212 balance_condition_2_fails
  balance_condition_2_fails: block 1288402, pointer 168: The left
  delimiting key [1412453424 1412453424 0x10001000 ??? (15)] of the block
  (170046457) is wrong,the item cannot be found
--------------------------------------------

What just doens't make any sens to me.

I'm using version  3.6.19.

If you have any ideas ...

Thanx.

kind regardes,


Matthias.




Dan Oglesby wrote:
 > Matthias Barremaecker wrote:
 >
 >> Hi,
 >>
 >> The first time, I did a bad block check, and feeded that list to the
 >> reiserfsck --rebuild-tree.
 >>
 >> The reiserfsck FAILED in phase 2.
 >>
 >> Now I have done a bad block check again and the list is the same, so
 >> no new bad blocks have occoured.
 >>
 >> Is it sane to do the reiserfsck --rebuild-tree again, or will it fail
 >> again?
 >>
 >> thanx.
 >>
 >> kind regardes,
 >>
 >> Matthias.
 >
 >
 >
 > What error did you get?  Same as before?
 >
 > FWIW, I was able to successfully recover my corrupted array!  Had to
 > update all hardware to latest BIOS/Firmware, all software and drivers to
 > latest versions.  Took two days to run, but it completed, and I ended up
 > only losing 2 files out of over 1.1 million files on a 1TB RAID-5
 > array.  That's not too bad, considering how many times the machine went
 > up and down due to bad power in the building.
 >
 > --Dan
 >
 >

-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.


Dan Oglesby wrote:
> Matthias Barremaecker wrote:
> 
>> Hi,
>>
>> The first time, I did a bad block check, and feeded that list to the 
>> reiserfsck --rebuild-tree.
>>
>> The reiserfsck FAILED in phase 2.
>>
>> Now I have done a bad block check again and the list is the same, so 
>> no new bad blocks have occoured.
>>
>> Is it sane to do the reiserfsck --rebuild-tree again, or will it fail 
>> again?
>>
>> thanx.
>>
>> kind regardes,
>>
>> Matthias.
> 
> 
> 
> What error did you get?  Same as before?
> 
> FWIW, I was able to successfully recover my corrupted array!  Had to 
> update all hardware to latest BIOS/Firmware, all software and drivers to 
> latest versions.  Took two days to run, but it completed, and I ended up 
> only losing 2 files out of over 1.1 million files on a 1TB RAID-5 
> array.  That's not too bad, considering how many times the machine went 
> up and down due to bad power in the building.
> 
> --Dan
> 
> 

-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

Re: 13000Gig partition badblock check is the same -- do a reiserfsck again ?

[Dan Oglesby]

What kind of kernel are you running?  I had to dump my RedHat kernel and 
use the latest kernel from http://www.kernel.org (2.4.30) for my system.

The problem I was having was due to the kernel being a bit old, and the 
hardware not handling bad blocks properly on an array.  Updating the 
software, drivers, firmware and BIOS on everything in my system allowed 
reiserfsck to work properly, and fix my filesystem.

--Dan

Matthias Barremaecker wrote:
> Hi Dan,
> 
> I'm glad you have your data back and just lost 2 files. It gives me a 
> bit of ... hope :))
> 
> The error I've got was :
> --------------------------------------------
> 2 directory entries were hashed with not set hash.
>  731305 directory entries were hashed with "r5" hash.
>           "r5" hash is selected
>  Flushing..finished
>           Read blocks (but not data blocks) 313507947
>                   Leaves among those 439626
>                           - corrected leaves 46
>                           - leaves all contents of which could not be
>  saved and deleted 56
>                   pointers in indirect items to wrong area 3997 (zeroed)
>                   Objectids found 731997
> 
>  Pass 1 (will try to insert 439570 leaves):
>  ####### Pass 1 #######
>  Looking for allocable blocks .. finished
>  0%....20%....40%pass1.c 212 balance_condition_2_fails
>  balance_condition_2_fails: block 1288402, pointer 168: The left
>  delimiting key [1412453424 1412453424 0x10001000 ??? (15)] of the block
>  (170046457) is wrong,the item cannot be found
> --------------------------------------------
> 
> What just doens't make any sens to me.
> 
> I'm using version  3.6.19.
> 
> If you have any ideas ...
> 
> Thanx.
> 
> kind regardes,
> 
> 
> Matthias.
> 
> 
> 
> 
> Dan Oglesby wrote:
>  > Matthias Barremaecker wrote:
>  >
>  >> Hi,
>  >>
>  >> The first time, I did a bad block check, and feeded that list to the
>  >> reiserfsck --rebuild-tree.
>  >>
>  >> The reiserfsck FAILED in phase 2.
>  >>
>  >> Now I have done a bad block check again and the list is the same, so
>  >> no new bad blocks have occoured.
>  >>
>  >> Is it sane to do the reiserfsck --rebuild-tree again, or will it fail
>  >> again?
>  >>
>  >> thanx.
>  >>
>  >> kind regardes,
>  >>
>  >> Matthias.
>  >
>  >
>  >
>  > What error did you get?  Same as before?
>  >
>  > FWIW, I was able to successfully recover my corrupted array!  Had to
>  > update all hardware to latest BIOS/Firmware, all software and drivers to
>  > latest versions.  Took two days to run, but it completed, and I ended up
>  > only losing 2 files out of over 1.1 million files on a 1TB RAID-5
>  > array.  That's not too bad, considering how many times the machine went
>  > up and down due to bad power in the building.
>  >
>  > --Dan
>  >
>  >
> 

Re: 13000Gig partition badblock check is the same -- do a reiserfsck again ?

[Matthias Barremaecker]

Hi Dan,

I'm using Gentoo kernel

Linux hyperspace 2.6.11-gentoo-r6 #6 SMP Tue Apr 19 23:40:58 CEST 2005 
i686 Pentium III (Coppermine) GenuineIntel GNU/Linux

Kind regardes,

Matthias.


Dan Oglesby wrote:
> What kind of kernel are you running?  I had to dump my RedHat kernel and 
> use the latest kernel from http://www.kernel.org (2.4.30) for my system.
> 
> The problem I was having was due to the kernel being a bit old, and the 
> hardware not handling bad blocks properly on an array.  Updating the 
> software, drivers, firmware and BIOS on everything in my system allowed 
> reiserfsck to work properly, and fix my filesystem.
> 
> --Dan
> 
> Matthias Barremaecker wrote:
> 
>> Hi Dan,
>>
>> I'm glad you have your data back and just lost 2 files. It gives me a 
>> bit of ... hope :))
>>
>> The error I've got was :
>> --------------------------------------------
>> 2 directory entries were hashed with not set hash.
>>  731305 directory entries were hashed with "r5" hash.
>>           "r5" hash is selected
>>  Flushing..finished
>>           Read blocks (but not data blocks) 313507947
>>                   Leaves among those 439626
>>                           - corrected leaves 46
>>                           - leaves all contents of which could not be
>>  saved and deleted 56
>>                   pointers in indirect items to wrong area 3997 (zeroed)
>>                   Objectids found 731997
>>
>>  Pass 1 (will try to insert 439570 leaves):
>>  ####### Pass 1 #######
>>  Looking for allocable blocks .. finished
>>  0%....20%....40%pass1.c 212 balance_condition_2_fails
>>  balance_condition_2_fails: block 1288402, pointer 168: The left
>>  delimiting key [1412453424 1412453424 0x10001000 ??? (15)] of the block
>>  (170046457) is wrong,the item cannot be found
>> --------------------------------------------
>>
>> What just doens't make any sens to me.
>>
>> I'm using version  3.6.19.
>>
>> If you have any ideas ...
>>
>> Thanx.
>>
>> kind regardes,
>>
>>
>> Matthias.
>>
>>
>>
>>
>> Dan Oglesby wrote:
>>  > Matthias Barremaecker wrote:
>>  >
>>  >> Hi,
>>  >>
>>  >> The first time, I did a bad block check, and feeded that list to the
>>  >> reiserfsck --rebuild-tree.
>>  >>
>>  >> The reiserfsck FAILED in phase 2.
>>  >>
>>  >> Now I have done a bad block check again and the list is the same, so
>>  >> no new bad blocks have occoured.
>>  >>
>>  >> Is it sane to do the reiserfsck --rebuild-tree again, or will it fail
>>  >> again?
>>  >>
>>  >> thanx.
>>  >>
>>  >> kind regardes,
>>  >>
>>  >> Matthias.
>>  >
>>  >
>>  >
>>  > What error did you get?  Same as before?
>>  >
>>  > FWIW, I was able to successfully recover my corrupted array!  Had to
>>  > update all hardware to latest BIOS/Firmware, all software and 
>> drivers to
>>  > latest versions.  Took two days to run, but it completed, and I 
>> ended up
>>  > only losing 2 files out of over 1.1 million files on a 1TB RAID-5
>>  > array.  That's not too bad, considering how many times the machine 
>> went
>>  > up and down due to bad power in the building.
>>  >
>>  > --Dan
>>  >
>>  >
>>
> 
> 
> 

-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.

Re: 13000Gig partition badblock check is the same -- do a reiserfsck again ?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

On Thu, 02 Jun 2005 09:28:50 CDT, Dan Oglesby said:

> latest versions.  Took two days to run, but it completed, and I ended up 
> only losing 2 files out of over 1.1 million files on a 1TB RAID-5 
> array.  That's not too bad, considering how many times the machine went 
> up and down due to bad power in the building.

Buy a UPS. Now.  Even if it's just a big battery that will only keep you
running for 10 mins - at least that will give you enough time to do a clean
shutdown -h rather than get stuff trashed.

If you can't get money for it, just point at the lost-productivity costs
the *next* time the terabyte takes 2 days to recover.. and remind the boss that
you could be down for 2 days every time the lights flicker.. ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 13000Gig partition badblock check is the same -- do a reiserfsck again ?

[Dan Oglesby]

Valdis.Kletnieks@vt.edu wrote:
> On Thu, 02 Jun 2005 09:28:50 CDT, Dan Oglesby said:
> 
> 
>>latest versions.  Took two days to run, but it completed, and I ended up 
>>only losing 2 files out of over 1.1 million files on a 1TB RAID-5 
>>array.  That's not too bad, considering how many times the machine went 
>>up and down due to bad power in the building.
> 
> 
> Buy a UPS. Now.  Even if it's just a big battery that will only keep you
> running for 10 mins - at least that will give you enough time to do a clean
> shutdown -h rather than get stuff trashed.
> 
> If you can't get money for it, just point at the lost-productivity costs
> the *next* time the terabyte takes 2 days to recover.. and remind the boss that
> you could be down for 2 days every time the lights flicker.. ;)

That's just it...  Bad power in the building was due to the building's 
UPS (big sucker, attached to a Cummins diesel generator) failing. 
That's been fixed, so the power in the building is OK again.

Previous to the UPS going down, this building hadn't lost power in many 
years.  We don't have to deal with this kind of problem very often.  ;-)

I've been using the array, and it's going great.  ReiserFS proves to be 
robust, as long as the hardware, power, and software on the system are 
in good working order.

--Dan

Re: 13000Gig partition badblock check is the same -- do a reiserfsck again ? -> ran badblocks

[Matthias Barremaecker]

Hi,

I've ran
   badblocks -v -b 4096 -o badblocks2 /dev/hyperspace/hyperspace1

It said :
   Checking blocks 0 to 322256896
   Checking for bad blocks (read-only test): done
   Pass completed, 312 bad blocks found.

the file contains
   36434712
   36434756
   36434757
   36448896
   36448916
   38853584
   38853616
   38854632
   38854635
   38854636

That aren't 312 numbers... Or am I seeing it wrong ?

thanx!!


Dan Oglesby wrote:
> Valdis.Kletnieks@vt.edu wrote:
> 
>> On Thu, 02 Jun 2005 09:28:50 CDT, Dan Oglesby said:
>>
>>
>>> latest versions.  Took two days to run, but it completed, and I ended 
>>> up only losing 2 files out of over 1.1 million files on a 1TB RAID-5 
>>> array.  That's not too bad, considering how many times the machine 
>>> went up and down due to bad power in the building.
>>
>>
>>
>> Buy a UPS. Now.  Even if it's just a big battery that will only keep you
>> running for 10 mins - at least that will give you enough time to do a 
>> clean
>> shutdown -h rather than get stuff trashed.
>>
>> If you can't get money for it, just point at the lost-productivity costs
>> the *next* time the terabyte takes 2 days to recover.. and remind the 
>> boss that
>> you could be down for 2 days every time the lights flicker.. ;)
> 
> 
> That's just it...  Bad power in the building was due to the building's 
> UPS (big sucker, attached to a Cummins diesel generator) failing. That's 
> been fixed, so the power in the building is OK again.
> 
> Previous to the UPS going down, this building hadn't lost power in many 
> years.  We don't have to deal with this kind of problem very often.  ;-)
> 
> I've been using the array, and it's going great.  ReiserFS proves to be 
> robust, as long as the hardware, power, and software on the system are 
> in good working order.
> 
> --Dan
> 
> 

-- 
   Matthias Barremaecker, MH.BE - Arta nv
   0495 30 31 72

   http://mh.be/

   SERVER HOUSING per 1HE € 50 per maand
     20Gig traffic, 100Mbit netwerk
     Center te Antwerpen.