Problems with "--rebuild-tree" on network (ENBD) storage

Problems with "--rebuild-tree" on network (ENBD) storage

[Bas van Schaik]

[-- Attachment #1: Type: text/plain, Size: 2027 bytes --]

Hi all,

I'm having severe problems with reiserfsck --rebuild-tree on a
CryptoLoop over LVM over RAID5 over ENBD (Enhanced Network Block Device)
device. The first pass is no problem (finds errors, but runs perfectly),
but the second pass hangs my whole system (load increasing to values
like 30, 40, 50) after being active for about 20 minutes. Attached,
you'll find two graphs of this behaviour.

We're talking about a cluster of 5 machines, 4 of them are filled with
in total about 3TB of harddisks, the 5th one imports those devices using
ENBD and performs 4x RAID5 over it. LVM combines those 4 arrays to one
device, and the cryptoloop over LVM ensures safe storage. In the normal
situation, there should a mount point /backups (from /dev/loop0) with
2.4TB total space.

However, about a week ago I added a new RAID-array to LVM, and started
resizing my /backups partition to the maximum available space within
LVM. During this resize, my new RAID5-array dropped out due to a disk
failure (I didn't let md finish syncing the array...) and the resize
failed. At that point, I had a corrupt filesystem, and I'm trying to run
reiserfsck --rebuild-tree for a week now.

I don't know exactly what is happening, but someone hinted me that
reiserfsck might be filling up my TCP buffers (remember, it's a
networked block device!) which will lock-up all the I/O to the network
block device.

For your information: I'm running Debian Sarge with a 2.6.17 kernel from
Debian Etch and reiserfsprogs version 3.6.19 from Debian Sarge. The 5th
system (frontend) contains a P4 3.0GHz and 1GB of RAM.

Has anyone seen something like this before? Or does someone have an idea
how I can solve this problem? Might it be worth a try to "upgrade" to
Reiser4? If there's no other way, I am willing to give up my data
(there's a partial backup of this backup anyway), but I do need to be
sure that this won't happen again!

BTW, I didn't find out how to subscribe to this list, so please cc. me
in your reply! Thanks!

Regards,

 -- Bas van Schaik

Re: Problems with "--rebuild-tree" on network (ENBD) storage

[Vladimir V. Saveliev]

Hello

On Thursday 05 October 2006 12:07, you wrote:
> Hi all,
> 
> I'm having severe problems with reiserfsck --rebuild-tree on a
> CryptoLoop over LVM over RAID5 over ENBD (Enhanced Network Block Device)
> device. The first pass is no problem (finds errors, but runs perfectly),
> but the second pass hangs my whole system (load increasing to values
> like 30, 40, 50) after being active for about 20 minutes. 

Please be precise: which pass hangs? Pass 1 or pass 2? 
Note that reiserfsck --rebuild-tree starts with pass 0.
Please clarify what does "hangs whole system" mean. If the system hangs so that it has to be hard rebooted -
it is very likely that your problem has nothing to do with reiserfsck.
If reiserfsck just consumes 100% CPU on pass2 - there is experimental version of reiserfsck which improves pass 2 performance
substantially in some cases. 

> Attached, 
> you'll find two graphs of this behaviour.
> 
I see nothing attached.

> We're talking about a cluster of 5 machines, 4 of them are filled with
> in total about 3TB of harddisks, the 5th one imports those devices using
> ENBD and performs 4x RAID5 over it. LVM combines those 4 arrays to one
> device, and the cryptoloop over LVM ensures safe storage. In the normal
> situation, there should a mount point /backups (from /dev/loop0) with
> 2.4TB total space.
> 
> However, about a week ago I added a new RAID-array to LVM, and started
> resizing my /backups partition to the maximum available space within
> LVM. During this resize, my new RAID5-array dropped out due to a disk
> failure (I didn't let md finish syncing the array...) and the resize
> failed. At that point, I had a corrupt filesystem, and I'm trying to run
> reiserfsck --rebuild-tree for a week now.
> 
> I don't know exactly what is happening, but someone hinted me that
> reiserfsck might be filling up my TCP buffers (remember, it's a
> networked block device!) which will lock-up all the I/O to the network
> block device.
> 
> For your information: I'm running Debian Sarge with a 2.6.17 kernel from
> Debian Etch and reiserfsprogs version 3.6.19 from Debian Sarge. The 5th
> system (frontend) contains a P4 3.0GHz and 1GB of RAM.
> 
> Has anyone seen something like this before? Or does someone have an idea
> how I can solve this problem? Might it be worth a try to "upgrade" to
> Reiser4? If there's no other way, I am willing to give up my data
> (there's a partial backup of this backup anyway), but I do need to be
> sure that this won't happen again!
> 
> BTW, I didn't find out how to subscribe to this list, so please cc. me
> in your reply! Thanks!
> 
> Regards,
> 
>  -- Bas van Schaik
> 

Re: Problems with "--rebuild-tree" on network (ENBD) storage

[Bas van Schaik]

Hi Vladimir,


> On Thursday 05 October 2006 12:07, you wrote:
>> Hi all,
>>
>> I'm having severe problems with reiserfsck --rebuild-tree on a
>> CryptoLoop over LVM over RAID5 over ENBD (Enhanced Network Block Device)
>> device. The first pass is no problem (finds errors, but runs perfectly),
>> but the second pass hangs my whole system (load increasing to values
>> like 30, 40, 50) after being active for about 20 minutes. 
> 
> Please be precise: which pass hangs? Pass 1 or pass 2? 
> Note that reiserfsck --rebuild-tree starts with pass 0.
I'm sorry: it hangs during the second pass, which is indeed called "pass 1".

> Please clarify what does "hangs whole system" mean. If the system hangs so that it has to be hard rebooted -
Like I said: loads increases dramatically and renders the machine unusable.

> it is very likely that your problem has nothing to do with reiserfsck.
I do think it has something to do with reiserfsck, since the system was
functioning fine until I had to repair my filesystem! I've tried it for
many times now, but it hangs every time during the rebuild-tree.

> If reiserfsck just consumes 100% CPU on pass2 - there is experimental version of reiserfsck which improves pass 2 performance
> substantially in some cases. 
It's not a matter of CPU usage, it's about I/O. I suspect that ReiserFS
fills my memory (TCP buffers) faster than they can flush, which causes
starvation of the buffers.

>> Attached, 
>> you'll find two graphs of this behaviour.
>>
> I see nothing attached.
I think the mailing list doesn't support attachments, but there's not
much too see anyway. Just a graph indicating an increasing load.

However, thanks for your thoughts!

 -- Bas



>> We're talking about a cluster of 5 machines, 4 of them are filled with
>> in total about 3TB of harddisks, the 5th one imports those devices using
>> ENBD and performs 4x RAID5 over it. LVM combines those 4 arrays to one
>> device, and the cryptoloop over LVM ensures safe storage. In the normal
>> situation, there should a mount point /backups (from /dev/loop0) with
>> 2.4TB total space.
>>
>> However, about a week ago I added a new RAID-array to LVM, and started
>> resizing my /backups partition to the maximum available space within
>> LVM. During this resize, my new RAID5-array dropped out due to a disk
>> failure (I didn't let md finish syncing the array...) and the resize
>> failed. At that point, I had a corrupt filesystem, and I'm trying to run
>> reiserfsck --rebuild-tree for a week now.
>>
>> I don't know exactly what is happening, but someone hinted me that
>> reiserfsck might be filling up my TCP buffers (remember, it's a
>> networked block device!) which will lock-up all the I/O to the network
>> block device.
>>
>> For your information: I'm running Debian Sarge with a 2.6.17 kernel from
>> Debian Etch and reiserfsprogs version 3.6.19 from Debian Sarge. The 5th
>> system (frontend) contains a P4 3.0GHz and 1GB of RAM.
>>
>> Has anyone seen something like this before? Or does someone have an idea
>> how I can solve this problem? Might it be worth a try to "upgrade" to
>> Reiser4? If there's no other way, I am willing to give up my data
>> (there's a partial backup of this backup anyway), but I do need to be
>> sure that this won't happen again!
>>
>> BTW, I didn't find out how to subscribe to this list, so please cc. me
>> in your reply! Thanks!
>>
>> Regards,
>>
>>  -- Bas van Schaik
>>

Re: Problems with "--rebuild-tree" on network (ENBD) storage

[Vladimir V. Saveliev]

Hello

On Friday 06 October 2006 01:59, Bas van Schaik wrote:
> Hi Vladimir,
> 
> 
> > On Thursday 05 October 2006 12:07, you wrote:
> >> Hi all,
> >>
> >> I'm having severe problems with reiserfsck --rebuild-tree on a
> >> CryptoLoop over LVM over RAID5 over ENBD (Enhanced Network Block Device)
> >> device. The first pass is no problem (finds errors, but runs perfectly),
> >> but the second pass hangs my whole system (load increasing to values
> >> like 30, 40, 50) after being active for about 20 minutes. 
> > 
> > Please be precise: which pass hangs? Pass 1 or pass 2? 
> > Note that reiserfsck --rebuild-tree starts with pass 0.
> I'm sorry: it hangs during the second pass, which is indeed called "pass 1".
> 
> > Please clarify what does "hangs whole system" mean. If the system hangs so that it has to be hard rebooted -
> Like I said: loads increases dramatically and renders the machine unusable.
> 
> > it is very likely that your problem has nothing to do with reiserfsck.
> I do think it has something to do with reiserfsck, since the system was
> functioning fine until I had to repair my filesystem! 

ok, may I ask you to run badblocks on that device? reiserfsck wants to be able to read and write filesystem device.
badblocks will show us whether your device is in good shape. 

> I've tried it for 
> many times now, but it hangs every time during the rebuild-tree.
> 
> > If reiserfsck just consumes 100% CPU on pass2 - there is experimental version of reiserfsck which improves pass 2 performance
> > substantially in some cases. 
> It's not a matter of CPU usage, it's about I/O. I suspect that ReiserFS
> fills my memory (TCP buffers) faster than they can flush, which causes
> starvation of the buffers.
> 
> >> Attached, 
> >> you'll find two graphs of this behaviour.
> >>
> > I see nothing attached.
> I think the mailing list doesn't support attachments, but there's not
> much too see anyway. Just a graph indicating an increasing load.
> 
> However, thanks for your thoughts!
> 
>  -- Bas
> 
> 
> 
> >> We're talking about a cluster of 5 machines, 4 of them are filled with
> >> in total about 3TB of harddisks, the 5th one imports those devices using
> >> ENBD and performs 4x RAID5 over it. LVM combines those 4 arrays to one
> >> device, and the cryptoloop over LVM ensures safe storage. In the normal
> >> situation, there should a mount point /backups (from /dev/loop0) with
> >> 2.4TB total space.
> >>
> >> However, about a week ago I added a new RAID-array to LVM, and started
> >> resizing my /backups partition to the maximum available space within
> >> LVM. During this resize, my new RAID5-array dropped out due to a disk
> >> failure (I didn't let md finish syncing the array...) and the resize
> >> failed. At that point, I had a corrupt filesystem, and I'm trying to run
> >> reiserfsck --rebuild-tree for a week now.
> >>
> >> I don't know exactly what is happening, but someone hinted me that
> >> reiserfsck might be filling up my TCP buffers (remember, it's a
> >> networked block device!) which will lock-up all the I/O to the network
> >> block device.
> >>
> >> For your information: I'm running Debian Sarge with a 2.6.17 kernel from
> >> Debian Etch and reiserfsprogs version 3.6.19 from Debian Sarge. The 5th
> >> system (frontend) contains a P4 3.0GHz and 1GB of RAM.
> >>
> >> Has anyone seen something like this before? Or does someone have an idea
> >> how I can solve this problem? Might it be worth a try to "upgrade" to
> >> Reiser4? If there's no other way, I am willing to give up my data
> >> (there's a partial backup of this backup anyway), but I do need to be
> >> sure that this won't happen again!
> >>
> >> BTW, I didn't find out how to subscribe to this list, so please cc. me
> >> in your reply! Thanks!
> >>
> >> Regards,
> >>
> >>  -- Bas van Schaik
> >>
> 
> 
> 

Re: Problems with "--rebuild-tree" on network (ENBD) storage

[Bas van Schaik]

Vladimir V. Saveliev wrote:
> Hello
> 
> On Friday 06 October 2006 01:59, Bas van Schaik wrote:
>> Hi Vladimir,
>>
>>
>>> On Thursday 05 October 2006 12:07, you wrote:
>>>> Hi all,
>>>>
>>>> I'm having severe problems with reiserfsck --rebuild-tree on a
>>>> CryptoLoop over LVM over RAID5 over ENBD (Enhanced Network Block Device)
>>>> device. The first pass is no problem (finds errors, but runs perfectly),
>>>> but the second pass hangs my whole system (load increasing to values
>>>> like 30, 40, 50) after being active for about 20 minutes. 
>>> Please be precise: which pass hangs? Pass 1 or pass 2? 
>>> Note that reiserfsck --rebuild-tree starts with pass 0.
>> I'm sorry: it hangs during the second pass, which is indeed called "pass 1".
>>
>>> Please clarify what does "hangs whole system" mean. If the system hangs so that it has to be hard rebooted -
>> Like I said: loads increases dramatically and renders the machine unusable.
>>
>>> it is very likely that your problem has nothing to do with reiserfsck.
>> I do think it has something to do with reiserfsck, since the system was
>> functioning fine until I had to repair my filesystem! 
> 
> ok, may I ask you to run badblocks on that device? reiserfsck wants to be able to read and write filesystem device.
> badblocks will show us whether your device is in good shape. 

Of course you may ask me this, but I really don't think it's relevant.
ReiserFS is on top of (in this specific order) CryptoLoop, LVM, RAID5
and ENBD. If there are bad blocks on one of the 12 (!) disks, then one
of my storage servers in the ENBD-cluster would report a bunch of I/O
errors, RAID5 would drop the device and ReiserFS won't even notice that
a hard drive failed.
Furthermore, every RAID5 device has had a resync since the filesystem
resize operation, which implies that every bit has been checked at least
once.

I think the problem lies within the way reiserfsck reads and writes to
the underlying block device. Maybe reiserfsck isn't opening the device
in direct I/O (O_DIRECT) mode? I think it should, because it's safer,
though slower. Maybe O_DIRECT can be set optionally on (or off) using a
commandline switch?

Regards,

  -- Bas


> 
>> I've tried it for 
>> many times now, but it hangs every time during the rebuild-tree.
>>
>>> If reiserfsck just consumes 100% CPU on pass2 - there is experimental version of reiserfsck which improves pass 2 performance
>>> substantially in some cases. 
>> It's not a matter of CPU usage, it's about I/O. I suspect that ReiserFS
>> fills my memory (TCP buffers) faster than they can flush, which causes
>> starvation of the buffers.
>>
>>>> Attached, 
>>>> you'll find two graphs of this behaviour.
>>>>
>>> I see nothing attached.
>> I think the mailing list doesn't support attachments, but there's not
>> much too see anyway. Just a graph indicating an increasing load.
>>
>> However, thanks for your thoughts!
>>
>>  -- Bas
>>
>>
>>
>>>> We're talking about a cluster of 5 machines, 4 of them are filled with
>>>> in total about 3TB of harddisks, the 5th one imports those devices using
>>>> ENBD and performs 4x RAID5 over it. LVM combines those 4 arrays to one
>>>> device, and the cryptoloop over LVM ensures safe storage. In the normal
>>>> situation, there should a mount point /backups (from /dev/loop0) with
>>>> 2.4TB total space.
>>>>
>>>> However, about a week ago I added a new RAID-array to LVM, and started
>>>> resizing my /backups partition to the maximum available space within
>>>> LVM. During this resize, my new RAID5-array dropped out due to a disk
>>>> failure (I didn't let md finish syncing the array...) and the resize
>>>> failed. At that point, I had a corrupt filesystem, and I'm trying to run
>>>> reiserfsck --rebuild-tree for a week now.
>>>>
>>>> I don't know exactly what is happening, but someone hinted me that
>>>> reiserfsck might be filling up my TCP buffers (remember, it's a
>>>> networked block device!) which will lock-up all the I/O to the network
>>>> block device.
>>>>
>>>> For your information: I'm running Debian Sarge with a 2.6.17 kernel from
>>>> Debian Etch and reiserfsprogs version 3.6.19 from Debian Sarge. The 5th
>>>> system (frontend) contains a P4 3.0GHz and 1GB of RAM.
>>>>
>>>> Has anyone seen something like this before? Or does someone have an idea
>>>> how I can solve this problem? Might it be worth a try to "upgrade" to
>>>> Reiser4? If there's no other way, I am willing to give up my data
>>>> (there's a partial backup of this backup anyway), but I do need to be
>>>> sure that this won't happen again!
>>>>
>>>> BTW, I didn't find out how to subscribe to this list, so please cc. me
>>>> in your reply! Thanks!
>>>>
>>>> Regards,
>>>>
>>>>  -- Bas van Schaik
>>>>
>>
>>

Re: Problems with "--rebuild-tree" on network (ENBD) storage

[Bas van Schaik]

Hi Vladimir,
>>> ok, may I ask you to run badblocks on that device? reiserfsck wants to be able to read and write filesystem device.
>>> badblocks will show us whether your device is in good shape. 
>>>       
>> Of course you may ask me this, but I really don't think it's relevant.
>> ReiserFS is on top of (in this specific order) CryptoLoop, LVM, RAID5
>> and ENBD. If there are bad blocks on one of the 12 (!) disks, then one
>> of my storage servers in the ENBD-cluster would report a bunch of I/O
>> errors, RAID5 would drop the device and ReiserFS won't even notice that
>> a hard drive failed.
>> Furthermore, every RAID5 device has had a resync since the filesystem
>> resize operation, which implies that every bit has been checked at least
>> once.
>>
>> I think the problem lies within the way reiserfsck reads and writes to
>> the underlying block device. Maybe reiserfsck isn't opening the device
>> in direct I/O (O_DIRECT) mode? 
>>     
> Yes, it does not. But why would it have to?
>
>   
>> I think it should, because it's safer, 
>> though slower. Maybe O_DIRECT can be set optionally on (or off) using a
>> commandline switch?
>>
>>     
> Maybe O_DIRECT should be used, I do not argue. But there is nothing wrong in not using O_DIRECT.
> Why would user land application make a computer unusable?
> reiserfsck uses standard libc's low level i/o functions to read and write a device, it also analyses and modify read data before writing them back.
> The worst thing reiserfsck can do is 100% CPU consumption. But that also should not hurt a system.
>
> I hope you understand what I mean: if user land application makes a box unusable - something is wrong in kernel.
> I have never dealt with setup like yours. There are so many layers, why there can not be any errors?
>   
That's true, of course. But there's (at least) one place in the kernel
where userland touches kernel space: buffering. In my case, I think
reiserfsck is causing starvation of my TCP buffers, because it doesn't
use direct I/O but buffered I/O. Of course, this is a normal (and maybe
wise) thing to do when the bottom layer is ATA or SATA (or something
like that), but in my case there's a network somewhere between
reiserfsck and ATA/SATA. So, I don't expect reiserfsck to use direct I/O
by default, but it would be a nice feature for me (and the few others
with the same problem?) if direct I/O can be enabled by a commandline
switch.

> Can you dd_rescue your filesystem to a spare device which has less underlaying layers (linear raid or oven plain hard disk)
> and try reiserfsck --rebuild-tree oin it?
I'm sorry, the system is built upon 12 harddrives, with a total of more
than 3TB of disk space. I don't have that amount of drives available for
creating a backup!

Thanks for you thoughts,

  -- Bas

Re: Problems with "--rebuild-tree" on network (ENBD) storage

[Vladimir V. Saveliev]

Hello

On Friday 06 October 2006 17:10, Bas van Schaik wrote:
> Hi Vladimir,
> >>> ok, may I ask you to run badblocks on that device? reiserfsck wants to be able to read and write filesystem device.
> >>> badblocks will show us whether your device is in good shape. 
> >>>       
> >> Of course you may ask me this, but I really don't think it's relevant.
> >> ReiserFS is on top of (in this specific order) CryptoLoop, LVM, RAID5
> >> and ENBD. If there are bad blocks on one of the 12 (!) disks, then one
> >> of my storage servers in the ENBD-cluster would report a bunch of I/O
> >> errors, RAID5 would drop the device and ReiserFS won't even notice that
> >> a hard drive failed.
> >> Furthermore, every RAID5 device has had a resync since the filesystem
> >> resize operation, which implies that every bit has been checked at least
> >> once.
> >>
> >> I think the problem lies within the way reiserfsck reads and writes to
> >> the underlying block device. Maybe reiserfsck isn't opening the device
> >> in direct I/O (O_DIRECT) mode? 
> >>     
> > Yes, it does not. But why would it have to?
> >
> >   
> >> I think it should, because it's safer, 
> >> though slower. Maybe O_DIRECT can be set optionally on (or off) using a
> >> commandline switch?
> >>
> >>     
> > Maybe O_DIRECT should be used, I do not argue. But there is nothing wrong in not using O_DIRECT.
> > Why would user land application make a computer unusable?
> > reiserfsck uses standard libc's low level i/o functions to read and write a device, it also analyses and modify read data before writing them back.
> > The worst thing reiserfsck can do is 100% CPU consumption. But that also should not hurt a system.
> >
> > I hope you understand what I mean: if user land application makes a box unusable - something is wrong in kernel.
> > I have never dealt with setup like yours. There are so many layers, why there can not be any errors?
> >   
> That's true, of course. But there's (at least) one place in the kernel
> where userland touches kernel space: buffering. In my case, I think
> reiserfsck is causing starvation of my TCP buffers, because it doesn't
> use direct I/O but buffered I/O. Of course, this is a normal (and maybe
> wise) thing to do when the bottom layer is ATA or SATA (or something
> like that), but in my case there's a network somewhere between
> reiserfsck and ATA/SATA. So, I don't expect reiserfsck to use direct I/O
> by default, but it would be a nice feature for me (and the few others
> with the same problem?) if direct I/O can be enabled by a commandline
> switch.
> 

I am going to send you a patch to try later today (I hope to complete debugging by that time).

> > Can you dd_rescue your filesystem to a spare device which has less underlaying layers (linear raid or oven plain hard disk)
> > and try reiserfsck --rebuild-tree oin it?
> I'm sorry, the system is built upon 12 harddrives, with a total of more
> than 3TB of disk space. I don't have that amount of drives available for
> creating a backup!
> 
> Thanks for you thoughts,
> 
>   -- Bas
> 
> 
>