The Infamous Reiser4-randomly-blocks-for-ages-and-writes-the-hd-continously-in-the-mean-while now with a btrace log! (hope it helps)

The Infamous Reiser4-randomly-blocks-for-ages-and-writes-the-hd-continously-in-the-mean-while now with a btrace log! (hope it helps)

[rvalles]

[-- Attachment #1.1: Type: text/plain, Size: 1558 bytes --]

The bug is as I've explained a thousand times. (and it does only affect
kernels newer than 2.6.12, all of them, that's one thing I'm sure)

Newer kernels have a nice feature called "blktrace" to trace the block
layer activity. I include with this mail a log of the whole block (this
time it was just about 30 seconds, the previous one, when I was sending
a mail to a maillist, it was 10 minutes or maybe more).

How did I do it:
- Write a mail to myself (small mail, btw).
- Start the btrace.
- Send it. (I pressed 'y' at mutt mail sending screen)
- Look at the HD led.
- When it stopped, the crap @ btrace stopped too. I then stopped btrace.

I hope that this log helps enlighten someone.

Now, to add to the data about the bug:
- My new desktop uses reiser4. It is affected, too.
- Just by typing "reboot" at my old desktop, the bug triggers inmediatly
  after the wall message is sent, and lasts about 10 minutes.
- Latest reiser4 fsck was run with --build-fs on my old desktop the day
  before; The FS had got, before that, some corrupcion (probably a bug)
  that caused kernel panics, so the FS is quite clean now, yet I can
  reproduce the bugs.

I will be happy to help further in any way.

I also have many friends who use reiser4 and are experiencing it; it
would be a shame if reiser4 finally got merged into the kernel with
this bug still there.

Roc Vallès Domènech
-- 
- Do you study theories? - Oh yes! Theories are fun. - How do you use them? - When I must act, theories are too slow. To act, I must know.

[-- Attachment #1.2: trace.gz --]
[-- Type: application/octet-stream, Size: 61977 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: The Infamous Reiser4-randomly-blocks-for-ages-and-writes-the-hd-continously-in-the-mean-while now with a btrace log! (hope it helps)

[Nate Diller]

On 8/6/06, rvalles <rvalles@es.gnu.org> wrote:
> The bug is as I've explained a thousand times. (and it does only affect
> kernels newer than 2.6.12, all of them, that's one thing I'm sure)
>
> Newer kernels have a nice feature called "blktrace" to trace the block
> layer activity. I include with this mail a log of the whole block (this
> time it was just about 30 seconds, the previous one, when I was sending
> a mail to a maillist, it was 10 minutes or maybe more).
>
> How did I do it:
> - Write a mail to myself (small mail, btw).
> - Start the btrace.
> - Send it. (I pressed 'y' at mutt mail sending screen)
> - Look at the HD led.
> - When it stopped, the crap @ btrace stopped too. I then stopped btrace.
>
> I hope that this log helps enlighten someone.
>
> Now, to add to the data about the bug:
> - My new desktop uses reiser4. It is affected, too.
> - Just by typing "reboot" at my old desktop, the bug triggers inmediatly
>   after the wall message is sent, and lasts about 10 minutes.
> - Latest reiser4 fsck was run with --build-fs on my old desktop the day
>   before; The FS had got, before that, some corrupcion (probably a bug)
>   that caused kernel panics, so the FS is quite clean now, yet I can
>   reproduce the bugs.
>
> I will be happy to help further in any way.
>
> I also have many friends who use reiser4 and are experiencing it; it
> would be a shame if reiser4 finally got merged into the kernel with
> this bug still there.

91% of the requests are 4K in size, 77% of requests are write
barriers.  looks like there's something that causes bitmap blocks to
be written synchronously.

there's also a LOT of duplication, blocks that are written and then
immediately RE-written.  the 4k block at sector 23246207 is written
226 times over the course of this trace, each time seemingly in a pair
(write it, rewrite it, do other stuff, write it, re-write it, etc).
this is pathological behavior, it's a real bug even without the
performance loss.

NATE

Re: The Infamous Reiser4-randomly-blocks-for-ages-and-writes-the-hd-continously-in-the-mean-while now with a btrace log! (hope it helps)

[Vesa Kaihlavirta]

Nate Diller wrote:
> On 8/6/06, rvalles <rvalles@es.gnu.org> wrote:
>> The bug is as I've explained a thousand times. (and it does only affect
>> kernels newer than 2.6.12, all of them, that's one thing I'm sure)
>>
>> Newer kernels have a nice feature called "blktrace" to trace the block
>> layer activity. I include with this mail a log of the whole block (this
>> time it was just about 30 seconds, the previous one, when I was sending
>> a mail to a maillist, it was 10 minutes or maybe more).
>>
>> How did I do it:
>> - Write a mail to myself (small mail, btw).
>> - Start the btrace.
>> - Send it. (I pressed 'y' at mutt mail sending screen)
>> - Look at the HD led.
>> - When it stopped, the crap @ btrace stopped too. I then stopped btrace.
>>
>> I hope that this log helps enlighten someone.
>>
>> Now, to add to the data about the bug:
>> - My new desktop uses reiser4. It is affected, too.
>> - Just by typing "reboot" at my old desktop, the bug triggers inmediatly
>>   after the wall message is sent, and lasts about 10 minutes.
>> - Latest reiser4 fsck was run with --build-fs on my old desktop the day
>>   before; The FS had got, before that, some corrupcion (probably a bug)
>>   that caused kernel panics, so the FS is quite clean now, yet I can
>>   reproduce the bugs.
>>
>> I will be happy to help further in any way.
>>
>> I also have many friends who use reiser4 and are experiencing it; it
>> would be a shame if reiser4 finally got merged into the kernel with
>> this bug still there.
>
> 91% of the requests are 4K in size, 77% of requests are write
> barriers.  looks like there's something that causes bitmap blocks to
> be written synchronously.
>
> there's also a LOT of duplication, blocks that are written and then
> immediately RE-written.  the 4k block at sector 23246207 is written
> 226 times over the course of this trace, each time seemingly in a pair
> (write it, rewrite it, do other stuff, write it, re-write it, etc).
> this is pathological behavior, it's a real bug even without the
> performance loss.
>
> NATE

I hope no-one says reiser4 is "2 times faster than ext3" before this
issue is cleared.
Could some developer (if Nate's one, pardon me) comment on this issue?
It seems to me
much more important than that endless thread about "official point of view".

Incidentally, I've witnessed similar behaviour in various simple tasks,
e.g. writing
entries to an sqlite database, or receiving mail from pop3 in thunderbird.

On kernel-2.6.17.7 currently.

--vk

Re: The Infamous Reiser4-randomly-blocks-for-ages-and-writes-the-hd-continously-in-the-mean-while now with a btrace log! (hope it helps)

[David Masover]

Vesa Kaihlavirta wrote:

> Incidentally, I've witnessed similar behaviour in various simple tasks,
> e.g. writing
> entries to an sqlite database, or receiving mail from pop3 in thunderbird.

Sounds like fsync issues.  That is being worked on.

Re: The Infamous Reiser4-randomly-blocks-for-ages-and-writes-the-hd-continously-in-the-mean-while now with a btrace log! (hope it helps)

[Łukasz Mierzwa]

Dnia Thu, 10 Aug 2006 20:48:59 +0200, David Masover <ninja@slaphack.com>  
napisa³:

> Vesa Kaihlavirta wrote:
>
>> Incidentally, I've witnessed similar behaviour in various simple tasks,
>> e.g. writing
>> entries to an sqlite database, or receiving mail from pop3 in  
>> thunderbird.
>
> Sounds like fsync issues.  That is being worked on.

I'm think it's writeout that's involved, I tried to disable fsync and it  
helped for apps that are calling fsync to keep data integrity (like  
sqlite) but it also happens when I'm downloading files using rtorrent  
which does not call fsync but generetes many little writes.

Re: The Infamous Reiser4-randomly-blocks-for-ages-and-writes-the-hd-continously-in-the-mean-while now with a btrace log! (hope it helps)

[David Masover]

£ukasz Mierzwa wrote:
> Dnia Thu, 10 Aug 2006 20:48:59 +0200, David Masover <ninja@slaphack.com> 
> napisa³:
> 
>> Vesa Kaihlavirta wrote:
>>
>>> Incidentally, I've witnessed similar behaviour in various simple tasks,
>>> e.g. writing
>>> entries to an sqlite database, or receiving mail from pop3 in 
>>> thunderbird.
>>
>> Sounds like fsync issues.  That is being worked on.
> 
> I'm think it's writeout that's involved, I tried to disable fsync and it 
> helped for apps that are calling fsync to keep data integrity (like 
> sqlite) but it also happens when I'm downloading files using rtorrent 
> which does not call fsync but generetes many little writes.

Hmm.  Fragmentation, maybe?  Is this easily reproduceable with a 
freshly-formatted fs?

I'm just guessing here...

Re: The Infamous Reiser4-randomly-blocks-for-ages-and-writes-the-hd-continously-in-the-mean-while now with a btrace log! (hope it helps)

[Łukasz Mierzwa]

Dnia Thu, 10 Aug 2006 22:59:30 +0200, David Masover <ninja@slaphack.com>  
napisa³:

> £ukasz Mierzwa wrote:
>> Dnia Thu, 10 Aug 2006 20:48:59 +0200, David Masover  
>> <ninja@slaphack.com> napisa³:
>>
>>> Vesa Kaihlavirta wrote:
>>>
>>>> Incidentally, I've witnessed similar behaviour in various simple  
>>>> tasks,
>>>> e.g. writing
>>>> entries to an sqlite database, or receiving mail from pop3 in  
>>>> thunderbird.
>>>
>>> Sounds like fsync issues.  That is being worked on.
>>  I'm think it's writeout that's involved, I tried to disable fsync and  
>> it helped for apps that are calling fsync to keep data integrity (like  
>> sqlite) but it also happens when I'm downloading files using rtorrent  
>> which does not call fsync but generetes many little writes.
>
> Hmm.  Fragmentation, maybe?  Is this easily reproduceable with a  
> freshly-formatted fs?
>
> I'm just guessing here...

That's hard for me to try as I'm using single reiser4 partition for all my  
data, after pushing my reset button several times I'm quite sure that my  
files are safe (if I won't hit any crazy bug ;) ).
But maybe You are right, it's slowly getting full and I did rebuild entire  
system few times. When I will have some free space for backup I'll  
reformat my disk.