Reiser4/2.6.11 dbench hang

Reiser4/2.6.11 dbench hang

[Christian Mayrhuber]

Hi,

If I'm running plain 2.6.11(.0) patched with 
reiser4-for-2.6.11-broken-out.2.gz I can hang reiser4 when running
ftp://ftp.samba.org/pub/tridge/dbench/dbench-3.02.tar.gz
for about an hour with the following script.

----------------------------------------------------------------------
#!/bin/sh

RUNTIME=200

run()
{
     t=$1
     n=$2
     sync
     date
     if [ $t = xattr ]; then
         ./dbench -c client.txt -x -D /tmp -t$RUNTIME $n
     else
         ./dbench -c client.txt -D /tmp -t$RUNTIME $n
     fi
}


while :; do
   for n in 90 100 110 120 130 140 150 160 170 180 190 200; do
      run noxattr $n
   done
done
----------------------------------------------------------------------

After a hard reset reiser4progs 1.0.4 reported semantic tree 
corruptions. (No, I didn't write down the fsck.reiser4 output.)

I tried it out because I found the following on linux-kernel:
http://marc.theaimsgroup.com/?l=linux-kernel&m=110099542612492&w=2

This behaviour is reproducible. I tried ext3,reiserfs,xfs and jfs
on the same disk with the same data, all ran dbench fine till
user abort without locking the machine.

# cat /proc/version
Linux version 2.6.11-k7-reiser4 (root@byblos) (gcc-Version 3.3.5 (Debian 
1:3.3.5-5)) #1 Mon Mar 28 20:23:57 CEST 2005


Quote from namesys.com/download.html
 >Reiser4 has just been released. Namesys has the most stringent QA
 >process of any Linux maintainer, a firm belief that no user should hit
 >a bug that developer testing could find first if the effort is made, a
 >by now grand archive of filesystem tests, and no one on our mailing
 >list can make it crash any more, and all of this is why it took us so
 >long to release it, but until a program has seen a few million real
 >users you should not use it for a mission critical server.

I guess dbench-3.x stress testing is missing from your grand archive ;-)

-- 
lg, Chris

Re: Reiser4/2.6.11 dbench hang

[Hans Reiser]

Christian Mayrhuber wrote:

>
>
> I guess dbench-3.x stress testing is missing from your grand archive ;-)
>
Yes, actually, I suspect it is.  Elena, please comment.

Re: Reiser4/2.6.11 dbench hang

[E.Gryaznova]

Hello.

We use dbench-2.0 in our testing.
Well, dbench-3.02 is added to stressing suit too.

Thanks,
Lena.

Hans Reiser wrote:

>Christian Mayrhuber wrote:
>
>  
>
>>I guess dbench-3.x stress testing is missing from your grand archive ;-)
>>
>>    
>>
>Yes, actually, I suspect it is.  Elena, please comment.
>  
>

Re: Reiser4/2.6.11 dbench hang

[Christian Mayrhuber]

Hi!

Thanks. I noticed there is already dbench-3.03 available:
ftp://ftp.samba.org/pub/tridge/dbench/dbench-3.03.tar.gz


E.Gryaznova wrote:
> Hello.
> 
> We use dbench-2.0 in our testing.
> Well, dbench-3.02 is added to stressing suit too.
> 
> Thanks,
> Lena.

-- 
lg, Chris

Re: Reiser4/2.6.11 dbench hang

[Vladimir Saveliev]

Hello

On Wed, 2005-03-30 at 02:23, Christian Mayrhuber wrote:
> Hi,
> 
> If I'm running plain 2.6.11(.0) patched with 
> reiser4-for-2.6.11-broken-out.2.gz I can hang reiser4 when running

When you say "hang reiser4" do you mean that system freezes completely
(does not respord to key pressing, etc)?

Is it SMP box? How much RAM does the box have? 

> ftp://ftp.samba.org/pub/tridge/dbench/dbench-3.02.tar.gz
> for about an hour with the following script.
> 
> ----------------------------------------------------------------------
> #!/bin/sh
> 
> RUNTIME=200
> 
> run()
> {
>      t=$1
>      n=$2
>      sync
>      date
>      if [ $t = xattr ]; then
>          ./dbench -c client.txt -x -D /tmp -t$RUNTIME $n
>      else
>          ./dbench -c client.txt -D /tmp -t$RUNTIME $n
>      fi
> }
> 
> 
> while :; do
>    for n in 90 100 110 120 130 140 150 160 170 180 190 200; do
>       run noxattr $n
>    done
> done
> ----------------------------------------------------------------------
> 
> After a hard reset reiser4progs 1.0.4 reported semantic tree 
> corruptions. (No, I didn't write down the fsck.reiser4 output.)
> 
> I tried it out because I found the following on linux-kernel:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=110099542612492&w=2
> 
> This behaviour is reproducible. I tried ext3,reiserfs,xfs and jfs
> on the same disk with the same data, all ran dbench fine till
> user abort without locking the machine.
> 
> # cat /proc/version
> Linux version 2.6.11-k7-reiser4 (root@byblos) (gcc-Version 3.3.5 (Debian 
> 1:3.3.5-5)) #1 Mon Mar 28 20:23:57 CEST 2005
> 
> 
> Quote from namesys.com/download.html
>  >Reiser4 has just been released. Namesys has the most stringent QA
>  >process of any Linux maintainer, a firm belief that no user should hit
>  >a bug that developer testing could find first if the effort is made, a
>  >by now grand archive of filesystem tests, and no one on our mailing
>  >list can make it crash any more, and all of this is why it took us so
>  >long to release it, but until a program has seen a few million real
>  >users you should not use it for a mission critical server.
> 
> I guess dbench-3.x stress testing is missing from your grand archive ;-)

Re: Reiser4/2.6.11 dbench hang

[Christian Mayrhuber]

Vladimir Saveliev schrieb:
> Hello
> 
> On Wed, 2005-03-30 at 02:23, Christian Mayrhuber wrote:
> 
>>Hi,
>>
>>If I'm running plain 2.6.11(.0) patched with 
>>reiser4-for-2.6.11-broken-out.2.gz I can hang reiser4 when running
> 
> 
> When you say "hang reiser4" do you mean that system freezes completely
> (does not respord to key pressing, etc)?

dbench-3.02 stuck in its cleanup phase when simulating 200 clients but
it kept reporting the benchmark statistics every second.

All X11 applications were useable till they performed some disk io. 
After that they stopped reacting to user interaction.

Next I tried to switch to vt01, which failed.
I waited for about 10 minutes but the system did not react to any
keyboard events. Even the keyboard leds (num lock, caps lock) stopped
working.

Finally I performed a cold reboot.

> 
> Is it SMP box? How much RAM does the box have? 
> 

No, it's an UP Athlon XP 1800+ Palomino system with 512MB RAM.
The mainboard uses the Nvidia NFORCE2 chipset and has a 60GB
seagate harddrive attached to its primary ide controller.
The kernel was optimized for the AMD K7 platform by gcc-3.3.5.
No other patches than the reiser4 patch were applied.

-- 
lg, Chris

Re: Reiser4/2.6.11 dbench hang

[Vladimir Saveliev]

Hello

On Thu, 2005-03-31 at 00:21, Christian Mayrhuber wrote:
> Vladimir Saveliev schrieb:
> > Hello
> > 
> > On Wed, 2005-03-30 at 02:23, Christian Mayrhuber wrote:
> > 
> >>Hi,
> >>
> >>If I'm running plain 2.6.11(.0) patched with 
> >>reiser4-for-2.6.11-broken-out.2.gz I can hang reiser4 when running
> > 
> > 
> > When you say "hang reiser4" do you mean that system freezes completely
> > (does not respord to key pressing, etc)?
> 
> dbench-3.02 stuck in its cleanup phase when simulating 200 clients but
> it kept reporting the benchmark statistics every second.
> 

ok, the problem is found already. I hope to make a fix tomorrow

> All X11 applications were useable till they performed some disk io. 
> After that they stopped reacting to user interaction.
> 
> Next I tried to switch to vt01, which failed.
> I waited for about 10 minutes but the system did not react to any
> keyboard events. Even the keyboard leds (num lock, caps lock) stopped
> working.
> 
> Finally I performed a cold reboot.
> 
> > 
> > Is it SMP box? How much RAM does the box have? 
> > 
> 
> No, it's an UP Athlon XP 1800+ Palomino system with 512MB RAM.
> The mainboard uses the Nvidia NFORCE2 chipset and has a 60GB
> seagate harddrive attached to its primary ide controller.
> The kernel was optimized for the AMD K7 platform by gcc-3.3.5.
> No other patches than the reiser4 patch were applied.

Re: Reiser4/2.6.11 dbench hang

[Christian Mayrhuber]

Vladimir Saveliev schrieb:
> Hello
> 
> On Thu, 2005-03-31 at 00:21, Christian Mayrhuber wrote:
> 
>>Vladimir Saveliev schrieb:
>>
>>>Hello
>>>
>>>On Wed, 2005-03-30 at 02:23, Christian Mayrhuber wrote:
>>>
>>>
>>>>Hi,
>>>>
>>>>If I'm running plain 2.6.11(.0) patched with 
>>>>reiser4-for-2.6.11-broken-out.2.gz I can hang reiser4 when running
>>>
>>>
>>>When you say "hang reiser4" do you mean that system freezes completely
>>>(does not respord to key pressing, etc)?
>>
>>dbench-3.02 stuck in its cleanup phase when simulating 200 clients but
>>it kept reporting the benchmark statistics every second.
>>
> 
> 
> ok, the problem is found already. I hope to make a fix tomorrow

Wow. Fantastic job.

-- 
lg, Chris

Re: Reiser4/2.6.11 dbench hang

[Vladimir Saveliev]

Hello

On Thu, 2005-03-31 at 02:35, Christian Mayrhuber wrote:
> Vladimir Saveliev schrieb:
> > Hello
> > 
> > On Thu, 2005-03-31 at 00:21, Christian Mayrhuber wrote:
> > 
> >>Vladimir Saveliev schrieb:
> >>
> >>>Hello
> >>>
> >>>On Wed, 2005-03-30 at 02:23, Christian Mayrhuber wrote:
> >>>
> >>>
> >>>>Hi,
> >>>>
> >>>>If I'm running plain 2.6.11(.0) patched with 
> >>>>reiser4-for-2.6.11-broken-out.2.gz I can hang reiser4 when running
> >>>
> >>>
> >>>When you say "hang reiser4" do you mean that system freezes completely
> >>>(does not respord to key pressing, etc)?
> >>
> >>dbench-3.02 stuck in its cleanup phase when simulating 200 clients but
> >>it kept reporting the benchmark statistics every second.
> >>
> > 
> > 
> > ok, the problem is found already. I hope to make a fix tomorrow
> 

This problem is fixed, but running your test during several hours caused
reiser4 crash. It just happened, so, tomorrow I will take a look at it.

Re: Reiser4/2.6.11 dbench hang

[Christian Mayrhuber]

Vladimir Saveliev schrieb:
> 
> This problem is fixed, but running your test during several hours caused
> reiser4 crash. It just happened, so, tomorrow I will take a look at it.
> 

I was able to deadlock kernel 2.6.11 + reiser4-for-2.6.11-3.patch.gz
+ elevator=cfq.
This time it happend during the first dbench-3.02 run with 90 clients in 
the cleanup phase of the benchmark.
Maybe the reproduction of the deadlock is easier when using the cfq
io scheduler.

No fs corruption reported by fsck.reiser4 after a cold reboot :-)

-- 
lg, Chris

Re: Reiser4/2.6.11 dbench hang

[sergey ivanov]

So how about this deadlock? Is there any expectation about fix for it?
     Wbr,
         Sergey.

Christian Mayrhuber wrote:

> Vladimir Saveliev schrieb:
>
>>
>> This problem is fixed, but running your test during several hours caused
>> reiser4 crash. It just happened, so, tomorrow I will take a look at it.
>>
>
> I was able to deadlock kernel 2.6.11 + reiser4-for-2.6.11-3.patch.gz
> + elevator=cfq.
> This time it happend during the first dbench-3.02 run with 90 clients
> in the cleanup phase of the benchmark.
> Maybe the reproduction of the deadlock is easier when using the cfq
> io scheduler.
>
> No fs corruption reported by fsck.reiser4 after a cold reboot :-)
>

Re: Reiser4/2.6.11 dbench hang

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

sergey ivanov wrote:
> So how about this deadlock? Is there any expectation about fix for it?
>     Wbr,
>         Sergey.
> 
> Christian Mayrhuber wrote:
> 
>> Vladimir Saveliev schrieb:
>>
>>>
>>> This problem is fixed, but running your test during several hours caused
>>> reiser4 crash. It just happened, so, tomorrow I will take a look at it.
>>>
>>
>> I was able to deadlock kernel 2.6.11 + reiser4-for-2.6.11-3.patch.gz
>> + elevator=cfq.

Is it just with elevator=cfq?

BTW, I heard something about anticipatory being bad (IO only goes one
way at a time).  Is this true?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQlSeqngHNmZLgCUhAQJasQ/+PEUD4RS4+vsRR0Raekx5RXNyyUspvqOO
8boVKC9fjUYxfFI14cD38TqOXvTgEVvoMobUkV6KaiYf/MdIFGFYx4G9M5/YsNCh
3jZ9ENKp8UzfmClX4ZkXtgySPTun85zk4yklv+GoFT56pdMzSbNUHXx/2u/ttRyv
/SMKAddqHRSwM0R6Svy33CBApDXmcyjCzzSWowBinO4u6vfZH6KJS/vdA9GYomTa
UreczCnx/iUN89fL1yPwLDamF787Wgp0This5ivbzxrq9KTeUPmsbS3dNJu1HuYs
spQ946jJW8gN15abszJuXG8qUFHr3Cy9t5FOBExcjHa25pYIJZ+lFjI5Euo5jaDl
1rLBAB2Ax3QdEEP4uiMfied+ynLvcGy1bOs9fS0jp0Ou53WXZlMuQ6jJTyF+XQJ0
MmLboyjZslg3JkyQSu2As5A3KwgN+rOWAjBqh1mbuHlZ4f727WSSaGbt4WRhyfxr
hrS8fW6sOSwK+50F/7ZnPuAwPuGZCWuQArNoEARE1CPwCQCduAs+bMOkM1p3Sowy
yLVxrJztGD5dssYmKJuE9UavYG+3ZPDk5hXG/FRczSJ0LHDu99z6bWR1BfEnmBoL
uMZMf0DhZwoQdtp99CE0hNL//wyrSrodtAnAx9q3YHL0qoQWCyYYFhnFWobsO/en
8ROwkSEKgyY=
=dHG5
-----END PGP SIGNATURE-----

Re: Reiser4/2.6.11 dbench hang

[Christian Mayrhuber]

David Masover schrieb:
> sergey ivanov wrote:
> 
>>>So how about this deadlock? Is there any expectation about fix for it?
>>>    Wbr,
>>>        Sergey.
>>>
>>>Christian Mayrhuber wrote:
>>>
>>>
>>>>Vladimir Saveliev schrieb:
>>>>
>>>>
>>>>>This problem is fixed, but running your test during several hours caused
>>>>>reiser4 crash. It just happened, so, tomorrow I will take a look at it.
>>>>>
>>>>
>>>>I was able to deadlock kernel 2.6.11 + reiser4-for-2.6.11-3.patch.gz
>>>>+ elevator=cfq.
> 
> 
> Is it just with elevator=cfq?
> 
> BTW, I heard something about anticipatory being bad (IO only goes one
> way at a time).  Is this true?
> 

I couldn't reproduce the deadlock with
kernel 2.6.11 + reiser4-for-2.6.11-3.patch.gz + anticipatory
scheduler while running dbench for 4 hours.
With cfq it deadlocked during the first cleanup phase of dbench
(running with 90 clients).

Note: With AS the system was nearly unuseable when running dbench.
Issuing "su" on the shell lasted 34 seconds till the password
prompt appeared.

-- 
lg, Chris