Re: [BUG] reiser4-for-2.6.22-2 Large stalls when using atime mount option

Re: [BUG] reiser4-for-2.6.22-2 Large stalls when using atime mount option

[Edward Shishkin]

Hello.

On 11/7/07, Dushan Tcholich <dusanc@gmail.com> wrote:
> Kernel: Gentoo 2.6.22-r2 + reiser4-for-2.6.22-2.patch
> fstab:
> /dev/sda2        /boot        ext2        noauto,noatime    1 2
> /dev/sda5        /        reiser4                0 0
> /dev/sda3        none        swap        sw        0 0
> /dev/sda6        /home        ext3        noatime        0 0
>
> reiser4 partition is with cryptocompress (lzo) default settings.
>
> I usually mount r4 with -noatime, but after some debbuging with some users
> on irc (rvalles), who have intermitent large stalls, we found that only
> difference between our setups was my -noatime flag in fstab.

Plus rvalles doesn't use compression (isn't it?)

> I tried to mount / without -noatime, started gnome, and then in about same
> time evolution, firefox, played some mp3s, avis, did a kernel compile,
> emerge -uD world (compiled few packages), acessed some random fles in
> /usr/src/linux, started gnome disk usage analyser etc. and had no stalls at
> all.
> And then I tried to reboot. System stalled for 3min 25 sec (i was ready with
> a watch :)) before printing the first message, with heavy disk activity.
> Upon reboot I just changed line in fstab with nano to mount r4 with -noatime
> and rebooted agin. Now there were no stalls.
>

Currently I am preparing a patch for this problem.
Not sure about its efficiency: we'll take a look..

Thanks,
Edward.

Re: [BUG] reiser4-for-2.6.22-2 Large stalls when using atime mount option

[Dushan Tcholich]

Hi

On Nov 7, 2007 10:31 PM, Edward Shishkin <edward.shishkin@gmail.com> wrote:
> Hello.
>
> On 11/7/07, Dushan Tcholich <dusanc@gmail.com> wrote:
> > Kernel: Gentoo 2.6.22-r2 + reiser4-for-2.6.22-2.patch
> > fstab:
> > /dev/sda2        /boot        ext2        noauto,noatime    1 2
> > /dev/sda5        /        reiser4                0 0
> > /dev/sda3        none        swap        sw        0 0
> > /dev/sda6        /home        ext3        noatime        0 0
> >
> > reiser4 partition is with cryptocompress (lzo) default settings.
> >
> > I usually mount r4 with -noatime, but after some debbuging with some users
> > on irc (rvalles), who have intermitent large stalls, we found that only
> > difference between our setups was my -noatime flag in fstab.
>
> Plus rvalles doesn't use compression (isn't it?)

He had stalls without compression, but now switched to compression and
still has them.
But I can't reproduce them with -noatime option.
>
> > I tried to mount / without -noatime, started gnome, and then in about same
> > time evolution, firefox, played some mp3s, avis, did a kernel compile,
> > emerge -uD world (compiled few packages), acessed some random fles in
> > /usr/src/linux, started gnome disk usage analyser etc. and had no stalls at
> > all.
> > And then I tried to reboot. System stalled for 3min 25 sec (i was ready with
> > a watch :)) before printing the first message, with heavy disk activity.
> > Upon reboot I just changed line in fstab with nano to mount r4 with -noatime
> > and rebooted agin. Now there were no stalls.
> >

To my inexperienced eyes it looks like lazy writes of atime are adding
with that "reiser4 writes continuously to same sector" bug.

>
> Currently I am preparing a patch for this problem.
> Not sure about its efficiency: we'll take a look..

Great

>
> Thanks,
> Edward.
>

Dushan

Fwd: [BUG] reiser4-for-2.6.22-2 Large stalls when using atime mount option

[Dushan Tcholich]

---------- Forwarded message ----------
From: Dushan Tcholich <dusanc@gmail.com>
Date: Nov 9, 2007 3:03 PM
Subject: Re: [BUG] reiser4-for-2.6.22-2 Large stalls when using atime
mount option
To: Edward Shishkin <edward.shishkin@gmail.com>


There were some stalls, but a lot shorter than before (~7sec on
shutdown with patch, ~3.5min without), and no kernel messages.
Stalls that were encountered during normal use were short 1-3sec.
Take all this with a grain of salt because, for me, these are hard to reproduce.
kernel2.6.22-r8+reiser-for-2.6.22-2.patch+reiser4-discard-entd.patch
reiser4 mounted as / without -noatime option.

Dusan




On Nov 9, 2007 2:04 AM, Edward Shishkin <edward.shishkin@gmail.com> wrote:
> Hello guys.
> This is the promised patch against (2.6.22 + reiser4-for-2.6.22-2).
>
> Note, that this patch may cause problems, so keep also a stable kernel
> to boot on.
> The goal is to detect any progress in preventing the nasty stalls:
> please boot to the
> patched kernel and try to reproduce them. If you encounter oops, then
> reboot to the
> same kernel and try again. If oopses are annoying, then boot to the
> stable kernel and
> let me know about it.
>
> Waiting for your results..
> Thanks,
> Edward.
>
>
> On 11/8/07, Dushan Tcholich <dusanc@gmail.com> wrote:
> > Hi
> >
> > On Nov 7, 2007 10:31 PM, Edward Shishkin <edward.shishkin@gmail.com> wrote:
> > > Hello.
> > >
> > > On 11/7/07, Dushan Tcholich <dusanc@gmail.com> wrote:
> > > > Kernel: Gentoo 2.6.22-r2 + reiser4-for-2.6.22-2.patch
> > > > fstab:
> > > > /dev/sda2        /boot        ext2        noauto,noatime    1 2
> > > > /dev/sda5        /        reiser4                0 0
> > > > /dev/sda3        none        swap        sw        0 0
> > > > /dev/sda6        /home        ext3        noatime        0 0
> > > >
> > > > reiser4 partition is with cryptocompress (lzo) default settings.
> > > >
> > > > I usually mount r4 with -noatime, but after some debbuging with some users
> > > > on irc (rvalles), who have intermitent large stalls, we found that only
> > > > difference between our setups was my -noatime flag in fstab.
> > >
> > > Plus rvalles doesn't use compression (isn't it?)
> >
> > He had stalls without compression, but now switched to compression and
> > still has them.
> > But I can't reproduce them with -noatime option.
> > >
> > > > I tried to mount / without -noatime, started gnome, and then in about same
> > > > time evolution, firefox, played some mp3s, avis, did a kernel compile,
> > > > emerge -uD world (compiled few packages), acessed some random fles in
> > > > /usr/src/linux, started gnome disk usage analyser etc. and had no stalls at
> > > > all.
> > > > And then I tried to reboot. System stalled for 3min 25 sec (i was ready with
> > > > a watch :)) before printing the first message, with heavy disk activity.
> > > > Upon reboot I just changed line in fstab with nano to mount r4 with -noatime
> > > > and rebooted agin. Now there were no stalls.
> > > >
> >
> > To my inexperienced eyes it looks like lazy writes of atime are adding
> > with that "reiser4 writes continuously to same sector" bug.
> >
> > >
> > > Currently I am preparing a patch for this problem.
> > > Not sure about its efficiency: we'll take a look..
> >
> > Great
> >
> > >
> > > Thanks,
> > > Edward.
> > >
> >
> > Dushan
> >
>

Re: [BUG] reiser4-for-2.6.22-2 Large stalls when using atime mount option

[Edward Shishkin]

On 11/14/07, Dushan Tcholich <dusanc@gmail.com> wrote:
> Managed to find them in the logs:

I would say the below is stacktrace of another thread:
Here we can see that kupdate won the commit races
and performs write_logs, whereas at your screen we
can see a stacktrace of the final sync, which is waiting
(I guess) for this atom to be committed. I see nothing
criminal here. So we need to figure out why the commit
takes so much time. Unfortunately sysrq-t dumps aren't
able to provide such details, I'll think how to do it with
your assistance.
Good work!

Thanks,
Edward.

> Nov 14 21:01:31 krshina3 00000000 f708b200 00047863 00000003 00000000
> 00000000 c1809900 00000000
> Nov 14 21:01:31 krshina3 Call Trace:
> Nov 14 21:01:31 krshina3 [<c03ac56a>] io_schedule+0x1d/0x27
> Nov 14 21:01:31 krshina3 [<c01360d3>] sync_page+0x3d/0x40
> Nov 14 21:01:31 krshina3 [<c03ac7f1>] __wait_on_bit+0x33/0x58
> Nov 14 21:01:31 krshina3 [<c0136096>] sync_page+0x0/0x40
> Nov 14 21:01:31 krshina3 [<c0136214>] wait_on_page_bit+0x61/0x66
> Nov 14 21:01:31 krshina3 [<c0128f16>] wake_bit_function+0x0/0x3c
> Nov 14 21:01:31 krshina3 [<c0182e33>] jwait_io+0x23/0x34
> Nov 14 21:01:31 krshina3 [<c01920e9>] reiser4_write_logs+0x940/0xbd1
> Nov 14 21:01:31 krshina3 [<c018ccf6>] reiser4_txn_end+0x317/0x46c
> Nov 14 21:01:31 krshina3 [<c018ce53>] reiser4_txn_restart+0x8/0x15
> Nov 14 21:01:31 krshina3 [<c018d5a8>] flush_some_atom+0x3bf/0x3e0
> Nov 14 21:01:31 krshina3 [<c0198675>] reiser4_writeout+0x9c/0x109
> Nov 14 21:01:31 krshina3 [<c018b723>] reiser4_txn_begin+0x1d/0x37
> Nov 14 21:01:31 krshina3 [<c0167145>] generic_sync_sb_inodes+0x1d/0x222
> Nov 14 21:01:31 krshina3 [<c0195c59>] reiser4_sync_inodes+0x80/0x99
> Nov 14 21:01:31 krshina3 [<c03abfa1>] __sched_text_start+0x6e9/0x780
> Nov 14 21:01:31 krshina3 [<c0167145>] generic_sync_sb_inodes+0x1d/0x222
> Nov 14 21:01:31 krshina3 [<c0195bd9>] reiser4_sync_inodes+0x0/0x99
> Nov 14 21:01:31 krshina3 [<c016735b>] sync_sb_inodes+0x11/0x1b
> Nov 14 21:01:31 krshina3 [<c0167679>] writeback_inodes+0x59/0x94
> Nov 14 21:01:31 krshina3 [<c013b3c6>] pdflush+0x0/0x19d
> Nov 14 21:01:31 krshina3 [<c013b0e8>] wb_kupdate+0x7b/0xdf
> Nov 14 21:01:31 krshina3 [<c013b4cf>] pdflush+0x109/0x19d
> Nov 14 21:01:31 krshina3 [<c013b06d>] wb_kupdate+0x0/0xdf
> Nov 14 21:01:31 krshina3 [<c0128e1b>] kthread+0x38/0x5d
> Nov 14 21:01:31 krshina3 [<c0128de3>] kthread+0x0/0x5d
> Nov 14 21:01:31 krshina3 [<c0103c83>] kernel_thread_helper+0x7/0x10
> Nov 14 21:01:31 krshina3 =======================
>

[BUG] reiser4-for-2.6.22-2 Large stalls when using atime mount option

[Dushan Tcholich]

Kernel: Gentoo 2.6.22-r2 + reiser4-for-2.6.22-2.patch
fstab:
/dev/sda2        /boot        ext2        noauto,noatime    1 2
/dev/sda5        /        reiser4                0 0
/dev/sda3        none        swap        sw        0 0 
/dev/sda6        /home        ext3        noatime        0 0

reiser4 partition is with cryptocompress (lzo) default settings.

I usually mount r4 with -noatime, but after some debbuging with some
users on irc (rvalles), who have intermitent large stalls, we found that
only difference between our setups was my -noatime flag in fstab. 
I tried to mount / without -noatime, started gnome, and then in about
same time evolution, firefox, played some mp3s, avis, did a kernel
compile, emerge -uD world (compiled few packages), acessed some random
fles in /usr/src/linux, started gnome disk usage analyser etc. and had
no stalls at all. 
And then I tried to reboot. System stalled for 3min 25 sec (i was ready
with a watch :)) before printing the first message, with heavy disk
activity.
Upon reboot I just changed line in fstab with nano to mount r4 with
-noatime and rebooted agin. Now there were no stalls. 


Related/same bugreports:
http://marc.info/?l=reiserfs-devel&m=113081021609510&w=2
http://marc.info/?l=reiserfs-devel&m=114368644227582&w=2
http://marc.info/?l=reiserfs-devel&m=115488109712570&w=2
http://marc.info/?l=reiserfs-devel&m=117507704927976&w=2 this one is
interesting - Reiser4 does a disk write with even no disk activity
during every fsync()
http://marc.info/?l=reiserfs-devel&m=115285478906088&w=2

As I can see this started in 2.6.12->2.6.13 transition time.

Have fun :)