[PATCH] updated data=ordered patch for 2.6.3

[PATCH] updated data=ordered patch for 2.6.3

[Chris Mason]

Hello everyone,

I've updated the reiserfs logging speedups and data=ordered support to
2.6.3, and fixed a few bugs:

i_block count not properly updated on files with holes
oops when the disk is full
small files were not being packed into tails

Also, the code has some extra performance tweaks to smooth out
performance both with and without data=ordered.  There are new
mechanisms to trigger metadata/commit block writeback and to help
throttle writers.  The goal is to reduce the huge bursts of io during a
commit and during data=ordered writeback.

I'm very interested in benchmarks here and info about how the patches
feel for interactive performance.

Please note that data=ordered is not the default yet, you have to mount
with -o data=ordered to get it.

You can get the patches from:

ftp.suse.com/pub/people/mason/patches/data-logging/experimental/2.6.3

(once the suse mirror copies it over there)

-chris

Re: [PATCH] updated data=ordered patch for 2.6.3

[Dieter Nützel]

Am Donnerstag, 19. Februar 2004 05:36 schrieb Chris Mason:
> Hello everyone,
>
> I've updated the reiserfs logging speedups and data=ordered support to
> 2.6.3, and fixed a few bugs:
>
> i_block count not properly updated on files with holes
> oops when the disk is full
> small files were not being packed into tails
>
> Also, the code has some extra performance tweaks to smooth out
> performance both with and without data=ordered.  There are new
> mechanisms to trigger metadata/commit block writeback and to help
> throttle writers.  The goal is to reduce the huge bursts of io during a
> commit and during data=ordered writeback.

On-top of SuSE 9.0, kernel 2.6.3-0:

01-reiserfs-journal-writer

Apply clean.


02-reiserfs-nesting
03-reiserfs-iosize

Are in.


04-reiserfs-logging
05-reiserfs-jh-3

Need some handwork due to ACLs.


You've forgotten in linux/fs/buffer.c
EXPORT_SYMBOL(try_to_release_page);
again ;-)


And the FIX in stree.c:

--- linux.test/fs/reiserfs/stree.c.1    2004-02-16 09:26:33.643567752 -0500
+++ linux.test/fs/reiserfs/stree.c      2004-02-16 09:27:24.166887040 -0500
@@ -1754,7 +1754,7 @@
          reiserfs_update_sd(th, p_s_inode) ;
 
          journal_end(th, p_s_inode->i_sb, orig_len_alloc) ;
-         journal_begin(th, p_s_inode->i_sb, orig_len_alloc) ;
+         journal_begin(th, p_s_inode->i_sb, JOURNAL_PER_BALANCE_CNT * 5) ;
          reiserfs_update_inode_transaction(p_s_inode) ;
        }
     } while ( n_file_size > ROUND_UP (n_new_file_size) &&


> I'm very interested in benchmarks here and info about how the patches
> feel for interactive performance.

Not, yet.
But same as with stree.c on 2.6.2-6.

Greetings,
	Dieter

Re: [PATCH] updated data=ordered patch for 2.6.3

[Chris Mason]

On Mon, 2004-02-23 at 04:42, Dieter Nützel wrote:

> You've forgotten in linux/fs/buffer.c
> EXPORT_SYMBOL(try_to_release_page);
> again ;-)
> 
Fixed in a new upload.
> 
> And the FIX in stree.c:

Also fixed.  Patches 00* will get submitted to Andrew as soon as namesys
reviews them.

There are going into the suse kernel now too, it's time for some broader
testing.

-chris

Re: [PATCH] updated data=ordered patch for 2.6.3

[Vince]

Chris Mason wrote:

> Also fixed.  Patches 00* will get submitted to Andrew as soon as namesys
> reviews them.
> 
> There are going into the suse kernel now too, it's time for some broader
> testing.
> 
> -chris
> 
> 

  Those patches make use of the ->journal_info field ... which is on the 
other hand removed in the reiser4 kernel patch, so both patches conflict :-/
  Is there any way to get both reiser4 and reiser3+data-logging in the 
same kernel ? Would adding back the journal_info field be enough ?

Compilation with a reiser4-patched kernel fails with:

/usr/src/linux$ make
   CHK     include/linux/version.h
   CC      scripts/empty.o
   MKELF   scripts/elfconfig.h
   HOSTCC  scripts/file2alias.o
   HOSTCC  scripts/modpost.o
   HOSTCC  scripts/sumversion.o
   HOSTLD  scripts/modpost
   CC      arch/i386/kernel/asm-offsets.s
   CHK     include/asm-i386/asm_offsets.h
   CHK     include/linux/compile.h
   CC      fs/buffer.o
   CC      fs/reiserfs/bitmap.o
In file included from fs/reiserfs/bitmap.c:8:
include/linux/reiserfs_fs.h: In function `reiserfs_transaction_running':
include/linux/reiserfs_fs.h:1753: error: structure has no member named 
`journal_info'
make[2]: *** [fs/reiserfs/bitmap.o] Error 1
make[1]: *** [fs/reiserfs] Error 2
make: *** [fs] Error 2


/usr/src/linux$ zgrep journal_info ../2004.02.06.all.diff.gz
-       current->journal_info = handle;
-               current->journal_info = NULL;
-       current->journal_info = NULL;
-       .journal_info   = NULL,
-       return current->journal_info;
-       void *journal_info;
+       result = init_journal_info(s);
+               done_journal_info(s);
+       done_journal_info(s);

Re: [PATCH] updated data=ordered patch for 2.6.3

[Christophe Saout]

Hi,

> Also, the code has some extra performance tweaks to smooth out
> performance both with and without data=ordered.  There are new
> mechanisms to trigger metadata/commit block writeback and to help
> throttle writers.  The goal is to reduce the huge bursts of io during a
> commit and during data=ordered writeback.

It seems you introduced a bug here. I installed the patches yesterday
and found a lockup on my notebook when running lilo (with /boot on the
root reiserfs filesystem).

A SysRq-T showed that lilo is stuck in fsync:

lilo:
schedule
sleep_on
default_wake_function
sys_sched_yield
let_transaction_grow
__commit_trans_jl
reiserfs_sync_file
sys_fsync

reiserfs/0:
worker_thread
flush_async_commits
default_wake_function
...

The problem is 100% reproducable here. Perhaps it has something to do
with tail unpacking? This is strange:

After copying the bzImage file to /boot and running lilo he can add the
copied bzImage file, add the old file and then hangs on the third one.
But after rebooting and running lilo he already hangs on the first one.
After copying the file again he again passes the first two and hangs on
the third kernel to add.

The whole filesystem locks up when this happens, not just the lilo
process.

Re: [PATCH] updated data=ordered patch for 2.6.3

[Chris Mason]

On Mon, 2004-03-01 at 08:30, Christophe Saout wrote:
> Hi,
> 
> > Also, the code has some extra performance tweaks to smooth out
> > performance both with and without data=ordered.  There are new
> > mechanisms to trigger metadata/commit block writeback and to help
> > throttle writers.  The goal is to reduce the huge bursts of io during a
> > commit and during data=ordered writeback.
> 
> It seems you introduced a bug here. I installed the patches yesterday
> and found a lockup on my notebook when running lilo (with /boot on the
> root reiserfs filesystem).
> 
> A SysRq-T showed that lilo is stuck in fsync:
> 

Ugh, I use grub so I haven't tried lilo.  Could you please send me the
full sysrq-t, this is probably something stupid.

-chris

Re: [PATCH] updated data=ordered patch for 2.6.3

[Chris Mason]

On Mon, 2004-03-01 at 09:30, Christophe Saout wrote:
> Am Mo, den 01.03.2004 schrieb Chris Mason um 15:01:
> 
> > > It seems you introduced a bug here. I installed the patches yesterday
> > > and found a lockup on my notebook when running lilo (with /boot on the
> > > root reiserfs filesystem).
> > > 
> > > A SysRq-T showed that lilo is stuck in fsync:
> > 
> > Ugh, I use grub so I haven't tried lilo.  Could you please send me the
> > full sysrq-t, this is probably something stupid.
> 
> Yes. I could reproduce it by simply creating a dummy /boot volume on
> reiserfs. I copied the content of /boot, ran lilo and it hung again. The
> other reiserfs filesystems were still usable (but a global sync hangs
> afterwards). I also attached a bzipped strace of the lilo process.
> 

Ok, thanks.  The problem is in reiserfs_unpack(), which needs updating
for the patch.  Fixing.

-chris

Re: [PATCH] updated data=ordered patch for 2.6.3

[Dieter Nützel]

Am Montag, 1. März 2004 15:38 schrieb Chris Mason:
> On Mon, 2004-03-01 at 09:30, Christophe Saout wrote:
> > Am Mo, den 01.03.2004 schrieb Chris Mason um 15:01:
> > > > It seems you introduced a bug here. I installed the patches yesterday
> > > > and found a lockup on my notebook when running lilo (with /boot on
> > > > the root reiserfs filesystem).
> > > >
> > > > A SysRq-T showed that lilo is stuck in fsync:
> > >
> > > Ugh, I use grub so I haven't tried lilo.  Could you please send me the
> > > full sysrq-t, this is probably something stupid.
> >
> > Yes. I could reproduce it by simply creating a dummy /boot volume on
> > reiserfs. I copied the content of /boot, ran lilo and it hung again. The
> > other reiserfs filesystems were still usable (but a global sync hangs
> > afterwards). I also attached a bzipped strace of the lilo process.
>
> Ok, thanks.  The problem is in reiserfs_unpack(), which needs updating
> for the patch.  Fixing.

I'll test this under SuSE 2.6.3-7 (with lilo).
Or is it in?

Thanks,
	Dieter

Re: [PATCH] updated data=ordered patch for 2.6.3

[Dieter Nützel]

Am Montag, 1. März 2004 15:38 schrieb Chris Mason:
> On Mon, 2004-03-01 at 09:30, Christophe Saout wrote:
> > Am Mo, den 01.03.2004 schrieb Chris Mason um 15:01:
> > > > It seems you introduced a bug here. I installed the patches yesterday
> > > > and found a lockup on my notebook when running lilo (with /boot on
> > > > the root reiserfs filesystem).
> > > >
> > > > A SysRq-T showed that lilo is stuck in fsync:
> > >
> > > Ugh, I use grub so I haven't tried lilo.  Could you please send me the
> > > full sysrq-t, this is probably something stupid.
> >
> > Yes. I could reproduce it by simply creating a dummy /boot volume on
> > reiserfs. I copied the content of /boot, ran lilo and it hung again. The
> > other reiserfs filesystems were still usable (but a global sync hangs
> > afterwards). I also attached a bzipped strace of the lilo process.
>
> Ok, thanks.  The problem is in reiserfs_unpack(), which needs updating
> for the patch.  Fixing.

Progress?

I'll try on SuSE 2.6.3-16.

Thanks,
	Dieter

Re: [PATCH] updated data=ordered patch for 2.6.3

[Chris Mason]

On Tue, 2004-03-02 at 13:14, Dieter Nützel wrote:
> Am Montag, 1. März 2004 15:38 schrieb Chris Mason:
> > On Mon, 2004-03-01 at 09:30, Christophe Saout wrote:
> > > Am Mo, den 01.03.2004 schrieb Chris Mason um 15:01:
> > > > > It seems you introduced a bug here. I installed the patches yesterday
> > > > > and found a lockup on my notebook when running lilo (with /boot on
> > > > > the root reiserfs filesystem).
> > > > >
> > > > > A SysRq-T showed that lilo is stuck in fsync:
> > > >
> > > > Ugh, I use grub so I haven't tried lilo.  Could you please send me the
> > > > full sysrq-t, this is probably something stupid.
> > >
> > > Yes. I could reproduce it by simply creating a dummy /boot volume on
> > > reiserfs. I copied the content of /boot, ran lilo and it hung again. The
> > > other reiserfs filesystems were still usable (but a global sync hangs
> > > afterwards). I also attached a bzipped strace of the lilo process.
> >
> > Ok, thanks.  The problem is in reiserfs_unpack(), which needs updating
> > for the patch.  Fixing.
> 
> Progress?
> 
> I'll try on SuSE 2.6.3-16.

Not fixed there yet.  Slight delay yesterday since I'm trying to roll
out the fix and some quota patches.  Should be ready this afternoon.

-chris

Re: [PATCH] updated data=ordered patch for 2.6.3

[Dieter Nützel]

Am Dienstag, 2. März 2004 19:16 schrieb Chris Mason:
> On Tue, 2004-03-02 at 13:14, Dieter Nützel wrote:
> > Am Montag, 1. März 2004 15:38 schrieb Chris Mason:
> > > On Mon, 2004-03-01 at 09:30, Christophe Saout wrote:
> > > > Am Mo, den 01.03.2004 schrieb Chris Mason um 15:01:
> > > > > > It seems you introduced a bug here. I installed the patches
> > > > > > yesterday and found a lockup on my notebook when running lilo
> > > > > > (with /boot on the root reiserfs filesystem).
> > > > > >
> > > > > > A SysRq-T showed that lilo is stuck in fsync:
> > > > >
> > > > > Ugh, I use grub so I haven't tried lilo.  Could you please send me
> > > > > the full sysrq-t, this is probably something stupid.
> > > >
> > > > Yes. I could reproduce it by simply creating a dummy /boot volume on
> > > > reiserfs. I copied the content of /boot, ran lilo and it hung again.
> > > > The other reiserfs filesystems were still usable (but a global sync
> > > > hangs afterwards). I also attached a bzipped strace of the lilo
> > > > process.
> > >
> > > Ok, thanks.  The problem is in reiserfs_unpack(), which needs updating
> > > for the patch.  Fixing.
> >
> > Progress?
> >
> > I'll try on SuSE 2.6.3-16.
>
> Not fixed there yet.  Slight delay yesterday since I'm trying to roll
> out the fix and some quota patches.  Should be ready this afternoon.

OK, I start with the older...

-Dieter

Re: [PATCH] updated data=ordered patch for 2.6.3

[Dieter Nützel]

Am Dienstag, 2. März 2004 19:22 schrieb Dieter Nützel:
> Am Dienstag, 2. März 2004 19:16 schrieb Chris Mason:
> > On Tue, 2004-03-02 at 13:14, Dieter Nützel wrote:
> > > Am Montag, 1. März 2004 15:38 schrieb Chris Mason:
> > > > On Mon, 2004-03-01 at 09:30, Christophe Saout wrote:
> > > > > Am Mo, den 01.03.2004 schrieb Chris Mason um 15:01:
> > > > > > > It seems you introduced a bug here. I installed the patches
> > > > > > > yesterday and found a lockup on my notebook when running lilo
> > > > > > > (with /boot on the root reiserfs filesystem).
> > > > > > >
> > > > > > > A SysRq-T showed that lilo is stuck in fsync:
> > > > > >
> > > > > > Ugh, I use grub so I haven't tried lilo.  Could you please send
> > > > > > me the full sysrq-t, this is probably something stupid.
> > > > >
> > > > > Yes. I could reproduce it by simply creating a dummy /boot volume
> > > > > on reiserfs. I copied the content of /boot, ran lilo and it hung
> > > > > again. The other reiserfs filesystems were still usable (but a
> > > > > global sync hangs afterwards). I also attached a bzipped strace of
> > > > > the lilo process.
> > > >
> > > > Ok, thanks.  The problem is in reiserfs_unpack(), which needs
> > > > updating for the patch.  Fixing.
> > >
> > > Progress?
> > >
> > > I'll try on SuSE 2.6.3-16.
> >
> > Not fixed there yet.  Slight delay yesterday since I'm trying to roll
> > out the fix and some quota patches.  Should be ready this afternoon.
>
> OK, I start with the older...

Ups, data-logging IS already in SuSE's 2.6.3-16 kernel ;-)

What's needed? --- The reiserfs_unpack() fix and your speed up patch?

[-]
> Also, the code has some extra performance tweaks to smooth out
> performance both with and without data=ordered.  There are new
> mechanisms to trigger metadata/commit block writeback and to help
> throttle writers.  The goal is to reduce the huge bursts of io during a
> commit and during data=ordered writeback.
[-]

00x-xxx?

Thanks,
	Dieter

Re: [PATCH] updated data=ordered patch for 2.6.3

[Chris Mason]

On Tue, 2004-03-02 at 14:53, Dieter Nützel wrote:

> Ups, data-logging IS already in SuSE's 2.6.3-16 kernel ;-)
> 
Yes ;-)

> What's needed? --- The reiserfs_unpack() fix and your speed up patch?
> 
suse only needs the reiserfs_unpack() fix.

-chris

Re: [PATCH] updated data=ordered patch for 2.6.3

[Dieter Nützel]

Am Dienstag, 2. März 2004 21:06 schrieb Chris Mason:
> On Tue, 2004-03-02 at 14:53, Dieter Nützel wrote:
> > Ups, data-logging IS already in SuSE's 2.6.3-16 kernel ;-)
>
> Yes ;-)

Brave men.

Some numbers:
time dd if=/dev/zero of=sparse bs=1 seek=200G count=1
1+0 Records ein
1+0 Records aus
0.001u 2.592s 0:04.39 58.9%     0+0k 0+0io 0pf+0w
time rm sparse
0.000u 4.516s 0:04.52 99.7%     0+0k 0+0io 0pf+0w

Mouse movement and X server (screen update) works but is very slow (choppy) 
for the "know" four seconds.
But much better than before.

> > What's needed? --- The reiserfs_unpack() fix and your speed up patch?
>
> suse only needs the reiserfs_unpack() fix.

But it works like a charm without it. ;-)

Now, let's see to get it in...

Thank you!

-Dieter

Re: [PATCH] updated data=ordered patch for 2.6.3

[Dieter Nützel]

Am Dienstag, 2. März 2004 22:06 schrieb Dieter Nützel:
> Am Dienstag, 2. März 2004 21:06 schrieb Chris Mason:
> > On Tue, 2004-03-02 at 14:53, Dieter Nützel wrote:
> > > Ups, data-logging IS already in SuSE's 2.6.3-16 kernel ;-)
> >
> > Yes ;-)
>
> Brave men.
>
> Some numbers:
> time dd if=/dev/zero of=sparse bs=1 seek=200G count=1
> 1+0 Records ein
> 1+0 Records aus
> 0.001u 2.592s 0:04.39 58.9%     0+0k 0+0io 0pf+0w
> time rm sparse
> 0.000u 4.516s 0:04.52 99.7%     0+0k 0+0io 0pf+0w
>
> Mouse movement and X server (screen update) works but is very slow (choppy)
> for the "know" four seconds.
> But much better than before.
>
> > > What's needed? --- The reiserfs_unpack() fix and your speed up patch?
> >
> > suse only needs the reiserfs_unpack() fix.
>
> But it works like a charm without it. ;-)

Argh, ... partially... 8-(

lilo... do NOT work with the "new" (2.6.3-16) kernel as we know...
But I could boot with my old one and could create the new initrd.
Let me know if you have something to test.

-Dieter

Re: [PATCH] updated data=ordered patch for 2.6.3

[Chris Mason]

On Tue, 2004-03-02 at 16:06, Dieter Nützel wrote:
> Am Dienstag, 2. März 2004 21:06 schrieb Chris Mason:
> > On Tue, 2004-03-02 at 14:53, Dieter Nützel wrote:
> > > Ups, data-logging IS already in SuSE's 2.6.3-16 kernel ;-)
> >
> > Yes ;-)
> 
> Brave men.
> 
> Some numbers:
> time dd if=/dev/zero of=sparse bs=1 seek=200G count=1
> 1+0 Records ein
> 1+0 Records aus
> 0.001u 2.592s 0:04.39 58.9%     0+0k 0+0io 0pf+0w
> time rm sparse
> 0.000u 4.516s 0:04.52 99.7%     0+0k 0+0io 0pf+0w
> 
> Mouse movement and X server (screen update) works but is very slow (choppy) 
> for the "know" four seconds.
> But much better than before.
> 
Oh, part of the delete code was missed in the latency fixes.  I'll add a
few more schedules.

Sorry for the delay, I'm chasing a strange bug, not sure if it is mine
yet.  

-chris

Re: [PATCH] updated data=ordered patch for 2.6.3

[Marc-Christian Petersen]

On Tuesday 02 March 2004 20:53, Dieter Nützel wrote:

Hi Dieter,

> > > > I'll try on SuSE 2.6.3-16.
sorry for my ignorance, but where do you find 2.6.3-_16_? I only find -0

ciao, Marc

Re: [PATCH] updated data=ordered patch for 2.6.3

[Dieter Nützel]

Am Mittwoch, 3. März 2004 10:13 schrieb Marc-Christian Petersen:
> On Tuesday 02 March 2004 20:53, Dieter Nützel wrote:
>
> Hi Dieter,
>
> > > > > I'll try on SuSE 2.6.3-16.
>
> sorry for my ignorance, but where do you find 2.6.3-_16_? I only find -0

ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/suse/people/kraxel

Or the "original site" and its mirrors.

Regards,
	Dieter