merge 2.4.21-pre4 with Direct IO bug fix

merge 2.4.21-pre4 with Direct IO bug fix

[Philippe Gramoullé]

Hi Oleg,

Could you please rediff the patch for the DIRECT IO bug fix (pre5) against 2.4.21-pre4 + 
data-logging patches ? ( I've tried to merge it manually, but there are too many non trivial rejects
for me to dare feeling comfortable :o)

I'd like to give these new quota patches a try as since our kernels date back from 2.4.19-pre6
(with quota), the journal overflow bug keep hitting us more and more every day.

Thanks much,

Philippe

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Oleg Drokin]

Hello!

On Tue, Mar 04, 2003 at 01:34:50AM +0100, Philippe Gramoull? wrote:

> Could you please rediff the patch for the DIRECT IO bug fix (pre5) against 2.4.21-pre4 + 
> data-logging patches ? ( I've tried to merge it manually, but there are too many non trivial rejects
> for me to dare feeling comfortable :o)

Is it possible for you to use 2.4.21-pre5 instead of 2.4.21-pre4?
Patches from ftp://ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21
should apply just fine, except that you need to apply my version of 05-data-logging... patch.
You can get it from ftp://namesys.com/pub/reiserfs-for-2.4/testing/05-2.4.21-pre5-data-logging-36.diff.gz

Also we have not merged that journal overflow patch (deleting highly-fragmented files) anywhere yet,
as you reported you had hangs with it and running quotacheck. Below is the patch you should apply on top of
all Chris' patches for this overflow fix.

Also another patch you probably want to apply is my iget5_locked bugfix for 2.4,
You can find it at ftp://namesys.com/pub/reiserfs-for-2.4/testing/iget5_locked_for_2.4.21-pre5-datalogging.diff.gz
it should be applied on top of datalogging/quota stuff I mentioned above.

Bye,
    Oleg

--- fs/reiserfs/stree.c.orig	Tue Mar  4 10:55:33 2003
+++ fs/reiserfs/stree.c	Tue Mar  4 10:57:30 2003
@@ -1125,6 +1125,19 @@
 		put_block_num(p_n_unfm_pointer, 0, 0);
 		journal_mark_dirty (th, p_s_sb, p_s_bh);
 		reiserfs_free_block(th, inode, tmp, 1);
+		/* In case of big fragmentation it is possible that each block
+		   freed will cause dirtying of one more bitmap and then we will
+		   quickly overflow our transaction space. This is a
+		   counter-measure against that scenario */
+		if (journal_transaction_should_end(th, th->t_blocks_allocated)) {
+		    int len = th->t_blocks_allocated;
+		    pathrelse(p_s_path) ;
+		    reiserfs_restart_transaction(th, len) ;
+		    reiserfs_update_inode_transaction(inode) ;
+		    need_research = 1;
+		    break;
+		}
+
 		if ( item_moved (&s_ih, p_s_path) )  {
 			need_research = 1;
 			break ;

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Marc-Christian Petersen]

On Tuesday 04 March 2003 09:18, Oleg Drokin wrote:

Hi Oleg,

> Also another patch you probably want to apply is my iget5_locked bugfix for
> 2.4, You can find it at
> ftp://namesys.com/pub/reiserfs-for-2.4/testing/iget5_locked_for_2.4.21-pre5
>-datalogging.diff.gz it should be applied on top of datalogging/quota stuff
> I mentioned above.
is it also possible to do a patch ontop of the XFS v1.2 tree? It seems XFS 
uses a similar approach.

ciao, Marc

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Oleg Drokin]

Hello!

On Tue, Mar 04, 2003 at 03:23:43PM +0100, Marc-Christian Petersen wrote:

> > Also another patch you probably want to apply is my iget5_locked bugfix for
> > 2.4, You can find it at
> > ftp://namesys.com/pub/reiserfs-for-2.4/testing/iget5_locked_for_2.4.21-pre5
> >-datalogging.diff.gz it should be applied on top of datalogging/quota stuff
> > I mentioned above.
> is it also possible to do a patch ontop of the XFS v1.2 tree? It seems XFS 
> uses a similar approach.

This patch's idea came from XFS. Then it was implemented for 2.5.
And then I just backported changes to 2.4
First we need to make this patch into 2.4, then XFS people would need to catch up
(in fact it should be pretty easy for them, I think).

Bye,
    Oleg

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Philippe Gramoullé]

Hi Oleg,

On Tue, 4 Mar 2003 11:18:07 +0300
Oleg Drokin <green@namesys.com> wrote:


  | Hello!
  | 
  | On Tue, Mar 04, 2003 at 01:34:50AM +0100, Philippe Gramoull? wrote:
  | 
  | > Could you please rediff the patch for the DIRECT IO bug fix (pre5) against 2.4.21-pre4 + 
  | > data-logging patches ? ( I've tried to merge it manually, but there are too many non trivial rejects
  | > for me to dare feeling comfortable :o)
  | 
  | Is it possible for you to use 2.4.21-pre5 instead of 2.4.21-pre4?
[snip]

Kernel is up and running with all patches applied for about 1h30, and so far it rocks ;)
while at the same time other non patched kernels still continue oops ( normal ,as nothing changed )

The other good news are : Load average is significantly lower, down from 4 or 6 to less than 1
on the filer that caused us most of the problems.

I didn't changed the journaling default (ordered) and FYI, mount options are as follow:
defaults,noatime,nodiratime,notail,grpquota

I don't want to say "hurra" too early but signs are really positive that the bugs that we hit
have been fixed and i can't congratulate you more are people were starting to get mad around me ;o)

I'll let it run for a little while now and upgrade all other servers so we'll be almost sure wether ot not
this problem has been fixed for good :o)

I'll also run a quotacheck early tomorrow at the most off hours and will send a report then.

Thanks again for all efforts that were put to solve this issue.

Philippe

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Ookhoi]

Oleg Drokin wrote (ao):
> Patches from
> ftp://ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 should
> apply just fine, except that you need to apply my version of
> 05-data-logging... patch.  You can get it from
> ftp://namesys.com/pub/reiserfs-for-2.4/testing/05-2.4.21-pre5-data-logging-36.diff.gz
> 
> Also we have not merged that journal overflow patch (deleting
> highly-fragmented files) anywhere yet, as you reported you had hangs
> with it and running quotacheck. Below is the patch you should apply on
> top of all Chris' patches for this overflow fix.
> 
> Also another patch you probably want to apply is my iget5_locked
> bugfix for 2.4, You can find it at
> ftp://namesys.com/pub/reiserfs-for-2.4/testing/iget5_locked_for_2.4.21-pre5-datalogging.diff.gz
> it should be applied on top of datalogging/quota stuff I mentioned
> above.
> 
> --- fs/reiserfs/stree.c.orig	Tue Mar  4 10:55:33 2003
> +++ fs/reiserfs/stree.c	Tue Mar  4 10:57:30 2003
> @@ -1125,6 +1125,19 @@
>  		put_block_num(p_n_unfm_pointer, 0, 0);
>  		journal_mark_dirty (th, p_s_sb, p_s_bh);
>  		reiserfs_free_block(th, inode, tmp, 1);
> +		/* In case of big fragmentation it is possible that each block
> +		   freed will cause dirtying of one more bitmap and then we will
> +		   quickly overflow our transaction space. This is a
> +		   counter-measure against that scenario */
> +		if (journal_transaction_should_end(th, th->t_blocks_allocated)) {
> +		    int len = th->t_blocks_allocated;
> +		    pathrelse(p_s_path) ;
> +		    reiserfs_restart_transaction(th, len) ;
> +		    reiserfs_update_inode_transaction(inode) ;
> +		    need_research = 1;
> +		    break;
> +		}
> +
>  		if ( item_moved (&s_ih, p_s_path) )  {
>  			need_research = 1;
>  			break ;

Wow. I now apply the following patches to linux-2.4.20:

patch-2.4.21-pre5
patch-2.4.21-pre5-ac2
03-relocation-6.diff
04-reiserfs-sync_fs-1.diff
05-data-logging-36-ac5.diff
06-logging-export.diff
08-reiserfs-quota-26.diff
kinoded-9.diff
10-reiserfs-quota-link-fix.diff
iget5_locked_for_2.4.21-pre5-datalogging.diff
oleg.patch

And they all just apply cleanly. They must like each other very much :-)

For some reason, I do not have quota support though.

# cat /etc/fstab
# /etc/fstab: static file system information.
#
#<partition> <mountpoint>  <type>    <options>                      <dump><pass>
/dev/hda2  /               reiserfs  noatime,usrquota,grpquota          0  0
/dev/hda3  /www            reiserfs  noatime,usrquota,grpquota          0  0
/dev/hda5  /tmp            reiserfs  noatime,usrquota,grpquota          0  0
/dev/hda6  /var            reiserfs  noatime,usrquota,grpquota          0  0

$ mount
rootfs on / type rootfs (rw)
/dev/root on / type reiserfs (rw,noatime)
/dev/hda3 on /www type reiserfs (rw,noatime)
/dev/hda5 on /tmp type reiserfs (rw,noatime)
/dev/hda6 on /var type reiserfs (rw,noatime)

$ dmesg | grep -i quota
VFS: Disk quotas vdquot_6.5.1

On a different system, where quota did not work because there 
was no support for reiserfs, I got this:
~# dmesg | grep -i quota
VFS: Disk quotas vdquot_6.5.1
reiserfs_getopt: unknown option "usrquota"

Because this "unknown option" line is missing on the system, 
I assume the kernel does think there is quota support for reiserfs.

$ grep -i quota config.progress.10-mar-2003
CONFIG_QUOTA=y

Any quick ideas?

Thanks in advance!

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Oleg Drokin]

Hello!

On Mon, Mar 10, 2003 at 01:15:28PM +0100, Ookhoi wrote:

> Wow. I now apply the following patches to linux-2.4.20:
> patch-2.4.21-pre5
> patch-2.4.21-pre5-ac2
> 03-relocation-6.diff
> 04-reiserfs-sync_fs-1.diff
> 05-data-logging-36-ac5.diff
> 06-logging-export.diff
> 08-reiserfs-quota-26.diff
> kinoded-9.diff
> 10-reiserfs-quota-link-fix.diff
> iget5_locked_for_2.4.21-pre5-datalogging.diff
> oleg.patch
> And they all just apply cleanly. They must like each other very much :-)
> For some reason, I do not have quota support though.

How do you know you do not have quota support?

> # cat /etc/fstab
> # /etc/fstab: static file system information.
> #<partition> <mountpoint>  <type>    <options>                      <dump><pass>
> /dev/hda2  /               reiserfs  noatime,usrquota,grpquota          0  0
> /dev/hda3  /www            reiserfs  noatime,usrquota,grpquota          0  0
> /dev/hda5  /tmp            reiserfs  noatime,usrquota,grpquota          0  0
> /dev/hda6  /var            reiserfs  noatime,usrquota,grpquota          0  0
> $ mount
> rootfs on / type rootfs (rw)
> /dev/root on / type reiserfs (rw,noatime)
> /dev/hda3 on /www type reiserfs (rw,noatime)
> /dev/hda5 on /tmp type reiserfs (rw,noatime)
> /dev/hda6 on /var type reiserfs (rw,noatime)
> $ dmesg | grep -i quota
> VFS: Disk quotas vdquot_6.5.1
> Any quick ideas?

Have you tried to actually enable quotas (with quotaon) and see if it works?

Bye,
    Oleg

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Ookhoi]

Oleg Drokin wrote (ao):
> On Mon, Mar 10, 2003 at 01:15:28PM +0100, Ookhoi wrote:
> > Wow. I now apply the following patches to linux-2.4.20:
> > patch-2.4.21-pre5
> > patch-2.4.21-pre5-ac2
> > 03-relocation-6.diff
> > 04-reiserfs-sync_fs-1.diff
> > 05-data-logging-36-ac5.diff
> > 06-logging-export.diff
> > 08-reiserfs-quota-26.diff
> > kinoded-9.diff
> > 10-reiserfs-quota-link-fix.diff
> > iget5_locked_for_2.4.21-pre5-datalogging.diff
> > oleg.patch
> > And they all just apply cleanly. They must like each other very much :-)
> > For some reason, I do not have quota support though.
> 
> How do you know you do not have quota support?

o  The options are not listed in /proc/mounts
   IIRC they should before quota is switched on (can't check that right
                                                 now though)
o  quotacheck gives this: 
   # quotacheck /dev/hda3
   quotacheck: Can't stat() mounted device /dev/root: No such file or directory
o  quotaon gives this:
   # quotaon -ugv /www
   quotaon: Can't stat() mounted device /dev/root: No such file or directory

I don't have /dev/root on any of my systems, so I don't understand the
error.

I tried both debian quota tools, and fresh ones from sourceforge:
# quota -V
Quota utilities version 3.08.
Compiled with RPC
Bugs to mvw@planets.elm.net, jack@suse.cz


> > # cat /etc/fstab
> > # /etc/fstab: static file system information.
> > #<partition> <mountpoint>  <type>    <options>                      <dump><pass>
> > /dev/hda2  /               reiserfs  noatime,usrquota,grpquota          0  0
> > /dev/hda3  /www            reiserfs  noatime,usrquota,grpquota          0  0
> > /dev/hda5  /tmp            reiserfs  noatime,usrquota,grpquota          0  0
> > /dev/hda6  /var            reiserfs  noatime,usrquota,grpquota          0  0
> > $ mount
> > rootfs on / type rootfs (rw)
> > /dev/root on / type reiserfs (rw,noatime)
> > /dev/hda3 on /www type reiserfs (rw,noatime)
> > /dev/hda5 on /tmp type reiserfs (rw,noatime)
> > /dev/hda6 on /var type reiserfs (rw,noatime)
> > $ dmesg | grep -i quota
> > VFS: Disk quotas vdquot_6.5.1
> > Any quick ideas?
> 
> Have you tried to actually enable quotas (with quotaon) and see if it works?

Yes :-)

On a system where quota does not work either (due to missing patches:
2.4.21-pre4-ac4):
# mount
/dev/md1 on / type reiserfs (rw,noatime,usrquota,grpquota)

On the system I compiled the last kernel for (2.4.21-pre5-ac2):
# mount
rootfs on / type rootfs (rw)
/dev/root on / type reiserfs (rw,noatime)

For some reason, the mount options are not listed.

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Oleg Drokin]

Hello!

On Mon, Mar 10, 2003 at 02:16:05PM +0100, Ookhoi wrote:

> > > And they all just apply cleanly. They must like each other very much :-)
> > > For some reason, I do not have quota support though.
> > How do you know you do not have quota support?
> o  The options are not listed in /proc/mounts
>    IIRC they should before quota is switched on (can't check that right

From the quotaon man page, it seems that these usrquota stuff are checked by
quota tools in /etc/fstab.

> o  quotacheck gives this: 
>    # quotacheck /dev/hda3
>    quotacheck: Can't stat() mounted device /dev/root: No such file or directory
> o  quotaon gives this:
>    # quotaon -ugv /www
>    quotaon: Can't stat() mounted device /dev/root: No such file or directory

Hm, I always thought that quota tools work with /etc/mtab,
don't you have /etc/mtab to be a symlink to /proc/mounts?
This might confuse it.

> > Have you tried to actually enable quotas (with quotaon) and see if it works?
> Yes :-)
> On a system where quota does not work either (due to missing patches:
> 2.4.21-pre4-ac4):
> # mount
> /dev/md1 on / type reiserfs (rw,noatime,usrquota,grpquota)

This output looks correct.

> On the system I compiled the last kernel for (2.4.21-pre5-ac2):
> # mount
> rootfs on / type rootfs (rw)
> /dev/root on / type reiserfs (rw,noatime)

This one suggests you have something strange with /etc/mtab.
Here's what I usually see (no quota enabled this time)
> mount
/dev/hda2 on / type reiserfs (rw)
...

> For some reason, the mount options are not listed.

This is because reiserfs is lacking necessary callback to print it's current
mount options in /proc/mounts, I think.

Bye,
    Oleg

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Ookhoi]

Oleg Drokin wrote (ao):
> On Mon, Mar 10, 2003 at 02:16:05PM +0100, Ookhoi wrote:
> > > > And they all just apply cleanly. They must like each other very much :-)
> > > > For some reason, I do not have quota support though.
> > > How do you know you do not have quota support?
> > o  The options are not listed in /proc/mounts
> >    IIRC they should before quota is switched on (can't check that right
> 
> >From the quotaon man page, it seems that these usrquota stuff are checked by
> quota tools in /etc/fstab.
> 
> > o  quotacheck gives this: 
> >    # quotacheck /dev/hda3
> >    quotacheck: Can't stat() mounted device /dev/root: No such file or directory
> > o  quotaon gives this:
> >    # quotaon -ugv /www
> >    quotaon: Can't stat() mounted device /dev/root: No such file or directory
> 
> Hm, I always thought that quota tools work with /etc/mtab,
> don't you have /etc/mtab to be a symlink to /proc/mounts?
> This might confuse it.

Damned. You are right ..
After rm /etc/mtab and reboot:
/dev/hda2 on / type reiserfs (rw,noatime,usrquota,grpquota)
/dev/hda3 on /www type reiserfs (rw,noatime,usrquota,grpquota)
/dev/hda5 on /tmp type reiserfs (rw,noatime,usrquota,grpquota)
/dev/hda6 on /var type reiserfs (rw,noatime,usrquota,grpquota)

I owe you a beer Oleg!

> > For some reason, the mount options are not listed.
> 
> This is because reiserfs is lacking necessary callback to print it's
> current mount options in /proc/mounts, I think.

That would fix setups which have a link to /proc/mounts most likely?

Anyway, let me know when you come to amsterdam, ok? ;-)

Btw, quota works (just tested it).

Thanks!!

Re: merge 2.4.21-pre4 with Direct IO bug fix

[Oleg Drokin]

Hello!

On Mon, Mar 10, 2003 at 03:23:53PM +0100, Ookhoi wrote:
> > > For some reason, the mount options are not listed.
> > This is because reiserfs is lacking necessary callback to print it's
> > current mount options in /proc/mounts, I think.
> That would fix setups which have a link to /proc/mounts most likely?

Not likely.
quota tools seems to be confused by non-existent /dev/root reported as mounted.
According to man page, quota tools look for "usr/grpquota" in /etc/fstab.

> Anyway, let me know when you come to amsterdam, ok? ;-)

This is not likely to happen any time soon unfortunatelly.

> Btw, quota works (just tested it).

That's good. I do not need to fix it then ;)

Bye,
    Oleg