Re: 2.4.20 at kernel.org and data logging

Re: 2.4.20 at kernel.org and data logging

[Sewell, Cassandra D (Cassandra)]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

Is the data logging patch for 2.4.20 kernel considered 
complete?  I see a patch at ftp.suse.com/pub/people/mason/patches/data-logging/2.4.20.  Also will this patch be considered working for the last stable 2.4.20 posted at the kernel.org location?  

I am in the process of upgrading some machines from 2.2.x (I know way overdue), and need the data-logging capability.  


Thanks, 

C. Sewell

[-- Attachment #2: Type: text/html, Size: 1069 bytes --]

Re: 2.4.20 at kernel.org and data logging

[Manuel Krause]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

On 01/17/2003 12:51 AM, Sewell, Cassandra D (Cassandra) wrote:
> Is the data logging patch for 2.4.20 kernel considered
> complete?  I see a patch at 
> ftp.suse.com/pub/people/mason/patches/data-logging/2.4.20.  Also will 
> this patch be considered working for the last stable 2.4.20 posted at 
> the kernel.org location? 
> 
> I am in the process of upgrading some machines from 2.2.x (I know way 
> overdue), and need the data-logging capability. 
> 
> 
> Thanks,
> 
> C. Sewell
> 

Hi Cassandra,

just apply these patches in the row they're numbered and mount the 
related partitions like you want, -o journal _or_ -o ordered (-o 
writeback represents the non data-logging standard/default so far).

If you want to mount a root fs you still need to pass 
rootflags=data=ordered (or =journal) to the kernel. Until the testing on 
2.4.21-preX is finished if I understood Chris correctly.

Chris posted a patch to the list as solution for an overflow bug hitting 
2.4.20-final and its -pres -- I'll attach it at the end so you needn't 
dig too far.

For my home system these patches have been stable since posting of that 
set and they're working with the related preempt-kernel-patch as well.


Currently I'm using the 2.4.21 version of these patches that have more 
improvements. Please, refer to Chris Masons original regarding mail to 
the list "[PATCH] data logging patches available for 2.4.21-preX".
BTW, I'm not getting system freezes like Anders Widman posted recently.

So to say, the patches are not really complete, but they seem very 
close... ;-))


Best regards,

Manuel


[-- Attachment #2: patch-2.4.20.rfs.06.05-transaction-overflow-fix-0.diff --]
[-- Type: text/plain, Size: 4816 bytes --]

--- 1.40/fs/reiserfs/inode.c	Wed Nov 13 17:47:56 2002
+++ linux/fs/reiserfs/inode.c	Wed Nov 13 19:20:39 2002
@@ -205,14 +205,15 @@
 }
 
 static void restart_transaction(struct reiserfs_transaction_handle *th,
-				struct inode *inode, struct path *path) {
+				struct inode *inode, struct path *path,
+				int jbegin_count) {
   /* we cannot restart while nested unless the parent allows it */
   if (!reiserfs_restartable_handle(th) && th->t_refcount > 1) {
       return  ;
   }
   pathrelse(path) ;
   reiserfs_update_sd(th, inode) ;
-  reiserfs_restart_transaction(th) ;
+  reiserfs_restart_transaction(th, jbegin_count) ;
   reiserfs_update_inode_transaction(inode) ;
 }
 
@@ -652,7 +653,7 @@
 	    ** some blocks.  releases the path, so we have to go back to
 	    ** research if we succeed on the second try
 	    */
-	    restart_transaction(th, inode, &path) ; 
+	    restart_transaction(th, inode, &path, jbegin_count) ; 
 	    repeat = _allocate_block(th, block, inode, &allocated_block_nr, NULL, create);
 
 	    if (repeat != NO_DISK_SPACE) {
@@ -882,8 +883,8 @@
 	** this only happens when inserting holes into the file, so it
 	** does not affect data=ordered safety at all
 	*/
-	if (journal_transaction_should_end(th, th->t_blocks_allocated)) {
-	    restart_transaction(th, inode, &path) ; 
+	if (journal_transaction_should_end(th, jbegin_count)) {
+	    restart_transaction(th, inode, &path, jbegin_count) ; 
 	}
 	/* inserting indirect pointers for a hole can take a 
 	** long time.  reschedule if needed
--- 1.25/fs/reiserfs/journal.c	Wed Nov 13 17:47:56 2002
+++ linux/fs/reiserfs/journal.c	Wed Nov 13 21:23:47 2002
@@ -2532,8 +2532,6 @@
 */
 int journal_transaction_should_end(struct reiserfs_transaction_handle *th, int new_alloc) {
   time_t now = CURRENT_TIME ;
-  if (reiserfs_dont_log(th->t_super)) 
-    return 0 ;
 
   /* cannot restart while nested unless the parent allows it */
   if (!reiserfs_restartable_handle(th) && th->t_refcount > 1)
@@ -2545,13 +2543,20 @@
        SB_JOURNAL(th->t_super)->j_cnode_free < (SB_JOURNAL_TRANS_MAX(th->t_super) * 3)) { 
     return 1 ;
   }
+
+  /* we are allowing them to continue in the current transaction, so
+  * we have to bump the blocks allocated now.
+  */
+  th->t_blocks_allocated += new_alloc;
+  SB_JOURNAL(th->t_super)->j_len_alloc += new_alloc;
+
   return 0 ;
 }
 
-int reiserfs_restart_transaction(struct reiserfs_transaction_handle *th) {
+int 
+reiserfs_restart_transaction(struct reiserfs_transaction_handle *th, int num) {
     int refcount = th->t_refcount ;
     struct super_block *s = th->t_super ;
-    int num = th->t_blocks_allocated ;
     int flags = th->t_flags ;
     int parent_flags = 0;
     struct reiserfs_transaction_handle *saved_th = current->journal_info ;
@@ -2568,7 +2573,7 @@
 	parent_flags = saved_th->t_flags ;
     }
     th->t_flags = 0 ;
-    journal_end(th, s, num) ;
+    journal_end(th, s, th->t_blocks_allocated) ;
     journal_begin(th, s, num) ;
     th->t_flags = flags; 
     if (refcount > 1) {
--- 1.21/fs/reiserfs/stree.c	Wed Nov 13 17:47:56 2002
+++ linux/fs/reiserfs/stree.c	Wed Nov 13 19:29:01 2002
@@ -1705,6 +1705,7 @@
 	n_new_file_size;/* New file size. */
     int                   n_deleted;      /* Number of deleted or truncated bytes. */
     int retval;
+    int jbegin_count = th->t_blocks_allocated;
 
     if ( ! (S_ISREG(p_s_inode->i_mode) || S_ISDIR(p_s_inode->i_mode) || S_ISLNK(p_s_inode->i_mode)) )
 	return;
@@ -1784,14 +1785,14 @@
 	** sure the file is consistent before ending the current trans
 	** and starting a new one
 	*/
-        if (journal_transaction_should_end(th, th->t_blocks_allocated)) {
+        if (journal_transaction_should_end(th, jbegin_count)) {
 	  decrement_counters_in_path(&s_search_path) ;
 
 	  if (update_timestamps) {
 	      p_s_inode->i_mtime = p_s_inode->i_ctime = CURRENT_TIME;
 	  } 
 	  reiserfs_update_sd(th, p_s_inode) ;
-	  reiserfs_restart_transaction(th) ;
+	  reiserfs_restart_transaction(th, jbegin_count) ;
 	  reiserfs_update_inode_transaction(p_s_inode) ;
 	}
     } while ( n_file_size > ROUND_UP (n_new_file_size) &&
--- 1.26/include/linux/reiserfs_fs.h	Wed Nov 13 17:47:56 2002
+++ linux/include/linux/reiserfs_fs.h	Wed Nov 13 19:29:32 2002
@@ -1806,7 +1806,7 @@
 int push_journal_writer(char *w) ;
 int pop_journal_writer(int windex) ;
 int journal_transaction_should_end(struct reiserfs_transaction_handle *, int) ;
-int reiserfs_restart_transaction(struct reiserfs_transaction_handle *) ;
+int reiserfs_restart_transaction(struct reiserfs_transaction_handle *, int) ;
 int reiserfs_in_journal(struct super_block *p_s_sb, kdev_t dev, int bmap_nr, int bit_nr, int size, int searchall, unsigned int *next) ;
 int journal_begin(struct reiserfs_transaction_handle *, struct super_block *p_s_sb, unsigned long) ;
 

Re: 2.4.20 at kernel.org and data logging

[Russell Coker]

On Fri, 17 Jan 2003 00:51, Sewell, Cassandra D (Cassandra) wrote:
> Is the data logging patch for 2.4.20 kernel considered
> complete?  I see a patch at
> ftp.suse.com/pub/people/mason/patches/data-logging/2.4.20.  Also will this
> patch be considered working for the last stable 2.4.20 posted at the
> kernel.org location?
>
> I am in the process of upgrading some machines from 2.2.x (I know way
> overdue), and need the data-logging capability.

I am using those patches on some 2.4.20 machines running as Qmail mail 
servers.  I have the /var file system mounted with data journalling.

I have 4 machines running each with >250,000 mail boxes with continual heavy 
IO load (the load never goes below 100 disk writes per second).

I found the benefit to be much smaller than I had hoped.  The patches in 
question have a number of other performance improvements which made a huge 
and noticable difference.  data=journal seemed to make a small difference, 
but nothing exciting.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: 2.4.20 at kernel.org and data logging

[Chris Mason]

On Sat, 2003-01-18 at 05:25, Russell Coker wrote:
> On Fri, 17 Jan 2003 00:51, Sewell, Cassandra D (Cassandra) wrote:
> > Is the data logging patch for 2.4.20 kernel considered
> > complete?  I see a patch at
> > ftp.suse.com/pub/people/mason/patches/data-logging/2.4.20.  Also will this
> > patch be considered working for the last stable 2.4.20 posted at the
> > kernel.org location?
> >
> > I am in the process of upgrading some machines from 2.2.x (I know way
> > overdue), and need the data-logging capability.
> 
> I am using those patches on some 2.4.20 machines running as Qmail mail 
> servers.  I have the /var file system mounted with data journalling.
> 
> I have 4 machines running each with >250,000 mail boxes with continual heavy 
> IO load (the load never goes below 100 disk writes per second).
> 
> I found the benefit to be much smaller than I had hoped.  The patches in 
> question have a number of other performance improvements which made a huge 
> and noticable difference.  data=journal seemed to make a small difference, 
> but nothing exciting.

The benefit from data=journal varies largely by the size of the files in
use.  But Russell, you might be hitting other bottlenecks having to do
with reading the directory items off disk (that many directories means
lots of inode).  

Also, we should probably try switching you to a highmem io enabled
kernel, I'm almost done with a patch to put the 2.4.21 data logging code
in the -aa kernel.

-chris

Re: 2.4.20 at kernel.org and data logging

[Russell Coker]

On Sat, 18 Jan 2003 16:57, Chris Mason wrote:
> > I found the benefit to be much smaller than I had hoped.  The patches in
> > question have a number of other performance improvements which made a
> > huge and noticable difference.  data=journal seemed to make a small
> > difference, but nothing exciting.
>
> The benefit from data=journal varies largely by the size of the files in
> use.  But Russell, you might be hitting other bottlenecks having to do
> with reading the directory items off disk (that many directories means
> lots of inode).

The kswapd issue is part of it.  I am considering the aa kernels...

> Also, we should probably try switching you to a highmem io enabled
> kernel, I'm almost done with a patch to put the 2.4.21 data logging code
> in the -aa kernel.

I have the following enabled already, is this what you refer to or is it 
something else?

HIGHMEM I/O support
CONFIG_HIGHIO
  If you want to be able to do I/O to high memory pages, say Y.
  Otherwise low memory pages are used as bounce buffers causing a
  degrade in performance.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

slightly [OT] highmem (was Re: 2.4.20 at kernel.org and data logging)

[Manuel Krause]

On 01/18/2003 04:57 PM, Chris Mason wrote:
> On Sat, 2003-01-18 at 05:25, Russell Coker wrote:
> 
>>On Fri, 17 Jan 2003 00:51, Sewell, Cassandra D (Cassandra) wrote:
>>
>>>Is the data logging patch for 2.4.20 kernel considered
>>>complete?  I see a patch at
>>>ftp.suse.com/pub/people/mason/patches/data-logging/2.4.20.  Also will this
>>>patch be considered working for the last stable 2.4.20 posted at the
>>>kernel.org location?
>>>
>>>I am in the process of upgrading some machines from 2.2.x (I know way
>>>overdue), and need the data-logging capability.
>>
>>I am using those patches on some 2.4.20 machines running as Qmail mail 
>>servers.  I have the /var file system mounted with data journalling.
>>
>>I have 4 machines running each with >250,000 mail boxes with continual heavy 
>>IO load (the load never goes below 100 disk writes per second).
>>
>>I found the benefit to be much smaller than I had hoped.  The patches in 
>>question have a number of other performance improvements which made a huge 
>>and noticable difference.  data=journal seemed to make a small difference, 
>>but nothing exciting.
> 
> 
> The benefit from data=journal varies largely by the size of the files in
> use.  But Russell, you might be hitting other bottlenecks having to do
> with reading the directory items off disk (that many directories means
> lots of inode).  
> 
> Also, we should probably try switching you to a highmem io enabled
> kernel, I'm almost done with a patch to put the 2.4.21 data logging code
> in the -aa kernel.
> 

Some OT(?) things to ask:

* Does enabled highmem support make any sense for mem<1G ?
* I especially wondered about the advantages of processors'
   "pae" feature in this context.
* VMware WS (I depend on) unfortunately doesn't support
   systems with enabled "pae" :-((

For the moment I only found out that highmem 64G didn't bring so to say 
"subjectively" measurable advantages on here (mem=512M) and -- VMware 
wouldn't run...

Thanks in advance,

Manuel

Re: slightly [OT] highmem (was Re: 2.4.20 at kernel.org and data logging)

[Russell Coker]

On Fri, 24 Jan 2003 00:10, Manuel Krause wrote:
> * Does enabled highmem support make any sense for mem<1G ?

You need highmem if you want more than 980M.  People usually refer to 1G, and 
if you don't mind wasting 20M then that's near enough.

Which of the overhead of highmem or the loss of 20M of RAM out of 1G is 
something that could be debated.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: slightly [OT] highmem (was Re: 2.4.20 at kernel.org and data logging)

[Manuel Krause]

On 01/24/2003 12:40 AM, Russell Coker wrote:
> On Fri, 24 Jan 2003 00:10, Manuel Krause wrote:
> 
>>* Does enabled highmem support make any sense for mem<1G ?
> 
> You need highmem if you want more than 980M.  People usually refer to 1G, and 
> if you don't mind wasting 20M then that's near enough.

Mmh. So I wasn't really clear with my questions?

Does it give advantages for 512M systems like mine if I enabled 
higmem4GB / highmem64GB with "pae" or does it produce more overhead that 
you mention below?

Thank you,

Manuel

> Which of the overhead of highmem or the loss of 20M of RAM out of 1G is 
> something that could be debated.

Re: slightly [OT] highmem (was Re: 2.4.20 at kernel.org and data logging)

[Russell Coker]

On Fri, 24 Jan 2003 01:18, Manuel Krause wrote:
> On 01/24/2003 12:40 AM, Russell Coker wrote:
> > On Fri, 24 Jan 2003 00:10, Manuel Krause wrote:
> >>* Does enabled highmem support make any sense for mem<1G ?
> >
> > You need highmem if you want more than 980M.  People usually refer to 1G,
> > and if you don't mind wasting 20M then that's near enough.
>
> Mmh. So I wasn't really clear with my questions?
>
> Does it give advantages for 512M systems like mine if I enabled
> higmem4GB / highmem64GB with "pae" or does it produce more overhead that
> you mention below?

The sole advantage of highmem is the ability to access more than 980M of RAM.  
If you don't have >980M of RAM and don't plan to get more then there's no 
benefit for you.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: slightly [OT] highmem (was Re: 2.4.20 at kernel.org and data logging)

[Oleg Drokin]

Hello!

On Fri, Jan 24, 2003 at 01:18:44AM +0100, Manuel Krause wrote:
> >You need highmem if you want more than 980M.  People usually refer to 1G, 
> >and if you don't mind wasting 20M then that's near enough.
> Mmh. So I wasn't really clear with my questions?
> Does it give advantages for 512M systems like mine if I enabled 
> higmem4GB / highmem64GB with "pae" or does it produce more overhead that 
> you mention below?

You get no advantage of course.
But lots of overhead. Rumours have it that 256M systems with highmem enabled
kernels (default for RedHat beta it seems) are swapping much more
then when the same kernel is built with highmem off.

Bye,
    Oleg

Re: slightly [OT] highmem (was Re: 2.4.20 at kernel.org and data logging)

[Dieter Nützel]

Am Freitag, 24. Januar 2003 08:50 schrieb Oleg Drokin:
> Hello!
>
> On Fri, Jan 24, 2003 at 01:18:44AM +0100, Manuel Krause wrote:
> > >You need highmem if you want more than 980M.  People usually refer to
> > > 1G, and if you don't mind wasting 20M then that's near enough.
> >
> > Mmh. So I wasn't really clear with my questions?
> > Does it give advantages for 512M systems like mine if I enabled
> > higmem4GB / highmem64GB with "pae" or does it produce more overhead that
> > you mention below?
>
> You get no advantage of course.
> But lots of overhead. Rumours have it that 256M systems with highmem
> enabled kernels (default for RedHat beta it seems) are swapping much more
> then when the same kernel is built with highmem off.

But that could be because they have forgotten to enabled HIGHMEM IO?
See Andrea Ancangeli's -aa kernels.

Greetings,
	Dieter

Re: slightly [OT] highmem (was Re: 2.4.20 at kernel.org and data logging)

[Oleg Drokin]

Hello!

On Fri, Jan 24, 2003 at 06:00:19PM +0100, Dieter N?tzel wrote:
> > > higmem4GB / highmem64GB with "pae" or does it produce more overhead that
> > > you mention below?
> > You get no advantage of course.
> > But lots of overhead. Rumours have it that 256M systems with highmem
> > enabled kernels (default for RedHat beta it seems) are swapping much more
> > then when the same kernel is built with highmem off.
> But that could be because they have forgotten to enabled HIGHMEM IO?
> See Andrea Ancangeli's -aa kernels.

What HIGHMEM IO? There is exactly NO highmem, so sighmem IO code won't be used.

Bye,
    Oleg

Re: slightly [OT] highmem (was Re: 2.4.20 at kernel.org and data logging)

[Dieter Nützel]

Am Freitag, 24. Januar 2003 18:03 schrieb Oleg Drokin:
> Hello!
>
> On Fri, Jan 24, 2003 at 06:00:19PM +0100, Dieter N?tzel wrote:
> > > > higmem4GB / highmem64GB with "pae" or does it produce more overhead
> > > > that you mention below?
> > >
> > > You get no advantage of course.
> > > But lots of overhead. Rumours have it that 256M systems with highmem
> > > enabled kernels (default for RedHat beta it seems) are swapping much
> > > more then when the same kernel is built with highmem off.
> >
> > But that could be because they have forgotten to enabled HIGHMEM IO?
> > See Andrea Ancangeli's -aa kernels.
>
> What HIGHMEM IO? There is exactly NO highmem, so sighmem IO code won't be
> used.

Yes, you are right. There isn't any physical highmem.
Shows how stupid the above idea is.

Manuel's system haven't have highmem, too.
So he should see the performance degradation, too.

Thanks,
	Dieter


2.4.20-aa1 for example:

CONFIG_HIGHIO:

If you want to be able to do I/O to high memory pages, say Y.
Otherwise low memory pages are used as bounce buffers causing a
degrade in performance.