1352 NUL bytes at the end of a page?

1352 NUL bytes at the end of a page?

[Andy Isaacson]

We've got a user who's reporting BK problems which we've traced down to
the fact that his s.ChangeSet file has a hole, filled with '\0' bytes,
that's so far always 1352 bytes long, and the end is page-aligned.  (In
fact, the two cases we've seen so far have been 8k-aligned.)  The
correct file data picks up again after the hole.

bk is writing the data using stdio in a child process (fork, exec,
wait), then mmaping the result.  The corruption is persistent; he sent
us the s.ChangeSet file and there it was (not a cache or buffering
problem, therefore).

2.6.6-bk (current head of tree from whenever he built), UP PIII, symptom
observed on both ext3 and reiserfs.  (However, we've explicitly verified
the hole only on reiser.)

The problem is intermittent, having happened "several" times over the
last few months, and doesn't appear to be tied to any particular kernel
version.

To me, this looks awfully close to an Ethernet frame size... but that's
just a wild guess.  And I don't think he's running Ethernet (still
waiting for dmesg and .config).

I've asked for more info, memtest86, and will attempt to reproduce it on
another box.

Does anyone have insight into this peculiar problem?

-andy

Re: 1352 NUL bytes at the end of a page?

[Andrew Morton]

Andy Isaacson <adi@bitmover.com> wrote:
>
>  We've got a user who's reporting BK problems which we've traced down to
>  the fact that his s.ChangeSet file has a hole, filled with '\0' bytes,
>  that's so far always 1352 bytes long, and the end is page-aligned.  (In
>  fact, the two cases we've seen so far have been 8k-aligned.)  The
>  correct file data picks up again after the hole.

When the reporter has a PIII machine it's often useful to find out the clock
frequency - the lower it is, the older it is and the more likely it is that
some component has rotted.

If this one cannot be reproduced on any other machine I'd say it's a
hardware failure.

Re: 1352 NUL bytes at the end of a page?

[Steven Cole]

Andrew Morton wrote:
>Andy Isaacson <adi@bitmover.com> wrote:
>>
>>  We've got a user who's reporting BK problems which we've traced down to
>>  the fact that his s.ChangeSet file has a hole, filled with '\0' bytes,
>>  that's so far always 1352 bytes long, and the end is page-aligned.  (In
>>  fact, the two cases we've seen so far have been 8k-aligned.)  The
>>  correct file data picks up again after the hole.
>
>When the reporter has a PIII machine it's often useful to find out the clock
>frequency - the lower it is, the older it is and the more likely it is that
>some component has rotted.
>
>If this one cannot be reproduced on any other machine I'd say it's a
>hardware failure.

Hi Andrew,

The user is me.  The machine is a 450 Mhz P-III, about five years old now.
Andy mentioned ethernet, but I don't have that here, just 56k dialup.  The
extra information he requested was sent a couple of hours ago, and in the
meantime I ran two full passes of memtest86 3.1 with zero errors.

<slight detour>
I do occasionally have problems with pppd, and the following message always
appears in /var/log/messages:

May 13 18:09:30 spc kernel: serial8250: too much work for irq10
May 13 18:09:30 spc kernel: serial8250: too much work for irq10

The message is always doubled as above.  This has never yet occurred
at the same time as the bk failure, so the two seem unrelated.  I have
to kill -9 the pppd process and reconnect when the above happens.
This problem never happened with a 2.4.x kernel, and was first detected
during the middle of 2.5.x development.
</slight detour>

The only reason the above was at all possibly relevant to the bk failtures,
is that I've only noticed the failures when pulling over the net via ppp.
I've never gotten the failure when pulling from another repository
on the same disk (I've only got one).

If you have any ideas about narrowing down the potentially rotted
component, please let me know.

I cut and pasted the above from a lkml archive, so sorry if this
messes up your mail thread.  I'm not on lkml here at home, so
please cc me on any replies.

Steven

Re: 1352 NUL bytes at the end of a page?

[Wayne Scott]

From: Steven Cole <elenstev@mesatop.com>
> [steven@spc BK]$ ls -ls foo/RESYNC/SCCS/*
> 40048 -r--r--r--  1 steven steven 41007273 May 14 18:08 foo/RESYNC/SCCS/s.ChangeSet
>    68 -r--r--r--  1 steven steven    67791 May 14 18:11 foo/RESYNC/SCCS/s.CREDITS
>    76 -r--r--r--  1 steven steven    75264 May 14 18:11 foo/RESYNC/SCCS/s.MAINTAINERS
>   124 -r--r--r--  1 steven steven   124747 May 14 18:11 foo/RESYNC/SCCS/s.Makefile
>   
> Let me know if you want any of these files.  I can compress them and send them
> the usual way.

scan the ChangeSet file for nulls.

-Wayne

Re: 1352 NUL bytes at the end of a page?

[Wayne Scott]

From: Andrew Morton <akpm@osdl.org>
> Well we can stop right there, because the only way someone can get some
> more non-zero user data into this page before we memset and write it is by
> locking the page beforehand, and block_write_full_page() has the page lock.
> (Or they can write stuff into it via mmap, but writing to the page outside
> i_size is an application bug).

BTW: BitKeeper never opens a writable mmap to a file.  The files are
read with mmap() and written by fwriting to a tmp file and then
renaming over the target.  And since we run on Windows, no process has
the file open when we are updating it.

Still catching up on this thread.

-Wayne

Re: 1352 NUL bytes at the end of a page?

[Theodore Ts'o]

On Mon, May 17, 2004 at 08:56:40AM -0500, Wayne Scott wrote:
> From: Andrew Morton <akpm@osdl.org>
> > Well we can stop right there, because the only way someone can get some
> > more non-zero user data into this page before we memset and write it is by
> > locking the page beforehand, and block_write_full_page() has the page lock.
> > (Or they can write stuff into it via mmap, but writing to the page outside
> > i_size is an application bug).
> 
> BTW: BitKeeper never opens a writable mmap to a file.  The files are
> read with mmap() and written by fwriting to a tmp file and then
> renaming over the target.  And since we run on Windows, no process has
> the file open when we are updating it.

Note though that the stdio library uses a writeable mmap to implement
fwrite.

						- Ted

Re: 1352 NUL bytes at the end of a page?

[Larry McVoy]

On Mon, May 17, 2004 at 11:17:38AM -0400, Theodore Ts'o wrote:
> On Mon, May 17, 2004 at 08:56:40AM -0500, Wayne Scott wrote:
> > From: Andrew Morton <akpm@osdl.org>
> > > Well we can stop right there, because the only way someone can get some
> > > more non-zero user data into this page before we memset and write it is by
> > > locking the page beforehand, and block_write_full_page() has the page lock.
> > > (Or they can write stuff into it via mmap, but writing to the page outside
> > > i_size is an application bug).
> > 
> > BTW: BitKeeper never opens a writable mmap to a file.  The files are
> > read with mmap() and written by fwriting to a tmp file and then
> > renaming over the target.  And since we run on Windows, no process has
> > the file open when we are updating it.
> 
> Note though that the stdio library uses a writeable mmap to implement
> fwrite.

That's news to me.  And we use fwrite.
-- 
---
Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com

Re: 1352 NUL bytes at the end of a page?

[Linus Torvalds]

On Mon, 17 May 2004, Theodore Ts'o wrote:
> 
> Note though that the stdio library uses a writeable mmap to implement
> fwrite.

It does? Whee. Then I'll have to agree with Andrew - if there is a path 
that is more likely to have bugs, it's trying to do writes with mmap and 
ftruncate.

Who came up with that braindead idea? Is it some crazed Mach developer 
that infiltrated the glibc development group?

		Linus

Re: 1352 NUL bytes at the end of a page?

[Larry McVoy]

On Mon, May 17, 2004 at 08:22:10AM -0700, Linus Torvalds wrote:
> On Mon, 17 May 2004, Theodore Ts'o wrote:
> > Note though that the stdio library uses a writeable mmap to implement
> > fwrite.
> 
> It does? Whee. Then I'll have to agree with Andrew - if there is a path 
> that is more likely to have bugs, it's trying to do writes with mmap and 
> ftruncate.
> 
> Who came up with that braindead idea? Is it some crazed Mach developer 
> that infiltrated the glibc development group?

You have my agreement on the craziness of this idea.  It's a lot easier for
the kernel to do smart things with write behind with write() rather than
mmap()-ed pages being touched.  SunOS is the only system I know which does
both "correctly" (correctly meaning the same way whether it is mmap or
write).

It's also a lose to do mmap() instead of read/write for small files.  Linux
is light weight enough that the cross over point is pretty small, probably
under 8K and certainly under 16K, but still.
-- 
---
Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com

Re: 1352 NUL bytes at the end of a page?

[viro]

On Mon, May 17, 2004 at 08:22:10AM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 17 May 2004, Theodore Ts'o wrote:
> > 
> > Note though that the stdio library uses a writeable mmap to implement
> > fwrite.
> 
> It does? Whee. Then I'll have to agree with Andrew - if there is a path 
> that is more likely to have bugs, it's trying to do writes with mmap and 
> ftruncate.
> 
> Who came up with that braindead idea? Is it some crazed Mach developer 
> that infiltrated the glibc development group?

IIRC, that idiocy had been disabled by default (note that it's inherently
broken, since truncate() between your mmap() and memcpy() will lead to
a coredump, which is not something fwrite() is allowed to do in such
situation).

strace should show if there such mmap calls are made, anyway.  Did they
show up in the traces?

Re: 1352 NUL bytes at the end of a page?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 330 bytes --]


> 
> Who came up with that braindead idea? Is it some crazed Mach developer 
> that infiltrated the glibc development 

afaik it's optional and off by default, for reads it sort of kinda makes
sense but it can't be on by default otherwise a truncate would cause
fscanf() to throw a sigbus, that's not legal posix wise.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 1352 NUL bytes at the end of a page?

[Steven Cole]

On May 17, 2004, at 9:40 AM, Arjan van de Ven wrote:

>
>>
>> Who came up with that braindead idea? Is it some crazed Mach developer
>> that infiltrated the glibc development
>
> afaik it's optional and off by default, for reads it sort of kinda 
> makes
> sense but it can't be on by default otherwise a truncate would cause
> fscanf() to throw a sigbus, that's not legal posix wise.
>
>

For what it's worth, here is the glibc information on a system which
has the same distribution as the system at home which hits this bug:

[steven@spc2 testing-2.6]$ /lib/libc.so.6
GNU C Library stable release version 2.3.3, by Roland McGrath et al.
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.3.2 (Mandrake Linux 10.0 3.3.2-4mdk).
Compiled on a Linux 2.6.0 system on 2004-02-16.
Available extensions:
         GNU libio by Per Bothner
         crypt add-on version 2.1 by Michael Glad and others
         linuxthreads-0.10 by Xavier Leroy
         BIND-8.2.3-T5B
         libthread_db work sponsored by Alpha Processor Inc
         NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Thread-local storage support included.
Report bugs using the `glibcbug' script to <bugs@gnu.org>.

	Steven

------------------------------------------------------------------------
Steven Cole <scole@lanl.gov>
MacOS X 10.3.3 Panther

Re: 1352 NUL bytes at the end of a page?

[Davide Libenzi]

On Mon, 17 May 2004, Theodore Ts'o wrote:

> On Mon, May 17, 2004 at 08:56:40AM -0500, Wayne Scott wrote:
> > From: Andrew Morton <akpm@osdl.org>
> > > Well we can stop right there, because the only way someone can get some
> > > more non-zero user data into this page before we memset and write it is by
> > > locking the page beforehand, and block_write_full_page() has the page lock.
> > > (Or they can write stuff into it via mmap, but writing to the page outside
> > > i_size is an application bug).
> > 
> > BTW: BitKeeper never opens a writable mmap to a file.  The files are
> > read with mmap() and written by fwriting to a tmp file and then
> > renaming over the target.  And since we run on Windows, no process has
> > the file open when we are updating it.
> 
> Note though that the stdio library uses a writeable mmap to implement
> fwrite.

Strange, it uses read/write but it also opens an mmap(private and 
anonymous):


#include <stdio.h>

int main(int ac, char **av) {
	size_t rd;
	FILE *fp;
	static char buf[1024 * 64];

	fp = fopen(av[1], "r+");
	rd = fread(buf, 1, sizeof(buf), fp);
	fseek(fp, 0, SEEK_SET);
	fwrite(buf, 1, rd, fp);
	fflush(fp);
	fclose(fp);

	return 0;
}


[davide@bigblue davide]$ strace ./foo zzzzzzzzzzzzzzzzzzz
execve("./foo", ["./foo", "zzzzzzzzzzzzzzzzzzz"], [/* 30 vars */]) = 0
uname({sys="Linux", node="bigblue.dev.mdolabs.com", ...}) = 0
brk(0)                                  = 0x9cc8000
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or 
directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=84014, ...}) = 0
old_mmap(NULL, 84014, PROT_READ, MAP_PRIVATE, 3, 0) = 0xbf50b000
close(3)                                = 0
open("/lib/tls/libc.so.6", O_RDONLY)    = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`\350\270"..., 
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1578228, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xbf50a000
old_mmap(0xb79000, 1281996, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 
0xb79000
old_mmap(0xcac000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 
0x132000) = 0xcac000
old_mmap(0xcb0000, 8140, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xcb0000
close(3)                                = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0xbf50a740, 
limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, 
limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0xbf50b000, 84014)               = 0
brk(0)                                  = 0x9cc8000
brk(0x9ce9000)                          = 0x9ce9000
brk(0)                                  = 0x9ce9000
open("zzzzzzzzzzzzzzzzzzz", O_RDWR)     = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=188307, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0xbf51f000
read(3, "<!DOCTYPE HTML PUBLIC \"-//IETF//"..., 65536) = 65536
_llseek(3, 0, [0], SEEK_SET)            = 0
write(3, "<!DOCTYPE HTML PUBLIC \"-//IETF//"..., 65536) = 65536
close(3)                                = 0
munmap(0xbf51f000, 4096)                = 0
exit_group(0)



- Davide

Re: 1352 NUL bytes at the end of a page?

[Davide Libenzi]

On Mon, 17 May 2004, Davide Libenzi wrote:

> Strange, it uses read/write but it also opens an mmap(private and 
> anonymous):

That is not file related (I should have had breakfast before posting :)



- Davide

Re: 1352 NUL bytes at the end of a page?

[Steven Cole]

On May 17, 2004, at 9:37 AM, viro@parcelfarce.linux.theplanet.co.uk 
wrote:

> On Mon, May 17, 2004 at 08:22:10AM -0700, Linus Torvalds wrote:
>>
>>
>> On Mon, 17 May 2004, Theodore Ts'o wrote:
>>>
>>> Note though that the stdio library uses a writeable mmap to implement
>>> fwrite.
>>
>> It does? Whee. Then I'll have to agree with Andrew - if there is a 
>> path
>> that is more likely to have bugs, it's trying to do writes with mmap 
>> and
>> ftruncate.
>>
>> Who came up with that braindead idea? Is it some crazed Mach developer
>> that infiltrated the glibc development group?
>
> IIRC, that idiocy had been disabled by default (note that it's 
> inherently
> broken, since truncate() between your mmap() and memcpy() will lead to
> a coredump, which is not something fwrite() is allowed to do in such
> situation).
>
> strace should show if there such mmap calls are made, anyway.  Did they
> show up in the traces?
>
>

These calls show up in an strace which I did from a non-failing system,
but which has the same glibc as the failing system:

mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0x40018000
old_mmap(NULL, 19184, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000

The command was the following, with the result "Nothing to pull".
strace bk pull bk://linux.bkbits.net/linux-2.5

There were 52 instances of mmap2 or old_mmap in the saved script log.

	Steven

Re: 1352 NUL bytes at the end of a page?

[Steven Cole]

On Mon, 2004-05-17 at 11:40, viro@parcelfarce.linux.theplanet.co.uk
wrote:
> On Mon, May 17, 2004 at 11:30:36AM -0600, Steven Cole wrote:
>  
> > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
> > 0) = 0x40018000
> 
> rw anonymous - that has nothing to do with any IO.
> 
> > old_mmap(NULL, 19184, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000
> 
> read-only, whatever file that was.
> 
> Was there anything with PROT_WRITE and without MAP_ANONYMOUS?

Yes, seven of the 52 references to mmap in the strace output met
the above criteria: 

  old_mmap(0x4015f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x142000) = 0x4015f000
  old_mmap(0x40170000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x9000) = 0x40170000
  old_mmap(0x40180000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x9000) = 0x40180000
  old_mmap(0x40191000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x10000) = 0x40191000
  old_mmap(0x4019c000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x7000) = 0x4019c000
  old_mmap(0x401a0000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x3000) = 0x401a0000
  old_mmap(0x401af000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0xe000) = 0x401af000


Steven 

Re: 1352 NUL bytes at the end of a page?

[viro]

On Mon, May 17, 2004 at 11:30:36AM -0600, Steven Cole wrote:
 
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
> 0) = 0x40018000

rw anonymous - that has nothing to do with any IO.

> old_mmap(NULL, 19184, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000

read-only, whatever file that was.

Was there anything with PROT_WRITE and without MAP_ANONYMOUS?

Re: 1352 NUL bytes at the end of a page?

[viro]

On Mon, May 17, 2004 at 11:39:58AM -0600, Steven Cole wrote:
> Yes, seven of the 52 references to mmap in the strace output met
> the above criteria: 
> 
>   old_mmap(0x4015f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x142000) = 0x4015f000
>   old_mmap(0x40170000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x9000) = 0x40170000
>   old_mmap(0x40180000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x9000) = 0x40180000
>   old_mmap(0x40191000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x10000) = 0x40191000
>   old_mmap(0x4019c000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x7000) = 0x4019c000
>   old_mmap(0x401a0000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x3000) = 0x401a0000
>   old_mmap(0x401af000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0xe000) = 0x401af000

Shared libraries.  And no, those will never lead to any writes, no matter how
you modify them (MAP_PRIVATE).  Which is the point, since that's where we
are doing relocations and we definitely do not want that to hit the disk ;-)

IOW, we can remove writes of dirtied mmap'ed pages from consideration.

Re: 1352 NUL bytes at the end of a page?

[Wayne Scott]

From: Chris Mason <mason@suse.com>
> Good to hear.  We probably still need Andrew's truncate fix, this just
> isn't the right workload to show it.  Andrew, that reiserfs fix survived
> testing here, could you please include it?
> 
> -chris

BTW. We have had one other person report a similar failure.

http://db.bitkeeper.com/cgi-bin/bugdb.cgi?.page=view&id=2004-05-19-001

But if sounds like this problem is now understood.  It was a pleasure
to watch you guys, and someone should buy Steven a beer.  Or perhaps
order a pizza for his family because I suspect this took some of their
time.

-Wayne

Re: 1352 NUL bytes at the end of a page?

[Nick Piggin]

Wayne Scott wrote:
> From: Chris Mason <mason@suse.com>
> 
>>Good to hear.  We probably still need Andrew's truncate fix, this just
>>isn't the right workload to show it.  Andrew, that reiserfs fix survived
>>testing here, could you please include it?
>>
>>-chris
> 
> 
> BTW. We have had one other person report a similar failure.
> 
> http://db.bitkeeper.com/cgi-bin/bugdb.cgi?.page=view&id=2004-05-19-001
> 
> But if sounds like this problem is now understood.  It was a pleasure
> to watch you guys, and someone should buy Steven a beer.  Or perhaps
> order a pizza for his family because I suspect this took some of their
> time.
> 

Yep. Thanks for your help Steven.

I don't think anyone has cleared up the performance regression
problem yet though, so I'll have to bug you a bit more.

Steven, with all else being equal, you said you found a 2.6.3 SuSE
kernel to significantly outperform 2.6.6, is that right? If so can
you try the same test with plain 2.6.3 please? We'll go from there.

This one isn't urgent, because I suspect it could be something
specific to the SuSE kernel rather than a regression in Linus' tree
- we've heard no other complaints... so just whenever you get the
chance.

Nick

Re: 1352 NUL bytes at the end of a page?

[Steven Cole]

On May 19, 2004, at 6:42 AM, Nick Piggin wrote:

> Wayne Scott wrote:
>> From: Chris Mason <mason@suse.com>
>>> Good to hear.  We probably still need Andrew's truncate fix, this 
>>> just
>>> isn't the right workload to show it.  Andrew, that reiserfs fix 
>>> survived
>>> testing here, could you please include it?
>>>
>>> -chris
>> BTW. We have had one other person report a similar failure.
>> http://db.bitkeeper.com/cgi-bin/bugdb.cgi?.page=view&id=2004-05-19-001

I received a report from James H. Cloos Jr. (cc'ed to the rest of you),
but apparently, that report never made it to linux-kernel (I haven't
seen it it the archives or in my lmkl file).  His report was regarding
similar file corruption on xfs.

<OT for this thread>
I also made a report yesterday regarding a compile problem with
lib/kobject.c and no CONFIG_SYSFS.  That post also never made it
to the list. If that happens again, I'll report it to the right folks.
</OT>

>> But if sounds like this problem is now understood.  It was a pleasure
>> to watch you guys, and someone should buy Steven a beer.  Or perhaps
>> order a pizza for his family because I suspect this took some of their
>> time.
>
> Yep. Thanks for your help Steven.
>
> I don't think anyone has cleared up the performance regression
> problem yet though, so I'll have to bug you a bit more.
>
> Steven, with all else being equal, you said you found a 2.6.3 SuSE
> kernel to significantly outperform 2.6.6, is that right? If so can
> you try the same test with plain 2.6.3 please? We'll go from there.

Actually, it was a Mandrake kernel, 2.6.3-4mdk IIRC.  Whatever is
the default with MDK 10.  One salient difference with the vendor
kernel is that everything which can be a module is, and I wasn't
using any modules with my kernels.  BTW, I was careful to have the
same hdparm settings during the performance testing.

The performance difference was very repeatable.  Using the script
provided by Andy Isaacson, the 2.6.3-4mdk did the clone in about
11 minutes total, while the various current kernels took about
15 minutes total.  The user times were the same, and the difference
was in system time.  Those numbers are from memory, the actual
results should be in the archive.

>
> This one isn't urgent, because I suspect it could be something
> specific to the SuSE kernel rather than a regression in Linus' tree
> - we've heard no other complaints... so just whenever you get the
> chance.
>

I may be able to do some some performance testing here at work,
where I have a greater variety (and much faster) machines to use.

	Steven

Re: 1352 NUL bytes at the end of a page?

[Chris Mason]

On Wed, 2004-05-19 at 09:28, Steven Cole wrote:

> > Steven, with all else being equal, you said you found a 2.6.3 SuSE
> > kernel to significantly outperform 2.6.6, is that right? If so can
> > you try the same test with plain 2.6.3 please? We'll go from there.
> 
> Actually, it was a Mandrake kernel, 2.6.3-4mdk IIRC.  Whatever is
> the default with MDK 10.  One salient difference with the vendor
> kernel is that everything which can be a module is, and I wasn't
> using any modules with my kernels.  BTW, I was careful to have the
> same hdparm settings during the performance testing.
> 
> The performance difference was very repeatable.  Using the script
> provided by Andy Isaacson, the 2.6.3-4mdk did the clone in about
> 11 minutes total, while the various current kernels took about
> 15 minutes total.  The user times were the same, and the difference
> was in system time.  Those numbers are from memory, the actual
> results should be in the archive.

Was this regression only reiserv3 or both v3 and ext3?

-chris

Re: 1352 NUL bytes at the end of a page?

[Steven Cole]

On May 19, 2004, at 7:36 AM, Chris Mason wrote:

> On Wed, 2004-05-19 at 09:28, Steven Cole wrote:
>
>>> Steven, with all else being equal, you said you found a 2.6.3 SuSE
>>> kernel to significantly outperform 2.6.6, is that right? If so can
>>> you try the same test with plain 2.6.3 please? We'll go from there.
>>
>> Actually, it was a Mandrake kernel, 2.6.3-4mdk IIRC.  Whatever is
>> the default with MDK 10.  One salient difference with the vendor
>> kernel is that everything which can be a module is, and I wasn't
>> using any modules with my kernels.  BTW, I was careful to have the
>> same hdparm settings during the performance testing.
>>
>> The performance difference was very repeatable.  Using the script
>> provided by Andy Isaacson, the 2.6.3-4mdk did the clone in about
>> 11 minutes total, while the various current kernels took about
>> 15 minutes total.  The user times were the same, and the difference
>> was in system time.  Those numbers are from memory, the actual
>> results should be in the archive.
>
> Was this regression only reiserv3 or both v3 and ext3?
>
> -chris
>

I went back through the archive to make sure, and since I didn't
specify where I did the timed tests, those timing tests would have
been done on my /home partition, which is reiserfs v3.

Since I was using different partitions for ext3 and reiserfs on
/dev/hda, a direct comparison between ext3 and reiserfs wouldn't
be completely fair, but a "watching the paint dry" observation
seemed to indicate that reiserfs was significantly faster for this
load.  I did press my backup disk into service for this testing,
to eliminate the possibility that this was due to a finicky disk,
and I have a 3.9 G partition which I've formatted first reiserfs,
then ext3, so I could do some fair tests between reiserfs and
ext3 on that disk.  But I think the results are already known;
reiserfs opens a can of whoopass for this kind of load.

	Steven

Re: 1352 NUL bytes at the end of a page?

[Wayne Scott]

From: Steven Cole <scole@lanl.gov>
>  But I think the results are already known;
> reiserfs opens a can of whoopass for this kind of load.

I can confirm that bk really likes reiserfs.

-Wayne

Re: 1352 NUL bytes at the end of a page?

[Chris Mason]

On Wed, 2004-05-19 at 09:59, Steven Cole wrote:

> I went back through the archive to make sure, and since I didn't
> specify where I did the timed tests, those timing tests would have
> been done on my /home partition, which is reiserfs v3.
> 
> Since I was using different partitions for ext3 and reiserfs on
> /dev/hda, a direct comparison between ext3 and reiserfs wouldn't
> be completely fair, but a "watching the paint dry" observation
> seemed to indicate that reiserfs was significantly faster for this
> load.  I did press my backup disk into service for this testing,
> to eliminate the possibility that this was due to a finicky disk,
> and I have a 3.9 G partition which I've formatted first reiserfs,
> then ext3, so I could do some fair tests between reiserfs and
> ext3 on that disk.  But I think the results are already known;
> reiserfs opens a can of whoopass for this kind of load.

While this is the kind of thing I like to hear, it wasn't really what I
was asking ;-)

There was a regression between a 2.6.3 mandrake kernel and 2.6.6, was
this regression just for reiserfs or was it for all filesystems?

If just reiserfs, it might be from the data=ordered and logging changes
that went into 2.6.6, so I'm quite interested in figuring things out.

-chris

Re: 1352 NUL bytes at the end of a page?

[Steven Cole]

On May 19, 2004, at 8:08 AM, Chris Mason wrote:

> On Wed, 2004-05-19 at 09:59, Steven Cole wrote:
>
>> I went back through the archive to make sure, and since I didn't
>> specify where I did the timed tests, those timing tests would have
>> been done on my /home partition, which is reiserfs v3.
>>
>> Since I was using different partitions for ext3 and reiserfs on
>> /dev/hda, a direct comparison between ext3 and reiserfs wouldn't
>> be completely fair, but a "watching the paint dry" observation
>> seemed to indicate that reiserfs was significantly faster for this
>> load.  I did press my backup disk into service for this testing,
>> to eliminate the possibility that this was due to a finicky disk,
>> and I have a 3.9 G partition which I've formatted first reiserfs,
>> then ext3, so I could do some fair tests between reiserfs and
>> ext3 on that disk.  But I think the results are already known;
>> reiserfs opens a can of whoopass for this kind of load.
>
> While this is the kind of thing I like to hear, it wasn't really what I
> was asking ;-)
>
> There was a regression between a 2.6.3 mandrake kernel and 2.6.6, was
> this regression just for reiserfs or was it for all filesystems?
>
> If just reiserfs, it might be from the data=ordered and logging changes
> that went into 2.6.6, so I'm quite interested in figuring things out.
>
> -chris

Sorry for distracting you with the second paragraph, but to reemphasize
the first paragraph, I did the timing comparisons between the
various kernel versions on reiserfs v3 _only_, and the /home reiserfs
wasn't mounted with anything special.  I don't have the /etc/fstab
right here in front of me, but I can get that later if needed.  The
.config file was posted in the very first post of this thread, and
the only deviations from that were to add some DEBUG config options
suggested by Andrew or Linus and I dropped PREEMPT (before your patch) 
to
keep the thing from going splaat.

The kernel versions I tested were 2.6.6-current a few days ago,
2.6.3-4mdk, and 2.6.5-aa5.

	Steven

Re: 1352 NUL bytes at the end of a page?

[Steven Cole]

On May 19, 2004, at 8:08 AM, Chris Mason wrote:

> On Wed, 2004-05-19 at 09:59, Steven Cole wrote:
>
>> I went back through the archive to make sure, and since I didn't
>> specify where I did the timed tests, those timing tests would have
>> been done on my /home partition, which is reiserfs v3.
>>
>> Since I was using different partitions for ext3 and reiserfs on
>> /dev/hda, a direct comparison between ext3 and reiserfs wouldn't
>> be completely fair, but a "watching the paint dry" observation
>> seemed to indicate that reiserfs was significantly faster for this
>> load.  I did press my backup disk into service for this testing,
>> to eliminate the possibility that this was due to a finicky disk,
>> and I have a 3.9 G partition which I've formatted first reiserfs,
>> then ext3, so I could do some fair tests between reiserfs and
>> ext3 on that disk.  But I think the results are already known;
>> reiserfs opens a can of whoopass for this kind of load.
>
> While this is the kind of thing I like to hear, it wasn't really what I
> was asking ;-)
>
> There was a regression between a 2.6.3 mandrake kernel and 2.6.6, was
> this regression just for reiserfs or was it for all filesystems?
>
> If just reiserfs, it might be from the data=ordered and logging changes
> that went into 2.6.6, so I'm quite interested in figuring things out.
>
> -chris

2nd reply:

It was just reiserfs, and I'm preparing to repeat some of those timing
tests on one of my test boxes at work, a dual P-III with IDE in this
case.  I can test both reiserfs v3 and ext3 and 2.6.3-4mdk (smp version)
versus other kernel versions and whatever reiserfs mount options you
want to try.  I have SCSI boxes too if needed.

	Steven