Slowdown is gone & apt-get works with updated reiser4. So nevermind...

Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[John Gilmore]

I grabbed the reiser4 patch for 2.6.14-rc5 and compiled it. Thanks to 

Vladimir V. Saveliev's comments about

EXPORT_SYMBOL(clear_page_dirty_for_io);

in mm/page-writeback.c, I was able to get it working as a module, and it seems 
to have taken care of it.

I would have waited for the official 2.6.14-mm1 reiser patch before upgrading 
to 2.6.14, but CD burning didn't work with 2.6.12.


<rant>
There is, btw, one main reason that I've decided that whatever trouble it may 
cause, and whatever growing pangs I may experience along the way, root 
reiser4 is worth it. Does anybody remember GoBack? It was a versioning system 
for windows 95/98 that was incredibly flexible and useful. Tracked all 
changes to the whole disk. Old versions of a file? no problem. grab an old 
version of a directory for referance temporarily? easy. Got a virus? revert 
the whole HD, and then grab the newer copies of your documents and saved 
games as needed.

Microsoft includes an almost useless version of the same ability with their 
"system restore" facility on XP, but I've never seen or heard of anybody 
using it. And rightfully so, it majorly stinks. It doesn't track all files, 
it's interface is opaque, the fact that it even exists is hidden seven layers 
deep, you can't control which files are restored, you can't list previous 
versions of a file, you can't copy an old version of a subdirectory and it's 
contents out without wiping the new version. You can bet that in 10 years or 
so, Microsoft will come out with a version of system restore that doesn't 
suck though. Integrated into the file manager, right click access, and 
everything else too.

Goback is the only thing that I missed when I switched over to linux, and 
reiser4 is the only thing I've found that even hints at a similar ability. 
Even if it takes another 10 years to reach the same point of usability that 
GoBack had, it'll be well worth it.

And when that day comes, I won't even have to reformat (you didn't have to 
reformat to install GoBack, either.) It's been 10 years or so since my last 
format (Hrmm... a little over eight, actually) and I figured that as long as 
my HD was trashed (another reason to love reiser4 - any fs that has a 
standard tool that commonly trashes file systems beyond any hope of 
recovery... darn fsck.ext3) I might as well prepare for the future, and get 
better performance while I'm at it.

Note though, that features are definitely the first thing for me, performance 
is nice but not something that I'll notice too much, and I'd definitely be 
willing to sacrifice some to get enhanced semantics or versioning. Compiles 
take forever no matter what you do, and as long as the little things (like 
starting vim) don't take longer than a second or two, that's good enough.

</rant>

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Hans Reiser]

John Gilmore wrote:

>
>Note though, that features are definitely the first thing for me, performance 
>is nice but not something that I'll notice too much, and I'd definitely be 
>willing to sacrifice some to get enhanced semantics or versioning. Compiles 
>take forever no matter what you do, and as long as the little things (like 
>starting vim) don't take longer than a second or two, that's good enough.
>
></rant>
>
>
>  
>
Thanks for an informative email.  I encourage people to write patches
for adding  versioning into reiser4 before we find the funding for it
(could be years, sigh).

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Jonathan Briggs]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

On Fri, 2005-11-11 at 13:59 +0000, John Gilmore wrote:
[snip]
> <rant>
> There is, btw, one main reason that I've decided that whatever trouble it may 
> cause, and whatever growing pangs I may experience along the way, root 
> reiser4 is worth it. Does anybody remember GoBack? It was a versioning system 
> for windows 95/98 that was incredibly flexible and useful.
[snip]

I have a "goback" utility for my Linux machines.  It's a 300 GB USB2
drive and rdiff-backup. :)

You could do a similar thing to GoBack with any recent Linux kernel and
inotify, actually.

Create a series of snapshot directories that contain copies of every
file.  Use inotify to watch all changes.  Every 15 minutes update the
snapshots with the changed files.

Or you might be able to hook it into rdiff-backup or something like it,
to update only the changed files every 15 minutes, without needing to do
full disk scans to find the changes.
-- 
Jonathan Briggs <jbriggs@esoft.com>
eSoft, Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[michael chang]

On 11/11/05, Jonathan Briggs <jbriggs@esoft.com> wrote:
> On Fri, 2005-11-11 at 13:59 +0000, John Gilmore wrote:
> [snip]
> > <rant>
> > There is, btw, one main reason that I've decided that whatever trouble it may
> > cause, and whatever growing pangs I may experience along the way, root
> > reiser4 is worth it. Does anybody remember GoBack? It was a versioning system
> > for windows 95/98 that was incredibly flexible and useful.
> [snip]
>
> I have a "goback" utility for my Linux machines.  It's a 300 GB USB2
> drive and rdiff-backup. :)
>
> You could do a similar thing to GoBack with any recent Linux kernel and
> inotify, actually.
>
> Create a series of snapshot directories that contain copies of every
> file.  Use inotify to watch all changes.  Every 15 minutes update the
> snapshots with the changed files.
>
> Or you might be able to hook it into rdiff-backup or something like it,
> to update only the changed files every 15 minutes, without needing to do
> full disk scans to find the changes.

That would probably be more flexable than any versioning plugin - but
if GoBack was a Win9x program, I'd presume it was double-click, setup,
and go -- the vision of a Reiser4 plugin with that ability makes it
seem like I can simply push "y" in kernel menuconfig, recompile with
Reiser4, and be off to the races.  Besides, if it's transparent
enough, when it runs out of room, it can automatically delete instead
of me running head first into the out-of-space error and then MANUALLY
going to delete a backup (assuming backup is stored in the same
parititon).  Just because you can afford a seperate backup HD doesn't
mean everyone can, and USB just doesn't transfer fast enough for my
needs (as a consumer with 4 kids in the family, I can't be backing
things up all night, or at least that's my impression of it).

Of course, no plugin will be a substitute for hard backups - on
optical media, tapes, or external drives, as the case may be.  [Best
bet, have all three, on off-site locations... but that's a hassle for
a anyone just trying to get his 10 and 14 year old using Linux.]

--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Hans Reiser]

Being seamless, cleanly implemented, and requiring little or no admin
work, matters a lot to end users.

Yes, users can do what you said with rsync, but it is important that it
be no more work than specifying a --use-versioning mount option, and
even that is beyond most users (but that is where defaults come in to
help them).

The namespace for the past versions should be as cleanly done as WAFL
does them.  Whether space gets freed automatically when space gets <10%
is another mount option.  Where we might do better than WAFL is in
allowing touching filename/..../checkin to cause a version to get
recorded, rather than doing it at particular times.

Hans

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[John Gilmore]

On Saturday 12 November 2005 06:38, Hans Reiser wrote:
> Being seamless, cleanly implemented, and requiring little or no admin
> work, matters a lot to end users.
Amen, Brother!


> Yes, users can do what you said with rsync, but it is important that it
> be no more work than specifying a --use-versioning mount option, and
> even that is beyond most users (but that is where defaults come in to
> help them).
>
> The namespace for the past versions should be as cleanly done as WAFL
> does them.  Whether space gets freed automatically when space gets <10%
> is another mount option.  Where we might do better than WAFL is in
> allowing touching filename/..../checkin to cause a version to get
> recorded, rather than doing it at particular times.
>
> Hans
Of particular concern is that the name space should (somehow) allow me to 
easily grab version by date, even if the file hadn't changed for the two 
weeks before that, and in fact still hasn't changed... Make it really easy to 
grab all or some files by wildcard and with a specific revision, even when 
not every file changed with that revision.

Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to 
soon. The specific symptom is that the effected process locks for a time, 
usually just a second or two, but sometimes a minute or two and and at least 
once for many many minutes. I think that the crash (soft lockup) that I 
reported earlier is related as well. And it sounds like the comment that 
rvalles had about lockups with mmaped files, except that it doesn't lock up 
permanently. Just for a second or three usually.

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Artur Makówka]

> On Saturday 12 November 2005 06:38, Hans Reiser wrote:
>   
>> Being seamless, cleanly implemented, and requiring little or no admin
>> work, matters a lot to end users.
>>     
> Amen, Brother!
>
>
>   
>> Yes, users can do what you said with rsync, but it is important that it
>> be no more work than specifying a --use-versioning mount option, and
>> even that is beyond most users (but that is where defaults come in to
>> help them).
>>
>> The namespace for the past versions should be as cleanly done as WAFL
>> does them.  Whether space gets freed automatically when space gets <10%
>> is another mount option.  Where we might do better than WAFL is in
>> allowing touching filename/..../checkin to cause a version to get
>> recorded, rather than doing it at particular times.
>>
>> Hans
>>     
> Of particular concern is that the name space should (somehow) allow me to 
> easily grab version by date, even if the file hadn't changed for the two 
> weeks before that, and in fact still hasn't changed... Make it really easy to 
> grab all or some files by wildcard and with a specific revision, even when 
> not every file changed with that revision.
>
> Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to 
> soon. The specific symptom is that the effected process locks for a time, 
> usually just a second or two, but sometimes a minute or two and and at least 
> once for many many minutes. I think that the crash (soft lockup) that I 
> reported earlier is related as well. And it sounds like the comment that 
> rvalles had about lockups with mmaped files, except that it doesn't lock up 
> permanently. Just for a second or three usually.
>
>
> .
>
>   
yes, its exactly the same with my case. it locks up just for few 
seconds, sometimes one/two minutes when using for example vim... but 
sometimes it locks up completly, machine is responding to ping but i 
cannot login, dns is not working and all services seems down. (but it is 
replying to ping)

it can happen up to 2-3 times per day. (usually only once per day or 
once per 2 days)

there is completly no info about anything in any log files.

i already installed latest kernel 2.6.14.2 and lates 2.6.14-1 reiser4 
patch and its the same.

My users are getting angry becouse of the downtimes... is this bug 
noticed by dev team ? This seems really serious, as it is causing 
sporadic crashesh and often short lock-ups. Any quick-fix please? It is 
happening for quite some time now so this is not new bug. thats why i 
keep asking... is it worked on or not?

thanks in advance

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Hans Reiser]

John Gilmore wrote:

>On Saturday 12 November 2005 06:38, Hans Reiser wrote:
>  
>
>>Being seamless, cleanly implemented, and requiring little or no admin
>>work, matters a lot to end users.
>>    
>>
>Amen, Brother!
>
>
>  
>
>>Yes, users can do what you said with rsync, but it is important that it
>>be no more work than specifying a --use-versioning mount option, and
>>even that is beyond most users (but that is where defaults come in to
>>help them).
>>
>>The namespace for the past versions should be as cleanly done as WAFL
>>does them.  Whether space gets freed automatically when space gets <10%
>>is another mount option.  Where we might do better than WAFL is in
>>allowing touching filename/..../checkin to cause a version to get
>>recorded, rather than doing it at particular times.
>>
>>Hans
>>    
>>
>Of particular concern is that the name space should (somehow) 
>
somehow = filename/versions/version_number and ls -l filename/versions ?

>allow me to 
>easily grab version by date, even if the file hadn't changed for the two 
>weeks before that, and in fact still hasn't changed... Make it really easy to 
>grab all or some files by wildcard and with a specific revision, even when 
>not every file changed with that revision.
>
>Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to 
>soon. The specific symptom is that the effected process locks for a time, 
>usually just a second or two, but sometimes a minute or two and and at least 
>once for many many minutes. I think that the crash (soft lockup) that I 
>reported earlier is related as well. And it sounds like the comment that 
>rvalles had about lockups with mmaped files, except that it doesn't lock up 
>permanently. Just for a second or three usually.
>
>  
>
zam, please comment.

>
>  
>

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Artur Makówka]

Hans Reiser wrote:
> John Gilmore wrote:
> 
>> On Saturday 12 November 2005 06:38, Hans Reiser wrote:
>>  
>>
>>> Being seamless, cleanly implemented, and requiring little or no admin
>>> work, matters a lot to end users.
>>>    
>>>
>> Amen, Brother!
>>
>>
>>  
>>
>>> Yes, users can do what you said with rsync, but it is important that it
>>> be no more work than specifying a --use-versioning mount option, and
>>> even that is beyond most users (but that is where defaults come in to
>>> help them).
>>>
>>> The namespace for the past versions should be as cleanly done as WAFL
>>> does them.  Whether space gets freed automatically when space gets <10%
>>> is another mount option.  Where we might do better than WAFL is in
>>> allowing touching filename/..../checkin to cause a version to get
>>> recorded, rather than doing it at particular times.
>>>
>>> Hans
>>>    
>>>
>> Of particular concern is that the name space should (somehow) 
>>
> somehow = filename/versions/version_number and ls -l filename/versions ?
> 
>> allow me to 
>> easily grab version by date, even if the file hadn't changed for the two 
>> weeks before that, and in fact still hasn't changed... Make it really easy to 
>> grab all or some files by wildcard and with a specific revision, even when 
>> not every file changed with that revision.
>>
>> Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to 
>> soon. The specific symptom is that the effected process locks for a time, 
>> usually just a second or two, but sometimes a minute or two and and at least 
>> once for many many minutes. I think that the crash (soft lockup) that I 
>> reported earlier is related as well. And it sounds like the comment that 
>> rvalles had about lockups with mmaped files, except that it doesn't lock up 
>> permanently. Just for a second or three usually.
>>
>>  
>>
> zam, please comment.
> 

one more thing im pretty sure of - the 2.6.13mm3 without any reiser4 
additional patches (just clean 2.6.13mm3 as it has reiser4 already 
built) is working fine.

i mean, im not sure if this bug still exists here, but im 100% sure i 
can write vim files easy without any downtime, so this is big difference.

so for everyone with this bug - try clean 2.6.13mm3 from kernel.org. It 
worked for me. i will wait with this kernel for a patch to stable line.

Also - i will test it a little more tomorrow to be sure that this 
version is bug free.

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Laurent Riffard]

Le 13.11.2005 01:55, Artur Makówka a écrit :
> Hans Reiser wrote:
> 
>> John Gilmore wrote:
>>
[snip]
>>>  
>>> Oh, BTW. "The slowdown" as I called it is still there. I guess I
>>> spoke to soon. The specific symptom is that the effected process
>>> locks for a time, usually just a second or two, but sometimes a
>>> minute or two and and at least once for many many minutes. I think
>>> that the crash (soft lockup) that I reported earlier is related as
>>> well. And it sounds like the comment that rvalles had about lockups
>>> with mmaped files, except that it doesn't lock up permanently. Just
>>> for a second or three usually.
>>>
>>>  
>>>
>> zam, please comment.
>>
> 
> one more thing im pretty sure of - the 2.6.13mm3 without any reiser4
> additional patches (just clean 2.6.13mm3 as it has reiser4 already
> built) is working fine.
> 
> i mean, im not sure if this bug still exists here, but im 100% sure i
> can write vim files easy without any downtime, so this is big difference.
> 
> so for everyone with this bug - try clean 2.6.13mm3 from kernel.org. It
> worked for me. i will wait with this kernel for a patch to stable line.
> 
> Also - i will test it a little more tomorrow to be sure that this
> version is bug free.
> 

Hello,

It seems that fsync could be really slow when there is a lot of
activity on the FS.

I've done the following test :
- on a reiser4 FS, unpack a fresh kernel tarball and start a compilation
- open a new terminal, wait 1 minute and start editing a file on the
same FS with vi.
- hit ":w" and wait...

$ strace -T vi foo 2>strace_vi.log
$ grep fsync strace_vi.log
fsync(3)                                = 0 <30.923808>
$

My computer is a desktop with an Athlon XP 1600 and 512 Mb RAM.
Write cache is disabled on the disk (hdparm -W0 /dev/hda).

~~
laurent

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Laurent Riffard]

Le 13.11.2005 13:18, Laurent Riffard a écrit :
> Le 13.11.2005 01:55, Artur Makówka a écrit :
> 
>>Hans Reiser wrote:
>>
>>
>>>John Gilmore wrote:
>>>
> 
> [snip]
> 
>>>> 
>>>>Oh, BTW. "The slowdown" as I called it is still there. I guess I
>>>>spoke to soon. The specific symptom is that the effected process
>>>>locks for a time, usually just a second or two, but sometimes a
>>>>minute or two and and at least once for many many minutes. I think
>>>>that the crash (soft lockup) that I reported earlier is related as
>>>>well. And it sounds like the comment that rvalles had about lockups
>>>>with mmaped files, except that it doesn't lock up permanently. Just
>>>>for a second or three usually.
>>>>
>>>> 
>>>>
>>>
>>>zam, please comment.
>>>
>>
>>one more thing im pretty sure of - the 2.6.13mm3 without any reiser4
>>additional patches (just clean 2.6.13mm3 as it has reiser4 already
>>built) is working fine.
>>
>>i mean, im not sure if this bug still exists here, but im 100% sure i
>>can write vim files easy without any downtime, so this is big difference.
>>
>>so for everyone with this bug - try clean 2.6.13mm3 from kernel.org. It
>>worked for me. i will wait with this kernel for a patch to stable line.
>>
>>Also - i will test it a little more tomorrow to be sure that this
>>version is bug free.
>>
> 
> 
> Hello,
> 
> It seems that fsync could be really slow when there is a lot of
> activity on the FS.
> 
> I've done the following test :
> - on a reiser4 FS, unpack a fresh kernel tarball and start a compilation
> - open a new terminal, wait 1 minute and start editing a file on the
> same FS with vi.
> - hit ":w" and wait...
> 
> $ strace -T vi foo 2>strace_vi.log
> $ grep fsync strace_vi.log
> fsync(3)                                = 0 <30.923808>
> $
> 
> My computer is a desktop with an Athlon XP 1600 and 512 Mb RAM.
> Write cache is disabled on the disk (hdparm -W0 /dev/hda).
> 

I forgot to mention: kernel version is 2.6.14-mm2.
-- 
laurent

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Artur Makówka]

Laurent Riffard wrote:
> Le 13.11.2005 01:55, Artur Makówka a écrit :
> [snip]
>> one more thing im pretty sure of - the 2.6.13mm3 without any reiser4
>> additional patches (just clean 2.6.13mm3 as it has reiser4 already
>> built) is working fine.
>>
>> i mean, im not sure if this bug still exists here, but im 100% sure i
>> can write vim files easy without any downtime, so this is big difference.
>>
>> so for everyone with this bug - try clean 2.6.13mm3 from kernel.org. It
>> worked for me. i will wait with this kernel for a patch to stable line.
>>
>> Also - i will test it a little more tomorrow to be sure that this
>> version is bug free.
>>
> 
> Hello,
> 
> It seems that fsync could be really slow when there is a lot of
> activity on the FS.
> 
> I've done the following test :
> - on a reiser4 FS, unpack a fresh kernel tarball and start a compilation
> - open a new terminal, wait 1 minute and start editing a file on the
> same FS with vi.
> - hit ":w" and wait...
> 
> $ strace -T vi foo 2>strace_vi.log
> $ grep fsync strace_vi.log
> fsync(3)                                = 0 <30.923808>
> $
> 
> My computer is a desktop with an Athlon XP 1600 and 512 Mb RAM.
> Write cache is disabled on the disk (hdparm -W0 /dev/hda).
> 
> ~~
> laurent
> 

it crashed today morning, so 2.6.13mm is not much better than 2.6.14.

My fs is heavly used (but not overloaded) by many apache process or ftp 
process or postfix process.

I have free hosting server and i think i will have to close this thing 
becouse for over a month its often downtime everyday. Could you please 
tell us if there is any idea of how to fix it? i would move to another 
more stable fs but its too much data...

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Thomas Kuther]

[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]

On Sun, 13 Nov 2005 13:29:25 +0100
Artur Makówka <juice@ursynow.2a.pl> wrote:

> Laurent Riffard wrote:
> > Le 13.11.2005 01:55, Artur Makówka a écrit :
> > [snip]
> >> one more thing im pretty sure of - the 2.6.13mm3 without any
> >> reiser4 additional patches (just clean 2.6.13mm3 as it has reiser4
> >> already built) is working fine.
> >>
> >> i mean, im not sure if this bug still exists here, but im 100%
> >> sure i can write vim files easy without any downtime, so this is
> >> big difference.
> >>
> >> so for everyone with this bug - try clean 2.6.13mm3 from
> >> kernel.org. It worked for me. i will wait with this kernel for a
> >> patch to stable line.
> >>
> >> Also - i will test it a little more tomorrow to be sure that this
> >> version is bug free.
> >>
> > 
> > Hello,
> > 
> > It seems that fsync could be really slow when there is a lot of
> > activity on the FS.
> > 
> > I've done the following test :
> > - on a reiser4 FS, unpack a fresh kernel tarball and start a
> > compilation
> > - open a new terminal, wait 1 minute and start editing a file on the
> > same FS with vi.
> > - hit ":w" and wait...
> > 
> > $ strace -T vi foo 2>strace_vi.log
> > $ grep fsync strace_vi.log
> > fsync(3)                                = 0 <30.923808>
> > $
> > 
> > My computer is a desktop with an Athlon XP 1600 and 512 Mb RAM.
> > Write cache is disabled on the disk (hdparm -W0 /dev/hda).
> > 
> > ~~
> > laurent
> > 
> 
> it crashed today morning, so 2.6.13mm is not much better than 2.6.14.
> 
> My fs is heavly used (but not overloaded) by many apache process or
> ftp process or postfix process.
> 
> I have free hosting server and i think i will have to close this
> thing becouse for over a month its often downtime everyday. Could you
> please tell us if there is any idea of how to fix it? i would move to
> another more stable fs but its too much data..

hmm, wondering which I/O scheduler you guys are using. 
I had the same slowdowns and soft lockups till 10 minutes ago, when i
switched from cfq to anticipatory I/O. I'm currently usig 2.6.15-rc1 +
reiser4 from 2.6.14-mm2
I was in the middle of a big gentoo'ish emerge, desktop almost
unusable. but now with anticipatory all seems fine so far. 
the compile seems a lot faster. no soft lockup trying to use vim or
locate... hmmm

regards
tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Artur Makówka]

Thomas Kuther wrote:
> On Sun, 13 Nov 2005 13:29:25 +0100
> Artur Makówka <juice@ursynow.2a.pl> wrote:
> 
>> it crashed today morning, so 2.6.13mm is not much better than 2.6.14.
>>
>> My fs is heavly used (but not overloaded) by many apache process or
>> ftp process or postfix process.
>>
>> I have free hosting server and i think i will have to close this
>> thing becouse for over a month its often downtime everyday. Could you
>> please tell us if there is any idea of how to fix it? i would move to
>> another more stable fs but its too much data..
> 
> hmm, wondering which I/O scheduler you guys are using. 
> I had the same slowdowns and soft lockups till 10 minutes ago, when i
> switched from cfq to anticipatory I/O. I'm currently usig 2.6.15-rc1 +
> reiser4 from 2.6.14-mm2
> I was in the middle of a big gentoo'ish emerge, desktop almost
> unusable. but now with anticipatory all seems fine so far. 
> the compile seems a lot faster. no soft lockup trying to use vim or
> locate... hmmm
> 
> regards
> tom

i was using anticipatory I/O as default in debian. i just did echo cfq > 
  /sys/block/hda/queue/scheduler. Hope its enough to change it.

no noticed effects yet, i will give report, if anything changes.

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Pat Double]

[-- Attachment #1: Type: text/plain, Size: 666 bytes --]

On Sunday 13 November 2005 08:05, Artur Makówka wrote:
> Thomas Kuther wrote:
>
> i was using anticipatory I/O as default in debian. i just did echo cfq >
>   /sys/block/hda/queue/scheduler. Hope its enough to change it.
>
> no noticed effects yet, i will give report, if anything changes.

I tried changing the scheduler like this and it worked the first time with low 
load and my X session was not started. I tried it again and it locked up the 
filesystem I had to reboot and rebuild the filesystem. I was changing from 
cfq to anticipatory. Be careful!

-- 
Pat Double, pat@patdouble.com
"In the beginning God created the heaven and the earth."

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Artur Makówka]

Pat Double wrote:
> On Sunday 13 November 2005 08:05, Artur Makówka wrote:
>> Thomas Kuther wrote:
>>
>> i was using anticipatory I/O as default in debian. i just did echo cfq >
>>   /sys/block/hda/queue/scheduler. Hope its enough to change it.
>>
>> no noticed effects yet, i will give report, if anything changes.
> 
> I tried changing the scheduler like this and it worked the first time with low 
> load and my X session was not started. I tried it again and it locked up the 
> filesystem I had to reboot and rebuild the filesystem. I was changing from 
> cfq to anticipatory. Be careful!
> 

its ok, i changed from anticipatory to cfq also with append option in 
lilo (elevator=).

but i'm afraid it didn't help. I got another crash today. as always - 
completely nothing in syslog or anywhere.

i can't turn on reiser4 debugging - it will lower performance too much.

i'm not sure what to do now. Is there any tool that will convert reiser4 
to another fs ? (a stable one i mean...)

Re: Slowdown is gone & apt-get works with updated reiser4. So nevermind...

[Ingo Bormuth]

On 2005-11-13 14:12, Thomas Kuther wrote:
>
> hmm, wondering which I/O scheduler you guys are using. 
> I had the same slowdowns and soft lockups till 10 minutes ago, when i
> switched from cfq to anticipatory I/O. I'm currently usig 2.6.15-rc1 +
> reiser4 from 2.6.14-mm2
> I was in the middle of a big gentoo'ish emerge, desktop almost
> unusable. but now with anticipatory all seems fine so far. 
> the compile seems a lot faster. no soft lockup trying to use vim or
> locate... hmmm
> 

Same here. 

I used to be hit by loads of 20 (!) and a systen being practically unusable 
for half an hour, everytime I synced the gentoo tree. After switching from 
2-6-14-archck5 to reiser4-for-vanilla-2.6.14-2.patch applied to vanilla
linux-2.6.14.2 everything seems to work just fine.

BTW find ebuild at: http://public.efil.de/gentoo/sys-kernel/reiser4-sources/


-- 
Ingo Bormuth, voicebox & telefax: +49-12125-10226517          '(~o-o~)'
public key 86326EC9, http://ibormuth.efil.de/contact     ---ooO--(.)--Ooo---

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

On Sat, 2005-11-12 at 09:06 +0000, John Gilmore wrote:
> Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to 
> soon. The specific symptom is that the effected process locks for a time, 
> usually just a second or two, but sometimes a minute or two and and at least 
> once for many many minutes. I think that the crash (soft lockup) that I 
> reported earlier is related as well. And it sounds like the comment that 
> rvalles had about lockups with mmaped files, except that it doesn't lock up 
> permanently. Just for a second or three usually.
> 

Hi,

I am sorry to say that I am having the same slowdown issue with Reiser4
and kernel 2.6.14.2. It seems that my problem is not related to the I/O
scheduler algorithm as I have tried both cfq and anticipatory.

The I/O activity seems to come in long bursts lasting up to 15 seconds,
and is unpredictable. The system almost grinds to a halt while the burst
of I/O is taking place, which can be quite annoying while typing.
There are no warnings in dmesg/syslog and the filesystem is clean.

Sometimes the burst of I/O activity occurs in a different form, with
short bursts every second, lasting about 10 seconds. Approx 30% duty
cycle. When a burst occurs in this form, it tends not to lock processes
as much.

The general sound of the drive when a burst is in progress is that of
heavy head movement, as if the drive is repeatedly seeking between two
or three locations.

Also, a thing which I noticed a while back, and may not be related in
any way to Reiser4.. When I re-size the columns in Evolution Mail, I get
continuous hard disk activity during the re-size. This hard disk
activity stops either when I release the mouse or hold it still with the
button down. Just wondering if anyone else gets the same thing.

Regards,
-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[jp]

Hi,

For me reiser4 is working perfectly on Vanilla 2.6.14.2, all the testers 
who I sent the new kernel to are reporting excellent performance and 
reliability.

Could you explain a bit further the way I could reproduce this ?

Thanks

JP

Craig Shelley wrote:

>On Sat, 2005-11-12 at 09:06 +0000, John Gilmore wrote:
>  
>
>>Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to 
>>soon. The specific symptom is that the effected process locks for a time, 
>>usually just a second or two, but sometimes a minute or two and and at least 
>>once for many many minutes. I think that the crash (soft lockup) that I 
>>reported earlier is related as well. And it sounds like the comment that 
>>rvalles had about lockups with mmaped files, except that it doesn't lock up 
>>permanently. Just for a second or three usually.
>>
>>    
>>
>
>Hi,
>
>I am sorry to say that I am having the same slowdown issue with Reiser4
>and kernel 2.6.14.2. It seems that my problem is not related to the I/O
>scheduler algorithm as I have tried both cfq and anticipatory.
>
>The I/O activity seems to come in long bursts lasting up to 15 seconds,
>and is unpredictable. The system almost grinds to a halt while the burst
>of I/O is taking place, which can be quite annoying while typing.
>There are no warnings in dmesg/syslog and the filesystem is clean.
>
>Sometimes the burst of I/O activity occurs in a different form, with
>short bursts every second, lasting about 10 seconds. Approx 30% duty
>cycle. When a burst occurs in this form, it tends not to lock processes
>as much.
>
>The general sound of the drive when a burst is in progress is that of
>heavy head movement, as if the drive is repeatedly seeking between two
>or three locations.
>
>Also, a thing which I noticed a while back, and may not be related in
>any way to Reiser4.. When I re-size the columns in Evolution Mail, I get
>continuous hard disk activity during the re-size. This hard disk
>activity stops either when I release the mouse or hold it still with the
>button down. Just wondering if anyone else gets the same thing.
>
>Regards,
>  
>

Re: More Slowdown

[Christian Iversen]

On Monday 14 November 2005 20:41, Craig Shelley wrote:
> On Sat, 2005-11-12 at 09:06 +0000, John Gilmore wrote:
> > Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to
> > soon. The specific symptom is that the effected process locks for a time,
> > usually just a second or two, but sometimes a minute or two and and at
> > least once for many many minutes. I think that the crash (soft lockup)
> > that I reported earlier is related as well. And it sounds like the
> > comment that rvalles had about lockups with mmaped files, except that it
> > doesn't lock up permanently. Just for a second or three usually.
>
> Hi,
>
> I am sorry to say that I am having the same slowdown issue with Reiser4
> and kernel 2.6.14.2. It seems that my problem is not related to the I/O
> scheduler algorithm as I have tried both cfq and anticipatory.
>
> The I/O activity seems to come in long bursts lasting up to 15 seconds,
> and is unpredictable. The system almost grinds to a halt while the burst
> of I/O is taking place, which can be quite annoying while typing.
> There are no warnings in dmesg/syslog and the filesystem is clean.
>
> Sometimes the burst of I/O activity occurs in a different form, with
> short bursts every second, lasting about 10 seconds. Approx 30% duty
> cycle. When a burst occurs in this form, it tends not to lock processes
> as much.
>
> The general sound of the drive when a burst is in progress is that of
> heavy head movement, as if the drive is repeatedly seeking between two
> or three locations.
>
> Also, a thing which I noticed a while back, and may not be related in
> any way to Reiser4.. When I re-size the columns in Evolution Mail, I get
> continuous hard disk activity during the re-size. This hard disk
> activity stops either when I release the mouse or hold it still with the
> button down. Just wondering if anyone else gets the same thing.

That's _serisouly_ odd. I've seen something like this happen before, when X 
programs generated some (non-harmful) X warnings. These were then written 
into a log file, typically .xsession-errors or similar. Any chance that's 
what you are seeing?

-- 
Regards,
Christian Iversen

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

On Mon, 2005-11-14 at 21:47 +0100, Christian Iversen wrote:
> That's _serisouly_ odd. I've seen something like this happen before, when X 
> programs generated some (non-harmful) X warnings. These were then written 
> into a log file, typically .xsession-errors or similar. Any chance that's 
> what you are seeing?
> 

This explains the problem with Evolution, it looks a bit trigger happy
with the fsync call while resizing columns.
craig@teratron.lan.etheus.net:~$ strace -p 8002 2>&1 | grep fsync
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0

while true; do lsof  -p 8002 2>&1 | grep 37w ; done
evolution 8002 craig   37w   REG        3,4     417
1901436 /mnt/storage/craig-home/.evolution/mail/views/.#custom_view-mbox:_home_craig_.evolution_mail_local#Lists_Reiser4.xmlwhile 

The file listed above seems to get moved, but is only a few hundred
bytes.

ls -lh /mnt/storage/craig-home/.evolution/mail/views/custom_view-mbox
\:_home_craig_.evolution_mail_local#Lists_Reiser4.xml
-rw-------  1 craig craig 417 2005-11-15
14:19 /mnt/storage/craig-home/.evolution/mail/views/custom_view-mbox:_home_craig_.evolution_mail_local#Lists_Reiser4.xml

Why does calling fsync on this one small file cause so much hard disk
activity?


-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[Laurent Riffard]

Le 15.11.2005 15:27, Craig Shelley a écrit :
> On Mon, 2005-11-14 at 21:47 +0100, Christian Iversen wrote:
> 
>>That's _serisouly_ odd. I've seen something like this happen before, when X 
>>programs generated some (non-harmful) X warnings. These were then written 
>>into a log file, typically .xsession-errors or similar. Any chance that's 
>>what you are seeing?
>>
> 
> This explains the problem with Evolution, it looks a bit trigger happy
> with the fsync call while resizing columns.
> craig@teratron.lan.etheus.net:~$ strace -p 8002 2>&1 | grep fsync
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0

Please, could you do it again with the -T option for strace? It will
show the time spent in system  calls.

> while true; do lsof  -p 8002 2>&1 | grep 37w ; done
> evolution 8002 craig   37w   REG        3,4     417
> 1901436 /mnt/storage/craig-home/.evolution/mail/views/.#custom_view-mbox:_home_craig_.evolution_mail_local#Lists_Reiser4.xmlwhile 
> 
> The file listed above seems to get moved, but is only a few hundred
> bytes.
> 
> ls -lh /mnt/storage/craig-home/.evolution/mail/views/custom_view-mbox
> \:_home_craig_.evolution_mail_local#Lists_Reiser4.xml
> -rw-------  1 craig craig 417 2005-11-15
> 14:19 /mnt/storage/craig-home/.evolution/mail/views/custom_view-mbox:_home_craig_.evolution_mail_local#Lists_Reiser4.xml
> 
> Why does calling fsync on this one small file cause so much hard disk
> activity?

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 3095 bytes --]

On Tue, 2005-11-15 at 19:04 +0100, Laurent Riffard wrote:
> Please, could you do it again with the -T option for strace? It will
> show the time spent in system  calls.
> 
craig@teratron.lan.etheus.net:~$ strace -T -p 8002 2>&1 | grep fsync
fsync(26)                               = 0 <0.566159>
fsync(26)                               = 0 <0.159852>
fsync(26)                               = 0 <0.179659>
fsync(26)                               = 0 <0.153406>
fsync(26)                               = 0 <0.136685>
fsync(26)                               = 0 <0.171057>
fsync(26)                               = 0 <0.165733>
fsync(26)                               = 0 <0.178846>
fsync(26)                               = 0 <0.186595>
fsync(26)                               = 0 <0.184368>
fsync(26)                               = 0 <0.177273>
fsync(26)                               = 0 <0.178634>
fsync(26)                               = 0 <0.181036>
fsync(26)                               = 0 <0.178750>
fsync(26)                               = 0 <0.324480>
fsync(26)                               = 0 <0.177226>
fsync(26)                               = 0 <0.184625>
fsync(26)                               = 0 <0.186224>
fsync(26)                               = 0 <0.177282>
fsync(26)                               = 0 <0.177146>
fsync(26)                               = 0 <0.176000>
fsync(26)                               = 0 <0.192434>
fsync(26)                               = 0 <0.177482>
fsync(26)                               = 0 <0.183357>
fsync(26)                               = 0 <0.176528>
fsync(26)                               = 0 <0.184334>
fsync(26)                               = 0 <0.176622>
fsync(26)                               = 0 <0.177977>
fsync(26)                               = 0 <0.178636>
fsync(26)                               = 0 <0.259863>
fsync(26)                               = 0 <0.172789>
fsync(26)                               = 0 <0.171739>
fsync(26)                               = 0 <0.179641>
fsync(26)                               = 0 <0.178157>
fsync(26)                               = 0 <0.180272>
fsync(26)                               = 0 <0.170426>
fsync(26)                               = 0 <0.187630>
fsync(26)                               = 0 <0.171589>
fsync(26)                               = 0 <0.178135>


I seem to be finding that the performance of the system degrades with
time and file system usage.
Openoffice really starts to run slow after a while. Just as an
experiment, I did 
ls -R /usr
The directory listing appeared very quickly with hardly any hard disk
access. Then shortly later when switching back to Openoffice, the
process froze for about 3 minutes with continuous hard disk access.

One thing which may be relevant is that my kernel is patched with
Suspend2 and FBsplash, although I can't remember having any problems
like this before.

Many Thanks,

-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 4113 bytes --]

On Wed, 2005-11-16 at 01:05 -0800, Hans Reiser wrote:
> Craig Shelley wrote:
> 
> >On Tue, 2005-11-15 at 19:04 +0100, Laurent Riffard wrote:
> >  
> >
> >>Please, could you do it again with the -T option for strace? It will
> >>show the time spent in system  calls.
> >>
> >>    
> >>
> >craig@teratron.lan.etheus.net:~$ strace -T -p 8002 2>&1 | grep fsync
> >fsync(26)                               = 0 <0.566159>
> >fsync(26)                               = 0 <0.159852>
> >fsync(26)                               = 0 <0.179659>
> >fsync(26)                               = 0 <0.153406>
> >fsync(26)                               = 0 <0.136685>
> >fsync(26)                               = 0 <0.171057>
> >fsync(26)                               = 0 <0.165733>
> >fsync(26)                               = 0 <0.178846>
> >fsync(26)                               = 0 <0.186595>
> >fsync(26)                               = 0 <0.184368>
> >fsync(26)                               = 0 <0.177273>
> >fsync(26)                               = 0 <0.178634>
> >fsync(26)                               = 0 <0.181036>
> >fsync(26)                               = 0 <0.178750>
> >fsync(26)                               = 0 <0.324480>
> >fsync(26)                               = 0 <0.177226>
> >fsync(26)                               = 0 <0.184625>
> >fsync(26)                               = 0 <0.186224>
> >fsync(26)                               = 0 <0.177282>
> >fsync(26)                               = 0 <0.177146>
> >fsync(26)                               = 0 <0.176000>
> >fsync(26)                               = 0 <0.192434>
> >fsync(26)                               = 0 <0.177482>
> >fsync(26)                               = 0 <0.183357>
> >fsync(26)                               = 0 <0.176528>
> >fsync(26)                               = 0 <0.184334>
> >fsync(26)                               = 0 <0.176622>
> >fsync(26)                               = 0 <0.177977>
> >fsync(26)                               = 0 <0.178636>
> >fsync(26)                               = 0 <0.259863>
> >fsync(26)                               = 0 <0.172789>
> >fsync(26)                               = 0 <0.171739>
> >fsync(26)                               = 0 <0.179641>
> >fsync(26)                               = 0 <0.178157>
> >fsync(26)                               = 0 <0.180272>
> >fsync(26)                               = 0 <0.170426>
> >fsync(26)                               = 0 <0.187630>
> >fsync(26)                               = 0 <0.171589>
> >fsync(26)                               = 0 <0.178135>
> >
> >
> >I seem to be finding that the performance of the system degrades with
> >time and file system usage.
> >Openoffice really starts to run slow after a while. Just as an
> >experiment, I did 
> >ls -R /usr
> >The directory listing appeared very quickly with hardly any hard disk
> >access. Then shortly later when switching back to Openoffice, the
> >process froze for about 3 minutes with continuous hard disk access.
> >
> >One thing which may be relevant is that my kernel is patched with
> >Suspend2 and FBsplash, although I can't remember having any problems
> >like this before.
> >
> >Many Thanks,
> >
> >  
> >
> could you define what the numbers mean (what units, what is measured)?

I believe the value inside < > is the time spent in the system call in
seconds. In the above list, evolution seemed to be calling fsync() as
fast as it could. Although the fd remains at 26 in this case, it closes
and re-opens the file between calls to fsync().
I seem to notice that the length of time spent in the fsync() calls
depends to some degree on how much hard disk activity there has been
since the last call. The strange thing is this seems to apply to reading
also, so performing an ls -R /, and stopping it after a few seconds.
Shortly later, some process will block for tens of seconds when it next
calls fsync().

-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[michael chang]

On 11/15/05, Craig Shelley <craig@microtron.org.uk> wrote:
> On Mon, 2005-11-14 at 21:47 +0100, Christian Iversen wrote:
> > That's _serisouly_ odd. I've seen something like this happen before, when X
> > programs generated some (non-harmful) X warnings. These were then written
> > into a log file, typically .xsession-errors or similar. Any chance that's
> > what you are seeing?
> >
>
> This explains the problem with Evolution, it looks a bit trigger happy
> with the fsync call while resizing columns.

> Why does calling fsync on this one small file cause so much hard disk
> activity?

If memory serves me right, it has been said time and time again (in
docs and on the list) that fsync performance on Reiser4 is horrendous
because it is (at the moment) unoptimized until ... other issues are
resolved and the optimizations can be implemented.

You should also realized the _reason_ why a majority of disk calls
take so little time - it's because Reiser4 puts them in memory, before
flushing to disk when memory is full.  That would explain any
jerkiness -- it's because Reiser4 is writing everything from the last
real write to the actual disk to now (the current real right) in one
go, and making everything wait.  (Now why this can't be done in
parallel is beyond me - or is it already done in parallel?)

This also means that if your hard disk can't handle writing the
contents of say 80% of your memory's size to disk or it takes 5
seconds for it to do so would explain the problem with the concepts
shown here.  Example: 1GB memory, hard disk can write at a max of 10
MB/s.  If it used 75% of the memory for a Reiser4 cache, [768 MB] it
would take maybe 77 or more seconds to flush. [Yes, a full minute.]  
For comparative purposes, I hear of consumer systems with 2 or 4 GB of
ram, and I know of brand new PC hard disks which have a throughput of
less than 5 MB/s (my hard disk maxees out at 20 MB/s on my PC.)  Do
you know the throughputs of your drives, and the size of the memory in
your machines?

When fsyncing, Reiser4, to my understanding, isn't allowed to put
stuff in it's memory cache - it must put it to disk right away.  And
maybe that's also why (partially) Reiser4 performance isn't as optimal
as it will eventually be.  [So, short answer: while there are
shortcomings, and this stuff shouldn't happen, part of this Reiser4FS
behaviour is, to my understanding, intentionally placed.]

Note, this is a my speculation based on what I've read on this mailing
list and on the namesys website, and could be inaccurate.  If any
professionals who are working with the Reiser4 code (e.g. Hans Reiser
and the like) correct me on what I say - their statements take
precedence.

Also, I've noticed in some newer computer's BIOSes (I know some recent
DELL desktop machines have this) sometimes you can find a setting to
change the noisiness/performance trade off of the hard disk.  Maybe
something similar is involved with your PC?  It should be said that
I've witnessed that in quieter mode settings, the disk seems to
operate slower - that could mean that a flush could be less noisy, but
might take longer.

--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: More Slowdown

[Hans Reiser]

fsync can be dramatically better optimized, and it will be after the
kernel merge work is completed.  This optimization will likely consist
of reducing the tendency to merge atoms and handling fsync by using
write twice algorithms to a fixed journal area.  Other improvements will
doubtless be found when time is spent on it.  I am sorry that vim and
evolution are suffering so much from our neglecting fsync until after
the merge.  fsync is one of the things I would have us working on if I
had my choice of what we improved in reiser4 rather than other persons
making that choice for us at the moment.  Oh well, such is life in the
society of men.;-)

It looks like akpm is going to start reading the reiser4 code.  I
suspect his remarks will be a step above the ones made so far, he wrote
such nice readahead code.

Re: More Slowdown

[Artur Makówka]

Hans Reiser wrote:
> fsync can be dramatically better optimized, and it will be after the
> kernel merge work is completed.  This optimization will likely consist
> of reducing the tendency to merge atoms and handling fsync by using
> write twice algorithms to a fixed journal area.  Other improvements will
> doubtless be found when time is spent on it.  I am sorry that vim and
> evolution are suffering so much from our neglecting fsync until after
> the merge.  fsync is one of the things I would have us working on if I
> had my choice of what we improved in reiser4 rather than other persons
> making that choice for us at the moment.  Oh well, such is life in the
> society of men.;-)
> 
> It looks like akpm is going to start reading the reiser4 code.  I
> suspect his remarks will be a step above the ones made so far, he wrote
> such nice readahead code.
> 

Just want to add, that for desktop systems its only vim and evolution, 
but for my free hosting server it was very frequent system crashes 
(because of that i lost part of my database) and so on. Just want to 
warn anyone deciding to use reiser4 on server, to use 2.6.12 kernels or 
lower as this bug is almost not existent there.

For some uses of reiser4 this bug is very dangerous. Not just slow work 
of some application, but like i said data loss, system instability (few 
crashes every day), and so on.

I would say such a bug/design choice should be top priority, number one. 
Of course thats not true if reiser4 is meant to be only for home use. 
Just want to point that out, because it seems everybody is using reiser4 
only at home, and i am the only one on earth using it on server 
environment (i guess that wasn't the best idea...)

Thanks in advance for consideration of taking this bug to top of your 
TODO list.

Re: More Slowdown

[Vladimir V. Saveliev]

[-- Attachment #1: Type: text/plain, Size: 2145 bytes --]

Hello

Artur Makówka wrote:
> Hans Reiser wrote:
>> fsync can be dramatically better optimized, and it will be after the
>> kernel merge work is completed.  This optimization will likely consist
>> of reducing the tendency to merge atoms and handling fsync by using
>> write twice algorithms to a fixed journal area.  Other improvements will
>> doubtless be found when time is spent on it.  I am sorry that vim and
>> evolution are suffering so much from our neglecting fsync until after
>> the merge.  fsync is one of the things I would have us working on if I
>> had my choice of what we improved in reiser4 rather than other persons
>> making that choice for us at the moment.  Oh well, such is life in the
>> society of men.;-)
>>
>> It looks like akpm is going to start reading the reiser4 code.  I
>> suspect his remarks will be a step above the ones made so far, he wrote
>> such nice readahead code.
>>
> 
> Just want to add, that for desktop systems its only vim and evolution,
> but for my free hosting server it was very frequent system crashes
> (because of that i lost part of my database) and so on. Just want to
> warn anyone deciding to use reiser4 on server, to use 2.6.12 kernels or
> lower as this bug is almost not existent there.
> 
> For some uses of reiser4 this bug is very dangerous. Not just slow work
> of some application, but like i said data loss, system instability (few
> crashes every day), and so on.
> 
> I would say such a bug/design choice should be top priority, number one.
> Of course thats not true if reiser4 is meant to be only for home use.
> Just want to point that out, because it seems everybody is using reiser4
> only at home, and i am the only one on earth using it on server
> environment (i guess that wasn't the best idea...)
> 
> Thanks in advance for consideration of taking this bug to top of your
> TODO list.
> 

Please try whether the attached patch improves anything. It simplifies fsync by
avoid commiting of transactions which do not modify file being fsync-ed.

The patch applied to 2.6.14-mm2 with warnings, but that can be ignored.



[-- Attachment #2: reiser4-fix-fsync.patch --]
[-- Type: text/plain, Size: 3987 bytes --]


This patch simplifies reiser4's fsync. 
It is an experimental patch to test whether it helps against slowdowns reported by users.


 fs/reiser4/plugin/file/file.c |   14 ++++++++++++++
 fs/reiser4/txnmgr.c           |   22 +++++++++++++++-------
 fs/reiser4/txnmgr.h           |    1 +
 3 files changed, 30 insertions(+), 7 deletions(-)

diff -puN fs/reiser4/plugin/file/file.c~reiser4-fix-fsync fs/reiser4/plugin/file/file.c
--- linux-2.6.14-mm2/fs/reiser4/plugin/file/file.c~reiser4-fix-fsync	2005-11-17 13:42:26.000000000 +0300
+++ linux-2.6.14-mm2-vs/fs/reiser4/plugin/file/file.c	2005-11-17 13:42:26.000000000 +0300
@@ -1633,11 +1633,25 @@ int sync_unix_file(struct file *file, st
 {
 	int result;
 	reiser4_context *ctx;
+	txn_atom *atom;
+	reiser4_block_nr reserve;
 
 	ctx = init_context(dentry->d_inode->i_sb);
 	if (IS_ERR(ctx))
 		return PTR_ERR(ctx);
 
+	reserve = estimate_update_common(dentry->d_inode);
+	if (reiser4_grab_space(reserve, BA_CAN_COMMIT))
+		return RETERR(-ENOSPC);
+	write_sd_by_inode_common(dentry->d_inode);
+
+	atom = get_current_atom_locked();
+	spin_lock_txnh(ctx->trans);
+	force_commit_atom(ctx->trans);
+	reiser4_exit_context(ctx);
+	return 0;
+
+
 	assert("nikita-3486", ctx->trans->atom == NULL);
 	result = commit_file_atoms(dentry->d_inode);
 	assert("nikita-3484", ergo(result == 0, ctx->trans->atom == NULL));
diff -puN fs/reiser4/txnmgr.c~reiser4-fix-fsync fs/reiser4/txnmgr.c
--- linux-2.6.14-mm2/fs/reiser4/txnmgr.c~reiser4-fix-fsync	2005-11-17 13:42:26.000000000 +0300
+++ linux-2.6.14-mm2-vs/fs/reiser4/txnmgr.c	2005-11-17 13:42:26.000000000 +0300
@@ -1173,9 +1173,14 @@ static int commit_current_atom(long *nr_
 
 /* TXN_TXNH */
 
-/* commit current atom and wait commit completion; atom and txn_handle should be
- * locked before call, this function unlocks them on exit. */
-static int force_commit_atom_nolock(txn_handle * txnh)
+/**
+ * force_commit_atom - commit current atom and wait commit completion
+ * @txnh:
+ *
+ * Commits current atom and wait commit completion; current atom and @txnh have
+ * to be spinlocked before call, this function unlocks them on exit.
+ */
+int force_commit_atom(txn_handle *txnh)
 {
 	txn_atom *atom;
 
@@ -1188,14 +1193,17 @@ static int force_commit_atom_nolock(txn_
 	assert("zam-834", atom != NULL);
 	assert_spin_locked(&(atom->alock));
 
-	/* Set flags for atom and txnh: forcing atom commit and waiting for
-	 * commit completion */
+	/*
+	 * Set flags for atom and txnh: forcing atom commit and waiting for
+	 * commit completion
+	 */
 	txnh->flags |= TXNH_WAIT_COMMIT;
 	atom->flags |= ATOM_FORCE_COMMIT;
 
 	spin_unlock_txnh(txnh);
 	spin_unlock_atom(atom);
 
+	/* commit is here */
 	txn_restart_current();
 	return 0;
 }
@@ -1240,7 +1248,7 @@ int txnmgr_force_commit_all(struct super
 					spin_lock_txnh(txnh);
 					/* Add force-context txnh */
 					capture_assign_txnh_nolock(atom, txnh);
-					ret = force_commit_atom_nolock(txnh);
+					ret = force_commit_atom(txnh);
 					if (ret)
 						return ret;
 				} else
@@ -2541,7 +2549,7 @@ int sync_atom(txn_atom * atom)
 		if (atom->stage < ASTAGE_PRE_COMMIT) {
 			spin_lock_txnh(txnh);
 			capture_assign_txnh_nolock(atom, txnh);
-			result = force_commit_atom_nolock(txnh);
+			result = force_commit_atom(txnh);
 		} else if (atom->stage < ASTAGE_POST_COMMIT) {
 			/* wait atom commit */
 			atom_wait_event(atom);
diff -puN fs/reiser4/txnmgr.h~reiser4-fix-fsync fs/reiser4/txnmgr.h
--- linux-2.6.14-mm2/fs/reiser4/txnmgr.h~reiser4-fix-fsync	2005-11-17 13:42:26.000000000 +0300
+++ linux-2.6.14-mm2-vs/fs/reiser4/txnmgr.h	2005-11-17 13:42:26.000000000 +0300
@@ -416,6 +416,7 @@ extern int current_atom_should_commit(vo
 extern jnode *find_first_dirty_jnode(txn_atom *, int);
 
 extern int commit_some_atoms(txn_mgr *);
+extern int force_commit_atom(txn_handle *);
 extern int flush_current_atom(int, long, long *, txn_atom **, jnode *);
 
 extern int flush_some_atom(jnode *, long *, const struct writeback_control *, int);

_

Re: More Slowdown

[John Gilmore]

On Thursday 17 November 2005 12:40, Vladimir V. Saveliev wrote:
> Please try whether the attached patch improves anything. It simplifies
> fsync by avoid commiting of transactions which do not modify file being
> fsync-ed.
>
> The patch applied to 2.6.14-mm2 with warnings, but that can be ignored.

I haven't tried this patch yet, but I did try the earlier 'disable fsync 
completely' patch. In fact, I'm using it right now.

It doesn't help. Therefore, your patch probably won't help either.

OK, disabling fsync does help a *little* bit. But I think that the issue (for 
me, anyway) isn't sync per se, it's just flat-out access time. Deleteing lots 
of small files is EXTREMELY slow, but even reading files is slower than it 
should be. It took no less that 10 minutes to delete an old kernel source 
tree, for instance. 

It's related to fragmentation, I think. I didn't really notice any speed 
problems until my hard drive got to about 95% (or so) full. But they haven't 
gone away, even though usage is now down around 54%.

Hrm... I should be able to check that...


About an hour later...

So maybe I was wrong. 6.5% data fragmentation doesn't seem that high. But 
19.8% tree fragmentation does seem a bit high. 

How should this data be interpreted?


#measurefs.reiser4 -T /dev/mapper/e-h -f
measurefs.reiser4 1.0.5
Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by 
reiser4progs/COPYING.

Tree fragmentation ... done
0.197747

#measurefs.reiser4 -S -D /dev/mapper/e-h -f
measurefs.reiser4 1.0.5
Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by 
reiser4progs/COPYING.

Data fragmentation ... done
0.065593
Tree statistics ... done
Packing statistics:
  Formatted nodes:    3721.29b (90.85%)
  Branch nodes:       1734.57b (42.35%)
  Twig nodes:         2886.97b (70.48%)
  Leaf nodes:         3814.82b (93.14%)

Node statistics:
  Total nodes:        8543553
  Formatted nodes:     146354
  Unformatted nodes:  8397199
  Branch nodes:           292
  Twig nodes:           10542
  Leaf nodes:         8532719

Item statistics:
  Total items:         637352
  Nodeptr items:       146353
  Statdata items:     191218
  Direntry items:       17512
  Tail items:          245018
  Extent items:         36869


#

Re: More Slowdown

[Hans Reiser]

6% fragmentation is enormous.  You very much need the repacker we have
not yet written.....

remember that one seek and rotate takes ~12 ms, and during that time you
could transfer 50MB*.012 bytes.....

Hans

John Gilmore wrote:

>On Thursday 17 November 2005 12:40, Vladimir V. Saveliev wrote:
>  
>
>>Please try whether the attached patch improves anything. It simplifies
>>fsync by avoid commiting of transactions which do not modify file being
>>fsync-ed.
>>
>>The patch applied to 2.6.14-mm2 with warnings, but that can be ignored.
>>    
>>
>
>I haven't tried this patch yet, but I did try the earlier 'disable fsync 
>completely' patch. In fact, I'm using it right now.
>
>It doesn't help. Therefore, your patch probably won't help either.
>
>OK, disabling fsync does help a *little* bit. But I think that the issue (for 
>me, anyway) isn't sync per se, it's just flat-out access time. Deleteing lots 
>of small files is EXTREMELY slow, but even reading files is slower than it 
>should be. It took no less that 10 minutes to delete an old kernel source 
>tree, for instance. 
>
>It's related to fragmentation, I think. I didn't really notice any speed 
>problems until my hard drive got to about 95% (or so) full. But they haven't 
>gone away, even though usage is now down around 54%.
>
>Hrm... I should be able to check that...
>
>
>About an hour later...
>
>So maybe I was wrong. 6.5% data fragmentation doesn't seem that high. But 
>19.8% tree fragmentation does seem a bit high. 
>
>How should this data be interpreted?
>
>
>#measurefs.reiser4 -T /dev/mapper/e-h -f
>measurefs.reiser4 1.0.5
>Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by 
>reiser4progs/COPYING.
>
>Tree fragmentation ... done
>0.197747
>
>#measurefs.reiser4 -S -D /dev/mapper/e-h -f
>measurefs.reiser4 1.0.5
>Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by 
>reiser4progs/COPYING.
>
>Data fragmentation ... done
>0.065593
>Tree statistics ... done
>Packing statistics:
>  Formatted nodes:    3721.29b (90.85%)
>  Branch nodes:       1734.57b (42.35%)
>  Twig nodes:         2886.97b (70.48%)
>  Leaf nodes:         3814.82b (93.14%)
>
>Node statistics:
>  Total nodes:        8543553
>  Formatted nodes:     146354
>  Unformatted nodes:  8397199
>  Branch nodes:           292
>  Twig nodes:           10542
>  Leaf nodes:         8532719
>
>Item statistics:
>  Total items:         637352
>  Nodeptr items:       146353
>  Statdata items:     191218
>  Direntry items:       17512
>  Tail items:          245018
>  Extent items:         36869
>
>
>#
>
>
>  
>

Re: More Slowdown

[Hans Reiser]

I would like to thank the users for doing so much to help us isolate
this bug.

Vladimir, it seems pretty clear that the problem is due to a code
change, not due to a lack of a feature.  Please isolate it to the
responsible code change, and fix the code change, before adding any
other code changes to speed things up.

Re: More Slowdown

[Ingo Bormuth]

If you are frequently hit by fsync (as some of the previosly posters told us) 
an option might be to say something as

  tmgr.atom_max_age=30     (or tmgr.atom_max_size=?)

in fstab (or remount the drive in vulnerable situations).

Of course that sacrifices large parts of reiser4's mean performance, but that
might be better than a largely unusable machine. One day reiser4 will be
fully optimized.




On 2005-11-16 22:08, michael chang wrote:
> That would explain any
> jerkiness -- it's because Reiser4 is writing everything from the last
> real write to the actual disk to now (the current real right) in one
> go, and making everything wait. 


-- 
Ingo Bormuth, voicebox & telefax: +49-12125-10226517          '(~o-o~)'
public key 86326EC9, http://ibormuth.efil.de/contact     ---ooO--(.)--Ooo---

Re: More Slowdown

[PFC]

	Just a few words about this "slowdown" thing.

	I use linux-2.6.11-cko1-swsusp2 with reiser4 included. I won't upgrade to  
a new version until Hans says the current one is at least as stable as it  
was before starting the merge...

	I get the "slowdown" once in a while, usually for 2-5 seconds, it's not  
that annoying.

	However :

	If I understand well, when memory is needed (memory pressure), reiser4  
triggers a flush and writes dirty pages to the disk in large quantities.
	This has many advantages like not writing at all the temporary files  
which already have been deleted, deferred allocation, etc, it's really  
cool.

	However when "memory pressure" (ie. need to free some RAM) occurs, it is  
usually that I am doing something interactively, like opening a document  
or starting some software. Or it might mean that the database server just  
received a big query which needs some sorting space in RAM for instance.
	Thus, "memory pressure" almost always means "I need memory NOW and  
someone is waiting in front of their screen for it"...
	And it is at this very moment that reiser4 has to flush (to free memory).
	Thus the "flush storm", by nature, always happens when you don't want it  
to happen. It almost never happens when you are away from the computer  
taking a leak, for instance.

	This is analogous to the problem caused to postgres by checkpointing...  
the postgres guys implemented a background writer to solve this.
	I wonder if reiser4 could do the same, ie. trickle down dirty pages to  
free up memory before it is actually needed, to improve reactivity. There  
is a balance to be found, of course, between flushing as late as possible  
(to benefit from all the nifty reiser4 features) and flushing earlier (to  
avoid triggering the "flush storm").
	Maybe this could be set via a few controls...
		- tell reiser4 to try to keep X amount of RAM immediately available  
without flush
		- when the CPU is idle, and/or when the disk is idle, start flushing,  
but stop doing it as soon as some CPU is needed, or a key is hit...
		- just do frequent, small partial flushes which will keep good locality  
of reference while being small
		- do it at a lower priority so that the keyboard does not stop  
responding !!!

	Huh, well, that's it... what do you think ?



> When fsyncing, Reiser4, to my understanding, isn't allowed to put
> stuff in it's memory cache - it must put it to disk right away.  And

	This is the whole point of fsync, yes.
	Now, it's pretty stupid for evlolution to issue a fsync() on every pixel  
move or whatever.
	fsync() is for databases or things which must survive a hard system  
crash...
	evolution could as well have used fflush() and everything would have been  
alright. Dumb.

> For comparative purposes, I hear of consumer systems with 2 or 4 GB of
> ram, and I know of brand new PC hard disks which have a throughput of
> less than 5 MB/s (my hard disk maxees out at 20 MB/s on my PC.)  Do
> you know the throughputs of your drives, and the size of the memory in
> your machines?

	Hum. A crap laptop harddrive will do 15 MBytes/s and a recent normal  
desktop drive (7200rpm ATA or SATA) will do 50 MBytes/s or more...
	If you get a lot less (like, 5 MB/s), you have a problem (DMA disabled,  
etc)

Re: More Slowdown

[Hans Reiser]

PFC wrote:

>
>     Just a few words about this "slowdown" thing.
>
>     I use linux-2.6.11-cko1-swsusp2 with reiser4 included. I won't
> upgrade to  a new version until Hans says the current one is at least
> as stable as it  was before starting the merge...
>
>     I get the "slowdown" once in a while, usually for 2-5 seconds,
> it's not  that annoying.
>
>     However :
>
>     If I understand well, when memory is needed (memory pressure),
> reiser4  triggers a flush and writes dirty pages to the disk in large
> quantities.
>     This has many advantages like not writing at all the temporary
> files  which already have been deleted, deferred allocation, etc, it's
> really  cool.
>
>     However when "memory pressure" (ie. need to free some RAM) occurs,
> it is  usually that I am doing something interactively, like opening a
> document  or starting some software. Or it might mean that the
> database server just  received a big query which needs some sorting
> space in RAM for instance.
>     Thus, "memory pressure" almost always means "I need memory NOW
> and  someone is waiting in front of their screen for it"...
>     And it is at this very moment that reiser4 has to flush (to free
> memory).
>     Thus the "flush storm", by nature, always happens when you don't
> want it  to happen. It almost never happens when you are away from the
> computer  taking a leak, for instance.
>
>     This is analogous to the problem caused to postgres by
> checkpointing...  the postgres guys implemented a background writer to
> solve this.
>     I wonder if reiser4 could do the same, ie. trickle down dirty
> pages to  free up memory before it is actually needed, to improve
> reactivity. There  is a balance to be found, of course, between
> flushing as late as possible  (to benefit from all the nifty reiser4
> features) and flushing earlier (to  avoid triggering the "flush storm").
>     Maybe this could be set via a few controls...
>         - tell reiser4 to try to keep X amount of RAM immediately
> available  without flush
>         - when the CPU is idle, and/or when the disk is idle, start
> flushing,  but stop doing it as soon as some CPU is needed, or a key
> is hit...
>         - just do frequent, small partial flushes which will keep good
> locality  of reference while being small
>         - do it at a lower priority so that the keyboard does not
> stop  responding !!!

I agree that smoothness needs working on.  After we merge.

>
>     Huh, well, that's it... what do you think ?
>
>
>
>> When fsyncing, Reiser4, to my understanding, isn't allowed to put
>> stuff in it's memory cache - it must put it to disk right away.  And
>
>
>     This is the whole point of fsync, yes.
>     Now, it's pretty stupid for evlolution to issue a fsync() on every
> pixel  move or whatever.
>     fsync() is for databases or things which must survive a hard
> system  crash...
>     evolution could as well have used fflush() and everything would
> have been  alright. Dumb.
>
>> For comparative purposes, I hear of consumer systems with 2 or 4 GB of
>> ram, and I know of brand new PC hard disks which have a throughput of
>> less than 5 MB/s (my hard disk maxees out at 20 MB/s on my PC.)  Do
>> you know the throughputs of your drives, and the size of the memory in
>> your machines?
>
>
>     Hum. A crap laptop harddrive will do 15 MBytes/s and a recent
> normal  desktop drive (7200rpm ATA or SATA) will do 50 MBytes/s or
> more...
>     If you get a lot less (like, 5 MB/s), you have a problem (DMA
> disabled,  etc)
>
>
>
>
PFC is right.

Re: More Slowdown

[michael chang]

On 11/17/05, Hans Reiser <reiser@namesys.com> wrote:
> PFC wrote:
>
> >
> >     Just a few words about this "slowdown" thing.
> >
> >     I use linux-2.6.11-cko1-swsusp2 with reiser4 included. I won't
> > upgrade to  a new version until Hans says the current one is at least
> > as stable as it  was before starting the merge...
> >
> >     I get the "slowdown" once in a while, usually for 2-5 seconds,
> > it's not  that annoying.
> >
> >     However :
> >
> >     If I understand well, when memory is needed (memory pressure),
> > reiser4  triggers a flush and writes dirty pages to the disk in large
> > quantities.
> >     This has many advantages like not writing at all the temporary
> > files  which already have been deleted, deferred allocation, etc, it's
> > really  cool.
> >
> >     However when "memory pressure" (ie. need to free some RAM) occurs,
> > it is  usually that I am doing something interactively, like opening a
> > document  or starting some software. Or it might mean that the
> > database server just  received a big query which needs some sorting
> > space in RAM for instance.
> >     Thus, "memory pressure" almost always means "I need memory NOW
> > and  someone is waiting in front of their screen for it"...
> >     And it is at this very moment that reiser4 has to flush (to free
> > memory).
> >     Thus the "flush storm", by nature, always happens when you don't
> > want it  to happen. It almost never happens when you are away from the
> > computer  taking a leak, for instance.
> >
> >     This is analogous to the problem caused to postgres by
> > checkpointing...  the postgres guys implemented a background writer to
> > solve this.
> >     I wonder if reiser4 could do the same, ie. trickle down dirty
> > pages to  free up memory before it is actually needed, to improve
> > reactivity. There  is a balance to be found, of course, between
> > flushing as late as possible  (to benefit from all the nifty reiser4
> > features) and flushing earlier (to  avoid triggering the "flush storm").
> >     Maybe this could be set via a few controls...
> >         - tell reiser4 to try to keep X amount of RAM immediately
> > available  without flush
> >         - when the CPU is idle, and/or when the disk is idle, start
> > flushing,  but stop doing it as soon as some CPU is needed, or a key
> > is hit...
> >         - just do frequent, small partial flushes which will keep good
> > locality  of reference while being small
> >         - do it at a lower priority so that the keyboard does not
> > stop  responding !!!
>
> I agree that smoothness needs working on.  After we merge.
>
> >
> >     Huh, well, that's it... what do you think ?
> >
> >
> >
> >> When fsyncing, Reiser4, to my understanding, isn't allowed to put
> >> stuff in it's memory cache - it must put it to disk right away.  And
> >
> >
> >     This is the whole point of fsync, yes.
> >     Now, it's pretty stupid for evlolution to issue a fsync() on every
> > pixel  move or whatever.
> >     fsync() is for databases or things which must survive a hard
> > system  crash...
> >     evolution could as well have used fflush() and everything would
> > have been  alright. Dumb.
> >
> >> For comparative purposes, I hear of consumer systems with 2 or 4 GB of
> >> ram, and I know of brand new PC hard disks which have a throughput of
> >> less than 5 MB/s (my hard disk maxees out at 20 MB/s on my PC.)  Do
> >> you know the throughputs of your drives, and the size of the memory in
> >> your machines?
> >
> >
> >     Hum. A crap laptop harddrive will do 15 MBytes/s and a recent
> > normal  desktop drive (7200rpm ATA or SATA) will do 50 MBytes/s or
> > more...
> >     If you get a lot less (like, 5 MB/s), you have a problem (DMA
> > disabled,  etc)
> >
> >
> >
> >
> PFC is right.
>

*shrugs* 15-30 MB/s on a recent low-end desktop from Dell (not that
old, maybe 6-12 months) that I own with DMA enabled; the other
machines are because people I know are using older PCs and can't
afford to upgrade them at the moment.  (Ironically, sometimes, those
PCs /feel/ faster, for some reason...)  Of course, a 7200rpm ATA/SATA
drive would be the best; but I just try and make do, as do people I
know.  But that's life; there will always be those on the bottom of
the heap. :)

--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Versioning Plugin

[Peter van Hardenberg]

On November 11, 2005 05:59 am, John Gilmore wrote:
> Does anybody remember GoBack? It was a versioning
> system for windows 95/98 that was incredibly flexible and useful. Tracked
> all changes to the whole disk. Old versions of a file? no problem. grab an
> old version of a directory for referance temporarily? easy. Got a virus?
> revert the whole HD, and then grab the newer copies of your documents and
> saved games as needed.

My thoughts on this:

The versioning would be an audit plugin. When the file is modified, tag the 
current version, copy it into a sub-directory (oh, I don't know, say 
file/.revisions/<number/date>), and disable write access to it. You might not 
even need extended filesystem attributes for this, but they would be handy 
for tagging particular versions.

Copy-on-write would make this action extremely cheap, only adding a couple of 
extra writes to make it work.

Given working resource directories, COW, and the ability to set plugins, this 
might be a relatively easy hack to implement. Given an efficient xpath shell, 
you could even create a view of your drive on a particular day. 

If you had a file that was changing often, perhaps you could set an attribute 
on that file which told it only to clone the file every once in a while. 

Come to think of it, a userspace daemon could run in the background and 
replace the need for a plugin, which is probably the better solution. Then 
you just need COW and files which can contain resources.

-pvh

-- 
Peter van Hardenberg (pvh@pvh.ca)
Victoria, BC, Canada

Re: Versioning Plugin

[Hans Reiser]

Peter van Hardenberg wrote:

>On November 11, 2005 05:59 am, John Gilmore wrote:
>  
>
>>Does anybody remember GoBack? It was a versioning
>>system for windows 95/98 that was incredibly flexible and useful. Tracked
>>all changes to the whole disk. Old versions of a file? no problem. grab an
>>old version of a directory for referance temporarily? easy. Got a virus?
>>revert the whole HD, and then grab the newer copies of your documents and
>>saved games as needed.
>>    
>>
>
>My thoughts on this:
>
>The versioning would be an audit plugin. When the file is modified, tag the 
>current version, copy it into a sub-directory (oh, I don't know, say 
>file/.revisions/<number/date>), and disable write access to it. You might not 
>even need extended filesystem attributes for this, but they would be handy 
>for tagging particular versions.
>
>Copy-on-write would make this action extremely cheap, only adding a couple of 
>extra writes to make it work.
>
>Given working resource directories, COW, and the ability to set plugins, this 
>might be a relatively easy hack to implement. Given an efficient xpath shell, 
>you could even create a view of your drive on a particular day. 
>
>If you had a file that was changing often, perhaps you could set an attribute 
>on that file which told it only to clone the file every once in a while. 
>
>Come to think of it, a userspace daemon could run in the background and 
>replace the need for a plugin, which is probably the better solution. Then 
>you just need COW and files which can contain resources.
>
>-pvh
>
>  
>
Yup, sounds good to me, it just needs someone to write it.   A later
version could implement a special compression plugin that spanned the
multiple file versions.....

Hans

Re: Versioning Plugin

[Ming Zhang]

On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
> On November 11, 2005 05:59 am, John Gilmore wrote:
> > Does anybody remember GoBack? It was a versioning
> > system for windows 95/98 that was incredibly flexible and useful. Tracked
> > all changes to the whole disk. Old versions of a file? no problem. grab an
> > old version of a directory for referance temporarily? easy. Got a virus?
> > revert the whole HD, and then grab the newer copies of your documents and
> > saved games as needed.
> 
> My thoughts on this:
> 
> The versioning would be an audit plugin. When the file is modified, tag the 
> current version, copy it into a sub-directory (oh, I don't know, say 
> file/.revisions/<number/date>), and disable write access to it. You might not 
> even need extended filesystem attributes for this, but they would be handy 
> for tagging particular versions.

if a file is opened, modified 2 times, then closed. u will only generate
1 version right? so "When the file is modified" is inaccurate.

> 
> Copy-on-write would make this action extremely cheap, only adding a couple of 
> extra writes to make it work.

add 1 line at the beginning of a 100MB text file will make this uncheap.
all data are shifted.


> 
> Given working resource directories, COW, and the ability to set plugins, this 
> might be a relatively easy hack to implement. Given an efficient xpath shell, 
> you could even create a view of your drive on a particular day. 
> 
> If you had a file that was changing often, perhaps you could set an attribute 
> on that file which told it only to clone the file every once in a while. 
> 
> Come to think of it, a userspace daemon could run in the background and 
> replace the need for a plugin, which is probably the better solution. Then 
> you just need COW and files which can contain resources.
> 
> -pvh
> 

Re: Versioning Plugin

[David Masover]

Ming Zhang wrote:
> On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
> 
>>On November 11, 2005 05:59 am, John Gilmore wrote:
>>
>>>Does anybody remember GoBack? It was a versioning
>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
>>>all changes to the whole disk. Old versions of a file? no problem. grab an
>>>old version of a directory for referance temporarily? easy. Got a virus?
>>>revert the whole HD, and then grab the newer copies of your documents and
>>>saved games as needed.
>>
>>My thoughts on this:
>>
>>The versioning would be an audit plugin. When the file is modified, tag the 
>>current version, copy it into a sub-directory (oh, I don't know, say 
>>file/.revisions/<number/date>), and disable write access to it. You might not 
>>even need extended filesystem attributes for this, but they would be handy 
>>for tagging particular versions.
> 
> 
> if a file is opened, modified 2 times, then closed. u will only generate
> 1 version right? so "When the file is modified" is inaccurate.

How about "When the transaction was completed?"  Why does it matter?

>>Copy-on-write would make this action extremely cheap, only adding a couple of 
>>extra writes to make it work.
> 
> 
> add 1 line at the beginning of a 100MB text file will make this uncheap.

Who has to work with 100 meg text files?  And why has this person not
broken them down into 100 kilobyte text files?  Storage efficiency isn't
really an issue there...

Anyway, I think the main win is from copy-on-write for the whole file.

Re: Versioning Plugin

[Ming Zhang]

On Sat, 2005-11-12 at 15:46 -0600, David Masover wrote:
> Ming Zhang wrote:
> > On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
> > 
> >>On November 11, 2005 05:59 am, John Gilmore wrote:
> >>
> >>>Does anybody remember GoBack? It was a versioning
> >>>system for windows 95/98 that was incredibly flexible and useful. Tracked
> >>>all changes to the whole disk. Old versions of a file? no problem. grab an
> >>>old version of a directory for referance temporarily? easy. Got a virus?
> >>>revert the whole HD, and then grab the newer copies of your documents and
> >>>saved games as needed.
> >>
> >>My thoughts on this:
> >>
> >>The versioning would be an audit plugin. When the file is modified, tag the 
> >>current version, copy it into a sub-directory (oh, I don't know, say 
> >>file/.revisions/<number/date>), and disable write access to it. You might not 
> >>even need extended filesystem attributes for this, but they would be handy 
> >>for tagging particular versions.
> > 
> > 
> > if a file is opened, modified 2 times, then closed. u will only generate
> > 1 version right? so "When the file is modified" is inaccurate.
> 
> How about "When the transaction was completed?"  Why does it matter?

then how u define a transaction? i mean we first need to choose a good
event/period to define what is a good meaningful version.


> 
> >>Copy-on-write would make this action extremely cheap, only adding a couple of 
> >>extra writes to make it work.
> > 
> > 
> > add 1 line at the beginning of a 100MB text file will make this uncheap.
> 
> Who has to work with 100 meg text files?  And why has this person not
> broken them down into 100 kilobyte text files?  Storage efficiency isn't
> really an issue there...

yes, 100MB/s text file is an extreme example, but a common case can be u
delete 1 frame in a streaming media file. basically, a cow is not good
for a data shift situation. u have >99% data unchanged, just their
offset in file is changed. this lead to all blocks changed, then COW
will need to copy a lot.


> 
> Anyway, I think the main win is from copy-on-write for the whole file.
> 

Re: Versioning Plugin

[David Masover]

Ming Zhang wrote:
> On Sat, 2005-11-12 at 15:46 -0600, David Masover wrote:
> 
>>Ming Zhang wrote:
>>
>>>On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
>>>
>>>
>>>>On November 11, 2005 05:59 am, John Gilmore wrote:
>>>>
>>>>
>>>>>Does anybody remember GoBack? It was a versioning
>>>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
>>>>>all changes to the whole disk. Old versions of a file? no problem. grab an
>>>>>old version of a directory for referance temporarily? easy. Got a virus?
>>>>>revert the whole HD, and then grab the newer copies of your documents and
>>>>>saved games as needed.
>>>>
>>>>My thoughts on this:
>>>>
>>>>The versioning would be an audit plugin. When the file is modified, tag the 
>>>>current version, copy it into a sub-directory (oh, I don't know, say 
>>>>file/.revisions/<number/date>), and disable write access to it. You might not 
>>>>even need extended filesystem attributes for this, but they would be handy 
>>>>for tagging particular versions.
>>>
>>>
>>>if a file is opened, modified 2 times, then closed. u will only generate
>>>1 version right? so "When the file is modified" is inaccurate.
>>
>>How about "When the transaction was completed?"  Why does it matter?
> 
> 
> then how u define a transaction? i mean we first need to choose a good
> event/period to define what is a good meaningful version.
> 
> 
> 
>>>>Copy-on-write would make this action extremely cheap, only adding a couple of 
>>>>extra writes to make it work.
>>>
>>>
>>>add 1 line at the beginning of a 100MB text file will make this uncheap.
>>
>>Who has to work with 100 meg text files?  And why has this person not
>>broken them down into 100 kilobyte text files?  Storage efficiency isn't
>>really an issue there...
> 
> 
> yes, 100MB/s text file is an extreme example, but a common case can be u
> delete 1 frame in a streaming media file.

What do you mean by "streaming"?  (To me, "streaming media" usually
means "over the Internet", which makes no sense here.)

> basically, a cow is not good
> for a data shift situation. u have >99% data unchanged, just their
> offset in file is changed. this lead to all blocks changed, then COW
> will need to copy a lot.

When do you have a data shift situation where this is significant enough
to impact COW, but not significant enough to affect normal performance?

As far as I know, *nix has no way to append to the beginning of a file,
so if you're editing a large video file, say several gigs of DVD, you
have to write out several gigs worth of data all over again because you
want it shifted.

The filesystem may eventually provide more intelligent ways of messing
with a file, and the COW system should be able to handle when a program
appends to or chops off the beginning of a file.

Until then, we can rely somewhat on programs optimizing for speed --
rather than rewrite several gigs, it could split the file into smaller
files (thus, only the file which was changed is copied), or make it a
sort of mini-FS in that it fragments the logical structure of the file
so that it writes as little as possible -- for instance, inserting a
clip in the middle might write to the end of a "project" file, instead
of shifting half of that file over first.  You'd keep versions of the
project file, not the stream (properly defragmented) you'd export when
you're done.

For cases where developers didn't have to deal with the speed issues, we
don't have to worry about it.  In the case of audio editing, if it's
actually messing with the sound itself, no COW in the world will catch
that.  If it's a mixing/sequencing program, that's usually stored as a
"project", accompanied by lots of little WAV files, which don't change,
and a tiny "project" file describing how they go together, which does
change.

And for text files and office documents, the sizes just aren't usually
enough for us to care.  My biggest OpenOffice.org document probably
isn't a hundred kilobytes, and my disk space is measured in gigabytes.
It'd take over ten thousand revisions to fill a gig with copies of one
of those files.  Sure, we could make an Oasis plugin for OO.o to use, so
all the contents of the document are stored as individual files, turned
into a zipfile on demand to match the current standard -- but that's not
worth it in the short term, and only really helps with presentations in
the long term.

Actually, while I think it'd be nice to be able to more advanced
splicing in a file (append or delete from the beginning or middle), I
think it's more important to come up with a sane way for a program to
access a file as a lot of little pieces, and to have a standard way of
serializing them for transport (email or otherwise).  Kind of like XML,
only it could be more efficient than the old model, instead of less.

Like XML in that XML allows programmers to dump internal structures to a
human-readable file without writing parsers and serializers.  Move the
serializing logic out to the FS, let it handle the performance, version
control, and export issues.

Re: Versioning Plugin

[Ming Zhang]

On Sat, 2005-11-12 at 20:56 -0600, David Masover wrote:
> Ming Zhang wrote:
> > On Sat, 2005-11-12 at 15:46 -0600, David Masover wrote:
> > 
> >>Ming Zhang wrote:
> >>
> >>>On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
> >>>
> >>>
> >>>>On November 11, 2005 05:59 am, John Gilmore wrote:
> >>>>
> >>>>
> >>>>>Does anybody remember GoBack? It was a versioning
> >>>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
> >>>>>all changes to the whole disk. Old versions of a file? no problem. grab an
> >>>>>old version of a directory for referance temporarily? easy. Got a virus?
> >>>>>revert the whole HD, and then grab the newer copies of your documents and
> >>>>>saved games as needed.
> >>>>
> >>>>My thoughts on this:
> >>>>
> >>>>The versioning would be an audit plugin. When the file is modified, tag the 
> >>>>current version, copy it into a sub-directory (oh, I don't know, say 
> >>>>file/.revisions/<number/date>), and disable write access to it. You might not 
> >>>>even need extended filesystem attributes for this, but they would be handy 
> >>>>for tagging particular versions.
> >>>
> >>>
> >>>if a file is opened, modified 2 times, then closed. u will only generate
> >>>1 version right? so "When the file is modified" is inaccurate.
> >>
> >>How about "When the transaction was completed?"  Why does it matter?
> > 
> > 
> > then how u define a transaction? i mean we first need to choose a good
> > event/period to define what is a good meaningful version.
> > 
> > 
> > 
> >>>>Copy-on-write would make this action extremely cheap, only adding a couple of 
> >>>>extra writes to make it work.
> >>>
> >>>
> >>>add 1 line at the beginning of a 100MB text file will make this uncheap.
> >>
> >>Who has to work with 100 meg text files?  And why has this person not
> >>broken them down into 100 kilobyte text files?  Storage efficiency isn't
> >>really an issue there...
> > 
> > 
> > yes, 100MB/s text file is an extreme example, but a common case can be u
> > delete 1 frame in a streaming media file.
> 
> What do you mean by "streaming"?  (To me, "streaming media" usually
> means "over the Internet", which makes no sense here.)

what i mean is frame is independent from each other, so when u delete
one frame, other frame data keep unchanged, like change ABCDEFG from
ACDEFG.


> 
> > basically, a cow is not good
> > for a data shift situation. u have >99% data unchanged, just their
> > offset in file is changed. this lead to all blocks changed, then COW
> > will need to copy a lot.
> 
> When do you have a data shift situation where this is significant enough
> to impact COW, but not significant enough to affect normal performance?
> 
> As far as I know, *nix has no way to append to the beginning of a file,
> so if you're editing a large video file, say several gigs of DVD, you
> have to write out several gigs worth of data all over again because you
> want it shifted.

yes, this is also what i know. thanks for u analysis, i now agree that
COW should be ok for this case, considering the overhead.

but another issue about COW is that when u have lots of versions, any
write to original data will lead a lot of new writings to these COW
storage.

any place i can find document about how to write a plugin for reiser?
sounds like interesting. :P

ming

> 
> The filesystem may eventually provide more intelligent ways of messing
> with a file, and the COW system should be able to handle when a program
> appends to or chops off the beginning of a file.
> 
> Until then, we can rely somewhat on programs optimizing for speed --
> rather than rewrite several gigs, it could split the file into smaller
> files (thus, only the file which was changed is copied), or make it a
> sort of mini-FS in that it fragments the logical structure of the file
> so that it writes as little as possible -- for instance, inserting a
> clip in the middle might write to the end of a "project" file, instead
> of shifting half of that file over first.  You'd keep versions of the
> project file, not the stream (properly defragmented) you'd export when
> you're done.
> 
> For cases where developers didn't have to deal with the speed issues, we
> don't have to worry about it.  In the case of audio editing, if it's
> actually messing with the sound itself, no COW in the world will catch
> that.  If it's a mixing/sequencing program, that's usually stored as a
> "project", accompanied by lots of little WAV files, which don't change,
> and a tiny "project" file describing how they go together, which does
> change.
> 
> And for text files and office documents, the sizes just aren't usually
> enough for us to care.  My biggest OpenOffice.org document probably
> isn't a hundred kilobytes, and my disk space is measured in gigabytes.
> It'd take over ten thousand revisions to fill a gig with copies of one
> of those files.  Sure, we could make an Oasis plugin for OO.o to use, so
> all the contents of the document are stored as individual files, turned
> into a zipfile on demand to match the current standard -- but that's not
> worth it in the short term, and only really helps with presentations in
> the long term.
> 
> Actually, while I think it'd be nice to be able to more advanced
> splicing in a file (append or delete from the beginning or middle), I
> think it's more important to come up with a sane way for a program to
> access a file as a lot of little pieces, and to have a standard way of
> serializing them for transport (email or otherwise).  Kind of like XML,
> only it could be more efficient than the old model, instead of less.
> 
> Like XML in that XML allows programmers to dump internal structures to a
> human-readable file without writing parsers and serializers.  Move the
> serializing logic out to the FS, let it handle the performance, version
> control, and export issues.

Re: Versioning Plugin

[Hans Reiser]

Ming Zhang wrote:

>On Sat, 2005-11-12 at 20:56 -0600, David Masover wrote:
>  
>
>>Ming Zhang wrote:
>>    
>>
>>>On Sat, 2005-11-12 at 15:46 -0600, David Masover wrote:
>>>
>>>      
>>>
>>>>Ming Zhang wrote:
>>>>
>>>>        
>>>>
>>>>>On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>On November 11, 2005 05:59 am, John Gilmore wrote:
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Does anybody remember GoBack? It was a versioning
>>>>>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
>>>>>>>all changes to the whole disk. Old versions of a file? no problem. grab an
>>>>>>>old version of a directory for referance temporarily? easy. Got a virus?
>>>>>>>revert the whole HD, and then grab the newer copies of your documents and
>>>>>>>saved games as needed.
>>>>>>>              
>>>>>>>
>>>>>>My thoughts on this:
>>>>>>
>>>>>>The versioning would be an audit plugin. When the file is modified, tag the 
>>>>>>current version, copy it into a sub-directory (oh, I don't know, say 
>>>>>>file/.revisions/<number/date>), and disable write access to it. You might not 
>>>>>>even need extended filesystem attributes for this, but they would be handy 
>>>>>>for tagging particular versions.
>>>>>>            
>>>>>>
>>>>>if a file is opened, modified 2 times, then closed. u will only generate
>>>>>1 version right? so "When the file is modified" is inaccurate.
>>>>>          
>>>>>
>>>>How about "When the transaction was completed?"  Why does it matter?
>>>>        
>>>>
>>>then how u define a transaction? i mean we first need to choose a good
>>>event/period to define what is a good meaningful version.
>>>
>>>
>>>
>>>      
>>>
>>>>>>Copy-on-write would make this action extremely cheap, only adding a couple of 
>>>>>>extra writes to make it work.
>>>>>>            
>>>>>>
>>>>>add 1 line at the beginning of a 100MB text file will make this uncheap.
>>>>>          
>>>>>
>>>>Who has to work with 100 meg text files?  And why has this person not
>>>>broken them down into 100 kilobyte text files?  Storage efficiency isn't
>>>>really an issue there...
>>>>        
>>>>
>>>yes, 100MB/s text file is an extreme example, but a common case can be u
>>>delete 1 frame in a streaming media file.
>>>      
>>>
>>What do you mean by "streaming"?  (To me, "streaming media" usually
>>means "over the Internet", which makes no sense here.)
>>    
>>
>
>what i mean is frame is independent from each other, so when u delete
>one frame, other frame data keep unchanged, like change ABCDEFG from
>ACDEFG.
>
>
>  
>
>>>basically, a cow is not good
>>>for a data shift situation. u have >99% data unchanged, just their
>>>offset in file is changed. this lead to all blocks changed, then COW
>>>will need to copy a lot.
>>>      
>>>
>>When do you have a data shift situation where this is significant enough
>>to impact COW, but not significant enough to affect normal performance?
>>
>>As far as I know, *nix has no way to append to the beginning of a file,
>>so if you're editing a large video file, say several gigs of DVD, you
>>have to write out several gigs worth of data all over again because you
>>want it shifted.
>>    
>>
>
>yes, this is also what i know. thanks for u analysis, i now agree that
>COW should be ok for this case, considering the overhead.
>
>but another issue about COW is that when u have lots of versions, any
>write to original data will lead a lot of new writings to these COW
>storage.
>
>any place i can find document about how to write a plugin for reiser?
>sounds like interesting. :P
>
>ming
>
>  
>
>>The filesystem may eventually provide more intelligent ways of messing
>>with a file, and the COW system should be able to handle when a program
>>appends to or chops off the beginning of a file.
>>
>>Until then, we can rely somewhat on programs optimizing for speed --
>>rather than rewrite several gigs, it could split the file into smaller
>>files (thus, only the file which was changed is copied), or make it a
>>sort of mini-FS in that it fragments the logical structure of the file
>>so that it writes as little as possible -- for instance, inserting a
>>clip in the middle might write to the end of a "project" file, instead
>>of shifting half of that file over first.  You'd keep versions of the
>>project file, not the stream (properly defragmented) you'd export when
>>you're done.
>>
>>For cases where developers didn't have to deal with the speed issues, we
>>don't have to worry about it.  In the case of audio editing, if it's
>>actually messing with the sound itself, no COW in the world will catch
>>that.  If it's a mixing/sequencing program, that's usually stored as a
>>"project", accompanied by lots of little WAV files, which don't change,
>>and a tiny "project" file describing how they go together, which does
>>change.
>>
>>And for text files and office documents, the sizes just aren't usually
>>enough for us to care.  My biggest OpenOffice.org document probably
>>isn't a hundred kilobytes, and my disk space is measured in gigabytes.
>>It'd take over ten thousand revisions to fill a gig with copies of one
>>of those files.  Sure, we could make an Oasis plugin for OO.o to use, so
>>all the contents of the document are stored as individual files, turned
>>into a zipfile on demand to match the current standard -- but that's not
>>worth it in the short term, and only really helps with presentations in
>>the long term.
>>
>>Actually, while I think it'd be nice to be able to more advanced
>>splicing in a file (append or delete from the beginning or middle), I
>>think it's more important to come up with a sane way for a program to
>>access a file as a lot of little pieces, and to have a standard way of
>>serializing them for transport (email or otherwise).  Kind of like XML,
>>only it could be more efficient than the old model, instead of less.
>>
>>Like XML in that XML allows programmers to dump internal structures to a
>>human-readable file without writing parsers and serializers.  Move the
>>serializing logic out to the FS, let it handle the performance, version
>>control, and export issues.
>>    
>>
>
>
>
>  
>
well, frames should be handled by inheritance, because there are times
you want to see them as separate objects....

Re: Versioning Plugin

[Hans Reiser]

Insertion optimized files are an ok plugin with me, just create an
insert operation in sys_reiser4 and an insertion optimized plugin.

Re: Versioning Plugin

[Hans Reiser]

David Masover wrote:

>Ming Zhang wrote:
>  
>
>>On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
>>
>>    
>>
>>>On November 11, 2005 05:59 am, John Gilmore wrote:
>>>
>>>      
>>>
>>>>Does anybody remember GoBack? It was a versioning
>>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
>>>>all changes to the whole disk. Old versions of a file? no problem. grab an
>>>>old version of a directory for referance temporarily? easy. Got a virus?
>>>>revert the whole HD, and then grab the newer copies of your documents and
>>>>saved games as needed.
>>>>        
>>>>
>>>My thoughts on this:
>>>
>>>The versioning would be an audit plugin. When the file is modified, tag the 
>>>current version, copy it into a sub-directory (oh, I don't know, say 
>>>file/.revisions/<number/date>), and disable write access to it. You might not 
>>>even need extended filesystem attributes for this, but they would be handy 
>>>for tagging particular versions.
>>>      
>>>
>>if a file is opened, modified 2 times, then closed. u will only generate
>>1 version right? so "When the file is modified" is inaccurate.
>>    
>>
one could do it for every file close, and that could be a state option
for the versioning plugin, but most users will want to do it everytime
they touch filename/..../checkin

>
>How about "When the transaction was completed?"  Why does it matter?
>
>  
>
>>>Copy-on-write would make this action extremely cheap, only adding a couple of 
>>>extra writes to make it work.
>>>      
>>>
>>add 1 line at the beginning of a 100MB text file will make this uncheap.
>>    
>>
>
>Who has to work with 100 meg text files?  And why has this person not
>broken them down into 100 kilobyte text files?  Storage efficiency isn't
>really an issue there...
>  
>
you need cross-version compression for this case. 

>Anyway, I think the main win is from copy-on-write for the whole file.
>
>
>
>  
>

Re: Versioning Plugin

[Ming Zhang]

On Sat, 2005-11-12 at 14:54 -0800, Hans Reiser wrote:
> David Masover wrote:
> 
> >Ming Zhang wrote:
> >  
> >
> >>On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
> >>
> >>    
> >>
> >>>On November 11, 2005 05:59 am, John Gilmore wrote:
> >>>
> >>>      
> >>>
> >>>>Does anybody remember GoBack? It was a versioning
> >>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
> >>>>all changes to the whole disk. Old versions of a file? no problem. grab an
> >>>>old version of a directory for referance temporarily? easy. Got a virus?
> >>>>revert the whole HD, and then grab the newer copies of your documents and
> >>>>saved games as needed.
> >>>>        
> >>>>
> >>>My thoughts on this:
> >>>
> >>>The versioning would be an audit plugin. When the file is modified, tag the 
> >>>current version, copy it into a sub-directory (oh, I don't know, say 
> >>>file/.revisions/<number/date>), and disable write access to it. You might not 
> >>>even need extended filesystem attributes for this, but they would be handy 
> >>>for tagging particular versions.
> >>>      
> >>>
> >>if a file is opened, modified 2 times, then closed. u will only generate
> >>1 version right? so "When the file is modified" is inaccurate.
> >>    
> >>
> one could do it for every file close, and that could be a state option
> for the versioning plugin, but most users will want to do it everytime
> they touch filename/..../checkin

what u mean touch filename? is "ls" a touch? i think close, unlink,
create, is likely to be good candidate.

> 
> >
> >How about "When the transaction was completed?"  Why does it matter?
> >
> >  
> >
> >>>Copy-on-write would make this action extremely cheap, only adding a couple of 
> >>>extra writes to make it work.
> >>>      
> >>>
> >>add 1 line at the beginning of a 100MB text file will make this uncheap.
> >>    
> >>
> >
> >Who has to work with 100 meg text files?  And why has this person not
> >broken them down into 100 kilobyte text files?  Storage efficiency isn't
> >really an issue there...
> >  
> >
> you need cross-version compression for this case. 

what u mean cross-version compression? interesting name. :P

ming

> 
> >Anyway, I think the main win is from copy-on-write for the whole file.
> >
> >
> >
> >  
> >
> 

Re: Versioning Plugin

[michael chang]

On 11/12/05, Ming Zhang <mingz@ele.uri.edu> wrote:
> On Sat, 2005-11-12 at 14:54 -0800, Hans Reiser wrote:
> > David Masover wrote:
> >
> > >Ming Zhang wrote:
> > >
> > >
> > >>On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
> > >>
> > >>
> > >>
> > >>>On November 11, 2005 05:59 am, John Gilmore wrote:
> > >>>
> > >>>
> > >>>
> > >>>>Does anybody remember GoBack? It was a versioning
> > >>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
> > >>>>all changes to the whole disk. Old versions of a file? no problem. grab an
> > >>>>old version of a directory for referance temporarily? easy. Got a virus?
> > >>>>revert the whole HD, and then grab the newer copies of your documents and
> > >>>>saved games as needed.
> > >>>>
> > >>>>
> > >>>My thoughts on this:
> > >>>
> > >>>The versioning would be an audit plugin. When the file is modified, tag the
> > >>>current version, copy it into a sub-directory (oh, I don't know, say
> > >>>file/.revisions/<number/date>), and disable write access to it. You might not
> > >>>even need extended filesystem attributes for this, but they would be handy
> > >>>for tagging particular versions.
> > >>>
> > >>>
> > >>if a file is opened, modified 2 times, then closed. u will only generate
> > >>1 version right? so "When the file is modified" is inaccurate.
> > >>
> > >>
> > one could do it for every file close, and that could be a state option
> > for the versioning plugin, but most users will want to do it everytime
> > they touch filename/..../checkin
>
> what u mean touch filename? is "ls" a touch? i think close, unlink,
> create, is likely to be good candidate.

What happens if I open, truncate, append, close?  You have to consider
that... in fact, what about just "open, append, close"?  Not every app
acts the same.

> >
> > >
> > >How about "When the transaction was completed?"  Why does it matter?
> > >
> > >
> > >
> > >>>Copy-on-write would make this action extremely cheap, only adding a couple of
> > >>>extra writes to make it work.
> > >>>
> > >>>
> > >>add 1 line at the beginning of a 100MB text file will make this uncheap.
> > >>
> > >>
> > >
> > >Who has to work with 100 meg text files?  And why has this person not
> > >broken them down into 100 kilobyte text files?  Storage efficiency isn't
> > >really an issue there...
> > >
> > >
> > you need cross-version compression for this case.
>
> what u mean cross-version compression? interesting name. :P

Compression of the files' different versions together; see one of
Hans' previous posts in this thread (in the archive or otherwise). 
-.-'

That way, if there is version X which is a file, and verison Y is just
a line at the top, the compression eliminates the duplication, so
instead of

(old version + new version)
ABCDEF + XABCDEF
 on disk
it becomes something like
Y=ABCDEF
X + XY
 on disk  [note, I don't know squat about this, so an expert might
tell differently -- if so, believe him, because he'll probably be the
guy implementing this, when it/if gets implemented]

[not literally... hopefully you get what i mean by that]

> >
> > >Anyway, I think the main win is from copy-on-write for the whole file.
> > >
> > >
> > >
> > >
> > >
> >
>
>


--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: Versioning Plugin

[michael chang]

On 11/12/05, michael chang <thenewme91@gmail.com> wrote:
> That way, if there is version X which is a file, and verison Y is just
> a line at the top, the compression eliminates the duplication, so
> instead of
>
> (old version + new version)
> ABCDEF + XABCDEF
>  on disk
> it becomes something like
> Y=ABCDEF

correction:
> X + XY

should be
Y + XY [sorry for any confusion]

--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: Versioning Plugin

[Ming Zhang]

On Sat, 2005-11-12 at 20:56 -0500, michael chang wrote:
> On 11/12/05, michael chang <thenewme91@gmail.com> wrote:
> > That way, if there is version X which is a file, and verison Y is just
> > a line at the top, the compression eliminates the duplication, so
> > instead of
> >
> > (old version + new version)
> > ABCDEF + XABCDEF
> >  on disk
> > it becomes something like
> > Y=ABCDEF
> 
> correction:
> > X + XY
> 
> should be
> Y + XY [sorry for any confusion]

thanks!

> 
> --
> ~Mike
>  - Just my two cents
>  - No man is an island, and no man is unable.

Re: Versioning Plugin

[Hans Reiser]

I should say, any write, to filename/version

Re: Versioning Plugin

[Ming Zhang]

On Sat, 2005-11-12 at 20:55 -0500, michael chang wrote:
> On 11/12/05, Ming Zhang <mingz@ele.uri.edu> wrote:
> > On Sat, 2005-11-12 at 14:54 -0800, Hans Reiser wrote:
> > > David Masover wrote:
> > >
> > > >Ming Zhang wrote:
> > > >
> > > >
> > > >>On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
> > > >>
> > > >>
> > > >>
> > > >>>On November 11, 2005 05:59 am, John Gilmore wrote:
> > > >>>
> > > >>>
> > > >>>
> > > >>>>Does anybody remember GoBack? It was a versioning
> > > >>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
> > > >>>>all changes to the whole disk. Old versions of a file? no problem. grab an
> > > >>>>old version of a directory for referance temporarily? easy. Got a virus?
> > > >>>>revert the whole HD, and then grab the newer copies of your documents and
> > > >>>>saved games as needed.
> > > >>>>
> > > >>>>
> > > >>>My thoughts on this:
> > > >>>
> > > >>>The versioning would be an audit plugin. When the file is modified, tag the
> > > >>>current version, copy it into a sub-directory (oh, I don't know, say
> > > >>>file/.revisions/<number/date>), and disable write access to it. You might not
> > > >>>even need extended filesystem attributes for this, but they would be handy
> > > >>>for tagging particular versions.
> > > >>>
> > > >>>
> > > >>if a file is opened, modified 2 times, then closed. u will only generate
> > > >>1 version right? so "When the file is modified" is inaccurate.
> > > >>
> > > >>
> > > one could do it for every file close, and that could be a state option
> > > for the versioning plugin, but most users will want to do it everytime
> > > they touch filename/..../checkin
> >
> > what u mean touch filename? is "ls" a touch? i think close, unlink,
> > create, is likely to be good candidate.
> 
> What happens if I open, truncate, append, close?  You have to consider
> that... in fact, what about just "open, append, close"?  Not every app
> acts the same.

yes, of course, i think we need to balance between these. anyway i think
each write a version is a bad idea with high overhead.

> 
> > >
> > > >
> > > >How about "When the transaction was completed?"  Why does it matter?
> > > >
> > > >
> > > >
> > > >>>Copy-on-write would make this action extremely cheap, only adding a couple of
> > > >>>extra writes to make it work.
> > > >>>
> > > >>>
> > > >>add 1 line at the beginning of a 100MB text file will make this uncheap.
> > > >>
> > > >>
> > > >
> > > >Who has to work with 100 meg text files?  And why has this person not
> > > >broken them down into 100 kilobyte text files?  Storage efficiency isn't
> > > >really an issue there...
> > > >
> > > >
> > > you need cross-version compression for this case.
> >
> > what u mean cross-version compression? interesting name. :P
> 
> Compression of the files' different versions together; see one of
> Hans' previous posts in this thread (in the archive or otherwise). 
> -.-'
> 
> That way, if there is version X which is a file, and verison Y is just
> a line at the top, the compression eliminates the duplication, so
> instead of
> 
> (old version + new version)
> ABCDEF + XABCDEF
>  on disk
> it becomes something like
> Y=ABCDEF
> X + XY
>  on disk  [note, I don't know squat about this, so an expert might
> tell differently -- if so, believe him, because he'll probably be the
> guy implementing this, when it/if gets implemented]
> 
> [not literally... hopefully you get what i mean by that]
> 
> > >
> > > >Anyway, I think the main win is from copy-on-write for the whole file.
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> 
> 
> --
> ~Mike
>  - Just my two cents
>  - No man is an island, and no man is unable.

Re: Versioning Plugin

[Hans Reiser]

maybe cat not touch.....

Re: Versioning Plugin

[Ming Zhang]

:P yes, cat will modify the last access time as well. but is that really
worthy a version? maybe so, since the version storage overhead is not
big anyway.




On Sat, 2005-11-12 at 18:27 -0800, Hans Reiser wrote:
> maybe cat not touch.....

Re: Versioning Plugin

[michael chang]

What about a per-file/directory/volume settable minimum interval [at
all three levels] at which files can be set as a new "version"?  So
let's say I want only a new version every 8 hours, I go and set that
somewhere and there will only be a new version if the file has been
changed in the last 8 hours... or if it has been 8 hours since the
last change, or something of the sort.

[That said, in that circumstance, if less than that, should we just
leave the last one we put in, or replace it with this one?]


On 11/12/05, Ming Zhang <mingz@ele.uri.edu> wrote:
> :P yes, cat will modify the last access time as well. but is that really
> worthy a version? maybe so, since the version storage overhead is not
> big anyway.
>
>
>
>
> On Sat, 2005-11-12 at 18:27 -0800, Hans Reiser wrote:
> > maybe cat not touch.....
>
>


--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: Versioning Plugin

[Ming Zhang]

agree, timing based should be ok as well. all these should be
configurable.

ming

On Sat, 2005-11-12 at 21:53 -0500, michael chang wrote:
> What about a per-file/directory/volume settable minimum interval [at
> all three levels] at which files can be set as a new "version"?  So
> let's say I want only a new version every 8 hours, I go and set that
> somewhere and there will only be a new version if the file has been
> changed in the last 8 hours... or if it has been 8 hours since the
> last change, or something of the sort.
> 
> [That said, in that circumstance, if less than that, should we just
> leave the last one we put in, or replace it with this one?]
> 
> 
> On 11/12/05, Ming Zhang <mingz@ele.uri.edu> wrote:
> > :P yes, cat will modify the last access time as well. but is that really
> > worthy a version? maybe so, since the version storage overhead is not
> > big anyway.
> >
> >
> >
> >
> > On Sat, 2005-11-12 at 18:27 -0800, Hans Reiser wrote:
> > > maybe cat not touch.....
> >
> >
> 
> 
> --
> ~Mike
>  - Just my two cents
>  - No man is an island, and no man is unable.

Re: Versioning Plugin

[Lares Moreau]

[-- Attachment #1: Type: text/plain, Size: 2214 bytes --]

Has a ;real' revision control system been considered? Something that
doesn't need to copy the entire file on change.  Taking the techiques
learned from CVS/Subverion et.al. and implimenting a variation of the
versioning database within the file system.

On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
> On November 11, 2005 05:59 am, John Gilmore wrote:
> > Does anybody remember GoBack? It was a versioning
> > system for windows 95/98 that was incredibly flexible and useful. Tracked
> > all changes to the whole disk. Old versions of a file? no problem. grab an
> > old version of a directory for referance temporarily? easy. Got a virus?
> > revert the whole HD, and then grab the newer copies of your documents and
> > saved games as needed.
> 
> My thoughts on this:
> 
> The versioning would be an audit plugin. When the file is modified, tag the 
> current version, copy it into a sub-directory (oh, I don't know, say 
> file/.revisions/<number/date>), and disable write access to it. You might not 
> even need extended filesystem attributes for this, but they would be handy 
> for tagging particular versions.
> 
> Copy-on-write would make this action extremely cheap, only adding a couple of 
> extra writes to make it work.
> 
> Given working resource directories, COW, and the ability to set plugins, this 
> might be a relatively easy hack to implement. Given an efficient xpath shell, 
> you could even create a view of your drive on a particular day. 
> 
> If you had a file that was changing often, perhaps you could set an attribute 
> on that file which told it only to clone the file every once in a while. 
> 
> Come to think of it, a userspace daemon could run in the background and 
> replace the need for a plugin, which is probably the better solution. Then 
> you just need COW and files which can contain resources.
> 
> -pvh
> 
-- 
Lares Moreau <lares.moreau@gmail.com>  | LRU: 400755 http://counter.li.org
Gentoo x86 Arch Tester                 |               ::0 Alberta, Canada
Public Key: 0D46BB6E @ subkeys.pgp.net |           Encrypted Mail Prefered
Key fingerprint = 0CA3 E40D F897 7709 3628  C5D4 7D94 483E 0D46 BB6E

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Versioning Plugin

[Hans Reiser]

Lares Moreau wrote:

>Has a ;real' revision control system been considered? Something that
>doesn't need to copy the entire file on change.  Taking the techiques
>learned from CVS/Subverion et.al. and implimenting a variation of the
>versioning database within the file system.
>  
>
all it needs is someone to write it.....

>On Fri, 2005-11-11 at 16:56 -0800, Peter van Hardenberg wrote:
>  
>
>>On November 11, 2005 05:59 am, John Gilmore wrote:
>>    
>>
>>>Does anybody remember GoBack? It was a versioning
>>>system for windows 95/98 that was incredibly flexible and useful. Tracked
>>>all changes to the whole disk. Old versions of a file? no problem. grab an
>>>old version of a directory for referance temporarily? easy. Got a virus?
>>>revert the whole HD, and then grab the newer copies of your documents and
>>>saved games as needed.
>>>      
>>>
>>My thoughts on this:
>>
>>The versioning would be an audit plugin. When the file is modified, tag the 
>>current version, copy it into a sub-directory (oh, I don't know, say 
>>file/.revisions/<number/date>), and disable write access to it. You might not 
>>even need extended filesystem attributes for this, but they would be handy 
>>for tagging particular versions.
>>
>>Copy-on-write would make this action extremely cheap, only adding a couple of 
>>extra writes to make it work.
>>
>>Given working resource directories, COW, and the ability to set plugins, this 
>>might be a relatively easy hack to implement. Given an efficient xpath shell, 
>>you could even create a view of your drive on a particular day. 
>>
>>If you had a file that was changing often, perhaps you could set an attribute 
>>on that file which told it only to clone the file every once in a while. 
>>
>>Come to think of it, a userspace daemon could run in the background and 
>>replace the need for a plugin, which is probably the better solution. Then 
>>you just need COW and files which can contain resources.
>>
>>-pvh
>>
>>    
>>