resizer?

resizer?

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

What happened to resizefs.reiser4?  I know repacking/shrinking is
gone/proprietary, but what about growing?  I distinctly remember that I
used to be able to grow a Reiser4 partition.

How much work would you expect a repacker to be?  (I might try writing a
proof-of-concept on my own...)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQlC6oHgHNmZLgCUhAQKxUw/+LX+KUaSWkm+iB9QbYx967tYU5/gNUApE
K/ITHVviCU2tC3MGsZY3e1mQ2KLzgKsLCmAtJUsyNlPujq8hP5tFnbTRkxso7Nb2
obEvPKOn7zTysRHAYVOaKbflv+8fr6Wz0RjD7gUe0LNuZC04I+rZP8tEjeJ4sqwi
wkvSNuazt0W9+Y+8DW9J5QfOMbRmsROFJmT2XCKxxSy0ncxmpK6FGx8v8WGLADaQ
77PY2EH5h1UpdURprR5PAHlzt9JuM2rVWSOWiTYeXOqOD4+HquXDCJVhgPW57eSN
eD8AA+v4gsYsHLVO5raoy/ibg8vLbmUCWpOPNMyKfl4WTGfyKtHlgFORo+ErdK3C
LTPZsIL7T6z4SKE/5yPeHT98gf1C7ThddUfTo8iGRN3leIAfErTyQ4T1SBO8k0f4
5FJ7AEzOl9dknCMJ90nrmUZRgqU3YUeuaceV1sucJkAGea4ulWxtAElX3zNsWYsY
kEZyhO5uq3EnQa7B1GOPHlsTV/DHy0rlFRQWAz5KC7kKjRKuqKNWYfZK7V9TForl
+SKsvTIdET6E5sjhY6Ex6FpfZwcezqJtg2SkZDXPL+tAdTLNgFirJGRx2Z5hOl3l
eBb0vmh2eCofyfprKDuQRflSkYfLKBmeU315w/Mh0XZmuDM653O5dXasa9Uz2MZN
6tyhOQRbHYc=
=JzL2
-----END PGP SIGNATURE-----

Re: resizer?

[Alex Zarochentsev]

hello,

On Sun, Apr 03, 2005 at 10:55:12PM -0500, David Masover wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> What happened to resizefs.reiser4?  I know repacking/shrinking is
> gone/proprietary, but what about growing?  I distinctly remember that I
> used to be able to grow a Reiser4 partition.

i guess it was reiserfs.

> 
> How much work would you expect a repacker to be?  (I might try writing a
> proof-of-concept on my own...)

no idea,  new version should be "smarter" then the old one, 
the algorithm is not scratched yet.

-- 
Alex.

Re: resizer?

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alex Zarochentsev wrote:
> hello,
> 
> On Sun, Apr 03, 2005 at 10:55:12PM -0500, David Masover wrote:
> 
>>-----BEGIN PGP SIGNED MESSAGE-----
>>Hash: SHA1
>>
>>What happened to resizefs.reiser4?  I know repacking/shrinking is
>>gone/proprietary, but what about growing?  I distinctly remember that I
>>used to be able to grow a Reiser4 partition.
> 
> 
> i guess it was reiserfs.

No.  It was Reiser4.  I remember because I booted with a normal install
CD, created partitions:

hda1	/boot		ext3
hda5	(no mount)	unformatted
hda6	/		reiserfs
hda7	swap		swap

After installing a Reiser4-enabled Gentoo Linux (and the reiser4progs)
on hda6, I booted hda6, formatted hda5 to reiser4 and copied hda6 to it.

After all that, I was able to boot a boot CD, copy the reiser4progs off
of hda6 to a tempfs (RAM), deleted hda6, resized hda5 to fill the disk,
and was able to use the reiser4 resizer to expand the filesystem on hda5.

I experimented, trying to shrink the FS back down, and it told me that
shrinking wasn't implemented.  But growing the FS?  That was in there
ever since my first download of reiser4progs, and it really seems like a
tiny job.

>>How much work would you expect a repacker to be?  (I might try writing a
>>proof-of-concept on my own...)
> 
> 
> no idea,  new version should be "smarter" then the old one, 
> the algorithm is not scratched yet.

An offline repacker would still be great, or at least one that assumes
no processes touching the FS.  Dirty a bunch of blocks towards the end
of the FS, shrink the FS, and let the normal packer (re: not repacker)
decide where to put them.

This seems like a weekend job to me, but my weekends are full and I
don't have a working knowledge of the FS internals, meaning I'd have to
spend two or three weekends reading source code...

I'm willing to do that, but it will have to wait till summer.

I realize that this may not be quite the industrial-strength repacker
that you wanted, but it should be immediately useful, which is a lot
better than "We might do it if you pay us."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQlHvsngHNmZLgCUhAQKb3A/+NQGDjvVLbMVS7u0BnWwLa/Y/tiRb1ZRm
5QuNw0On57Y8WYj5ZJMQD5xP7F/waMKRbN1aDZXpxLC2TDQ1NpYh5v4FrM/YDmK+
DXMA0ZfQ89SLbfqLgj5Plys+9QVa8hlH6sokneL+0YXVBRIl24f3LuD4jKiwgwGD
pBNL3q8AgCsomfJWdMrF4wqmtIfML/jKgooqRlwi28kiNH5yjgFPGXawbFlUz6Yf
OT66SdkBUSOKp+NbeT9qasFtVwkL63wUq1QkE+Cg3kpck42hgWszo2S1eUI76zmb
Kf5c0dC/d1OiLeD6T+fs0nImVo69FMft+Wl3OtDhgizJzLKAvXFV4A3iUdJJOPTo
jw9QO/QryvtyfI4aslsogvE3+A5CpW3+JEUx46csrDUUASh87xXVojwLUiPidTaS
sH0+eWSZahfc2FlLNoUemm1ODJTOew0p5jxfNz+q27OEatFXcMcY3ia1Uql9SGJe
efWr49re5a3pendkkcKkLdiv7mloP/jdqUp5sYWznqOKgKdfA+14VlUZtOxmaHq0
8WzI4nUbTh9owHU8GbFJhemsw2QTwqyssrnnmiiOrnqteFAL4Jyb6iDRr4IYfZbW
vEfJXNL0L4ryWhWtJupvQiQmO5mpUIGpbVYoMHKde0yMHXcG6jf3aiIepUILbCaM
5VV09nFd7V4=
=c0ko
-----END PGP SIGNATURE-----

Re: resizer?

[Hans Reiser]

David Masover wrote:

>
>
> I realize that this may not be quite the industrial-strength repacker
> that you wanted, but it should be immediately useful, which is a lot
> better than "We might do it if you pay us."

Just wait a little, and shortly after we go into the kernel we will work
on the repacker.

Hans

Re: resizer?

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
> David Masover wrote:
> 
> 
>>
>>I realize that this may not be quite the industrial-strength repacker
>>that you wanted, but it should be immediately useful, which is a lot
>>better than "We might do it if you pay us."
> 
> 
> Just wait a little, and shortly after we go into the kernel we will work
> on the repacker.
> 
> Hans

Disclaimer:  I've hardly read any of the Reiser4 code, and I'm not
really an authority on this subject.  I just like to pretend that I am.
 I would take this off-list, but I'm curious about whether I'm wrong.

The repacker (and the resizer) doesn't seem like a hugely complicated
concept, unless you're trying to streamline the user experience during
the process.  "On-line" means that I don't have to use a bootdisk and
stop all my servers.  It doesn't mean that I would do it at any time
other than 2 AM, when I do backups, when I generally expect almost 0
traffic.

Basically, I'm saying that an off-line or a slow on-line shrinker should
have been done by now.  In fact, it should have been done before the
meta-files, because meta-files benefit from a repacker, but not the
other way around.

Since you've told me to wait, I'm going to write this, because it's
easier for me to write documentation than to read code.  This is
probably the fault of school, and will likely disappear this summer.

Anyway, this is how I think the resizer should be done:

If we are growing the FS, we should lock everything necessary, then
change the size value for the FS and make the new blocks available.
Unless we're actually storing something in unused nodes, this should be
an instantaneous operation which requires very little hacking to add.  I
seem to remember that there was even an offline resizer (growing only)
awhile ago.

If we are shrinking the FS, we first set the new size of the FS in RAM,
so that nothing will try to write to the "chopped-off" portion until
we're done.

Next, we turn off the "write-in-the-middle" feature for large
database-like files (where a block in the middle of a huge file may be
written twice to avoid fragmentation), so that absolutely no new writes
will go to the chopped-off portion.

Basically, the filesystem should already think it's shrunken by now, we
just need to make sure it doesn't freak out when it _reads_ blocks past
the end of the FS.  We should capture warnings about this and dirty
those nodes on the spot (nodes which are being read and which are in the
chopped section) -- they are already in RAM, so it'll be faster that way.

Next, we start walking the tree (as you described), dirtying all the
blocks we find which are in the chopped portion and leaving the rest
alone.  We need to be careful about locking here, but that should just
mean "Lock the block we're dealing with, or if locks aren't that
granularity, lock the whole file."  Locking should block, and userland
shouldn't have to know about it except to notice that the FS seems a
little slow right then.

This isn't as dangerous as it seems.  If there is a crash, we just go
back to the old size -- automatically, since the new size hasn't been
written to disk anywhere yet -- with the only difference being that most
of the files will be already moved to where we want them.

Locking isn't as hard as it seems.  If this were a VFS-level operation,
we'd have to worry about a new directory being created, a file being
moved, or our current path being deleted out from under us, but we
aren't working on the semantic layer, we're working on the key/object
layer.  If I'm right, that means that all the things that we'd have to
worry about are merely seen as new writes, and would thus go to the new
places.

Metadata blocks may need a tiny bit of special treatment, since it may
be some small amount of data changing in-place.  All we do here is, when
we notice any attempted write outside the new FS size, but inside the
old FS size, we relocate before we flush it out to disk.  If this means
there's some parent metadata block we need to move, we do it afterwards,
as part of the same transaction.  When we finally get to a parent block
that does not need to be moved, we close the transaction.  This isn't as
elegant as the method for moving data blocks, but it works.  I think.

The nice thing about this is that for the most part, the net impact on
normal FS operation is about the same as that of doing a large "cp -a".

Thoughts?  How close to right is this?  Do you already have another
document on the same thing that I should be reading?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQl3wqXgHNmZLgCUhAQIIGw//SWn2lkNPAGrFcF/r+Vr3t84l/haxnDFL
AF9/xARb6vQ/Mu/AQEd/L8lNabLPymXdzfBUJan2mhLFH97SrlGrA3hdBDcd9xMi
LXlvernTOFcv63jTB2cEq4awnMpTih4mZFrp1qAJ0kcWSu8oaCBUaOk3htXBfuKU
YAkireyHU6EWV2HQlfHmJrd9G/Z0CR6JmmAfVeBKG1CkI0t4Y86GmbeMVqsLdSz1
VEHfdTsCWgcaaod5GOjMk7BbB1a+fvf2wDk3ZsTiCkk8KP1JYPjKnXpCgG3ts8np
hMH1CEDj2Ql+lga8s44fXc0zrez6OAMjzMc/erNc6eUA7iFedQhmQW5oPxMu7TNh
aDF8PekMeYF1cYR1gFXG7B2P5gFx/k2KqDCxzHFNGKZLtSDBvuVlotDD7oJspYpd
5qvVQ0Mj1iYe6bxnV11rCHOvE2f56JlrFJtmzmEI0vmsln0sE4WktxKFONddwf5H
FuEn0L6XB+HkA9gsvkrM3J5xTd2PP1G02oF1MQFRIe3+CsomlSOwE1ZjfEi81s/p
z3Lvz6+0AO8xS7L2et84/y6uCaTb2/z8LZUhKMKx2j+OaUSBqgrzTYjCcotYooYO
7OM4KrSwpjmYQkCtm+iGYy8eC9sv09ng+YsE7F0MJlV1YZ17wo0eRbOVoUJpIgFF
kyq/7WnIUwM=
=D55S
-----END PGP SIGNATURE-----

Re: resizer?

[Hans Reiser]

The current repacker code uses the allocate on flush code and the
transaction code, and walks through the tree sorting it, walking in both
directions.

Hans

David Masover wrote:

> Hans Reiser wrote:
>
> >David Masover wrote:
>
>
> >>I realize that this may not be quite the industrial-strength repacker
> >>that you wanted, but it should be immediately useful, which is a lot
> >>better than "We might do it if you pay us."
>
>
> >Just wait a little, and shortly after we go into the kernel we will work
> >on the repacker.
>
> >Hans
>
>
> Disclaimer:  I've hardly read any of the Reiser4 code, and I'm not
> really an authority on this subject.  I just like to pretend that I am.
>  I would take this off-list, but I'm curious about whether I'm wrong.
>
> The repacker (and the resizer) doesn't seem like a hugely complicated
> concept, unless you're trying to streamline the user experience during
> the process.  "On-line" means that I don't have to use a bootdisk and
> stop all my servers.  It doesn't mean that I would do it at any time
> other than 2 AM, when I do backups, when I generally expect almost 0
> traffic.
>
> Basically, I'm saying that an off-line or a slow on-line shrinker should
> have been done by now.  In fact, it should have been done before the
> meta-files, because meta-files benefit from a repacker, but not the
> other way around.
>
> Since you've told me to wait, I'm going to write this, because it's
> easier for me to write documentation than to read code.  This is
> probably the fault of school, and will likely disappear this summer.
>
> Anyway, this is how I think the resizer should be done:
>
> If we are growing the FS, we should lock everything necessary, then
> change the size value for the FS and make the new blocks available.
> Unless we're actually storing something in unused nodes, this should be
> an instantaneous operation which requires very little hacking to add.  I
> seem to remember that there was even an offline resizer (growing only)
> awhile ago.
>
> If we are shrinking the FS, we first set the new size of the FS in RAM,
> so that nothing will try to write to the "chopped-off" portion until
> we're done.
>
> Next, we turn off the "write-in-the-middle" feature for large
> database-like files (where a block in the middle of a huge file may be
> written twice to avoid fragmentation), so that absolutely no new writes
> will go to the chopped-off portion.
>
> Basically, the filesystem should already think it's shrunken by now, we
> just need to make sure it doesn't freak out when it _reads_ blocks past
> the end of the FS.  We should capture warnings about this and dirty
> those nodes on the spot (nodes which are being read and which are in the
> chopped section) -- they are already in RAM, so it'll be faster that way.
>
> Next, we start walking the tree (as you described), dirtying all the
> blocks we find which are in the chopped portion and leaving the rest
> alone.  We need to be careful about locking here, but that should just
> mean "Lock the block we're dealing with, or if locks aren't that
> granularity, lock the whole file."  Locking should block, and userland
> shouldn't have to know about it except to notice that the FS seems a
> little slow right then.
>
> This isn't as dangerous as it seems.  If there is a crash, we just go
> back to the old size -- automatically, since the new size hasn't been
> written to disk anywhere yet -- with the only difference being that most
> of the files will be already moved to where we want them.
>
> Locking isn't as hard as it seems.  If this were a VFS-level operation,
> we'd have to worry about a new directory being created, a file being
> moved, or our current path being deleted out from under us, but we
> aren't working on the semantic layer, we're working on the key/object
> layer.  If I'm right, that means that all the things that we'd have to
> worry about are merely seen as new writes, and would thus go to the new
> places.
>
> Metadata blocks may need a tiny bit of special treatment, since it may
> be some small amount of data changing in-place.  All we do here is, when
> we notice any attempted write outside the new FS size, but inside the
> old FS size, we relocate before we flush it out to disk.  If this means
> there's some parent metadata block we need to move, we do it afterwards,
> as part of the same transaction.  When we finally get to a parent block
> that does not need to be moved, we close the transaction.  This isn't as
> elegant as the method for moving data blocks, but it works.  I think.
>
> The nice thing about this is that for the most part, the net impact on
> normal FS operation is about the same as that of doing a large "cp -a".
>
> Thoughts?  How close to right is this?  Do you already have another
> document on the same thing that I should be reading?
>

Re: resizer?

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
> The current repacker code uses the allocate on flush code and the
> transaction code, and walks through the tree sorting it, walking in both
> directions.

I don't think we even have to sort it for a resizer (not repacker).
Faster that way.

That's almost exactly what I read from the website/whitepaper; most of
my message was my interpretation of that.  The reason I wrote it out is
that it still seems too simple.

By "the tree", we're talking about the storage tree, right?  Do we have
to do anything to the semantic tree?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQmBGqHgHNmZLgCUhAQILOxAAjs3xpREGU+rdoFiSmqQEtELzxSfgU9OM
H3BKtvXFXvhJFwrvJApCutQebMb2FjpTZ+bV7tkdgZ2c4KIL5Qm6PMBO07YSrZLG
LSeDCyDu4GN9kJbw6IwLK1iq9xMsGF+DiMfqGg5C6Lr+mYyTPVnr4EEV63QuVCtU
hrljURE6aza/52KMjbFwGeHFUfwxT8yu9qowDTg/7/oR4pbujmpLESi6aHlyjIrn
kzf00sT5yoJ9Uk8p6lZfAa7tR1jhRTQ+E9fDJTbor6XdpsUt/kRN4ViHdM1+OQge
BA2MVKeolrSm2pRPLWJlIj24D+zHLwYUaMknvuOt0ZP0lsQShHzu9nQIAUCuANyN
IHbRDL4fsxgWc0zza4cDrgEyuMUKAmv5tOFEthSMNm+6ZWR8/0WbXovUP1pouM+k
woJfF0LCT2hzud95gy31qNctqRqYoZG4YXf4a+5ofRn0C7GxJGk1LiqYV8oFwHJ8
weIxzxfJpz1ecuyHbFq5fetsgH1kuyyGr3SSy9eJOQ8yyYwD+im4dp09vOc3jIDO
Uu72D0sGOlsPwTP9fjD6OB/+NrAJH7RQO9DP9yIP2LQvexzRgH16JJgI0K6X9j3s
k7fW8R+lHyMUfaTflImmrT0Q2H4C/reoWNlMbV7CEh32g2yp6idFVRfinOXT9pd/
rnKuf+0yz80=
=XXnZ
-----END PGP SIGNATURE-----