reserving an IOCTL number; other details

reserving an IOCTL number; other details

[David Nicol]

I have written and am currently testing a patch that provides a new
ioctl that waits for snapshot deletion to finish before returning; The
work is being sponsored, and I can't distribute it until the
sponsorship details are more solid, but I'd like to reserve ioctl #21
for this purpose (the next one from btrfs-unstable) or receive advice
on what ioctl number to use instead.

Also, if you would like to test this function on a trade-secret basis
(until the sponsorship comes through) please contact me off-list.

David Nicol

--=20
l'=C3=A9galit=C3=A9 des droits pour les ambidextres
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reserving an IOCTL number; other details

[David Nicol]

this I can release, the user-facing API.


diff --git a/man/btrfsctl.8.in b/man/btrfsctl.8.in
index c2d4488..94403a4 100644
--- a/man/btrfsctl.8.in
+++ b/man/btrfsctl.8.in
@@ -35,6 +35,10 @@ Scans all devices present in the system for btrfs filesystem.
 .TP
 \fB\-c\fR
 Forces a filesystem sync.
+.TP
+\fB\-C\fR \fIdirectory [ms]\fR
+Wait until all the space from all deleted snapshots have been recovered.
+The optional timeout parameter is in milliseconds.
 .SH AVAILABILITY
 .B btrfsctl
 is part of btrfs-progs. Btrfs is currently under heavy development,

Re: reserving an IOCTL number; other details

[Goffredo Baroncelli]

Hi David,

please don't do that. "btrfsctl" is obsolete, now the develop should involve 
the "btrfs" command.
I you need help in updating the "btrfs" command source files, don't hesitate 
to contact me.

Regards
G.Baroncelli

On Monday, 13 September, 2010, David Nicol wrote:
> this I can release, the user-facing API.
> 
> 
> diff --git a/man/btrfsctl.8.in b/man/btrfsctl.8.in
> index c2d4488..94403a4 100644
> --- a/man/btrfsctl.8.in
> +++ b/man/btrfsctl.8.in
> @@ -35,6 +35,10 @@ Scans all devices present in the system for btrfs 
filesystem.
>  .TP
>  \fB\-c\fR
>  Forces a filesystem sync.
> +.TP
> +\fB\-C\fR \fIdirectory [ms]\fR
> +Wait until all the space from all deleted snapshots have been recovered.
> +The optional timeout parameter is in milliseconds.
>  .SH AVAILABILITY
>  .B btrfsctl
>  is part of btrfs-progs. Btrfs is currently under heavy development,
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
gpg key@ keyserver.linux.it: Goffredo Baroncelli (ghigo) <kreijack@inwind.it>
Key fingerprint = 4769 7E51 5293 D36C 814E  C054 BF04 F161 3DC5 0512

Re: reserving an IOCTL number; other details

[David Nicol]

https://btrfs.wiki.kernel.org/index.php/Category:Documentation does
not mention any btrfs command. the btrfsctl command has the -c sync
tool, and the tools for creating and deleting snapshots and
subvolumes, so it seems that unless this btrfs program does all of
that I have amended the right tool to provide a command line access to
a new ioctl that waits for deleted subvolume cleanups to finish.

Nothing on https://btrfs.wiki.kernel.org/index.php/Btrfsctl says
anything about deprecation or obsolescence.

On Mon, Sep 13, 2010 at 12:23 PM, Goffredo Baroncelli
<kreijack@gmail.com> wrote:
> Hi David,
>
> please don't do that. "btrfsctl" is obsolete, now the develop should involve
> the "btrfs" command.
> I you need help in updating the "btrfs" command source files, don't hesitate
> to contact me.
>
> Regards
> G.Baroncelli

Re: reserving an IOCTL number; other details

[David Nicol]

I see btrfs utility appears here:

http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-progs-unstable.git;a=commit;h=6d2cf042471cc728b5399b2beae54603739bc66a


I shall be patching it, too, with the new ioctl which I made this
thread to reserve.

Re: reserving an IOCTL number; other details

[Goffredo Baroncelli]

On Monday, 13 September, 2010, David Nicol wrote:
> https://btrfs.wiki.kernel.org/index.php/Category:Documentation does
> not mention any btrfs command. the btrfsctl command has the -c sync
> tool, and the tools for creating and deleting snapshots and
> subvolumes, so it seems that unless this btrfs program does all of
> that I have amended the right tool to provide a command line access to
> a new ioctl that waits for deleted subvolume cleanups to finish.

$ /sbin/btrfs help
Usage:
        btrfs subvolume snapshot <source> [<dest>/]<name>
                Create a writable snapshot of the subvolume <source> with
                the name <name> in the <dest> directory.
        btrfs subvolume delete <subvolume>
                Delete the subvolume <subvolume>.
        btrfs subvolume create [<dest>/]<name>
                Create a subvolume in <dest> (or the current directory if
                not passed).
        btrfs subvolume list <path>
                List the snapshot/subvolume of a filesystem.
        btrfs subvolume find-new <path> <last_gen>
                List the recently modified files in a filesystem.
        btrfs filesystem defragment [-vcf] [-s start] [-l len] [-t size] 
<file>|<dir> [<file>|<dir>...]
                Defragment a file or a directory.
        btrfs subvolume set-default <id> <path>
                Set the subvolume of the filesystem <path> which will be 
mounted
                as default.
        btrfs filesystem sync <path>
                Force a sync on the filesystem <path>.
        btrfs filesystem resize [+/-]<newsize>[gkm]|max <filesystem>
                Resize the file system. If 'max' is passed, the filesystem
                will occupe all available space on the device.
        btrfs filesystem show [<uuid>|<label>]
                Show the info of a btrfs filesystem. If no <uuid> or <label>
                is passed, info of all the btrfs filesystem are shown.
        btrfs filesystem df <path>
                Show space usage information for a mount point
                .
        btrfs filesystem balance <path>
                Balance the chunks across the device.
        btrfs device scan [<device> [<device>..]
                Scan all device for or the passed device for a btrfs
                filesystem.
        btrfs device add <dev> [<dev>..] <path>
                Add a device to a filesystem.
        btrfs device delete <dev> [<dev>..] <path>
                Remove a device from a filesystem.
                                                                                                 
        btrfs help|--help|-h                                                                     
                Show the help.                                                                   
                                                                                                 
Btrfs Btrfs v0.19                                            


> 
> Nothing on https://btrfs.wiki.kernel.org/index.php/Btrfsctl says
> anything about deprecation or obsolescence.
> 
> On Mon, Sep 13, 2010 at 12:23 PM, Goffredo Baroncelli
> <kreijack@gmail.com> wrote:
> > Hi David,
> >
> > please don't do that. "btrfsctl" is obsolete, now the develop should 
involve
> > the "btrfs" command.
> > I you need help in updating the "btrfs" command source files, don't 
hesitate
> > to contact me.
> >
> > Regards
> > G.Baroncelli
> 
> 


-- 
gpg key@ keyserver.linux.it: Goffredo Baroncelli (ghigo) <kreijack@inwind.it>
Key fingerprint = 4769 7E51 5293 D36C 814E  C054 BF04 F161 3DC5 0512

Updating the wiki pages adding btrfs command [Was Re: reserving an IOCTL number; other details]

[Goffredo Baroncelli]

Hi all,

On Monday, 13 September, 2010, David Nicol wrote:
> https://btrfs.wiki.kernel.org/index.php/Category:Documentation does
> not mention any btrfs command. 
[...]

I updated the wiki, inserting the btrfs command. For now I left the old 
commands in the examples (some distros have the old version of the btrfs-
progs); I put a "notice" saying:

	"the new command btrfs is replacing the old one btrfsctl."

I think to remove the old commands from the wiki when the next generation of 
linux distros will appear.

Thoughts ?

Regards
G.Baroncelli

-- 
gpg key@ keyserver.linux.it:Goffredo Baroncelli (ghigo) <kreijackATinwind.it>
Key fingerprint = 4769 7E51 5293 D36C 814E  C054 BF04 F161 3DC5 0512

Re: Updating the wiki pages adding btrfs command [Was Re: reserving an IOCTL number; other details]

[David Nicol]

Does the manpatch below describe an acceptable API for a hook to a
facility for waiting for the completion of deleted snapshot cleanup?

diff --git a/man/btrfs.8.in b/man/btrfs.8.in
index 26ef982..290d202 100644
--- a/man/btrfs.8.in
+++ b/man/btrfs.8.in
@@ -19,6 +19,8 @@ btrfs \- control a btrfs filesystem
 .PP
 \fBbtrfs\fP \fBfilesystem sync\fP\fI <path> \fP
 .PP
+\fBbtrfs\fP \fBfilesystem GC\fP\fI <path> [ms] \fP
+.PP
 \fBbtrfs\fP \fBfilesystem resize\fP\fI [+/\-]<size>[gkm]|max <filesystem>\fP
 .PP
 \fBbtrfs\fP \fBdevice scan\fP\fI [<device> [<device>..]]\fP
@@ -115,6 +117,12 @@ all the block devices.
 Force a sync for the filesystem identified by \fI<path>\fR.
 .TP

+\fBfilesystem GC\fR\fI <path> [ms] \fR
+wait for completion of salvage of deleted subvolumes on
+the filesystem identified by \fI<path>\fR. The optional
+timeout is in milliseconds.
+.TP
+
 .\"
 .\" Some wording are extracted by the resize2fs man page
 .\"

Re: Updating the wiki pages adding btrfs command [Was Re: reserving an IOCTL number; other details]

[Goffredo Baroncelli]

Hi David,

the btrfs command is smart enough to accept short form of the sub-comma=
nd if=20
there is no ambiguity. For example "btrfs fi" is a synonymous of "btrfs=
=20
filesystem".
Because that the acronyms are discouraged. Instead a long form is prefe=
rred.

So instead of CG (which means ? Garbage Collector ? ) I suggest=20

	btrfs filesystem wait-subvolume-deletion

or better
=09
	btrfs subvolume wait-deletion

which is a better self-explain form and may be shortened as

	btrfs s w

Only my 2=C2=A2

regards
G.Baroncelli




On Wednesday, 15 September, 2010, David Nicol wrote:
> Does the manpatch below describe an acceptable API for a hook to a
> facility for waiting for the completion of deleted snapshot cleanup?
>=20
> diff --git a/man/btrfs.8.in b/man/btrfs.8.in
> index 26ef982..290d202 100644
> --- a/man/btrfs.8.in
> +++ b/man/btrfs.8.in
> @@ -19,6 +19,8 @@ btrfs \- control a btrfs filesystem
>  .PP
>  \fBbtrfs\fP \fBfilesystem sync\fP\fI <path> \fP
>  .PP
> +\fBbtrfs\fP \fBfilesystem GC\fP\fI <path> [ms] \fP
> +.PP
>  \fBbtrfs\fP \fBfilesystem resize\fP\fI [+/\-]<size>[gkm]|max=20
<filesystem>\fP
>  .PP
>  \fBbtrfs\fP \fBdevice scan\fP\fI [<device> [<device>..]]\fP
> @@ -115,6 +117,12 @@ all the block devices.
>  Force a sync for the filesystem identified by \fI<path>\fR.
>  .TP
>=20
> +\fBfilesystem GC\fR\fI <path> [ms] \fR
> +wait for completion of salvage of deleted subvolumes on
> +the filesystem identified by \fI<path>\fR. The optional
> +timeout is in milliseconds.
> +.TP
> +
>  .\"
>  .\" Some wording are extracted by the resize2fs man page
>  .\"
>=20
>=20


--=20
gpg key@ keyserver.linux.it:Goffredo Baroncelli (ghigo) <kreijackATinwi=
nd.it>
Key fingerprint =3D 4769 7E51 5293 D36C 814E  C054 BF04 F161 3DC5 0512
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Updating the wiki pages adding btrfs command [Was Re: reserving an IOCTL number; other details]

[David Nicol]

btrfs s ... is imprecise, as the operation isn't on a subvolume, but
on the fi where the s used to be but isn't any more.

With your approval I'll go with "filesystem garbage-collection" which
can be abbrevd to

   btrfs fi g /butter 30000 || echo there should be some more space now \
   but it would have taken more than 30 sec to clean it all

and in some imaginary future might trigger something instead of simply
waiting for something.

Re: Updating the wiki pages adding btrfs command [Was Re: reserving an IOCTL number; other details]

[Goffredo Baroncelli]

On Wednesday, 15 September, 2010, David Nicol wrote:
> btrfs s ... is imprecise, as the operation isn't on a subvolume, but
> on the fi where the s used to be but isn't any more.

ok 
> 
> With your approval I'll go with "filesystem garbage-collection" which
> can be abbrevd to
> 
>    btrfs fi g /butter 30000 || echo there should be some more space now \
>    but it would have taken more than 30 sec to clean it all
> 
> and in some imaginary future might trigger something instead of simply
> waiting for something.

I don't like too much "garbage-collector". It seems that you can start a 
"garbage collector". Instead we wait the finish of the garbage collector 
activity.

I would like also to have the option to wait in "btrfs subvolume delete"

# btrfs subvolume delete --wait /path/to/subvolume

(instead of "--wait", we can use "--sync"...)

In any case could you highlight the behavior of your ioctl when two snapshot 
deletions are performed. Does the ioctl wait the end of both the snapshot 
deletion ? 

Regards
G.Baroncelli

-- 
gpg key@ keyserver.linux.it:Goffredo Baroncelli (ghigo) <kreijackATinwind.it>
Key fingerprint = 4769 7E51 5293 D36C 814E  C054 BF04 F161 3DC5 0512

Re: Updating the wiki pages adding btrfs command [Was Re: reserving an IOCTL number; other details]

[Josh Berry]

On Wed, Sep 15, 2010 at 07:10, David Nicol <davidnicol@gmail.com> wrote:
> Does the manpatch below describe an acceptable API for a hook to a
> facility for waiting for the completion of deleted snapshot cleanup?
...
> +\fBfilesystem GC\fR\fI <path> [ms] \fR
> +wait for completion of salvage of deleted subvolumes on
> +the filesystem identified by \fI<path>\fR. The optional
> +timeout is in milliseconds.
> +.TP

I'd suggest different wording for the manpage:

filesystem GC <filesystem> [timeout_in_ms]
Wait for btrfs to free any space that was used by deleted subvolumes
in <filesystem>.  If timeout_in_ms is specified, wait up to that many
milliseconds, then exit with status 1 [or whatever code is chosen] if
not all space could be freed in that time.

Personally, I like "gc" or "clean".  "wait-subvolume-deletion",
"garbage-collect" or similar is too verbose for my taste.

-- Josh

Re: Updating the wiki pages adding btrfs command [Was Re: reserving an IOCTL number; other details]

[David Nicol]

On Wed, Sep 15, 2010 at 1:41 PM, Josh Berry <des@condordes.net> wrote:
> Personally, I like "gc" or "clean". =C2=A0"wait-subvolume-deletion",
> "garbage-collect" or similar is too verbose for my taste.
>
> -- Josh

the question IMO is how much implementation details to saddle the user
with. Anything with the word "garbage" in it is just yucky; anything
with "wait" is revealing an implementation detail that hopefully will
change eventually.

adding a --wait or --sync switch to the delete command is certainly
possible, you'd want to add it to the last in a series of deletes.

I don't like "clean" for end-user API because it seems to imply
erasing data, like it's going to get overwritten with zeroes or
something like that to hide it from forensic analysis.

among the listings at http://thesaurus.com/browse/recycle "reclaim"
seems best: "salvage" connotes that the thing being salvaged is
damaged.

Is everyone happy with

  btrfs filesystem reclaim <path> [ms]

then? (I'm pleasantly surprised that I'm not having to defend milliseco=
nds.)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Updating the wiki pages adding btrfs command

[Chris Ball]

Hi,

   > adding a --wait or --sync switch to the delete command is
   > certainly possible, you'd want to add it to the last in a series
   > of deletes.

I think this (--sync) is a much better idea than the alternatives so far.

Thanks,

-- 
Chris Ball   <cjb@laptop.org>   <http://printf.net/>
One Laptop Per Child

Re: Updating the wiki pages adding btrfs command

[Josh Berry]

On Wed, Sep 15, 2010 at 14:07, Chris Ball <cjb@laptop.org> wrote:
> Hi,
>
> =C2=A0 > adding a --wait or --sync switch to the delete command is
> =C2=A0 > certainly possible, you'd want to add it to the last in a se=
ries
> =C2=A0 > of deletes.
>
> I think this (--sync) is a much better idea than the alternatives so =
far.

Or make --sync the default behavior.  This is probably what most
people are expecting anyway (similar to how standard filesystem
commands like "rm" work).  Add an --aysnc option for those that only
care about knowing when the subvolume is taken out of the tree.

But I agree, an option to "subvolume delete" is probably better.

-- Josh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Updating the wiki pages adding btrfs command

[Chris Ball]

Hi,

   > Or make --sync the default behavior.  This is probably what most
   > people are expecting anyway (similar to how standard filesystem
   > commands like "rm" work).  Add an --aysnc option for those that
   > only care about knowing when the subvolume is taken out of the
   > tree.

Yeah.  We've also talked about making snapshot _creation_ perform an
FS sync first by default, since otherwise you get a snapshot with stale
files, or without files that existed (not yet on disk) at creation-time.

- Chris.
-- 
Chris Ball   <cjb@laptop.org>   <http://printf.net/>
One Laptop Per Child

Re: Updating the wiki pages adding btrfs command

[Josh Berry]

On Wed, Sep 15, 2010 at 14:52, Chris Ball <cjb@laptop.org> wrote:
> Hi,
>
> =C2=A0 > Or make --sync the default behavior. =C2=A0This is probably =
what most
> =C2=A0 > people are expecting anyway (similar to how standard filesys=
tem
> =C2=A0 > commands like "rm" work). =C2=A0Add an --aysnc option for th=
ose that
> =C2=A0 > only care about knowing when the subvolume is taken out of t=
he
> =C2=A0 > tree.
>
> Yeah. =C2=A0We've also talked about making snapshot _creation_ perfor=
m an
> FS sync first by default, since otherwise you get a snapshot with sta=
le
> files, or without files that existed (not yet on disk) at creation-ti=
me.

Oh wow, I didn't know this.  Yes, sync-before-create would be nice.

*runs off to update his snapshot script*

-- Josh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Updating the wiki pages adding btrfs command

[Sage Weil]

On Wed, 15 Sep 2010, Chris Ball wrote:
> Hi,
> 
>    > Or make --sync the default behavior.  This is probably what most
>    > people are expecting anyway (similar to how standard filesystem
>    > commands like "rm" work).  Add an --aysnc option for those that
>    > only care about knowing when the subvolume is taken out of the
>    > tree.
> 
> Yeah.  We've also talked about making snapshot _creation_ perform an
> FS sync first by default, since otherwise you get a snapshot with stale
> files, or without files that existed (not yet on disk) at creation-time.

Actually, that was fixed in 2.6.34 (0bdb1db2).  Creating a snapshot syncs 
all dirty data and metadata to disk, so you get a fully consistent point 
in time snapshot.  No 'sync' is necessary (and doing a 'sync' yourself 
would be racy any).

For subvolume removal, racing doesn't seem like an issue.  Why not just 
do

# btrfs subvolume delete /path/to/subvolume1
# btrfs subvolume delete /path/to/subvolume2
# ...
# sync

?
sage

Re: Updating the wiki pages adding btrfs command

[Sage Weil]

On Wed, 15 Sep 2010, Sage Weil wrote:
> For subvolume removal, racing doesn't seem like an issue.  Why not just 
> do
> 
> # btrfs subvolume delete /path/to/subvolume1
> # btrfs subvolume delete /path/to/subvolume2
> # ...
> # sync
> 
> ?

Nevermind, read the whole thread this time.  I'll defer to others on the 
best way to wait for the old space to be freed.

sage

Re: Updating the wiki pages adding btrfs command

[Chris Ball]

Hi Sage,

   > Actually, that was fixed in 2.6.34 (0bdb1db2).  Creating a
   > snapshot syncs all dirty data and metadata to disk, so you get a
   > fully consistent point in time snapshot.  No 'sync' is necessary
   > (and doing a 'sync' yourself would be racy any).

Oh, thanks!  Am happy to see that.

   > Why not just do:
   > # btrfs subvolume delete /path/to/subvolume1
   > # btrfs subvolume delete /path/to/subvolume2
   > # sync

Or even better, "btrfs filesystem sync <path>".

I'm assuming that the caller wants to wait until the subvol delete
has happened, but is unwilling to wait until all other writes to the
FS have finished, and is even more unwilling to wait for all writes
on the rest of the system to finish.

I don't know whether that's actually a reasonable/plausible use case,
though.

- Chris.
-- 
Chris Ball   <cjb@laptop.org>   <http://printf.net/>
One Laptop Per Child

Re: Updating the wiki pages adding btrfs command

[Josh Berry]

On Wed, Sep 15, 2010 at 15:37, Chris Ball <cjb@laptop.org> wrote:
=2E..
> =C2=A0 > Why not just do:
> =C2=A0 > # btrfs subvolume delete /path/to/subvolume1
> =C2=A0 > # btrfs subvolume delete /path/to/subvolume2
> =C2=A0 > # sync
>
> Or even better, "btrfs filesystem sync <path>".
>
> I'm assuming that the caller wants to wait until the subvol delete
> has happened, but is unwilling to wait until all other writes to the
> FS have finished, and is even more unwilling to wait for all writes
> on the rest of the system to finish.
>
> I don't know whether that's actually a reasonable/plausible use case,
> though.

I think so, because I have that use case (sorta).  I have a home
fileserver with btrfs, and a cron job on that machine runs hourly,
taking snapshots.  However, once the disk fills up, I want to delete
the oldest snapshots:

while [[ disk > xx% full ]]; do
    btrfs subvol delete /the/oldest/snapshot
    wait for space to free up
done

If delete does not wait to free space, I'll delete too many snapshots.
 But if I force a filesystem-wide sync after every delete, the loop
may take longer, which on a system busier than mine could take more
than the hour to complete.

-- Josh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html