[PATCH] Add git-clean command

[PATCH] Add git-clean command

[Pavel Roskin]

This command removes untracked files from the working tree.  This
implementation is based on cg-clean with some simplifications.  The
documentation is included.

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 .gitignore                  |    1 +
 Documentation/git-clean.txt |   50 +++++++++++++++++++++++++++++++
 Makefile                    |    2 +
 git-clean.sh                |   70 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 122 insertions(+), 1 deletions(-)

diff --git a/.gitignore b/.gitignore
index 75891c3..b5959d6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -15,6 +15,7 @@ git-checkout
 git-checkout-index
 git-cherry
 git-cherry-pick
+git-clean
 git-clone
 git-clone-pack
 git-commit
diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt
new file mode 100644
index 0000000..0c671e1
--- /dev/null
+++ b/Documentation/git-clean.txt
@@ -0,0 +1,50 @@
+git-clean(1)
+============
+
+NAME
+----
+git-clean - Remove untracked files from the working tree
+
+SYNOPSIS
+--------
+[verse]
+'git-clean' [-d | -D] [-n] [-q] [-x]
+
+DESCRIPTION
+-----------
+Removes files unknown to git.  This allows to clean the working tree
+from files that are not under version control.  If the '-x' option is
+specified, ignored files are also removed, allowing to remove all
+build products.
+
+OPTIONS
+-------
+-d::
+	Remove untracked directories in addition to untracked files.
+
+-D::
+	Same as above, but use chmod first.  This is useful for
+	read-only directories used in some testsuites.
+
+-n::
+	Don't actually remove anything, just show what would be done.
+
+-q::
+	Be quiet, only report errors, but not the files that are
+	successfully removed.
+
+-x::
+	Don't use the ignore rules.  This allows removing all untracked
+	files, including build products.  This can be used (possibly in
+	conjunction with gitlink:git-reset[1]) to create a pristine
+	working directory to test a clean build.
+
+
+Author
+------
+Written by Pavel Roskin <proski@gnu.org>
+
+
+GIT
+---
+Part of the gitlink:git[7] suite
diff --git a/Makefile b/Makefile
index c79d646..221e59a 100644
--- a/Makefile
+++ b/Makefile
@@ -114,7 +114,7 @@ ### --- END CONFIGURATION SECTION ---
 
 SCRIPT_SH = \
 	git-add.sh git-bisect.sh git-branch.sh git-checkout.sh \
-	git-cherry.sh git-clone.sh git-commit.sh \
+	git-cherry.sh git-clean.sh git-clone.sh git-commit.sh \
 	git-count-objects.sh git-diff.sh git-fetch.sh \
 	git-format-patch.sh git-log.sh git-ls-remote.sh \
 	git-merge-one-file.sh git-parse-remote.sh \
diff --git a/git-clean.sh b/git-clean.sh
new file mode 100755
index 0000000..8470559
--- /dev/null
+++ b/git-clean.sh
@@ -0,0 +1,70 @@
+#!/bin/sh
+#
+# Copyright (c) 2005-2006 Pavel Roskin
+#
+
+USAGE="[-d | -D] [-n] [-q] [-x]"
+LONG_USAGE='Clean untracked files from the working directory
+	-d	remove directories as well
+	-D	remove directories, try harder (chmod first)
+	-n 	don'\''t remove anything, just show what would be done
+	-q	be quiet, only report errors
+	-x	remove ignored files as well'
+SUBDIRECTORY_OK=Yes
+. git-sh-setup
+
+noexclude=
+cleandir=
+cleandirhard=
+quiet=
+rmf="rm -f"
+rmrf="rm -rf"
+rm_refuse="echo Not removing"
+echo1="echo"
+
+for arg in "$@"; do
+	if [ "$arg" = "-d" ]; then
+		cleandir=1
+	elif [ "$arg" = "-D" ]; then
+		cleandir=1
+		cleandirhard=1
+	elif [ "$arg" = "-n" ]; then
+		quiet=1
+		rmf="echo Would remove"
+		rmrf="echo Would remove"
+		rm_refuse="echo Would not remove"
+		echo1=":"
+	elif [ "$arg" = "-q" ]; then
+		quiet=1
+	elif [ "$arg" = "-x" ]; then
+		noexclude=1
+	else
+		echo >&2 "Unknown option: \"$arg\""
+		exit 1
+	fi
+done
+
+excl1=
+excl2=
+if [ -z "$noexclude" ]; then
+	excl1="--exclude-per-directory=.gitignore"
+	if [ -f "$GIT_DIR/info/exclude" ]; then
+		excl2="--exclude-from=$GIT_DIR/info/exclude"
+	fi
+fi
+
+git-ls-files --others --directory "$excl1" "$excl2" |
+while read -r file; do
+	if [ -d "$file" -a ! -L "$file" ]; then
+		if [ -z "$cleandir" ]; then
+			$rm_refuse "$file"
+			continue
+		fi
+		$echo1 "Removing $file"
+		[ "$cleandirhard" ] && chmod -R 700 "$file"
+		$rmrf "$file"
+	else
+		$echo1 "Removing $file"
+		$rmf "$file"
+	fi
+done

Re: [PATCH] Add git-clean command

[Junio C Hamano]

Pavel Roskin <proski@gnu.org> writes:

> This command removes untracked files from the working tree.  This
> implementation is based on cg-clean with some simplifications.  The
> documentation is included.

I am not opposed to the command in the sense that I do not want
to forbid people from doing what they want to do, but on the
other hand I do not see why people (apparently many people) want
to have something like this.  Are their "make clean" broken?

Having said that, just some nitpicks.

> diff --git a/.gitignore b/.gitignore
>...
> +git-clean

I appreciate the attention to the detail; very nice to have
a .gitignore entry along with addition of a command.

> diff --git a/git-clean.sh b/git-clean.sh
>...
> +for arg in "$@"; do

	for arg
        do
        	...

> +	if [ "$arg" = "-d" ]; then

	case "$arg" in -d)...

> +excl1=
> +excl2=
> +if [ -z "$noexclude" ]; then
> +	excl1="--exclude-per-directory=.gitignore"
> +	if [ -f "$GIT_DIR/info/exclude" ]; then
> +		excl2="--exclude-from=$GIT_DIR/info/exclude"
> +	fi
> +fi
> +
> +git-ls-files --others --directory "$excl1" "$excl2" |
> +while read -r file; do
> ...

The $noexclude case passes two empty strings to git-ls-files,
which may happen to be harmless with the current implementation,
but does not feel quite right.

Maybe better to read ls-files -z to be really pathname safe, I
dunno.

> +		$echo1 "Removing $file"
> +		[ "$cleandirhard" ] && chmod -R 700 "$file"

I am not quite sure this chmod -R is a good idea.  If we are
trying really hard would we need to also make sure we can rmdir
the "$file" by chmod'ing its parent directory?  But once we
start doing that where would we stop?

Re: [PATCH] Add git-clean command

[Sam Ravnborg]

On Mon, Apr 03, 2006 at 05:06:36PM -0700, Junio C Hamano wrote:
> 
> I am not opposed to the command in the sense that I do not want
> to forbid people from doing what they want to do, but on the
> other hand I do not see why people (apparently many people) want
> to have something like this.  Are their "make clean" broken?

No reason to waste time on make clean.
git ls-files -o | xargs rm
Does the same job nicely.

Other typical usecases for me:
Remove temporaries that I created while trying out stuff.
Often I have a bunch of files named 'x', 'xx', 'fisk' etc.
around for no use. An easy way to remove these without breaking
my 'allmodconfig' build is nice. It anyway > 1 hour to build and
I like to get rid of the untracked stuff in an easy way.

So use cases goes like this:
- Remove everything not tracked by git (including .gitignore files)
- Remove everything except tracked by git or ignored
- Remove ignored files (replacement of make clean) (seldom)

Above should work both from top-level dir and in subdirectories.

That is my minimal expectations to git clean.
What Pavel came up with cover everything except the make clean
replacement part.

	Sam

Re: [PATCH] Add git-clean command

[Pavel Roskin]

On Tue, 2006-04-04 at 12:58 +0200, Sam Ravnborg wrote:
> On Mon, Apr 03, 2006 at 05:06:36PM -0700, Junio C Hamano wrote:
> > 
> > I am not opposed to the command in the sense that I do not want
> > to forbid people from doing what they want to do, but on the
> > other hand I do not see why people (apparently many people) want
> > to have something like this.  Are their "make clean" broken?

Normally it is not, but if it is, they want to know it.  If git-clean
remove something after "make clean" then maybe the later is incomplete.

To put it another way, if I make changes and share them with others, I
want to make sure that my changes will work for them.  If the mechanism
for sharing my changes is make-based, I rely on make to check them.  In
projects using Automake it's called "make distcheck".  If I'm sharing my
changes as a git patch, I want a git-based verification that the project
still builds from scratch.

Also, the files cleaned by "make clean" are normally build products.
Things that you _really_ don't want to be in the source tree at the
release time are other files, such as backup files with changes that you
decided to back out, sources that you may not share, firmware that cost
your company $50000 and so on.  "make clean" won't remove them.

> No reason to waste time on make clean.
> git ls-files -o | xargs rm
> Does the same job nicely.

This would remove ignored files.  It's not always what I want.

> Other typical usecases for me:
> Remove temporaries that I created while trying out stuff.
> Often I have a bunch of files named 'x', 'xx', 'fisk' etc.
> around for no use. An easy way to remove these without breaking
> my 'allmodconfig' build is nice. It anyway > 1 hour to build and
> I like to get rid of the untracked stuff in an easy way.
> 
> So use cases goes like this:
> - Remove everything not tracked by git (including .gitignore files)
> - Remove everything except tracked by git or ignored
> - Remove ignored files (replacement of make clean) (seldom)

I'll think about the later.

> Above should work both from top-level dir and in subdirectories.
> 
> That is my minimal expectations to git clean.
> What Pavel came up with cover everything except the make clean
> replacement part.

Exactly.  The "make clean" replacement is actually the one I didn't
implement.

-- 
Regards,
Pavel Roskin

unchecked uses of strdup

[Jim Meyering]

There are pretty many uses of strdup in git's sources.
Here's one that can cause trouble if it ever returns NULL:

    [from fsck-objects.c]
    static int fsck_head_link(void)
    {
            unsigned char sha1[20];
            const char *git_HEAD = strdup(git_path("HEAD"));
            const char *git_refs_heads_master = resolve_ref(git_HEAD, sha1, 1);

The problem is that resolve_ref does an unconditional `stat'
on the parameter corresponding to the maybe-NULL git_HEAD.

One solution is to change such uses of strdup to uses of xstrdup.

Re: unchecked uses of strdup

[Alex Riesen]

On 4/5/06, Jim Meyering <jim@meyering.net> wrote:
> There are pretty many uses of strdup in git's sources.
> Here's one that can cause trouble if it ever returns NULL:
>
>     [from fsck-objects.c]
>     static int fsck_head_link(void)
>     {
>             unsigned char sha1[20];
>             const char *git_HEAD = strdup(git_path("HEAD"));
>             const char *git_refs_heads_master = resolve_ref(git_HEAD, sha1, 1);
>
> The problem is that resolve_ref does an unconditional `stat'
> on the parameter corresponding to the maybe-NULL git_HEAD.

That's actually alright (aside a nice core file). Worse are the cases
where a NULL would cause some "normal" behaviour, e.g. arguments,
which have a meaning for NULL value.

Re: [PATCH] Add git-clean command

[Pavel Roskin]

On Mon, 2006-04-03 at 17:06 -0700, Junio C Hamano wrote:
> > diff --git a/git-clean.sh b/git-clean.sh
> >...
> > +for arg in "$@"; do
> 
> 	for arg
>         do
>         	...

I checked other git shell scripts and copied the "while" construct from
one of them - if it's good for other commands, it's good for git-clean.

> > +	if [ "$arg" = "-d" ]; then
> 
> 	case "$arg" in -d)...
> 
> > +excl1=
> > +excl2=
> > +if [ -z "$noexclude" ]; then
> > +	excl1="--exclude-per-directory=.gitignore"
> > +	if [ -f "$GIT_DIR/info/exclude" ]; then
> > +		excl2="--exclude-from=$GIT_DIR/info/exclude"
> > +	fi
> > +fi
> > +
> > +git-ls-files --others --directory "$excl1" "$excl2" |
> > +while read -r file; do
> > ...
> 
> The $noexclude case passes two empty strings to git-ls-files,
> which may happen to be harmless with the current implementation,
> but does not feel quite right.

Good catch.  This is needed since $GIT_DIR can contain spaces.  I
believe ${excl2:+"$excl2"} would do the trick.

> Maybe better to read ls-files -z to be really pathname safe, I
> dunno.

I think "xargs -0" has its own problems (argument length limitations),
and the other solution is to use perl.  While at that, I'd rather
rewrite the whole script in Perl, or maybe in even C.

I think this should eventually happen to all git scripts, but I have no
intention to do it right now unless you really want me to.

> > +		$echo1 "Removing $file"
> > +		[ "$cleandirhard" ] && chmod -R 700 "$file"
> 
> I am not quite sure this chmod -R is a good idea.  If we are
> trying really hard would we need to also make sure we can rmdir
> the "$file" by chmod'ing its parent directory?  But once we
> start doing that where would we stop?

OK, I was undecided on that.  I'm dropping this option.

-- 
Regards,
Pavel Roskin

Re: [PATCH] Add git-clean command

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

hoi :)

> +-x::
> +	Don't use the ignore rules.  This allows removing all untracked
> +	files, including build products.  This can be used (possibly in
> +	conjunction with gitlink:git-reset[1]) to create a pristine
> +	working directory to test a clean build.

as ignored files are generally generated files, doesn't it make sense
to clean up the "ignored" files, and leave other untracked files
alone?  That way you don't loose files which you forgot to add to git.

What is the use case of cleaning up all untracked files without also
cleaning ignored files?

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Add git-clean command

[Junio C Hamano]

Martin Waitz <tali@admingilde.org> writes:

>> +-x::
>> +	Don't use the ignore rules.  This allows removing all untracked
>> +	files, including build products.  This can be used (possibly in
>> +	conjunction with gitlink:git-reset[1]) to create a pristine
>> +	working directory to test a clean build.
>
> as ignored files are generally generated files, doesn't it make sense
> to clean up the "ignored" files, and leave other untracked files
> alone?  That way you don't loose files which you forgot to add to git.

Sounds like a sane suggestion, but my previous comment about
"make clean" broken for people who want this "git clean" feature
applies here as well.

One justification I can think of for "git clean" without -x flag
is to make a clean tree that has only the source and build
products, removing editor backup files, throwaway test output
files and friends, but as you pointed out, this risks losing
newly created source files that you forgot to add, so I would
need a bit of convincing before I use such a command myself.

Compared to that, removing only ignored files sounds like a much
safer operation -- they are explicitly listed as expendable, so
it is a lot less likely to lose anything important.  But again,
that is what "make clean" is there for...

Re: [PATCH] Add git-clean command

[Alex Riesen]

On 4/4/06, Martin Waitz <tali@admingilde.org> wrote:
> What is the use case of cleaning up all untracked files without also
> cleaning ignored files?

Thinks of git's .gitignore: it has config.mak in it. Are you sure you want
"clean" your build customizations?

Re: [PATCH] Add git-clean command

[Pavel Roskin]

On Tue, 2006-04-04 at 11:17 +0200, Alex Riesen wrote:
> On 4/4/06, Martin Waitz <tali@admingilde.org> wrote:
> > What is the use case of cleaning up all untracked files without also
> > cleaning ignored files?
> 
> Thinks of git's .gitignore: it has config.mak in it. Are you sure you want
> "clean" your build customizations?

In may case, I normally want to remove copies of the sources.  For
example, I take foo.c, make a clean copy of it, then I change and test
it.  If if doesn't work and I want to try another approach, I copy it to
foo.c-bad or foo.c-approach1.  I also make diffs between files to see
what exactly I changed.  I may also create files for output, valgrind
logs and so on.

At some point, I'm satisfied with foo.c, so I commit it.  Then I want to
remove copies, diffs and other stuff.  Yet I don't want to rebuild
everything.

It's very rare that I add a new file, and I always remember to add it to
the version control.  But I'll consider adding an option to only remove
ignored files.

-- 
Regards,
Pavel Roskin

[PATCH] Add git-clean command

[Pavel Roskin]

This command removes untracked files from the working tree.  This
implementation is based on cg-clean with some simplifications.  The
documentation is included.

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 .gitignore                  |    1 +
 Documentation/git-clean.txt |   50 +++++++++++++++++++++++++++
 Makefile                    |    2 +
 git-clean.sh                |   80 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 132 insertions(+), 1 deletions(-)

diff --git a/.gitignore b/.gitignore
index 75891c3..b5959d6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -15,6 +15,7 @@ git-checkout
 git-checkout-index
 git-cherry
 git-cherry-pick
+git-clean
 git-clone
 git-clone-pack
 git-commit
diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt
new file mode 100644
index 0000000..e91b2ed
--- /dev/null
+++ b/Documentation/git-clean.txt
@@ -0,0 +1,50 @@
+git-clean(1)
+============
+
+NAME
+----
+git-clean - Remove untracked files from the working tree
+
+SYNOPSIS
+--------
+[verse]
+'git-clean' [-d | -D] [-n] [-q] [-x]
+
+DESCRIPTION
+-----------
+Removes files unknown to git.  This allows to clean the working tree
+from files that are not under version control.  If the '-x' option is
+specified, ignored files are also removed, allowing to remove all
+build products.
+
+OPTIONS
+-------
+-d::
+	Remove untracked directories in addition to untracked files.
+
+-n::
+	Don't actually remove anything, just show what would be done.
+
+-q::
+	Be quiet, only report errors, but not the files that are
+	successfully removed.
+
+-x::
+	Don't use the ignore rules.  This allows removing all untracked
+	files, including build products.  This can be used (possibly in
+	conjunction with gitlink:git-reset[1]) to create a pristine
+	working directory to test a clean build.
+
+-X::
+	Remove only files ignored by git.  This may be useful to rebuild
+	everything from scratch, but keep manually created files.
+
+
+Author
+------
+Written by Pavel Roskin <proski@gnu.org>
+
+
+GIT
+---
+Part of the gitlink:git[7] suite
diff --git a/Makefile b/Makefile
index 145099a..3367b8c 100644
--- a/Makefile
+++ b/Makefile
@@ -114,7 +114,7 @@ ### --- END CONFIGURATION SECTION ---
 
 SCRIPT_SH = \
 	git-add.sh git-bisect.sh git-branch.sh git-checkout.sh \
-	git-cherry.sh git-clone.sh git-commit.sh \
+	git-cherry.sh git-clean.sh git-clone.sh git-commit.sh \
 	git-count-objects.sh git-diff.sh git-fetch.sh \
 	git-format-patch.sh git-log.sh git-ls-remote.sh \
 	git-merge-one-file.sh git-parse-remote.sh \
diff --git a/git-clean.sh b/git-clean.sh
new file mode 100755
index 0000000..b200868
--- /dev/null
+++ b/git-clean.sh
@@ -0,0 +1,80 @@
+#!/bin/sh
+#
+# Copyright (c) 2005-2006 Pavel Roskin
+#
+
+USAGE="[-d] [-n] [-q] [-x | -X]"
+LONG_USAGE='Clean untracked files from the working directory
+	-d	remove directories as well
+	-n 	don'\''t remove anything, just show what would be done
+	-q	be quiet, only report errors
+	-x	remove ignored files as well
+	-X	remove only ignored files as well'
+SUBDIRECTORY_OK=Yes
+. git-sh-setup
+
+ignored=
+ignoredonly=
+cleandir=
+quiet=
+rmf="rm -f"
+rmrf="rm -rf"
+rm_refuse="echo Not removing"
+echo1="echo"
+
+while case "$#" in 0) break ;; esac
+do
+	case "$1" in
+	-d)
+		cleandir=1
+		;;
+	-n)
+		quiet=1
+		rmf="echo Would remove"
+		rmrf="echo Would remove"
+		rm_refuse="echo Would not remove"
+		echo1=":"
+		;;
+	-q)
+		quiet=1
+		;;
+	-x)
+		ignored=1
+		;;
+	-X)
+		ignoredonly=1
+		;;
+	*)
+		usage
+	esac
+	shift
+done
+
+case "$ignored,$ignoredonly" in
+	1,1) usage;;
+esac
+
+if [ -z "$ignored" ]; then
+	excl="--exclude-per-directory=.gitignore"
+	if [ -f "$GIT_DIR/info/exclude" ]; then
+		excl_info="--exclude-from=$GIT_DIR/info/exclude"
+	fi
+	if [ "$ignoredonly" ]; then
+		excl="$excl --ignored"
+	fi
+fi
+
+git-ls-files --others --directory $excl ${excl_info:+"$excl_info"} |
+while read -r file; do
+	if [ -d "$file" -a ! -L "$file" ]; then
+		if [ -z "$cleandir" ]; then
+			$rm_refuse "$file"
+			continue
+		fi
+		$echo1 "Removing $file"
+		$rmrf "$file"
+	else
+		$echo1 "Removing $file"
+		$rmf "$file"
+	fi
+done

Re: [PATCH] Add git-clean command

[Jakub Narebski]

Pavel Roskin wrote:

> +SYNOPSIS
> +--------
> +[verse]
> +'git-clean' [-d | -D] [-n] [-q] [-x]

Shouldn't it be
> +'git-clean' [-d] [-n] [-q] [-x | -X]
like in below?

> +USAGE="[-d] [-n] [-q] [-x | -X]"

-- 
Jakub Narebski
Warsaw, Poland

[PATCH] Add git-clean command

[Pavel Roskin]

This command removes untracked files from the working tree.  This
implementation is based on cg-clean with some simplifications.  The
documentation is included.

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 0 files changed, 0 insertions(+), 0 deletions(-)