[RFC] Removing deleted files after checkout

[RFC] Removing deleted files after checkout

[Carl Baldwin]

[-- Attachment #1: Type: text/plain, Size: 2000 bytes --]

Hello,

I recently started using git to revision control the source for my
web-page.  I wrote a post-update hook to checkout the files when I push
to the 'live' repository.

In this particular context I decided that it was important to me to remove
deleted files after checking out the new HEAD.  I accomplished this by running
git-ls-files before and after the checkout.

Is there a better way?  Could there be some way built into git to easily
find out what files dissappear when replacing the current index with one
from a new tree?  Is there already?  The behavior of git should NOT
change to delete these files but I would argue that some way should
exist to query what files disappeared if removing them is desired.

Here is some code that I wrote for this.  It feels a bit hackish to me but I
couldn't think of anything better.  Comments and criticism are welcome.

#!/bin/sh

# HEAD changed so checkout the new HEAD deleted any files that should no longer
# be around.
oldlist=$(tempfile)
newlist=$(tempfile)
removedlist=$(tempfile)

git-ls-files | sort -r > $oldlist
git-checkout-script -f
git-ls-files | sort -r > $newlist

diff -u $oldlist $newlist |
  tail -n +4 |
  sed -n 's/^-//p' > $removedlist

# Remove each file
cat $removedlist | xargs -rl rm -f
# Remove the directories if empty
cat $removedlist | xargs -rl dirname | xargs -rl rmdir -p --ignore-fail-on-non-empty

rm -f $oldlist $newlist $removedlist

# --- snip ---

If you are interested I attached the full post-update hook script that I
actually use to do this.  Again, comments are welcome.

Thanks,
Carl

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        Systems VLSI Laboratory
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[-- Attachment #2: post-update --]
[-- Type: text/plain, Size: 827 bytes --]

#!/bin/sh

export PATH=/usr/local/bin:/usr/bin:/bin

# cd to the root of the project directory (assume one dir up from GIT_DIR)
cd $GIT_DIR/..
unset GIT_DIR

# Set up some temporary files and a trap to delete them
oldlist=$(tempfile)
newlist=$(tempfile)
removelist=$(tempfile)
trap "rm -f $oldlist $newlist $removelist" 0 1 2 3 15

# Get list of files from the current index
git-ls-files | sort -r > $oldlist

# Checkout the index to the working directory
git-checkout-script -f

# Get list of files from the current (new) index
git-ls-files | sort -r > $newlist

# Use diff to determine which files to remove from the working copy
diff -u $oldlist $newlist |
  tail -n +4 |
  sed -n 's/^-//p' > $removelist

cat $removelist | xargs -rl rm -f
cat $removelist | xargs -rl dirname | xargs -rl rmdir -p --ignore-fail-on-non-empty

Re: [RFC] Removing deleted files after checkout

[Daniel Barkalow]

On Tue, 23 Aug 2005, Carl Baldwin wrote:

> Hello,
>
> I recently started using git to revision control the source for my
> web-page.  I wrote a post-update hook to checkout the files when I push
> to the 'live' repository.
>
> In this particular context I decided that it was important to me to remove
> deleted files after checking out the new HEAD.  I accomplished this by running
> git-ls-files before and after the checkout.
>
> Is there a better way?  Could there be some way built into git to easily
> find out what files dissappear when replacing the current index with one
> from a new tree?  Is there already?  The behavior of git should NOT
> change to delete these files but I would argue that some way should
> exist to query what files disappeared if removing them is desired.

If you don't use -f, git-checkout-script removes deleted files. Using -f
tells it to ignore the old index, which means that it can't tell the
difference between removed files and files that weren't tracked at all.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Removing deleted files after checkout

[Carl Baldwin]

On Tue, Aug 23, 2005 at 03:43:56PM -0400, Daniel Barkalow wrote:
> On Tue, 23 Aug 2005, Carl Baldwin wrote:
> 
> > Hello,
> >
> > I recently started using git to revision control the source for my
> > web-page.  I wrote a post-update hook to checkout the files when I push
> > to the 'live' repository.
> >
> > In this particular context I decided that it was important to me to remove
> > deleted files after checking out the new HEAD.  I accomplished this by running
> > git-ls-files before and after the checkout.
> >
> > Is there a better way?  Could there be some way built into git to easily
> > find out what files dissappear when replacing the current index with one
> > from a new tree?  Is there already?  The behavior of git should NOT
> > change to delete these files but I would argue that some way should
> > exist to query what files disappeared if removing them is desired.
> 
> If you don't use -f, git-checkout-script removes deleted files. Using -f
> tells it to ignore the old index, which means that it can't tell the
> difference between removed files and files that weren't tracked at all.

Maybe I'm doing something wrong.  This does not happen for me.

I tried a simple test with git v0.99.4...

cd
mkdir test-git && cd test-git/
echo testing | cg-init
echo contents > file
git-add-script file
git-commit-script -m 'testing'
cd ..
cg-clone test-git/.git/ test-git2
cd test-git2
cg-rm file
git-commit-script -m 'testing'
ls
cg-push
cd ../test-git
git-checkout-script
ls
git-status-script

At this point, I want 'file' to be gone.  It is, however, still there.
That is the situation that my code was meant to handle.  Maybe you were
thinking of something different?  Maybe there is new code since 0.99.4?

Now, I think it would be wrong for git-checkout-script to actually
remove files unless an option were given to do so.  So, the behavior
that I observe is correct in my opinion.  I'm looking for a way to find
out what files should be deleted if that is desired.

Thanks,
Carl

> 	-Daniel
> *This .sig left intentionally blank*
> 

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        Systems VLSI Laboratory
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Re: [RFC] Removing deleted files after checkout

[Daniel Barkalow]

On Tue, 23 Aug 2005, Carl Baldwin wrote:

> On Tue, Aug 23, 2005 at 03:43:56PM -0400, Daniel Barkalow wrote:
> > On Tue, 23 Aug 2005, Carl Baldwin wrote:
> >
> > > Hello,
> > >
> > > I recently started using git to revision control the source for my
> > > web-page.  I wrote a post-update hook to checkout the files when I push
> > > to the 'live' repository.
> > >
> > > In this particular context I decided that it was important to me to remove
> > > deleted files after checking out the new HEAD.  I accomplished this by running
> > > git-ls-files before and after the checkout.
> > >
> > > Is there a better way?  Could there be some way built into git to easily
> > > find out what files dissappear when replacing the current index with one
> > > from a new tree?  Is there already?  The behavior of git should NOT
> > > change to delete these files but I would argue that some way should
> > > exist to query what files disappeared if removing them is desired.
> >
> > If you don't use -f, git-checkout-script removes deleted files. Using -f
> > tells it to ignore the old index, which means that it can't tell the
> > difference between removed files and files that weren't tracked at all.
>
> Maybe I'm doing something wrong.  This does not happen for me.
>
> I tried a simple test with git v0.99.4...
>
> cd
> mkdir test-git && cd test-git/
> echo testing | cg-init
> echo contents > file
> git-add-script file
> git-commit-script -m 'testing'

[point 1]

> cd ..
> cg-clone test-git/.git/ test-git2
> cd test-git2
> cg-rm file
> git-commit-script -m 'testing'
> ls

> cg-push
> cd ../test-git
> git-checkout-script

Ah, okay. I think "push" and "checkout" don't play that well together;
"push" changes the ref, which "checkout" uses to determine what it expects
for the old contents, and then it's confused.

What you probably actually want is:

cd ../test-git
git pull ../test-git2

which will correctly identify before and after, and remove any files that
were removed.

Alternatively, you could do, at point 1:

cp .git/refs/master .git/refs/deployed
git checkout deployed

Then, after the push and cd:

git checkout master
cp .git/refs/master .git/refs/deployed
git checkout deployed

because checkout does remove files if you switch from a branch with them
(e.g., deployed) to one without them (master, after the push).

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Removing deleted files after checkout

[Carl Baldwin]

On Tue, Aug 23, 2005 at 05:27:12PM -0400, Daniel Barkalow wrote:
> On Tue, 23 Aug 2005, Carl Baldwin wrote:
> 
> > On Tue, Aug 23, 2005 at 03:43:56PM -0400, Daniel Barkalow wrote:
> > > On Tue, 23 Aug 2005, Carl Baldwin wrote:
> > >
> > > > Hello,
> > > >
> > > > I recently started using git to revision control the source for my
> > > > web-page.  I wrote a post-update hook to checkout the files when I push
> > > > to the 'live' repository.
> > > >
> > > > In this particular context I decided that it was important to me to remove
> > > > deleted files after checking out the new HEAD.  I accomplished this by running
> > > > git-ls-files before and after the checkout.
> > > >
> > > > Is there a better way?  Could there be some way built into git to easily
> > > > find out what files dissappear when replacing the current index with one
> > > > from a new tree?  Is there already?  The behavior of git should NOT
> > > > change to delete these files but I would argue that some way should
> > > > exist to query what files disappeared if removing them is desired.
> > >
> > > If you don't use -f, git-checkout-script removes deleted files. Using -f
> > > tells it to ignore the old index, which means that it can't tell the
> > > difference between removed files and files that weren't tracked at all.
> >
> > Maybe I'm doing something wrong.  This does not happen for me.
> >
> > I tried a simple test with git v0.99.4...
> >
> > cd
> > mkdir test-git && cd test-git/
> > echo testing | cg-init
> > echo contents > file
> > git-add-script file
> > git-commit-script -m 'testing'
> 
> [point 1]
> 
> > cd ..
> > cg-clone test-git/.git/ test-git2
> > cd test-git2
> > cg-rm file
> > git-commit-script -m 'testing'
> > ls
> 
> > cg-push
> > cd ../test-git
> > git-checkout-script
> 
> Ah, okay. I think "push" and "checkout" don't play that well together;
> "push" changes the ref, which "checkout" uses to determine what it expects
> for the old contents, and then it's confused.
> 
> What you probably actually want is:
> 
> cd ../test-git
> git pull ../test-git2

The point is to push and use a post-update hook to do the checkout.  So,
this won't be possible.

> which will correctly identify before and after, and remove any files that
> were removed.
> 
> Alternatively, you could do, at point 1:
> 
> cp .git/refs/master .git/refs/deployed
> git checkout deployed

How to get a post-update hook to do this?  I suppose an update script
could set this up for the post-update to later use.

Thanks,
Carl

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        Systems VLSI Laboratory
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Re: [RFC] Removing deleted files after checkout

[Daniel Barkalow]

On Tue, 23 Aug 2005, Carl Baldwin wrote:

> The point is to push and use a post-update hook to do the checkout.  So,
> this won't be possible.

You could have the remote repository be something like
"~/git/website.git", and have a hook which does: "cd ~/www; git pull
~/git/website.git/". That is, have three things: the directory where you
work on stuff, the central storage location, and the area that the web
server serves, and have the storage location automatically update the web
server area. That's what I do with my website section that's still in CVS,
and the general concept is good (and means that the "real" repository
isn't somewhere the web server is poking around).

> > which will correctly identify before and after, and remove any files that
> > were removed.
> >
> > Alternatively, you could do, at point 1:
> >
> > cp .git/refs/master .git/refs/deployed
> > git checkout deployed
>
> How to get a post-update hook to do this?  I suppose an update script
> could set this up for the post-update to later use.

If you have "deployed" checked out, and you push to "master" in the same
repository, having the hook do "git resolve deployed master auto-update"
should work.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Removing deleted files after checkout

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

>> > If you don't use -f, git-checkout-script removes deleted files. Using -f
>> > tells it to ignore the old index, which means that it can't tell the
>> > difference between removed files and files that weren't tracked at all.

Yes and no.  "git checkout" assumes that the index file and the
working tree somewhat resembles what is in .git/HEAD commit.
Since push operation updates .git/HEAD commit without touching
the index file, that assumption does not hold.

The update hook gets old commit name and new commit name, so you
should be able to do (untested):

    git-read-tree -m -u $old_commit $new_commit

there, of course after making sure that you had old_commit (the
first time push happens to a new ref you would not have that one).

Re: [RFC] Removing deleted files after checkout

[Carl Baldwin]

Ok, the following is what I came up with based on your response.  This
is .git/hooks/update.  It mostly works in my situation.  See below for
my discussion on what didn't work.

#!/bin/sh

export PATH=/usr/local/bin:/usr/bin:/bin

# cd to the root of the project directory (assume one dir up from GIT_DIR)
cd $GIT_DIR/..
unset GIT_DIR

if expr "$2" : '0*$' >/dev/null; then
    git-read-tree --reset $3 &&
        git-checkout-cache -q -f -u -a
else
    git-read-tree -m -u $2 $3
fi

exit 0

# --- snip ---

The thing that this doesn't do is remove empty directories when the last
file is deleted.  I once expressed the opinion in a previous thread that
directories should be added and removed explicitly in git.  (Thus
allowing an empty directory to be added).  If this were to happen then
this case would get handled correctly.  However, if git stays with the
status quo then I think that git-read-tree -u should be changed to
remove the empty directory.  This would make it consistent.

What do you think?  Ideas?

Carl

On Tue, Aug 23, 2005 at 02:54:30PM -0700, Junio C Hamano wrote:
> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> >> > If you don't use -f, git-checkout-script removes deleted files. Using -f
> >> > tells it to ignore the old index, which means that it can't tell the
> >> > difference between removed files and files that weren't tracked at all.
> 
> Yes and no.  "git checkout" assumes that the index file and the
> working tree somewhat resembles what is in .git/HEAD commit.
> Since push operation updates .git/HEAD commit without touching
> the index file, that assumption does not hold.
> 
> The update hook gets old commit name and new commit name, so you
> should be able to do (untested):
> 
>     git-read-tree -m -u $old_commit $new_commit
> 
> there, of course after making sure that you had old_commit (the
> first time push happens to a new ref you would not have that one).
> 

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        Systems VLSI Laboratory
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Re: [RFC] Removing deleted files after checkout

[Daniel Barkalow]

On Tue, 23 Aug 2005, Carl Baldwin wrote:

> The thing that this doesn't do is remove empty directories when the last
> file is deleted.  I once expressed the opinion in a previous thread that
> directories should be added and removed explicitly in git.  (Thus
> allowing an empty directory to be added).  If this were to happen then
> this case would get handled correctly.  However, if git stays with the
> status quo then I think that git-read-tree -u should be changed to
> remove the empty directory.  This would make it consistent.

I think that git-read-tree -u ought to remove a directory if it removes
the last file (or directory) in it.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Removing deleted files after checkout

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> I think that git-read-tree -u ought to remove a directory if it removes
> the last file (or directory) in it.

I concur and do not have much objections to a patch that would
do so.