Selectively commit/publish files to GIT

Selectively commit/publish files to GIT

[Thomas Manson]

Hi,

  I've converting my Bazaar repository to GIT.

 I've successfully done this conversion and I now want to publish my
source code to github.

  The problem is that in Bazaar, I've commited some big files (63MB &
173MB), but this files are no longer in my project, only in the
revisions files of Bazaar and now Git.

  I don't need this files to be pushed on Github.

  How can I search git history for big files and remove them, or mark
them to be not published ?

I've tryed this solution found on the link in the error:

git filter-branch --force --index-filter   'git rm --cached
--ignore-unmatch giant_file'   --prune-empty --tag-name-filter cat --
--all
git commit --amend -CHEAD

Then I've tryed to Commit & Push from Github mac  application and I
had several network error, and finally get the same error on giant
files (maybe my multiple commit & push did override something, but I
understood that the git rm command would remove things once for
good...

can anybody help me ?
I'm blocked in my dev because of this, I can't share my project with a friend.
I'm publishing here : https://github.com/dev-mansonthomas/crf-rdp.git
(paying for storage is an option as I'm quite fed up loosing time for
filesize...)

Regards,
Thomas.

here is the error I have using the GitHub application on Mac :
(after that I intend to use Eclipse)

File Ressources/dwr/dwr-3.0.0.110.dev-src.zip is 67.69 MB; this is
larger than GitHub's recommended maximum file size of 50 MB
GH001: Large files detected.
Trace: 8f0259b29260f0c4d7ae4d4ae70e0306
See http://git.io/iEPt8g for more information.
File .bzr/repository/packs/a7bcd6ba235114ab88c80fe8a97adcfa.pack is
178.76 MB; this exceeds GitHub's file size limit of 100 MB

Re: Selectively commit/publish files to GIT

[Magnus Bäck]

On Friday, November 08, 2013 at 05:52 EST,
     Thomas Manson <dev.mansonthomas@gmail.com> wrote:

[...]

>   How can I search git history for big files and remove them, or mark
> them to be not published ?
> 
> I've tryed this solution found on the link in the error:
> 
> git filter-branch --force --index-filter   'git rm --cached
> --ignore-unmatch giant_file'   --prune-empty --tag-name-filter cat --
> --all
> git commit --amend -CHEAD

[...]

> here is the error I have using the GitHub application on Mac :
> (after that I intend to use Eclipse)
> 
> File Ressources/dwr/dwr-3.0.0.110.dev-src.zip is 67.69 MB; this is
> larger than GitHub's recommended maximum file size of 50 MB
> GH001: Large files detected.
> Trace: 8f0259b29260f0c4d7ae4d4ae70e0306
> See http://git.io/iEPt8g for more information.
> File .bzr/repository/packs/a7bcd6ba235114ab88c80fe8a97adcfa.pack is
> 178.76 MB; this exceeds GitHub's file size limit of 100 MB

Did you actually replace the 'giant_file' placeholder with the path to
*your* giant file? The error message indicates that you didn't, or that
something else went wrong during the filtering.

You can use 'git log' to look for commits touching particular pathnames,
making it easy to verify that a file has been eradicated.

git log --all -- path/to/big-honking-file.zip

-- 
Magnus Bäck
baeck@google.com

Re: Selectively commit/publish files to GIT

[Thomas Manson]

Hi Magnus,

 thanks for pointing that out. I must have overlooked the webpage,
when I saw that I thought it was some kind of alias that designated
files greater than a specific files (server side configurated as the
100MB limit)

  I'll try again ;)

Thomas.

On Fri, Nov 8, 2013 at 3:13 PM, Magnus Bäck <baeck@google.com> wrote:
> On Friday, November 08, 2013 at 05:52 EST,
>      Thomas Manson <dev.mansonthomas@gmail.com> wrote:
>
> [...]
>
>>   How can I search git history for big files and remove them, or mark
>> them to be not published ?
>>
>> I've tryed this solution found on the link in the error:
>>
>> git filter-branch --force --index-filter   'git rm --cached
>> --ignore-unmatch giant_file'   --prune-empty --tag-name-filter cat --
>> --all
>> git commit --amend -CHEAD
>
> [...]
>
>> here is the error I have using the GitHub application on Mac :
>> (after that I intend to use Eclipse)
>>
>> File Ressources/dwr/dwr-3.0.0.110.dev-src.zip is 67.69 MB; this is
>> larger than GitHub's recommended maximum file size of 50 MB
>> GH001: Large files detected.
>> Trace: 8f0259b29260f0c4d7ae4d4ae70e0306
>> See http://git.io/iEPt8g for more information.
>> File .bzr/repository/packs/a7bcd6ba235114ab88c80fe8a97adcfa.pack is
>> 178.76 MB; this exceeds GitHub's file size limit of 100 MB
>
> Did you actually replace the 'giant_file' placeholder with the path to
> *your* giant file? The error message indicates that you didn't, or that
> something else went wrong during the filtering.
>
> You can use 'git log' to look for commits touching particular pathnames,
> making it easy to verify that a file has been eradicated.
>
> git log --all -- path/to/big-honking-file.zip
>
> --
> Magnus Bäck
> baeck@google.com

Re: Selectively commit/publish files to GIT

[Thomas Manson]

Thanks Magnus, it did solve my issue.

I've noticed by the way that i've accidentally commited the whole .bzr
directory (that hold all my commit in Bazaar style).

I've deleted this folder as well using -r

git filter-branch --force --index-filter   'git rm -r --cached
--ignore-unmatch .bzr'   --prune-empty --tag-name-filter cat -- --all




for reference, an example of working run :

Thomass-MacBook-Pro:crf-rdp tmanson$ git filter-branch --force
--index-filter   'git rm --cached --ignore-unmatch
Ressources/dwr/dwr-3.0.0.110.dev-src.zip'   --prune-empty
--tag-name-filter cat -- --all
Rewrite 58952de4e10bc1df2f8c324e94d249a0366f7512 (65/158)rm
'Ressources/dwr/dwr-3.0.0.110.dev-src.zip'
Rewrite e661d08f1c33d7bd53e2c2231a3296b2ade5a2dd (66/158)rm
'Ressources/dwr/dwr-3.0.0.110.dev-src.zip'
Rewrite 8ff3f3f9ffb901b7cdc1dba220a121bab0698121 (67/158)rm
'Ressources/dwr/dwr-3.0.0.110.dev-src.zip'
Rewrite cfec6341565c35c410f248e5861543ac01b84f40 (158/158)
Ref 'refs/heads/master' was rewritten
Ref 'refs/tags/version-0-1_20081120' was rewritten
Ref 'refs/tags/version-0-5-1_20120512' was rewritten


On Fri, Nov 8, 2013 at 11:51 PM, Thomas Manson
<dev.mansonthomas@gmail.com> wrote:
> Hi Magnus,
>
>  thanks for pointing that out. I must have overlooked the webpage,
> when I saw that I thought it was some kind of alias that designated
> files greater than a specific files (server side configurated as the
> 100MB limit)
>
>   I'll try again ;)
>
> Thomas.
>
> On Fri, Nov 8, 2013 at 3:13 PM, Magnus Bäck <baeck@google.com> wrote:
>> On Friday, November 08, 2013 at 05:52 EST,
>>      Thomas Manson <dev.mansonthomas@gmail.com> wrote:
>>
>> [...]
>>
>>>   How can I search git history for big files and remove them, or mark
>>> them to be not published ?
>>>
>>> I've tryed this solution found on the link in the error:
>>>
>>> git filter-branch --force --index-filter   'git rm --cached
>>> --ignore-unmatch giant_file'   --prune-empty --tag-name-filter cat --
>>> --all
>>> git commit --amend -CHEAD
>>
>> [...]
>>
>>> here is the error I have using the GitHub application on Mac :
>>> (after that I intend to use Eclipse)
>>>
>>> File Ressources/dwr/dwr-3.0.0.110.dev-src.zip is 67.69 MB; this is
>>> larger than GitHub's recommended maximum file size of 50 MB
>>> GH001: Large files detected.
>>> Trace: 8f0259b29260f0c4d7ae4d4ae70e0306
>>> See http://git.io/iEPt8g for more information.
>>> File .bzr/repository/packs/a7bcd6ba235114ab88c80fe8a97adcfa.pack is
>>> 178.76 MB; this exceeds GitHub's file size limit of 100 MB
>>
>> Did you actually replace the 'giant_file' placeholder with the path to
>> *your* giant file? The error message indicates that you didn't, or that
>> something else went wrong during the filtering.
>>
>> You can use 'git log' to look for commits touching particular pathnames,
>> making it easy to verify that a file has been eradicated.
>>
>> git log --all -- path/to/big-honking-file.zip
>>
>> --
>> Magnus Bäck
>> baeck@google.com

Re: Selectively commit/publish files to GIT

[Keshav Kini]

Thomas Manson <dev.mansonthomas@gmail.com> writes:
>   The problem is that in Bazaar, I've commited some big files (63MB &
> 173MB), but this files are no longer in my project, only in the
> revisions files of Bazaar and now Git.
>
>   I don't need this files to be pushed on Github.
>
>   How can I search git history for big files and remove them, or mark
> them to be not published ?

I see you already solved your problem, but for future reference, there
is a nice tool that is more limited in functionality than git
filter-branch but also much faster and possibly easier to use:

    http://rtyley.github.io/bfg-repo-cleaner/

-Keshav