git apply fails with 'error: git apply: failed to read: No such file or directory'

git apply fails with 'error: git apply: failed to read: No such file or directory'

[Premek Vysoky]

What did you do before the bug happened? (Steps to reproduce your issue)
git clone https://github.com/dotnet/llvm-project
git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)**/*' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg'
mkdir foo
git -C foo init
git -C foo apply --cached --ignore-space-change ../llvm.patch

What did you expect to happen? (Expected behavior)
Patch should be created and applied to another folder

What happened instead? (Actual behavior)
git apply fails with 'error: git apply: failed to read: No such file or directory'
(and returns 128)

What's different between what you expected and what actually happened?
There is no error. I tried -v, --reject etc but wasn't able to get any details.

Anything else you want to add:
This happens in both Windows and Linux environments.

I tried analyzing file handlers via procmon.exe and could not see anything. git reads the whole patch and then shuts down. No indication of a file it is trying to open.

I tried excluding more files from the patch and then I was able to create/apply the patch, like so:

git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)**/*' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg' ':(exclude,glob)bolt' ':(exclude,glob)clang/docs' ':(exclude,glob)clang/www' ':(exclude,glob)flang' ':(exclude,glob)libclc' ':(exclude,glob)lldb' ':(exclude,glob)llvm/docs' ':(exclude,glob)mlir' ':(exclude,glob)openmp' ':(exclude,glob)polly' ':(exclude,glob)pstl' ':(exclude,glob)third-party' ':(exclude,glob)**/tests/**'

But if I do the opposite and create a patch with only the previously excluded files, I'd expect that it breaks. But this works as well!

git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)bolt' ':(glob)clang/docs' ':(glob)clang/www' ':(glob)flang' ':(glob)libclc' ':(glob)lldb' ':(glob)llvm/docs' ':(glob)mlir' ':(glob)openmp' ':(glob)polly' ':(glob)pstl' ':(glob)third-party' ':(glob)**/tests/**' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg'

Only when they are together, git fails. Size of the patch is 1GB but I've had 3GB patches apply successfully before.

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.


[System Info]
git version:
git version 2.41.0.windows.1
cpu: x86_64
built from commit: ff94e79c4724635915dbb3d4ba38f6bb91528260
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon
uname: Windows 10.0 22621 
compiler info: gnuc: 13.1
libc info: no libc information available
$SHELL (typically, interactive shell): <unset>


[Enabled Hooks]
not run from a git repository - no hooks to show

Re: git apply fails with 'error: git apply: failed to read: No such file or directory'

[Phillip Wood]

Hi Premek

Thanks for taking the time to report this issue

On 23/06/2023 13:42, Premek Vysoky wrote:
> What did you do before the bug happened? (Steps to reproduce your issue)
> git clone https://github.com/dotnet/llvm-project
> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)**/*' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg'
> mkdir foo
> git -C foo init
> git -C foo apply --cached --ignore-space-change ../llvm.patch
> 
> What did you expect to happen? (Expected behavior)
> Patch should be created and applied to another folder
> 
> What happened instead? (Actual behavior)
> git apply fails with 'error: git apply: failed to read: No such file or directory'
> (and returns 128)

This stems from commit f1c0e3946e (apply: reject patches larger than ~1 
GiB, 2022-10-25). Unfortunately it does not provide a very helpful error 
message. I'll submit a patch later to fix that in the next few days.

Best Wishes

Phillip

> What's different between what you expected and what actually happened?
> There is no error. I tried -v, --reject etc but wasn't able to get any details.
> 
> Anything else you want to add:
> This happens in both Windows and Linux environments.
> 
> I tried analyzing file handlers via procmon.exe and could not see anything. git reads the whole patch and then shuts down. No indication of a file it is trying to open.
> 
> I tried excluding more files from the patch and then I was able to create/apply the patch, like so:
> 
> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)**/*' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg' ':(exclude,glob)bolt' ':(exclude,glob)clang/docs' ':(exclude,glob)clang/www' ':(exclude,glob)flang' ':(exclude,glob)libclc' ':(exclude,glob)lldb' ':(exclude,glob)llvm/docs' ':(exclude,glob)mlir' ':(exclude,glob)openmp' ':(exclude,glob)polly' ':(exclude,glob)pstl' ':(exclude,glob)third-party' ':(exclude,glob)**/tests/**'
> 
> But if I do the opposite and create a patch with only the previously excluded files, I'd expect that it breaks. But this works as well!
> 
> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)bolt' ':(glob)clang/docs' ':(glob)clang/www' ':(glob)flang' ':(glob)libclc' ':(glob)lldb' ':(glob)llvm/docs' ':(glob)mlir' ':(glob)openmp' ':(glob)polly' ':(glob)pstl' ':(glob)third-party' ':(glob)**/tests/**' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg'
> 
> Only when they are together, git fails. Size of the patch is 1GB but I've had 3GB patches apply successfully before.
> 
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
> 
> 
> [System Info]
> git version:
> git version 2.41.0.windows.1
> cpu: x86_64
> built from commit: ff94e79c4724635915dbb3d4ba38f6bb91528260
> sizeof-long: 4
> sizeof-size_t: 8
> shell-path: /bin/sh
> feature: fsmonitor--daemon
> uname: Windows 10.0 22621
> compiler info: gnuc: 13.1
> libc info: no libc information available
> $SHELL (typically, interactive shell): <unset>
> 
> 
> [Enabled Hooks]
> not run from a git repository - no hooks to show

RE: [EXTERNAL] Re: git apply fails with 'error: git apply: failed to read: No such file or directory'

[Premek Vysoky]

Hi Phillip,

Thanks for the fast response! And thanks for the improvement if you decide to make this more obvious.
I must have used an older git version earlier as I was able to process even 3GB patches easily.

Too bad there is no override for this..
(we're working on a .NET monorepo - https://github.com/dotnet/dotnet - so one of a kind, and we hit these limits at times)

Cheers,
Premek


-----Original Message-----
From: Phillip Wood <phillip.wood123@gmail.com> 
Sent: Saturday, June 24, 2023 12:09 PM
To: Premek Vysoky <Premek.Vysoky@microsoft.com>; git@vger.kernel.org
Subject: [EXTERNAL] Re: git apply fails with 'error: git apply: failed to read: No such file or directory'

Hi Premek

Thanks for taking the time to report this issue

On 23/06/2023 13:42, Premek Vysoky wrote:
> What did you do before the bug happened? (Steps to reproduce your 
> issue) git clone 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fdotnet%2Fllvm-project&data=05%7C01%7CPremek.Vysoky%40microsof
> t.com%7Ca9684566250842b65be608db749b030b%7C72f988bf86f141af91ab2d7cd01
> 1db47%7C1%7C0%7C638231981398613951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7
> C&sdata=prrT8WKHGtvll2XG%2FKSrxSOplev6eLjjeBGu%2FRQzOMg%3D&reserved=0
> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)**/*' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg'
> mkdir foo
> git -C foo init
> git -C foo apply --cached --ignore-space-change ../llvm.patch
>
> What did you expect to happen? (Expected behavior) Patch should be 
> created and applied to another folder
>
> What happened instead? (Actual behavior) git apply fails with 'error: 
> git apply: failed to read: No such file or directory'
> (and returns 128)

This stems from commit f1c0e3946e (apply: reject patches larger than ~1 GiB, 2022-10-25). Unfortunately it does not provide a very helpful error message. I'll submit a patch later to fix that in the next few days.

Best Wishes

Phillip

> What's different between what you expected and what actually happened?
> There is no error. I tried -v, --reject etc but wasn't able to get any details.
>
> Anything else you want to add:
> This happens in both Windows and Linux environments.
>
> I tried analyzing file handlers via procmon.exe and could not see anything. git reads the whole patch and then shuts down. No indication of a file it is trying to open.
>
> I tried excluding more files from the patch and then I was able to create/apply the patch, like so:
>
> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)**/*' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg' ':(exclude,glob)bolt' ':(exclude,glob)clang/docs' ':(exclude,glob)clang/www' ':(exclude,glob)flang' ':(exclude,glob)libclc' ':(exclude,glob)lldb' ':(exclude,glob)llvm/docs' ':(exclude,glob)mlir' ':(exclude,glob)openmp' ':(exclude,glob)polly' ':(exclude,glob)pstl' ':(exclude,glob)third-party' ':(exclude,glob)**/tests/**'
>
> But if I do the opposite and create a patch with only the previously excluded files, I'd expect that it breaks. But this works as well!
>
> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)bolt' ':(glob)clang/docs' ':(glob)clang/www' ':(glob)flang' ':(glob)libclc' ':(glob)lldb' ':(glob)llvm/docs' ':(glob)mlir' ':(glob)openmp' ':(glob)polly' ':(glob)pstl' ':(glob)third-party' ':(glob)**/tests/**' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg'
>
> Only when they are together, git fails. Size of the patch is 1GB but I've had 3GB patches apply successfully before.
>
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
>
>
> [System Info]
> git version:
> git version 2.41.0.windows.1
> cpu: x86_64
> built from commit: ff94e79c4724635915dbb3d4ba38f6bb91528260
> sizeof-long: 4
> sizeof-size_t: 8
> shell-path: /bin/sh
> feature: fsmonitor--daemon
> uname: Windows 10.0 22621
> compiler info: gnuc: 13.1
> libc info: no libc information available $SHELL (typically, 
> interactive shell): <unset>
>
>
> [Enabled Hooks]
> not run from a git repository - no hooks to show

Re: [EXTERNAL] Re: git apply fails with 'error: git apply: failed to read: No such file or directory'

[Phillip Wood]

Hi Premek

On 24/06/2023 11:28, Premek Vysoky wrote:
> Hi Phillip,
> 
> Thanks for the fast response! And thanks for the improvement if you decide to make this more obvious.
> I must have used an older git version earlier as I was able to process even 3GB patches easily.
> 
> Too bad there is no override for this..

The limit was added because of concerns about integer overflows with 
larger patches (I've reproduced the commit message below). In principle 
the code could be updated to use size_t rather than int/long to 
accommodate larger patches though I haven't looked how difficult that 
would be and we'd still need to restrict the size of the diff for each 
file as the object store uses long rather than size_t which causes 
problems on LLP64 platforms like windows.

If you want to export a subset of files from a repository then 
git-filter-repo[1] might be a better bet. It uses fast-export and 
fast-import which should be more efficient than generating patches and 
applying them.

Best Wishes

Phillip

[1] https://github.com/newren/git-filter-repo

Here is the commit message from the commit that introduced the limit on 
patch sizes

apply: reject patches larger than ~1 GiB

The apply code is not prepared to handle extremely large files. It uses
"int" in some places, and "unsigned long" in others.

This combination leads to unfortunate problems when switching between
the two types. Using "int" prevents us from handling large files, since
large offsets will wrap around and spill into small negative values,
which can result in wrong behavior (like accessing the patch buffer with
a negative offset).

Converting from "unsigned long" to "int" also has truncation problems
even on LLP64 platforms where "long" is the same size as "int", since
the former is unsigned but the latter is not.

To avoid potential overflow and truncation issues in `git apply`, apply
similar treatment as in dcd1742e56 (xdiff: reject files larger than
~1GB, 2015-09-24), where the xdiff code was taught to reject large
files for similar reasons.

The maximum size was chosen somewhat arbitrarily, but picking a value
just shy of a gigabyte allows us to double it without overflowing 2^31-1
(after which point our value would wrap around to a negative number).
To give ourselves a bit of extra margin, the maximum patch size is a MiB
smaller than a full GiB, which gives us some slop in case we allocate
"(records + 1) * sizeof(int)" or similar.

Luckily, the security implications of these conversion issues are
relatively uninteresting, because a victim needs to be convinced to
apply a malicious patch.

Reported-by: 정재우 <thebound7@gmail.com>
Suggested-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>


> (we're working on a .NET monorepo - https://github.com/dotnet/dotnet - so one of a kind, and we hit these limits at times)
> 
> Cheers,
> Premek
> 
> 
> -----Original Message-----
> From: Phillip Wood <phillip.wood123@gmail.com>
> Sent: Saturday, June 24, 2023 12:09 PM
> To: Premek Vysoky <Premek.Vysoky@microsoft.com>; git@vger.kernel.org
> Subject: [EXTERNAL] Re: git apply fails with 'error: git apply: failed to read: No such file or directory'
> 
> Hi Premek
> 
> Thanks for taking the time to report this issue
> 
> On 23/06/2023 13:42, Premek Vysoky wrote:
>> What did you do before the bug happened? (Steps to reproduce your
>> issue) git clone
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
>> ub.com%2Fdotnet%2Fllvm-project&data=05%7C01%7CPremek.Vysoky%40microsof
>> t.com%7Ca9684566250842b65be608db749b030b%7C72f988bf86f141af91ab2d7cd01
>> 1db47%7C1%7C0%7C638231981398613951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
>> wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7
>> C&sdata=prrT8WKHGtvll2XG%2FKSrxSOplev6eLjjeBGu%2FRQzOMg%3D&reserved=0
>> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)**/*' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg'
>> mkdir foo
>> git -C foo init
>> git -C foo apply --cached --ignore-space-change ../llvm.patch
>>
>> What did you expect to happen? (Expected behavior) Patch should be
>> created and applied to another folder
>>
>> What happened instead? (Actual behavior) git apply fails with 'error:
>> git apply: failed to read: No such file or directory'
>> (and returns 128)
> 
> This stems from commit f1c0e3946e (apply: reject patches larger than ~1 GiB, 2022-10-25). Unfortunately it does not provide a very helpful error message. I'll submit a patch later to fix that in the next few days.
> 
> Best Wishes
> 
> Phillip
> 
>> What's different between what you expected and what actually happened?
>> There is no error. I tried -v, --reject etc but wasn't able to get any details.
>>
>> Anything else you want to add:
>> This happens in both Windows and Linux environments.
>>
>> I tried analyzing file handlers via procmon.exe and could not see anything. git reads the whole patch and then shuts down. No indication of a file it is trying to open.
>>
>> I tried excluding more files from the patch and then I was able to create/apply the patch, like so:
>>
>> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)**/*' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg' ':(exclude,glob)bolt' ':(exclude,glob)clang/docs' ':(exclude,glob)clang/www' ':(exclude,glob)flang' ':(exclude,glob)libclc' ':(exclude,glob)lldb' ':(exclude,glob)llvm/docs' ':(exclude,glob)mlir' ':(exclude,glob)openmp' ':(exclude,glob)polly' ':(exclude,glob)pstl' ':(exclude,glob)third-party' ':(exclude,glob)**/tests/**'
>>
>> But if I do the opposite and create a patch with only the previously excluded files, I'd expect that it breaks. But this works as well!
>>
>> git -C llvm-project diff --patch --binary --output ../llvm.patch 4b825dc642cb6eb9a060e54bf8d69288fbee4904..30e9e6bc2e9f04e0a75daf4b8088ee91f66069da -- ':(glob)bolt' ':(glob)clang/docs' ':(glob)clang/www' ':(glob)flang' ':(glob)libclc' ':(glob)lldb' ':(glob)llvm/docs' ':(glob)mlir' ':(glob)openmp' ':(glob)polly' ':(glob)pstl' ':(glob)third-party' ':(glob)**/tests/**' ':(exclude,glob)**/*.dll' ':(exclude,glob)**/*.Dll' ':(exclude,glob)**/*.exe' ':(exclude,glob)**/*.pdb' ':(exclude,glob)**/*.mdb' ':(exclude,glob)**/*.zip' ':(exclude,glob)**/*.nupkg'
>>
>> Only when they are together, git fails. Size of the patch is 1GB but I've had 3GB patches apply successfully before.
>>
>> Please review the rest of the bug report below.
>> You can delete any lines you don't wish to share.
>>
>>
>> [System Info]
>> git version:
>> git version 2.41.0.windows.1
>> cpu: x86_64
>> built from commit: ff94e79c4724635915dbb3d4ba38f6bb91528260
>> sizeof-long: 4
>> sizeof-size_t: 8
>> shell-path: /bin/sh
>> feature: fsmonitor--daemon
>> uname: Windows 10.0 22621
>> compiler info: gnuc: 13.1
>> libc info: no libc information available $SHELL (typically,
>> interactive shell): <unset>
>>
>>
>> [Enabled Hooks]
>> not run from a git repository - no hooks to show