RE: AI Textconv filter misconfiguration on Windows leads to silent corruption of diff output (ongoing investigation)

[<rsbecker@nexbridge.com>]

On September 1, 2026 4:14 PM, Skybuck Flying wrote:
>MORE GOD DAMN PROBLEMS WITH GIT AND CR/LF FILTERS.
>
>I DOWNLOADED/GIT CLONED:
>
>https://github.com/openai/openai-openapi/tree/main
>
>I NOTICED:
>
>https://github.com/openai/openai-openapi/tree/main/assets
>
>WAS CORRUPTED.
>
>(CORRECT DOWNLOAD METHOD USES TO PROVE FILE IS INTACT ON SERVER):
>
>curl -L --output "K:\Delphi\Specifications\OpenAI API\github version 3.1.0 (1
>september 2026)\assets\openai-api-referencev2.png"
>https://raw.githubusercontent.com/openai/openai-
>openapi/master/assets/openai-api-reference.png
>
>GOOD THING I INSPECTED IT JUST OUT OF CURIOSITY.
>
>I IMMEDIATELY EXPECTED GIT FILTER TO BE THE CAUSE.
>
>DIAGNOSIS COMMANDS.
>
>"
>Microsoft Windows [Version 10.0.22631.6199]
>(c) Microsoft Corporation. All rights reserved.
>
>C:\Users\skybu>git config --global core.autocrlf false
>
>C:\Users\skybu>git config --system core.autocrlf false
>
>C:\Users\skybu>git config --local core.autocrlf
>fatal: --local can only be used inside a git repository
>
>C:\Users\skybu>git config --global --get-regexp filter
>
>C:\Users\skybu>git config --local --get-regexp filter
>fatal: --local can only be used inside a git repository
>
>C:\Users\skybu>git check-attr -a openai-api-reference.png
>fatal: not a git repository (or any of the parent directories): .git
>
>C:\Users\skybu>type .gitattributes
>* text diff=lfclean
>C:\Users\skybu>git config --global --list core.autocrlf=false core.eol=crlf
>core.sshcommand=C:/Windows/System32/OpenSSH/ssh.exe
>core.attributesfile=C:\Users\skybu\.gitattributes
>user.email=skybuck2000@hotmail.com
>user.name=Skybuck Flying
>user.signingkey=I:\Informatie\Van mezelf\SSH Keys\PrivateKey\GitSigningKey
>gui.recentrepo=V:/FuckingWhore/vite-wallet
>cinnabar.version-check=1743733941
>credential.http://localhost:3000.provider=generic
>includeif.gitdir:V:/AI0001/.path=~/.gitconfigs/.gitconfig-ai0001-v2
>includeif.gitdir:V:/AI0002/.path=~/.gitconfigs/.gitconfig-ai0002-v2
>includeif.gitdir:V:/AI0003/.path=~/.gitconfigs/.gitconfig-ai0003-v2
>includeif.gitdir:V:/AI0004/.path=~/.gitconfigs/.gitconfig-ai0004-v2
>includeif.gitdir:V:/AI0005/.path=~/.gitconfigs/.gitconfig-ai0005-v2
>includeif.gitdir:V:/AI0006/.path=~/.gitconfigs/.gitconfig-ai0006-v2
>includeif.gitdir:V:/AI0007/.path=~/.gitconfigs/.gitconfig-ai0007-v2
>includeif.gitdir:V:/AI0008/.path=~/.gitconfigs/.gitconfig-ai0008-v2
>includeif.gitdir:V:/AI0009/.path=~/.gitconfigs/.gitconfig-ai0009-v2
>includeif.gitdir:V:/AI0010/.path=~/.gitconfigs/.gitconfig-ai0010-v2
>includeif.gitdir:V:/AI0011/.path=~/.gitconfigs/.gitconfig-ai0011-v2
>includeif.gitdir:V:/AI0012/.path=~/.gitconfigs/.gitconfig-ai0012-v2
>includeif.gitdir:V:/AI0013/.path=~/.gitconfigs/.gitconfig-ai0013-v2
>includeif.gitdir:V:/AI0014/.path=~/.gitconfigs/.gitconfig-ai0014-v2
>includeif.gitdir:V:/AI0015/.path=~/.gitconfigs/.gitconfig-ai0015-v2
>includeif.gitdir:V:/AI0016/.path=~/.gitconfigs/.gitconfig-ai0016-v2
>includeif.gitdir:V:/AI0017/.path=~/.gitconfigs/.gitconfig-ai0017-v2
>includeif.gitdir:V:/AI0018/.path=~/.gitconfigs/.gitconfig-ai0018-v2
>includeif.gitdir:V:/AI0019/.path=~/.gitconfigs/.gitconfig-ai0019-v2
>includeif.gitdir:V:/AI0020/.path=~/.gitconfigs/.gitconfig-ai0020-v2
>includeif.gitdir:V:/AIMain/.path=~/.gitconfigs/.gitconfig-aimain-v2
>merge.tool=meld
>mergetool.prompt=false
>mergetool.keepbackup=false
>gpg.format=ssh
>commit.gpgsign=true
>log.showsignature=true
>gpg.ssh.allowedsignersfile=I:/Informatie/Van mezelf/SSH
>Keys/GitAllowedSigners/GitAllowedSigners.txt
>diff.lfclean.textconv=sed -e s/\\r//
>diff.lfclean.binary=false
>
>C:\Users\skybu>git check-attr --all -- openai-api-reference.png
>fatal: not a git repository (or any of the parent directories): .git
>
>C:\Users\skybu>
>"
>
>HYPOTHESIS BY CO-PILOT AND POSSIBLE FIX:
>
>The **smoking gun is now visible**: your global config + your global attributes file
>**are applying a custom diff filter (`lfclean`) to *all files*** — including PNGs.
>That is exactly why your Git‑downloaded PNG is corrupted while the curl version is
>intact.
>
>Let me break it down cleanly and give you the exact commands to verify every part.
>
>---
>
>## 🎯 Core finding: your global `.gitattributes` is rewriting files You showed:
>
>```
>C:\Users\skybu>type .gitattributes
>* text diff=lfclean
>```
>
>This means:
>
>- `*` → **every file**, regardless of type
>- `text` → Git treats **every file as text**, even binaries
>- `diff=lfclean` → Git applies your custom diff driver
>- Your diff driver is defined as:
>
>```
>diff.lfclean.textconv=sed -e s/\\r//
>diff.lfclean.binary=false
>```
>
>This forces Git to:
>
>- run `sed -e s/\r//` on files
>- treat them as **non‑binary**
>- strip carriage returns
>- potentially corrupt binary formats (PNG, JPG, ZIP, EXE, DLL, etc.)
>
>Curl does not run Git filters → curl file stays intact.
>
>Git does run filters → Git file becomes corrupted.
>
>This is 100% the cause.
>
>---
>
>## 🧪 Commands to confirm the corruption mechanism
>
>### 1. Check what attributes Git applies to your PNG ``` git check-attr --all -- openai-
>api-reference.png ```
>
>Expected output (because of your global `.gitattributes`):
>
>```
>openai-api-reference.png: text: set
>openai-api-reference.png: diff: lfclean
>```
>
>If you see this → confirmed.
>
>---
>
>### 2. Check the raw blob stored in Git (to see if corruption happens on checkout)
>``` git show HEAD:openai-api-reference.png > raw.png ```
>
>Then compare:
>
>```
>fc /b raw.png openai-api-reference.png
>```
>
>If `raw.png` is intact but the working‑tree file is corrupted → the corruption
>happens during checkout due to filters.
>
>---
>
>### 3. Show all filters Git knows about
>```
>git config --show-origin --get-regexp diff git config --show-origin --get-regexp filter
>```
>
>You will see your `lfclean` diff driver.
>
>---
>
>### 4. Show your global attributes file path ``` git config --global core.attributesfile
>```
>
>It points to:
>
>```
>C:\Users\skybu\.gitattributes
>```
>
>This file is applied to **every repository** on your system.
>
>---
>
>## 🛠️ How to fix it (choose one)
>
>### Option A — Remove the global `.gitattributes` (recommended) ``` del
>C:\Users\skybu\.gitattributes ```
>
>### Option B — Restrict the rule to text files only
>Replace:
>
>```
>* text diff=lfclean
>```
>
>With:
>
>```
>*.txt text diff=lfclean
>*.pas text diff=lfclean
>*.dpr text diff=lfclean
>```
>
>### Option C — Mark binary files correctly
>Add:
>
>```
>*.png binary
>*.jpg binary
>*.zip binary
>*.exe binary
>*.dll binary
>```
>
>---
>
>## 📌 Why this happened
>You created a global `.gitattributes` with:
>
>```
>* text diff=lfclean
>```
>
>This is extremely dangerous because:
>
>- Git thinks **every file is text**
>- Git applies your custom diff filter to **every file**
>- Binary files get corrupted on checkout
>
>This is why:
>
>- **curl file = correct**
>- **git file = corrupted**
>
>---
>
>## 🔍 Want me to help rewrite your `.gitattributes` safely?
>I can generate a clean, safe version tailored to your Delphi workflow.
>Pick one:
>
>- fix global attributes
>- create safe Delphi attributes
>- audit all Git filters
>
>
>
>Apperently Co-Pilot now has the ability to send e-mails, funny functionality, risky
>too ! Expect another melissa ai virus soon ? ;)
>
>Anyway I copy & paste it's reply here in plain text:
>
>```
>Hi Git developers,
>
>I'm writing to report a reproducible corruption issue caused by global Git attributes
>and diff filters being applied to binary files during checkout.
>
>Summary of the problem
>A PNG file downloaded via Git becomes corrupted, while the same file downloaded
>via curl remains intact. The corruption is caused by a global `.gitattributes` file that
>marks all files as text and assigns a custom diff driver:
>
>* text diff=lfclean
>
>The diff driver is defined globally as:
>
>diff.lfclean.textconv=sed -e s/\\r//
>diff.lfclean.binary=false
>
>This forces Git to treat every file as text and run a textconv filter that strips carriage
>returns. When applied to binary formats (PNG, JPG, ZIP, EXE, etc.), the checkout
>process produces corrupted files.
>
>Reproduction steps
>1. Create a global `.gitattributes` file containing:
>   * text diff=lfclean
>2. Define the diff driver globally:
>   diff.lfclean.textconv=sed -e s/\\r//
>   diff.lfclean.binary=false
>3. Clone any repository containing binary files.
>4. Compare a binary file from the working tree with the raw blob:
>   git show HEAD:path/to/file > raw.bin
>   fc /b raw.bin path/to/file
>
>The working-tree file differs from the blob, confirming corruption during checkout.
>
>Expected behavior
>Binary files should remain untouched unless explicitly marked with filters.
>
>Actual behavior
>Git applies the global attributes and diff driver to all files, including binaries, causing
>corruption.
>
>Request
>I would appreciate guidance on whether Git should:
>- Warn users when global attributes force text filters onto binary files, or
>- Provide safeguards to prevent textconv filters from being applied to known binary
>formats.
>
>Thanks for your time and consideration.
>
>Best regards,
>Skybuck
>```
>
>version 2:
>
>Here is the **clean, corrected, plain‑text reply**, in a **single code block**, clearly
>explaining:
>
>- the **cause**
>- the **solution**
>- the **future risks**
>
>You can paste this directly into the Git mailing list.
>
>```
>Hi Git developers,
>
>I want to report an issue where Git corrupts binary files during checkout when a
>global .gitattributes file forces all paths to be treated as text and assigns a diff driver.
>
>Cause of the problem
>My global attributes file contained the line:
>
>    * text diff=lfclean
>
>Because the pattern "*" matches every file, Git applies "text" and "diff=lfclean" to all
>paths, including binary files such as PNGs. The "text" attribute forces Git to treat the
>file as text even when its content is clearly binary. The assigned diff driver runs a
>textconv filter:
>
>    diff.lfclean.textconv = sed -e s/\r//
>    diff.lfclean.binary = false
>
>This causes Git to rewrite the file during checkout, stripping carriage returns from
>binary data. As a result, the working-tree version of a PNG differs from the blob
>stored in the repository, and the file becomes corrupted. A copy downloaded via
>curl remains intact because curl does not apply Git filters.
>
>Reproduction steps
>1. Create a global .gitattributes file containing:
>       * text diff=lfclean
>2. Define the diff driver globally:
>       diff.lfclean.textconv=sed -e s/\r//
>       diff.lfclean.binary=false
>3. Clone any repository containing binary files.
>4. Compare the blob with the working-tree file:
>       git show HEAD:path/to/file > raw.bin
>       fc /b raw.bin path/to/file
>
>The working-tree file is corrupted while the blob is correct.
>
>Solution
>Remove the global rule or restrict it to actual text file extensions. For example:
>
>    *.txt text diff=lfclean
>    *.pas text diff=lfclean
>    *.md  text diff=lfclean
>
>Alternatively, explicitly mark binary formats:
>
>    *.png binary
>    *.jpg binary
>    *.zip binary
>    *.exe binary
>
>Deleting the global .gitattributes file entirely also resolves the issue.
>
>Future risks
>Global attribute rules are applied to every repository on the system. Using "*" with
>"text" or any filter can silently rewrite binary files, break archives, corrupt
>executables, and cause difficult-to-diagnose data loss. Users may not realize that Git
>is modifying files during checkout, especially when the corruption is subtle. A
>warning or safeguard when "text" is forced globally might help prevent accidental
>misuse.
>
>Thanks for your time and consideration.

Just some musings from my own frustration in this area.

Having gone through some similar things, I would ignore CoPilot. There are other words I would use as well but they are not for polite company.

Set autocrlf=input not false. Also it is a good idea to set ignorecase=true and filemode=false on Windows.

Git tends to give preferential treatment to text files, only looking at the first hunk (whatever that might be) looking for non-text characters. CR is text, so a file containing those near the front will probably be consider text unless explicitly marked as binary. If you are sure you have binary files, declare them. Do not assume git will always get it right - although .EXE, .ZIP, .JPG, and .PNG are pretty much always binary.

I am going to assume something there, that the clean/smudge and diff engines are not guaranteed to be subject to autocrlf processing before receiving the files. It might be or might not be, depending on what git feels like doing given the state of the file. You would have to go look in the code on the version you have to be certain, but don't count on it in future.

The other problem you may to face, and I have been there, is that clean/smudge and textconv filters definitely *do not like* binary files if not declared as binary, and sometimes even then. You are dealing with stdin and stdout, so have to know how the filter/textconv is opening the files. I have seen platforms that always open in "r" instead of "rb", which was a problem. I had to hack around that using %f in textconv. I have also seen people write textconv programs without awareness that they might get binary data, and that blows up runtimes badly when you hit a NUL in an input buffer after an fgets() in C.

I wish you luck in your adventure.
Randall