Programmatic patches (transform commits)

Programmatic patches (transform commits)

[Mark Fulton]

Is there a way to commit a transform script that programmatically
applies file changes rather than committing the file changes directly?

e.g. Imagine in a large repository that a contributor wants to replace
certain instances of "abc" with "xyz". A transform script might be
like the following:

```sh
#!/bin/sh

sed -i 's/abc/xyz/g' $(find .)
```

Applying such a "programmatic patch" will potentially edit many files.
Doing a code review on such a change is error prone due to authors
resolving merge conflicts manually, etc. while reviewing the patch in
some circumstances is much easier (especially tools for specifically
this type of file transformations are used to make it easy to parse
code, traverse abstract syntax trees, make edits, etc.).

Does anything like this exist today? Depending on the implementation I
could see there being cross-platform support challenges but maybe
there is something that already exists to assist with this which I can
learn about.

As an alternative to making this part of Git I can see tools like
GitHub Actions being used to look for commits of "programmatic patch"
files, pick those up, run them, and commit and push the change but
having a solution for this as part of Git itself would make it
independent of GitHub and more reusable, etc.

Re: Programmatic patches (transform commits)

[Ævar Arnfjörð Bjarmason]

On Thu, Aug 18 2022, Mark Fulton wrote:

> Is there a way to commit a transform script that programmatically
> applies file changes rather than committing the file changes directly?
>
> e.g. Imagine in a large repository that a contributor wants to replace
> certain instances of "abc" with "xyz". A transform script might be
> like the following:
>
> ```sh
> #!/bin/sh
>
> sed -i 's/abc/xyz/g' $(find .)
> ```
>
> Applying such a "programmatic patch" will potentially edit many files.
> Doing a code review on such a change is error prone due to authors
> resolving merge conflicts manually, etc. while reviewing the patch in
> some circumstances is much easier (especially tools for specifically
> this type of file transformations are used to make it easy to parse
> code, traverse abstract syntax trees, make edits, etc.).
>
> Does anything like this exist today? Depending on the implementation I
> could see there being cross-platform support challenges but maybe
> there is something that already exists to assist with this which I can
> learn about.
>
> As an alternative to making this part of Git I can see tools like
> GitHub Actions being used to look for commits of "programmatic patch"
> files, pick those up, run them, and commit and push the change but
> having a solution for this as part of Git itself would make it
> independent of GitHub and more reusable, etc.

Yes, e.g. for C (and used by linux.git and this project):
https://www.kernel.org/doc/html/v4.14/dev-tools/coccinelle.html

Generally speaking I've made such ad-hoc commits in the past, where I've
e.g. included the needed script (or one-liner) in the commit message,
along with manually adding the diff to the commit message that *isn't*
the part changed by the script or one-liner.

But I don't think there's any sort of widespread standard for this sort
of thing, other than doing that, and the same (sometimes leading to
non-atomic commits) of having the manual part of the change applied
before or after in a separate commit.

Re: Programmatic patches (transform commits)

[Johannes Schindelin]

Hi Mark,

On Thu, 18 Aug 2022, Mark Fulton wrote:

> Is there a way to commit a transform script that programmatically
> applies file changes rather than committing the file changes directly?
>
> e.g. Imagine in a large repository that a contributor wants to replace
> certain instances of "abc" with "xyz". A transform script might be
> like the following:
>
> ```sh
> #!/bin/sh
>
> sed -i 's/abc/xyz/g' $(find .)
> ```

I do have a script to do such a thing, and I am sure other users do, too.
It would be a simpler script if it did not have to accommodate for `sed`'s
`-i` option behaving differently between GNU and BSD variants. For the GNU
variant, I use

	git grep -zl "$regex" | xargs -0r sed -i "s/$regex/$replacement/g"

and for the BSD variant I use `sed -i ''` (i.e. an extra, empty argument)
and skip the `-r` option because BSD `xargs` does not understand it.

> Applying such a "programmatic patch" will potentially edit many files.
> Doing a code review on such a change is error prone due to authors
> resolving merge conflicts manually, etc. while reviewing the patch in
> some circumstances is much easier (especially tools for specifically
> this type of file transformations are used to make it easy to parse
> code, traverse abstract syntax trees, make edits, etc.).

In the Git project, we do have a track record of mentioning the exact
commands for automated transformation in the commit message, see e.g.
https://github.com/git/git/commit/8dcf73c5c940.

The idea is that any reviewer could run the same command, verify that the
outcome is the same, and give their ACK. Or, in case of merge conflicts,
they would re-run the command and commit the result using the original
commit message.

Ciao,
Dscho