[Buildroot] go: patching dependency versions w/ go modules?

[Buildroot] go: patching dependency versions w/ go modules?

[Christian Stewart via buildroot]

Hi all,

There's a particular issue I've run into with the Go implementation
while trying to fix the gocryptfs package.

Currently the process is:

 1. Download the package source code.
 2. Run "go mod vendor" and re-bundle into a .tar.gz.
 3. Hash the resulting .tar.gz
 4. Extract it to the build/ tree
 5. Apply any patches
 6. Run the build.

The problem with this is, if we try to patch the versions of any
dependencies, for example by editing go.mod and go.sum, this is after
the download step at #2.

So currently it is not possible to change the version of dependencies
with a patch.

Is there any clean way to fix this?

The only thing I can think of is to add a new variable which contains
a list of "go mod edit" commands to run just before "go mod vendor" in
the download & extract step.

go mod edit -replace
github.com/jacobsa/crypto=github.com/piggynl/jacobsa-crypto@xorblock-generic

Thanks & best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] go: patching dependency versions w/ go modules?

[Thomas Petazzoni via buildroot]

Hello,

On Sat, 25 Jun 2022 15:37:36 -0700
Christian Stewart <christian@paral.in> wrote:

> There's a particular issue I've run into with the Go implementation
> while trying to fix the gocryptfs package.
> 
> Currently the process is:
> 
>  1. Download the package source code.
>  2. Run "go mod vendor" and re-bundle into a .tar.gz.
>  3. Hash the resulting .tar.gz
>  4. Extract it to the build/ tree
>  5. Apply any patches
>  6. Run the build.
> 
> The problem with this is, if we try to patch the versions of any
> dependencies, for example by editing go.mod and go.sum, this is after
> the download step at #2.
> 
> So currently it is not possible to change the version of dependencies
> with a patch.

Correct.

> Is there any clean way to fix this?

No, not with the current infrastructure indeed.

> The only thing I can think of is to add a new variable which contains
> a list of "go mod edit" commands to run just before "go mod vendor" in
> the download & extract step.
> 
> go mod edit -replace
> github.com/jacobsa/crypto=github.com/piggynl/jacobsa-crypto@xorblock-generic

I don't really have a good idea here, but whatever solution we come up
with should not be Go-specific, as it should also work for the Cargo
use-case (and possibly other similar situations in the future).

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] go: patching dependency versions w/ go modules?

[Arnout Vandecappelle]

On 26/06/2022 01:20, Thomas Petazzoni via buildroot wrote:
> Hello,
> 
> On Sat, 25 Jun 2022 15:37:36 -0700
> Christian Stewart <christian@paral.in> wrote:
> 
>> There's a particular issue I've run into with the Go implementation
>> while trying to fix the gocryptfs package.
>>
>> Currently the process is:
>>
>>   1. Download the package source code.
>>   2. Run "go mod vendor" and re-bundle into a .tar.gz.
>>   3. Hash the resulting .tar.gz
>>   4. Extract it to the build/ tree
>>   5. Apply any patches
>>   6. Run the build.
>>
>> The problem with this is, if we try to patch the versions of any
>> dependencies, for example by editing go.mod and go.sum, this is after
>> the download step at #2.
>>
>> So currently it is not possible to change the version of dependencies
>> with a patch.
> 
> Correct.
> 
>> Is there any clean way to fix this?
> 
> No, not with the current infrastructure indeed.
> 
>> The only thing I can think of is to add a new variable which contains
>> a list of "go mod edit" commands to run just before "go mod vendor" in
>> the download & extract step.
>>
>> go mod edit -replace
>> github.com/jacobsa/crypto=github.com/piggynl/jacobsa-crypto@xorblock-generic
> 
> I don't really have a good idea here, but whatever solution we come up
> with should not be Go-specific, as it should also work for the Cargo
> use-case (and possibly other similar situations in the future).

  I vaguely remember something like this was discussed by some people and the 
idea we came up with was to add some kind of pre-post-process step that would be 
called before the {cargo,go}-post-process. Well, it would actually have to be 
called from within the post-process script, because the script itself is 
responsible for detecting if the tarball was already post-processed or not (and 
any patches should only be applied to the raw tarball, not the post-processed one).

  Come to think of it, because any patching anyway needs to be done in the 
post-process script itself, it's not so crazy to make the patching specific for 
the post-processor as well. We just need to add yet another argument to 
dl-wrapper that propagates down to the post-processor - similar to the arguments 
after -- that are passed down to the backend. With that, you could do e.g.

GOCRYPTFS_DOWNLOAD_POST_PROCESS_ARGS = \
	-m '-replace 
github.com/jacobsa/crypto=github.com/piggynl/jacobsa-crypto@xorblock-generic'

  Quoting is going to be complicated though...

  Regards,
  Arnout
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot