FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

[Nathan Royce]

Let me know if I should be submitting each of these feature-requests separately.

For projects I built, up to the git project (think LFS steps
(altered)), they've all provided OOT builds so the source directory
remains pristine (albeit those that needed autoconf to be run for
`configure` generation).

They've also all provided `--{build,host,target,with-sysroot}=`
parameters for ease of cross-compiling.

Looking at what the git project offers, and I'm not seeing either features.

Looking at the Makefile, I see mention of `HOST_CPU` which looks
pertinent, but being able to point to the path of the target root for
files to link is important and I'm not noticing anything for that.

Grepping through Documentation for `HOST_CPU` comes up with nothing.

Re: FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

[Eric Sunshine]

On Mon, Jul 8, 2024 at 12:38 PM Nathan Royce <nroycea+kernel@gmail.com> wrote:
> For projects I built, up to the git project (think LFS steps
> (altered)), they've all provided OOT builds so the source directory
> remains pristine (albeit those that needed autoconf to be run for
> `configure` generation).
>
> They've also all provided `--{build,host,target,with-sysroot}=`
> parameters for ease of cross-compiling.
>
> Looking at what the git project offers, and I'm not seeing either features.
>
> Looking at the Makefile, I see mention of `HOST_CPU` which looks
> pertinent, but being able to point to the path of the target root for
> files to link is important and I'm not noticing anything for that.
>
> Grepping through Documentation for `HOST_CPU` comes up with nothing.

Indeed, HOST_CPU was chosen intentionally for compatibility with
`autotools` in the event that support for cross-compilation was ever
added[*].

A few years ago, I had started adding cross-compilation support to the
project but never finished the task. I'm pretty sure I still have the
patches sitting around somewhere. I'll look for them, but I'm not sure
how much they will help. Aside from the obvious patch adding
`config.guess` and `config.sub`, I recall creating a patch to fool
`autoconf` into not demanding that the project also carry the bunch of
other scripts/tools `autoconf` normally wants (since we don't use
those tools in our build process).

[*] https://lore.kernel.org/git/20171209094310.GA60808@flurp.local/

Re: FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

[Nathan Royce]

Well goodness me, seems I spoke too soon.
I found the that zlib was required (looking for "zlib.h"), so I built
that first and that too wasn't all that cross-compile friendly.
I saw "CHOST" is used, and was surprised that it didn't seem to need
anything to link against from the target sysroot, so that turned out
better than I thought it would.

I then used `configure` prefixed with
the`CFLAGS="--sysroot=<pathToSysroot>"`, along with `HOST_CPU=<tuple>`
for `make`, and it worked out fine.
Before moving it to my device, I just nspawned/chrooted into it and
`git --help` worked. So looks good and easy steps. Of course, it'll
depend on whether or not a git function using zlib also passes (hoping
zlib actually built fine without needing any outside linkage).

I'd still suggest and prefer that git (and zlib) follows what others
have settled on doing to be cross-compile-friendly.

On Mon, Jul 8, 2024 at 11:52 AM Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> A few years ago, I had started adding cross-compilation support to the
> project but never finished the task. I'm pretty sure I still have the
> patches sitting around somewhere. I'll look for them, but I'm not sure
> how much they will help. Aside from the obvious patch adding
> `config.guess` and `config.sub`, I recall creating a patch to fool
> `autoconf` into not demanding that the project also carry the bunch of
> other scripts/tools `autoconf` normally wants (since we don't use
> those tools in our build process).
>
> [*] https://lore.kernel.org/git/20171209094310.GA60808@flurp.local/

Re: FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

[Eric Sunshine]

[please avoid top-posting on this mailing list(*)]

On Mon, Jul 8, 2024 at 2:33 PM Nathan Royce <nroycea+kernel@gmail.com> wrote:
> Well goodness me, seems I spoke too soon.
> I found the that zlib was required (looking for "zlib.h"), so I built
> that first and that too wasn't all that cross-compile friendly.
> I saw "CHOST" is used, and was surprised that it didn't seem to need
> anything to link against from the target sysroot, so that turned out
> better than I thought it would.
>
> I then used `configure` prefixed with
> the`CFLAGS="--sysroot=<pathToSysroot>"`, along with `HOST_CPU=<tuple>`
> for `make`, and it worked out fine.

The Git build system determines some aspects of the environment
dynamically, so if you were cross-compiling for a different
architecture, it is possible that this did not enable every feature of
Git. For instance, if you look inside `config.mak.uname` and
`Makefile`, you will find a number of invocations of $(shell ...)
which pluck some host system information at build time rather than at
configuration time.

> Before moving it to my device, I just nspawned/chrooted into it and
> `git --help` worked. So looks good and easy steps. Of course, it'll
> depend on whether or not a git function using zlib also passes (hoping
> zlib actually built fine without needing any outside linkage).

A successful `git --help` is one small victory. Running the full test
suite on the cross-compiled project will give a more complete picture
of whether or not the effort was successful.

> I'd still suggest and prefer that git (and zlib) follows what others
> have settled on doing to be cross-compile-friendly.

I can't speak for the zlib project, but for this to happen in Git,
someone with an interest in seeing such an outcome will need to submit
patches.

[*] https://lore.kernel.org/all/YQK0JuI1w1zsEHeC@kroah.com/

Re: FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

>> I'd still suggest and prefer that git (and zlib) follows what others
>> have settled on doing to be cross-compile-friendly.
>
> I can't speak for the zlib project, but for this to happen in Git,
> someone with an interest in seeing such an outcome will need to submit
> patches.

Sure.  

Somebody unknown to the community suddenly coming here and
suggesting a feature alone would not achieve anything.  If there
were infinite engineering resources and motivated contributors, and
if sufficient number of contributors thought something is worth
doing, it would already have been done.  And "cross compilation" is
one of the things that is so obvious "isn't it nice if we had..."
items.  At least the offer has to be a bit more, like "I'll help in
this and that area (e.g., organizing the effort, keeping track of
progress, researching dependencies, ...).  Any others who want to
join forces?"

Thanks.

Re: FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

[Đoàn Trần Công Danh]

On 2024-07-08 12:56:41-0700, Junio C Hamano <gitster@pobox.com> wrote:
> Eric Sunshine <sunshine@sunshineco.com> writes:
> 
> >> I'd still suggest and prefer that git (and zlib) follows what others
> >> have settled on doing to be cross-compile-friendly.
> >
> > I can't speak for the zlib project, but for this to happen in Git,
> > someone with an interest in seeing such an outcome will need to submit
> > patches.
> 
> Sure.  
> 
> Somebody unknown to the community suddenly coming here and
> suggesting a feature alone would not achieve anything.  If there
> were infinite engineering resources and motivated contributors, and
> if sufficient number of contributors thought something is worth
> doing, it would already have been done.  And "cross compilation" is
> one of the things that is so obvious "isn't it nice if we had..."
> items.  At least the offer has to be a bit more, like "I'll help in
> this and that area (e.g., organizing the effort, keeping track of
> progress, researching dependencies, ...).  Any others who want to
> join forces?"

I thought in Git project, Makefile is the official build system, and
the autotools build system is only an after-thought, no?

For cross-compilation, I think various project has been
cross-compiling Git from forever.  They only need to provide a file
named `config.mak' with proper information for that platform, e.g:

	cat <<-EOF
	prefix = /usr
	CC = $CC
	CFLAGS = $CFLAGS
	LDFLAGS = $LDFLAGS
	USE_LIBPCRE2 := $(if true; then echo Yes; fi)
	perllibdir=/usr/share/perl5/vendor_perl
	HOST_CPU = $(config.guess | cut -d- -f1)
	ICONV_OMITS_BOM = Yes
	NO_REGEX = Yes
	EOF

Those last values need to be specified manually because they can't be
detected by running a test program anyway.  Those keys are already
listed in Makefile.

-- 
Danh

Re: FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

[Nathan Royce]

On Tue, Jul 9, 2024 at 10:20 PM Đoàn Trần Công Danh
<congdanhqx@gmail.com> wrote:
>
> I thought in Git project, Makefile is the official build system, and
> the autotools build system is only an after-thought, no?
>
> For cross-compilation, I think various project has been
> cross-compiling Git from forever.  They only need to provide a file
> named `config.mak' with proper information for that platform, e.g:
>
>         cat <<-EOF
>         prefix = /usr
>         CC = $CC
>         CFLAGS = $CFLAGS
>         LDFLAGS = $LDFLAGS
>         USE_LIBPCRE2 := $(if true; then echo Yes; fi)
>         perllibdir=/usr/share/perl5/vendor_perl
>         HOST_CPU = $(config.guess | cut -d- -f1)
>         ICONV_OMITS_BOM = Yes
>         NO_REGEX = Yes
>         EOF
>
> Those last values need to be specified manually because they can't be
> detected by running a test program anyway.  Those keys are already
> listed in Makefile.
>
> --
> Danh

(noted the "top-most" comment earlier, I didn't realize that was a
thing. Even "ticket/issue" emails always say "Don't write *below* this
line")

Danh, I was referring to the "--' parameters that is common amongst
most of the configure/make-based projects.
While I believe I had already had a "config.auto.mak" file generated,
it didn't appear that I could do OOT builds with it.
If perhaps there was some var I could pass into "make" where I could
point to the file, that'd be good (though I'd still question whether
that would even be enough to get OOT builds).

Someone on the IRC channel pointed out to me that there IS a
CMakeLists.txt file and I found it is included in
"contrib/buildsystems".
Awesome, right? So I thought and hoped...
Before I get into it, it might be nice to have that mentioned in INSTALL/README.

While that gets me the OOT builds, it seems it forces the use of
pcre2, even when it isn't installed in sysroot.
If it's installed in the build system, it'll find it there and say
things are good to go even though the actual "make" will fail because
of it.
"configure" defaults pcre2 to "no", so it builds fine (apart from not
being OOT and other stuff I have to do).

Getting closer (after manually commenting that USE_PCRE2 stuff in the
file (which I also don't like since source is no longer pristine),
though I just now got:
*****
...
[ 76%] Built target scalar
make[2]: *** No rule to make target 'git-remote-http', needed by
'git-add'.  Stop.
*****
so I'll have to peek as to what that's about.

Re: FR: Provide Out-Of-Tree Building; Provide Cross-Compile Parameters

[Junio C Hamano]

Đoàn Trần Công Danh <congdanhqx@gmail.com> writes:

>> ...  And "cross compilation" is
>> one of the things that is so obvious "isn't it nice if we had..."
>> items.  At least the offer has to be a bit more, like "I'll help in
>> this and that area (e.g., organizing the effort, keeping track of
>> progress, researching dependencies, ...).  Any others who want to
>> join forces?"
>
> I thought in Git project, Makefile is the official build system, and
> the autotools build system is only an after-thought, no?

True.  In addition to autotools, there also is cmake "support" (only
meant to be used for Windows) in our tree, which us non-Windows
folks pretty much treat like how Makefile-only folks treat
autotools.

But even if they started as and still are after-thought, with enough
interest by other participant, there is nothing forbidding a
contributor to organize an effort to improve them.  The only
constraint from our side is not to butcher the main build system too
badly to make Makefile-only build less pleasant to work with and
harder to maintain.

Thanks.