[Buildroot] syslinux using host compiler.

[Buildroot] syslinux using host compiler.

[Gilles Chanteperdrix]

Hi,

I am new to buildroot, and have been working for about two weeks
with it, and everything seems to be working fine so far.

I have a question however: why is syslinux (and grub2 for that
matter, I guess) built with the host toolchain? I can understand
that some tools (such as isohybrid) need to be executed on the host,
but the final binaries are for the target. So, does not that call
for having a separate host-syslinux and syslinux packages like other
packages do? 

The current setup only works if the host is x86.

I am using an x86_64 toolchain generated with ct-ng, did not enable
multilib, and syslinux seems to have no problem compiling with that
toolchain, which seems normal, since the boot code should not be
using the libc anyway.

So, what am I missing?

Thanks in advance.
Regards.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] syslinux using host compiler.

[Gilles Chanteperdrix]

On Thu, Mar 24, 2016 at 11:30:00AM +0100, Gilles Chanteperdrix wrote:
> Hi,
> 
> I am new to buildroot, and have been working for about two weeks
> with it, and everything seems to be working fine so far.

Sorry for the duplicate.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] syslinux using host compiler.

[Gilles Chanteperdrix]

Hi,

I am new to buildroot, and have been working for a couple of weeks
with it, and everything seems to be working fine so far.

I have a question however: why is syslinux (and grub2 for that
matter, I guess) built with the host toolchain? I can understand
that some tools (such as isohybrid) need to be executed on the host,
but the final binaries are for the target. So, does not that call
for having a separate host-syslinux and syslinux packages like other
packages do? 

The current setup only works if the host is x86.

I am using an x86_64 toolchain generated with ct-ng, did not enable
multilib, and syslinux seems to have no problem compiling with that
toolchain, which seems normal, since the boot code should not be
using the libc anyway.

So, what am I missing?

Thanks in advance.
Regards.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] syslinux using host compiler.

[Thomas Petazzoni]

Hello,

On Thu, 24 Mar 2016 11:39:41 +0100, Gilles Chanteperdrix wrote:

> I am new to buildroot, and have been working for a couple of weeks
> with it, and everything seems to be working fine so far.

Great :-) Glad to see you on this mailing list!

> I have a question however: why is syslinux (and grub2 for that
> matter, I guess) built with the host toolchain?

For grub2, it is actually using the target compiler, see the comment in
grub2.mk:

# Grub2 is kind of special: it considers CC, LD and so on to be the
# tools to build the native tools (i.e to be executed on the build
# machine), and uses TARGET_CC, TARGET_CFLAGS, TARGET_CPPFLAGS,
# TARGET_LDFLAGS to build the bootloader itself. However, to add to
# the confusion, it also uses NM, OBJCOPY and STRIP to build the
# bootloader itself; none of these are used to build the native
# tools.

> I can understand that some tools (such as isohybrid) need to be
> executed on the host, but the final binaries are for the target. So,
> does not that call for having a separate host-syslinux and syslinux
> packages like other packages do? 

We used to have a host-syslinux package, but it was removed in commit
ac7f60d87266c30753cb45b820dc48ec834941c6 as it was no longer needed.

> The current setup only works if the host is x86.

No, because syslinux selects BR2_HOSTARCH_NEEDS_IA32_COMPILER, which
ensures you have a working x86 32-bits compiler on your host machine. If
you don't, then Buildroot will not even start the build.

The reason why we're using the host compiler rather than the target
compiler is because syslinux must be built as 32-bits x86 code, even if
your target system is x86_64.

> I am using an x86_64 toolchain generated with ct-ng, did not enable
> multilib, and syslinux seems to have no problem compiling with that
> toolchain, which seems normal, since the boot code should not be
> using the libc anyway.

Are you sure you're using this target compiler to build syslinux, and
not your host compiler? Back in 2014, when commit
09e7b4c89159a3e48f463b0a22e649c4fc12f4b3 was done, it wouldn't build
with a x86-64 only compiler.

If that actually works, then patches are welcome. But in this case, we
will indeed probably need separate host-syslinux and syslinux packages,
as you suggested.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] syslinux using host compiler.

[Gilles Chanteperdrix]

On Thu, Mar 24, 2016 at 01:50:53PM +0100, Thomas Petazzoni wrote:
> Hello,
> 
> On Thu, 24 Mar 2016 11:39:41 +0100, Gilles Chanteperdrix wrote:
> 
> > I am new to buildroot, and have been working for a couple of weeks
> > with it, and everything seems to be working fine so far.
> 
> Great :-) Glad to see you on this mailing list!
> 
> > I have a question however: why is syslinux (and grub2 for that
> > matter, I guess) built with the host toolchain?
> 
> For grub2, it is actually using the target compiler, see the comment in
> grub2.mk:

Ok, I thought I saw grub selected BR2_HOSTARCH_NEEDS_IA32_COMPILER
too, must I have misread.

> 
> # Grub2 is kind of special: it considers CC, LD and so on to be the
> # tools to build the native tools (i.e to be executed on the build
> # machine), and uses TARGET_CC, TARGET_CFLAGS, TARGET_CPPFLAGS,
> # TARGET_LDFLAGS to build the bootloader itself. However, to add to
> # the confusion, it also uses NM, OBJCOPY and STRIP to build the
> # bootloader itself; none of these are used to build the native
> # tools.
> 
> > I can understand that some tools (such as isohybrid) need to be
> > executed on the host, but the final binaries are for the target. So,
> > does not that call for having a separate host-syslinux and syslinux
> > packages like other packages do? 
> 
> We used to have a host-syslinux package, but it was removed in commit
> ac7f60d87266c30753cb45b820dc48ec834941c6 as it was no longer needed.
> 
> > The current setup only works if the host is x86.
> 
> No, because syslinux selects BR2_HOSTARCH_NEEDS_IA32_COMPILER, which
> ensures you have a working x86 32-bits compiler on your host machine. If
> you don't, then Buildroot will not even start the build.

I meant you can not build syslinux if your host is arm for instance,
and you have an arm/x86 cross-compilation toolchain.

> 
> The reason why we're using the host compiler rather than the target
> compiler is because syslinux must be built as 32-bits x86 code, even if
> your target system is x86_64.

Yes, but AFAIK, gcc for x86_64 always supports the -m32 switch. To
build 32 bits host binaries, you need a multilib installation with a
32 bits glibc, but a bootloader should not use anything in the glibc.

> 
> > I am using an x86_64 toolchain generated with ct-ng, did not enable
> > multilib, and syslinux seems to have no problem compiling with that
> > toolchain, which seems normal, since the boot code should not be
> > using the libc anyway.
> 
> Are you sure you're using this target compiler to build syslinux, and
> not your host compiler? Back in 2014, when commit
> 09e7b4c89159a3e48f463b0a22e649c4fc12f4b3 was done, it wouldn't build
> with a x86-64 only compiler.

Yes, I am sure of it. The only "trick" I had to apply was to force
LD="$(TARGET_LD) -m elf_i386". Now buildroot complains later because
isohybrid in the host directory is not compiled with the proper
rpath, but I guess it means we need host-syslinux.

> 
> If that actually works, then patches are welcome. But in this case, we
> will indeed probably need separate host-syslinux and syslinux packages,
> as you suggested.

Yes. Ok, will work on the patch.

Regards.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] syslinux using host compiler.

[Thomas Petazzoni]

Hello,

On Thu, 24 Mar 2016 14:08:56 +0100, Gilles Chanteperdrix wrote:

> > For grub2, it is actually using the target compiler, see the comment in
> > grub2.mk:
> 
> Ok, I thought I saw grub selected BR2_HOSTARCH_NEEDS_IA32_COMPILER
> too, must I have misread.

grub yes, not grub2. But your original comment was about grub2, hence I
looked at grub2. But you are right, grub is like syslinux, we use the
host compiler.

> > No, because syslinux selects BR2_HOSTARCH_NEEDS_IA32_COMPILER, which
> > ensures you have a working x86 32-bits compiler on your host machine. If
> > you don't, then Buildroot will not even start the build.
> 
> I meant you can not build syslinux if your host is arm for instance,
> and you have an arm/x86 cross-compilation toolchain.

Correct, but the fact that you select BR2_HOSTARCH_NEEDS_IA32_COMPILER
will test that you have a host compiler capable of producing x86 32
bits binaries. So if you're building on ARM, it will simply abort at
the beginning of the build. Of course, it would be nicer if we could be
on any host architecture, but on the other hand, virtually everyone is
using x86/x86-64 as build machines, so it wasn't really worth putting
more efforts to fix this.

> > The reason why we're using the host compiler rather than the target
> > compiler is because syslinux must be built as 32-bits x86 code, even if
> > your target system is x86_64.
> 
> Yes, but AFAIK, gcc for x86_64 always supports the -m32 switch. To
> build 32 bits host binaries, you need a multilib installation with a
> 32 bits glibc, but a bootloader should not use anything in the glibc.

Completely agree with the theory. In practice, I remember seeing some
problems, like it was anyway trying to build some actual user-space
applications with -m32. But (1) my analysis at the time might have been
wrong and (2) syslinux was upgraded several times since then so maybe
they have changed/fixed things. But that definitely doesn't mean I'm
not interested in seeing patches improving this stuff.

> > > I am using an x86_64 toolchain generated with ct-ng, did not enable
> > > multilib, and syslinux seems to have no problem compiling with that
> > > toolchain, which seems normal, since the boot code should not be
> > > using the libc anyway.
> > 
> > Are you sure you're using this target compiler to build syslinux, and
> > not your host compiler? Back in 2014, when commit
> > 09e7b4c89159a3e48f463b0a22e649c4fc12f4b3 was done, it wouldn't build
> > with a x86-64 only compiler.
> 
> Yes, I am sure of it. The only "trick" I had to apply was to force
> LD="$(TARGET_LD) -m elf_i386". Now buildroot complains later because
> isohybrid in the host directory is not compiled with the proper
> rpath, but I guess it means we need host-syslinux.

OK.

> > If that actually works, then patches are welcome. But in this case, we
> > will indeed probably need separate host-syslinux and syslinux packages,
> > as you suggested.
> 
> Yes. Ok, will work on the patch.

Great!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] syslinux using host compiler.

[Arnout Vandecappelle]

On 03/24/16 15:12, Thomas Petazzoni wrote:
> Hello,
>
> On Thu, 24 Mar 2016 14:08:56 +0100, Gilles Chanteperdrix wrote:
>
>>> For grub2, it is actually using the target compiler, see the comment in
>>> grub2.mk:
>>
>> Ok, I thought I saw grub selected BR2_HOSTARCH_NEEDS_IA32_COMPILER
>> too, must I have misread.
>
> grub yes, not grub2. But your original comment was about grub2, hence I
> looked at grub2. But you are right, grub is like syslinux, we use the
> host compiler.

  One reason for this difference is that the build system was improved a lot 
with grub2. grub1 made it impossible to distinguish host from target build, and 
it does need to build some host tools.

[snip]
>>> The reason why we're using the host compiler rather than the target
>>> compiler is because syslinux must be built as 32-bits x86 code, even if
>>> your target system is x86_64.
>>
>> Yes, but AFAIK, gcc for x86_64 always supports the -m32 switch. To
>> build 32 bits host binaries, you need a multilib installation with a
>> 32 bits glibc, but a bootloader should not use anything in the glibc.
>
> Completely agree with the theory. In practice, I remember seeing some
> problems, like it was anyway trying to build some actual user-space
> applications with -m32. But (1) my analysis at the time might have been
> wrong and (2) syslinux was upgraded several times since then so maybe
> they have changed/fixed things. But that definitely doesn't mean I'm
> not interested in seeing patches improving this stuff.

  I remember I looked at it as well at the time, I think I even posted some 
patches that attempted to get a working syslinux using the cross-compiler, but 
it failed in some cases. This is from memory, consult the mailing list archives 
to be really sure.

  It's possible that it was just a matter of the -m elf_i386, which I wouldn't 
have thought of at the time (it's three years ago, I was still young and 
innexperienced :-P ).


  Regards,
  Arnout

>
>>>> I am using an x86_64 toolchain generated with ct-ng, did not enable
>>>> multilib, and syslinux seems to have no problem compiling with that
>>>> toolchain, which seems normal, since the boot code should not be
>>>> using the libc anyway.
>>>
>>> Are you sure you're using this target compiler to build syslinux, and
>>> not your host compiler? Back in 2014, when commit
>>> 09e7b4c89159a3e48f463b0a22e649c4fc12f4b3 was done, it wouldn't build
>>> with a x86-64 only compiler.
>>
>> Yes, I am sure of it. The only "trick" I had to apply was to force
>> LD="$(TARGET_LD) -m elf_i386". Now buildroot complains later because
>> isohybrid in the host directory is not compiled with the proper
>> rpath, but I guess it means we need host-syslinux.
>
> OK.
>
>>> If that actually works, then patches are welcome. But in this case, we
>>> will indeed probably need separate host-syslinux and syslinux packages,
>>> as you suggested.
>>
>> Yes. Ok, will work on the patch.
>
> Great!
>
> Thomas
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF