tool stack tool chain dependencies (again)

tool stack tool chain dependencies (again)

[Jan Beulich]

Hello,

especially some of the changes late in the 4.8 cycle have caused me
to spend a good part of the morning trying to figure out how to build
the tool stack on an older system. Among other things I've run into
- ipxe all of the sudden not working with make 3.80 anymore (despite
  apparent attempts to do so in their Makefile)
- ipxe not working with gas prior to 2.18 anymore (requiring the
  .reloc directive)
- rombios causing ld to segfault when building with debug=y (later
  findings suggest this may be because overriding CC in ./.config
  works, but doing so for e.g. AS and LD doesn't affect namely
  the tools/firmware/ subtrees)
- -O0 causing fallout with an admittedly questionable sys/stat.h
None of this was a problem with an early October build.

I think we really need two things here: One is that I think we should
bump our minimal required tool chain component versions, or
alternatively have tools/configure properly disable sub-components
(taking into account overrides from ./.config) - I certainly could live
with qemu-trad being disabled on such old systems, but otoh
upstream qemu in the past has proven to not be much better, and
it would be of questionable use if both got disabled. The other is
that I think we should actually make sure things build with those
versions.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: tool stack tool chain dependencies (again)

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 2153 bytes --]

On 12/16/16 6:52 AM, Jan Beulich wrote:
> Hello,
> 
> especially some of the changes late in the 4.8 cycle have caused me
> to spend a good part of the morning trying to figure out how to build
> the tool stack on an older system. Among other things I've run into
> - ipxe all of the sudden not working with make 3.80 anymore (despite
>   apparent attempts to do so in their Makefile)
> - ipxe not working with gas prior to 2.18 anymore (requiring the
>   .reloc directive)
> - rombios causing ld to segfault when building with debug=y (later
>   findings suggest this may be because overriding CC in ./.config
>   works, but doing so for e.g. AS and LD doesn't affect namely
>   the tools/firmware/ subtrees)
> - -O0 causing fallout with an admittedly questionable sys/stat.h
> None of this was a problem with an early October build.
> 
> I think we really need two things here: One is that I think we should
> bump our minimal required tool chain component versions, or
> alternatively have tools/configure properly disable sub-components
> (taking into account overrides from ./.config) - I certainly could live
> with qemu-trad being disabled on such old systems, but otoh
> upstream qemu in the past has proven to not be much better, and
> it would be of questionable use if both got disabled. The other is
> that I think we should actually make sure things build with those
> versions.
> 
> Jan

I'm with you on wanting to see a solution for this. I want to get back
to my Travis CI or GitLab CI work to test build Xen in as many different
Docker environments as possible.

Some of this would be made better by not shipping code as part of Xen
and requiring the distro or the source builder to install the proper
dependencies from their distro. The reason you're having an issue with
iPXE is that it was bumped to a newer version to support GCC 6 and part
of the modernizing work of iPXE was to remove old hacks for toolchains
they no longer supported. I don't know of a single distro that ships the
Xen built iPXE (except for the CentOS packages but could and should be
fixed).

-- 
Doug Goldstein



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: tool stack tool chain dependencies (again)

[Wei Liu]

On Fri, Dec 16, 2016 at 05:52:29AM -0700, Jan Beulich wrote:
> Hello,
> 
> especially some of the changes late in the 4.8 cycle have caused me
> to spend a good part of the morning trying to figure out how to build
> the tool stack on an older system. Among other things I've run into
> - ipxe all of the sudden not working with make 3.80 anymore (despite
>   apparent attempts to do so in their Makefile)
> - ipxe not working with gas prior to 2.18 anymore (requiring the
>   .reloc directive)
> - rombios causing ld to segfault when building with debug=y (later
>   findings suggest this may be because overriding CC in ./.config
>   works, but doing so for e.g. AS and LD doesn't affect namely
>   the tools/firmware/ subtrees)
> - -O0 causing fallout with an admittedly questionable sys/stat.h
> None of this was a problem with an early October build.
> 
> I think we really need two things here: One is that I think we should
> bump our minimal required tool chain component versions, or

+1.

> alternatively have tools/configure properly disable sub-components
> (taking into account overrides from ./.config) - I certainly could live

If I'm not mistaken this requires we track tool chain requirements for
all dependencies and their dependencies.  This is going to be
unmanageable. 

> with qemu-trad being disabled on such old systems, but otoh
> upstream qemu in the past has proven to not be much better, and
> it would be of questionable use if both got disabled. The other is
> that I think we should actually make sure things build with those
> versions.
> 

Doug's work should help.

Wei.

> Jan
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: tool stack tool chain dependencies (again)

[Jan Beulich]

>>> On 05.01.17 at 12:47, <wei.liu2@citrix.com> wrote:
> On Fri, Dec 16, 2016 at 05:52:29AM -0700, Jan Beulich wrote:
>> Hello,
>> 
>> especially some of the changes late in the 4.8 cycle have caused me
>> to spend a good part of the morning trying to figure out how to build
>> the tool stack on an older system. Among other things I've run into
>> - ipxe all of the sudden not working with make 3.80 anymore (despite
>>   apparent attempts to do so in their Makefile)
>> - ipxe not working with gas prior to 2.18 anymore (requiring the
>>   .reloc directive)
>> - rombios causing ld to segfault when building with debug=y (later
>>   findings suggest this may be because overriding CC in ./.config
>>   works, but doing so for e.g. AS and LD doesn't affect namely
>>   the tools/firmware/ subtrees)
>> - -O0 causing fallout with an admittedly questionable sys/stat.h
>> None of this was a problem with an early October build.
>> 
>> I think we really need two things here: One is that I think we should
>> bump our minimal required tool chain component versions, or
> 
> +1.
> 
>> alternatively have tools/configure properly disable sub-components
>> (taking into account overrides from ./.config) - I certainly could live
> 
> If I'm not mistaken this requires we track tool chain requirements for
> all dependencies and their dependencies.  This is going to be
> unmanageable. 

Well, that (I think) depends on whether those other projects
document their requirements. If things are properly spelled out,
it shouldn't be that hard. And if they aren't properly spelled out,
perhaps it would be worth a try asking them to do so.

>> with qemu-trad being disabled on such old systems, but otoh
>> upstream qemu in the past has proven to not be much better, and
>> it would be of questionable use if both got disabled. The other is
>> that I think we should actually make sure things build with those
>> versions.
> 
> Doug's work should help.

Which parts are you thinking of? I'm not aware of anything that
has been done to test with specific tool chain component versions.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: tool stack tool chain dependencies (again)

[Wei Liu]

On Thu, Jan 05, 2017 at 04:57:12AM -0700, Jan Beulich wrote:
> >>> On 05.01.17 at 12:47, <wei.liu2@citrix.com> wrote:
> > On Fri, Dec 16, 2016 at 05:52:29AM -0700, Jan Beulich wrote:
> >> Hello,
> >> 
> >> especially some of the changes late in the 4.8 cycle have caused me
> >> to spend a good part of the morning trying to figure out how to build
> >> the tool stack on an older system. Among other things I've run into
> >> - ipxe all of the sudden not working with make 3.80 anymore (despite
> >>   apparent attempts to do so in their Makefile)
> >> - ipxe not working with gas prior to 2.18 anymore (requiring the
> >>   .reloc directive)
> >> - rombios causing ld to segfault when building with debug=y (later
> >>   findings suggest this may be because overriding CC in ./.config
> >>   works, but doing so for e.g. AS and LD doesn't affect namely
> >>   the tools/firmware/ subtrees)
> >> - -O0 causing fallout with an admittedly questionable sys/stat.h
> >> None of this was a problem with an early October build.
> >> 
> >> I think we really need two things here: One is that I think we should
> >> bump our minimal required tool chain component versions, or
> > 
> > +1.
> > 
> >> alternatively have tools/configure properly disable sub-components
> >> (taking into account overrides from ./.config) - I certainly could live
> > 
> > If I'm not mistaken this requires we track tool chain requirements for
> > all dependencies and their dependencies.  This is going to be
> > unmanageable. 
> 
> Well, that (I think) depends on whether those other projects
> document their requirements. If things are properly spelled out,
> it shouldn't be that hard. And if they aren't properly spelled out,
> perhaps it would be worth a try asking them to do so.
> 

Yes, we can try that. Which external components that you find break
easily? We can start with those.

> >> with qemu-trad being disabled on such old systems, but otoh
> >> upstream qemu in the past has proven to not be much better, and
> >> it would be of questionable use if both got disabled. The other is
> >> that I think we should actually make sure things build with those
> >> versions.
> > 
> > Doug's work should help.
> 
> Which parts are you thinking of? I'm not aware of anything that
> has been done to test with specific tool chain component versions.
> 

The Docker based CI part -- see his reply to your email. It would be
easy to add different configurations there.

That would help we catch regressions early and fix things properly.

Wei.

> Jan
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: tool stack tool chain dependencies (again)

[Jan Beulich]

>>> On 05.01.17 at 13:04, <wei.liu2@citrix.com> wrote:
> On Thu, Jan 05, 2017 at 04:57:12AM -0700, Jan Beulich wrote:
>> >>> On 05.01.17 at 12:47, <wei.liu2@citrix.com> wrote:
>> > If I'm not mistaken this requires we track tool chain requirements for
>> > all dependencies and their dependencies.  This is going to be
>> > unmanageable. 
>> 
>> Well, that (I think) depends on whether those other projects
>> document their requirements. If things are properly spelled out,
>> it shouldn't be that hard. And if they aren't properly spelled out,
>> perhaps it would be worth a try asking them to do so.
>> 
> 
> Yes, we can try that. Which external components that you find break
> easily? We can start with those.

I ran into the issue with the relatively recent ipxe update, and I've
been in a similar situation with qemu (upstream) a number of times
(going as far as finding it necessary to invent hacks to disable the
qemu upstream build automatically, as I don't want to always
manually invoke ./configure when re-basing my trees). I think there
was an issue at least once with seabios, too.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel