silo fails to build with recent ext2fs libraries

silo fails to build with recent ext2fs libraries

[Jurij Smakov]

Hello,

Recently I've discovered that silo fails to build with recent ext2fs 
libraries. Debian bug report [0] shows the relevant part of the build 
log. Someone also reported the same problem on gentoo forum just a few 
days ago [1].

As far as I can understand, the problem arises because newer ext2fs 
library versions introduced additional symbols, depending on libc 
functions not implemented in silo, into the same object files which 
contain the symbols used by the second stage. Since linker operates on 
object file level, it pulls in these symbols along with needed ones, 
then fails to resolve their dependencies because they are not 
implemented in silo and we are not linking against libc.

I was hoping that I can create a reduced ext2fs library containing 
only necessary symbols, but later recognized that this would require 
compiling and maintaining an explicit list of such symbols and their 
dependencies, which would be pretty tedious and error prone. I've also 
tried pruning the symbols causing the failure from the original 
library, however it's not possible in some cases, as some of them are 
mentioned in relocation entries. Another option would be to start 
linking against libc to work around this problem, but I don't know why 
it was not done in the first place, some size constraints perhaps?

[0] http://bugs.debian.org/655897
[1] http://forums.gentoo.org/viewtopic-p-6923830.html?sidtbeb3b8188dd2e6f33a39b6503479c4

Best regards,
-- 
Jurij Smakov                                           jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                      KeyID: C99E03CC

Re: silo fails to build with recent ext2fs libraries

[David Miller]

From: Jurij Smakov <jurij@wooyd.org>
Date: Sun, 15 Jan 2012 14:09:39 +0000

> Another option would be to start linking against libc to work around
> this problem, but I don't know why it was not done in the first
> place, some size constraints perhaps?

Not only size, but you then have to audit every symbol libext2fs uses
in libc to make sure the code in libc doesn't directly or indirectly
do system calls or other things we can't support in the context of
SILO's execution environment.

That's why I suggest we just keep adding whatever routines libext2fs
wants into SILO.  That way we can audit what it wants, explicitly, and
know that our implementation won't have unwanted dependencies upon
executing in a real userland environment.

Re: silo fails to build with recent ext2fs libraries

[David Miller]

From: Jurij Smakov <jurij@wooyd.org>
Date: Sun, 15 Jan 2012 14:09:39 +0000

> Recently I've discovered that silo fails to build with recent ext2fs 
> libraries. Debian bug report [0] shows the relevant part of the build 
> log. Someone also reported the same problem on gentoo forum just a few 
> days ago [1].

First of all, the sources used in that package are out of date,
current SILO in GIT has a posix_memalign() and it was added exactly
one year ago.

So that part of the build failure is just due to an out of date
copy of the SILO sources.

I'm looking into the rest.

Re: silo fails to build with recent ext2fs libraries

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 17 Jan 2012 12:56:56 -0500 (EST)

> I'm looking into the rest.

So just adding the stubs (or your idea, linking with libc) doesn't
work at all.

All the new features in libext2 that get linked in cause the second
stage bootloader image to grow too large.

I'll just have to extract the parts of libext2 that we actually need
into a stripped down local copy in SILO and I guess this will make
us immune to this issue showing up again in the future.

Re: silo fails to build with recent ext2fs libraries

[Julian Calaby]

On Wed, Jan 18, 2012 at 05:51, David Miller <davem@davemloft.net> wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 17 Jan 2012 12:56:56 -0500 (EST)
>
>> I'm looking into the rest.
>
> So just adding the stubs (or your idea, linking with libc) doesn't
> work at all.
>
> All the new features in libext2 that get linked in cause the second
> stage bootloader image to grow too large.
>
> I'll just have to extract the parts of libext2 that we actually need
> into a stripped down local copy in SILO and I guess this will make
> us immune to this issue showing up again in the future.

Would it make sense to get Grub working on Sparc so we can abandon
SILO and reduce our maintenance burden to just the Sparc-specific
parts of Grub?

Thanks,

-- 
Julian Calaby

Email: julian.calaby@gmail.com
Profile: http://www.google.com/profiles/julian.calaby/
.Plan: http://sites.google.com/site/juliancalaby/

Re: silo fails to build with recent ext2fs libraries

[Jurij Smakov]

On Tue, Jan 17, 2012 at 01:51:51PM -0500, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 17 Jan 2012 12:56:56 -0500 (EST)
> 
> > I'm looking into the rest.
> 
> So just adding the stubs (or your idea, linking with libc) doesn't
> work at all.
> 
> All the new features in libext2 that get linked in cause the second
> stage bootloader image to grow too large.
> 
> I'll just have to extract the parts of libext2 that we actually need
> into a stripped down local copy in SILO and I guess this will make
> us immune to this issue showing up again in the future.

If you haven't put too much effort into it yet, can you give me a 
few days to experiment with dropping unnecessary symbols from the 
built library using objcopy? If it turns out to be simple, I'd rather 
use this than duplicating code.

Best regards,
-- 
Jurij Smakov                                           jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                      KeyID: C99E03CC

Re: silo fails to build with recent ext2fs libraries

[David Miller]

From: Julian Calaby <julian.calaby@gmail.com>
Date: Wed, 18 Jan 2012 08:51:01 +1100

> On Wed, Jan 18, 2012 at 05:51, David Miller <davem@davemloft.net> wrote:
>> From: David Miller <davem@davemloft.net>
>> Date: Tue, 17 Jan 2012 12:56:56 -0500 (EST)
>>
>>> I'm looking into the rest.
>>
>> So just adding the stubs (or your idea, linking with libc) doesn't
>> work at all.
>>
>> All the new features in libext2 that get linked in cause the second
>> stage bootloader image to grow too large.
>>
>> I'll just have to extract the parts of libext2 that we actually need
>> into a stripped down local copy in SILO and I guess this will make
>> us immune to this issue showing up again in the future.
> 
> Would it make sense to get Grub working on Sparc so we can abandon
> SILO and reduce our maintenance burden to just the Sparc-specific
> parts of Grub?

The sparc64 port is already in there and mostly works but it needs
tons of bug fixing and testing.

However the GRUB maintainer is a prick and there is a lot of
infighting in the project so I stopped doing GRUB work a very long
time ago.

For the time being it's quicker to do add a quick piece of ext2 r/o
support code to SILO, and that's what I'm already working on.

Re: silo fails to build with recent ext2fs libraries

[David Miller]

From: Jurij Smakov <jurij@wooyd.org>
Date: Wed, 18 Jan 2012 00:20:08 +0000

> On Tue, Jan 17, 2012 at 01:51:51PM -0500, David Miller wrote:
>> From: David Miller <davem@davemloft.net>
>> Date: Tue, 17 Jan 2012 12:56:56 -0500 (EST)
>> 
>> > I'm looking into the rest.
>> 
>> So just adding the stubs (or your idea, linking with libc) doesn't
>> work at all.
>> 
>> All the new features in libext2 that get linked in cause the second
>> stage bootloader image to grow too large.
>> 
>> I'll just have to extract the parts of libext2 that we actually need
>> into a stripped down local copy in SILO and I guess this will make
>> us immune to this issue showing up again in the future.
> 
> If you haven't put too much effort into it yet, can you give me a 
> few days to experiment with dropping unnecessary symbols from the 
> built library using objcopy? If it turns out to be simple, I'd rather 
> use this than duplicating code.

We've been fighting this problem for years, it only gets worse and
all of these various hacks are error prone.

The only real solution is to write something made for a bootloader and
that's what I'm already doing.  The code isn't that hard at all.

Re: silo fails to build with recent ext2fs libraries

[Julian Calaby]

On Wed, Jan 18, 2012 at 15:38, David Miller <davem@davemloft.net> wrote:
> From: Julian Calaby <julian.calaby@gmail.com>
> Date: Wed, 18 Jan 2012 08:51:01 +1100
>
>> On Wed, Jan 18, 2012 at 05:51, David Miller <davem@davemloft.net> wrote:
>>> From: David Miller <davem@davemloft.net>
>>> Date: Tue, 17 Jan 2012 12:56:56 -0500 (EST)
>>>
>>>> I'm looking into the rest.
>>>
>>> So just adding the stubs (or your idea, linking with libc) doesn't
>>> work at all.
>>>
>>> All the new features in libext2 that get linked in cause the second
>>> stage bootloader image to grow too large.
>>>
>>> I'll just have to extract the parts of libext2 that we actually need
>>> into a stripped down local copy in SILO and I guess this will make
>>> us immune to this issue showing up again in the future.
>>
>> Would it make sense to get Grub working on Sparc so we can abandon
>> SILO and reduce our maintenance burden to just the Sparc-specific
>> parts of Grub?
>
> The sparc64 port is already in there and mostly works but it needs
> tons of bug fixing and testing.

As everything does.

> However the GRUB maintainer is a prick and there is a lot of
> infighting in the project so I stopped doing GRUB work a very long
> time ago.

But it's a GNU project, surely it's full of fairies and unicorns and
rainbows! .... </sarcasm>

> For the time being it's quicker to do add a quick piece of ext2 r/o
> support code to SILO, and that's what I'm already working on.

Fair enough.

Thanks,

-- 
Julian Calaby

Email: julian.calaby@gmail.com
Profile: http://www.google.com/profiles/julian.calaby/
.Plan: http://sites.google.com/site/juliancalaby/

Re: silo fails to build with recent ext2fs libraries

[Jurij Smakov]

On Tue, Jan 17, 2012 at 11:49:01PM -0500, David Miller wrote:
> 
> We've been fighting this problem for years, it only gets worse and
> all of these various hacks are error prone.
> 
> The only real solution is to write something made for a bootloader and
> that's what I'm already doing.  The code isn't that hard at all.

All right, I'll be watching git tree for commits then :-).

Thanks for taking care of this.
-- 
Jurij Smakov                                           jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                      KeyID: C99E03CC

Re: silo fails to build with recent ext2fs libraries

[David Miller]

From: Jurij Smakov <jurij@wooyd.org>
Date: Wed, 18 Jan 2012 19:53:33 +0000

> On Tue, Jan 17, 2012 at 11:49:01PM -0500, David Miller wrote:
>> 
>> We've been fighting this problem for years, it only gets worse and
>> all of these various hacks are error prone.
>> 
>> The only real solution is to write something made for a bootloader and
>> that's what I'm already doing.  The code isn't that hard at all.
> 
> All right, I'll be watching git tree for commits then :-).
> 
> Thanks for taking care of this.

Ok, I commited my work, but be careful.

I tested it on a SunBlade-2500 running squeeze, with disks on the onboard
MPT Fusion controller.

Make backups and have a way to netboot or CD boot your test machine if
things go wrong.

Let me know how it goes.