[dtc] breaking out libfdt from dtc so other progs can use it

[dtc] breaking out libfdt from dtc so other progs can use it

[Jerone Young]

Currently the dtc source code has libfdt integrated in it. This seems to
have become place for upstream libfdt changes. Now we all know everyone
(linux kernel, cuboot) also have their own versions over libfdt. But if
another userspace app wants to use libfdt , it has to copy it from the
dtc source and try to maintain it's own copy.

The question I have is can libfdt be split out from dtc source, and
become it's own thing. This way other userspace apps can easily download
it and link with it?

The reason I ask is I have added dynamic manipulation support of device
trees in memory into qemu for KVM. But the issue is keeping a copy of
libfdt in the KVM userspace repository, which is getting some opposition
(understandably). But this would be much easier if there was a libfdt
repo for the library so that we wouldn't need to keep our own copy.

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Josh Boyer]

On Wed, 27 Feb 2008 13:40:43 -0600
Jerone Young <jyoung5@us.ibm.com> wrote:

> Currently the dtc source code has libfdt integrated in it. This seems to
> have become place for upstream libfdt changes. Now we all know everyone
> (linux kernel, cuboot) also have their own versions over libfdt. But if
> another userspace app wants to use libfdt , it has to copy it from the
> dtc source and try to maintain it's own copy.
> 
> The question I have is can libfdt be split out from dtc source, and
> become it's own thing. This way other userspace apps can easily download
> it and link with it?

Downloading isn't changed at all by splitting it out to it's own repo.

> The reason I ask is I have added dynamic manipulation support of device
> trees in memory into qemu for KVM. But the issue is keeping a copy of
> libfdt in the KVM userspace repository, which is getting some opposition
> (understandably). But this would be much easier if there was a libfdt
> repo for the library so that we wouldn't need to keep our own copy.

It seems the real crux of your issue is that you want distros to
provide a libfdt package.  That can be done by creating a subpackage
off of the dtc package.  The harder part for certain distros will be
either convincing them to allow the static libfdt to exist, or creating
a shared library for it instead.

josh

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Jerone Young]

On Wed, 2008-02-27 at 14:31 -0600, Josh Boyer wrote:
> On Wed, 27 Feb 2008 13:40:43 -0600
> Jerone Young <jyoung5@us.ibm.com> wrote:
> 
> > Currently the dtc source code has libfdt integrated in it. This seems to
> > have become place for upstream libfdt changes. Now we all know everyone
> > (linux kernel, cuboot) also have their own versions over libfdt. But if
> > another userspace app wants to use libfdt , it has to copy it from the
> > dtc source and try to maintain it's own copy.
> > 
> > The question I have is can libfdt be split out from dtc source, and
> > become it's own thing. This way other userspace apps can easily download
> > it and link with it?
> 
> Downloading isn't changed at all by splitting it out to it's own repo.

Well it is, as we could then point to libfdt repository. Right now we
need to download dtc ..the do make libfdt ... and then we are rollin. So
there are some steps required. 

It would just be really really nice if it was more stand alone.

> 
> > The reason I ask is I have added dynamic manipulation support of device
> > trees in memory into qemu for KVM. But the issue is keeping a copy of
> > libfdt in the KVM userspace repository, which is getting some opposition
> > (understandably). But this would be much easier if there was a libfdt
> > repo for the library so that we wouldn't need to keep our own copy.
> 
> It seems the real crux of your issue is that you want distros to
> provide a libfdt package.  That can be done by creating a subpackage
> off of the dtc package.  The harder part for certain distros will be
> either convincing them to allow the static libfdt to exist, or creating
> a shared library for it instead.

Ultimately this would be the most optimal solution for the future. If
distros could distribute a library with headers that would be awesome. 
Since this is something that qemu in general can take advantage of (not
just KVM qemu)...having libs in the distro would be easier for
distributing.

But for now I just need a way for users to easily get a hold of libary
and just point cflags and ldflags at the directory where it is
compiled. 


> 
> josh

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[David Gibson]

On Wed, Feb 27, 2008 at 01:40:43PM -0600, Jerone Young wrote:
> Currently the dtc source code has libfdt integrated in it. This seems to
> have become place for upstream libfdt changes. Now we all know everyone
> (linux kernel, cuboot) also have their own versions over libfdt. But if
> another userspace app wants to use libfdt , it has to copy it from the
> dtc source and try to maintain it's own copy.
> 
> The question I have is can libfdt be split out from dtc source, and
> become it's own thing. This way other userspace apps can easily download
> it and link with it?
> 
> The reason I ask is I have added dynamic manipulation support of device
> trees in memory into qemu for KVM. But the issue is keeping a copy of
> libfdt in the KVM userspace repository, which is getting some opposition
> (understandably). But this would be much easier if there was a libfdt
> repo for the library so that we wouldn't need to keep our own copy.

Um.. libfdt was moved into the dtc repo for convenience, both for us
writing it (they help to test each other), and for those using it -
don't have to have separate pulls for these closely related tools.

I don't understand why you're finding the merged libfdt inconvenient.
"make" will build both dtc and libfdt, and libfdt can be easily taken
out and embedded on other projects.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Jerone Young]

On Thu, 2008-02-28 at 12:41 +1100, David Gibson wrote:
> On Wed, Feb 27, 2008 at 01:40:43PM -0600, Jerone Young wrote:
> > Currently the dtc source code has libfdt integrated in it. This seems to
> > have become place for upstream libfdt changes. Now we all know everyone
> > (linux kernel, cuboot) also have their own versions over libfdt. But if
> > another userspace app wants to use libfdt , it has to copy it from the
> > dtc source and try to maintain it's own copy.
> > 
> > The question I have is can libfdt be split out from dtc source, and
> > become it's own thing. This way other userspace apps can easily download
> > it and link with it?
> > 
> > The reason I ask is I have added dynamic manipulation support of device
> > trees in memory into qemu for KVM. But the issue is keeping a copy of
> > libfdt in the KVM userspace repository, which is getting some opposition
> > (understandably). But this would be much easier if there was a libfdt
> > repo for the library so that we wouldn't need to keep our own copy.
> 
> Um.. libfdt was moved into the dtc repo for convenience, both for us
> writing it (they help to test each other), and for those using it -
> don't have to have separate pulls for these closely related tools.
> 
> I don't understand why you're finding the merged libfdt inconvenient.
> "make" will build both dtc and libfdt, and libfdt can be easily taken
> out and embedded on other projects.

The reason for all the contention is that libfdt is more of a shared
userspace library. Even though dtc really has been the only user of it
in userspace at the moment, now I want to add it's use to qemu. What
many are thinking about is the maintenance of the library. There are
still fixes going in the libfdt in dtc. 

If it where broken out of dtc it would be easier to pickup and pull
fixes from it. Even package it so programs can easily build it
standalone.

It's not often any userspace app needs libfdt. But now with
virtualization userspace apps actually can see the device tree in guest
memory and do have all sort of fun with it easily because of libfdt. 

> 

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Josh Boyer]

On Thu, 28 Feb 2008 10:30:44 -0600
Jerone Young <jyoung5@us.ibm.com> wrote:

> 
> On Thu, 2008-02-28 at 12:41 +1100, David Gibson wrote:
> > On Wed, Feb 27, 2008 at 01:40:43PM -0600, Jerone Young wrote:
> > > Currently the dtc source code has libfdt integrated in it. This seems to
> > > have become place for upstream libfdt changes. Now we all know everyone
> > > (linux kernel, cuboot) also have their own versions over libfdt. But if
> > > another userspace app wants to use libfdt , it has to copy it from the
> > > dtc source and try to maintain it's own copy.
> > > 
> > > The question I have is can libfdt be split out from dtc source, and
> > > become it's own thing. This way other userspace apps can easily download
> > > it and link with it?
> > > 
> > > The reason I ask is I have added dynamic manipulation support of device
> > > trees in memory into qemu for KVM. But the issue is keeping a copy of
> > > libfdt in the KVM userspace repository, which is getting some opposition
> > > (understandably). But this would be much easier if there was a libfdt
> > > repo for the library so that we wouldn't need to keep our own copy.
> > 
> > Um.. libfdt was moved into the dtc repo for convenience, both for us
> > writing it (they help to test each other), and for those using it -
> > don't have to have separate pulls for these closely related tools.
> > 
> > I don't understand why you're finding the merged libfdt inconvenient.
> > "make" will build both dtc and libfdt, and libfdt can be easily taken
> > out and embedded on other projects.
> 
> The reason for all the contention is that libfdt is more of a shared
> userspace library. Even though dtc really has been the only user of it

Actually, in it's current form, it's not.  It's built to be statically
linked, not as a shared library (.a vs. .so).

> in userspace at the moment, now I want to add it's use to qemu. What
> many are thinking about is the maintenance of the library. There are
> still fixes going in the libfdt in dtc. 

You still haven't explained why maintenance is harder or somehow less
doable by having it in the dtc repo.  Maintenance is very much the
concern of the upstream developers, which seem to be saying it's not a
problem for them...

> If it where broken out of dtc it would be easier to pickup and pull
> fixes from it. Even package it so programs can easily build it
> standalone.

That's akin to saying libcrypto should be broken out to be completely
standalone from openssl.  That doesn't make sense either.

josh

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Jerone Young]

On Thu, 2008-02-28 at 12:59 -0600, Josh Boyer wrote:
> On Thu, 28 Feb 2008 10:30:44 -0600
> Jerone Young <jyoung5@us.ibm.com> wrote:
> 
> > 
> > On Thu, 2008-02-28 at 12:41 +1100, David Gibson wrote:
> > > On Wed, Feb 27, 2008 at 01:40:43PM -0600, Jerone Young wrote:
> > > > Currently the dtc source code has libfdt integrated in it. This seems to
> > > > have become place for upstream libfdt changes. Now we all know everyone
> > > > (linux kernel, cuboot) also have their own versions over libfdt. But if
> > > > another userspace app wants to use libfdt , it has to copy it from the
> > > > dtc source and try to maintain it's own copy.
> > > > 
> > > > The question I have is can libfdt be split out from dtc source, and
> > > > become it's own thing. This way other userspace apps can easily download
> > > > it and link with it?
> > > > 
> > > > The reason I ask is I have added dynamic manipulation support of device
> > > > trees in memory into qemu for KVM. But the issue is keeping a copy of
> > > > libfdt in the KVM userspace repository, which is getting some opposition
> > > > (understandably). But this would be much easier if there was a libfdt
> > > > repo for the library so that we wouldn't need to keep our own copy.
> > > 
> > > Um.. libfdt was moved into the dtc repo for convenience, both for us
> > > writing it (they help to test each other), and for those using it -
> > > don't have to have separate pulls for these closely related tools.
> > > 
> > > I don't understand why you're finding the merged libfdt inconvenient.
> > > "make" will build both dtc and libfdt, and libfdt can be easily taken
> > > out and embedded on other projects.
> > 
> > The reason for all the contention is that libfdt is more of a shared
> > userspace library. Even though dtc really has been the only user of it
> 
> Actually, in it's current form, it's not.  It's built to be statically
> linked, not as a shared library (.a vs. .so).

Right now the dtc makefiles do this. Though I have created a makefile so
that it does :-). 
> 
> > in userspace at the moment, now I want to add it's use to qemu. What
> > many are thinking about is the maintenance of the library. There are
> > still fixes going in the libfdt in dtc. 
> 
> You still haven't explained why maintenance is harder or somehow less
> doable by having it in the dtc repo.  Maintenance is very much the
> concern of the upstream developers, which seem to be saying it's not a
> problem for them...

I guess what I see libfdt as something like shared userspace library. At
the moment dtc is the only userspace project to use it.  So it make
perfect since to keep it with the source and not separated.

Though when other projects need it .. the option of having to try to
figure out what version of dtc to grab so understand what libfdt is
usable, can be a bit of a pain.

Though I can't really argue that you can't get around this by just
downloading dtc and grabbing out the libfdt package..though it does
cause some indirection.  


> > If it where broken out of dtc it would be easier to pickup and pull
> > fixes from it. Even package it so programs can easily build it
> > standalone.
> 
> That's akin to saying libcrypto should be broken out to be completely
> standalone from openssl.  That doesn't make sense either.

Actually ouch.. I think you got me dead on :-) .... so basically I am
making the same argument.


Thanks for the feedback guys!

> 
> josh

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Jerry Van Baren]

Jerone Young wrote:
> On Thu, 2008-02-28 at 12:59 -0600, Josh Boyer wrote:
>> On Thu, 28 Feb 2008 10:30:44 -0600
>> Jerone Young <jyoung5@us.ibm.com> wrote:

[big snip]

>> You still haven't explained why maintenance is harder or somehow less
>> doable by having it in the dtc repo.  Maintenance is very much the
>> concern of the upstream developers, which seem to be saying it's not a
>> problem for them...
> 
> I guess what I see libfdt as something like shared userspace library. At
> the moment dtc is the only userspace project to use it.  So it make
> perfect since to keep it with the source and not separated.
> 
> Though when other projects need it .. the option of having to try to
> figure out what version of dtc to grab so understand what libfdt is
> usable, can be a bit of a pain.
> 
> Though I can't really argue that you can't get around this by just
> downloading dtc and grabbing out the libfdt package..though it does
> cause some indirection.  

FWIIW, that is what the u-boot project is doing.  The last pass, I 
actually extracted the libfdt git patch(es) and then applied them to the 
u-boot tree so that the history would be carried over.  The libfdt 
portion is now quite stable and I don't see major changes coming that 
would cause this methodology to be a problem.

Best regards,
gvb

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Geert Uytterhoeven]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1543 bytes --]

On Thu, 28 Feb 2008, Josh Boyer wrote:
> On Thu, 28 Feb 2008 10:30:44 -0600
> Jerone Young <jyoung5@us.ibm.com> wrote:
> > If it where broken out of dtc it would be easier to pickup and pull
> > fixes from it. Even package it so programs can easily build it
> > standalone.
> 
> That's akin to saying libcrypto should be broken out to be completely
> standalone from openssl.  That doesn't make sense either.

Thanks, your openssl example triggered me posting this reply ;-)

I think people are confusing source and binary packages.

E.g. on Debian, the openssl source package is used to build 3 binary packages:
openssl, libssl0.9.8, and libssl-dev. Hence to install applications that use
libssl, you don't have to install all 3, just libssl0.9.8.

But this doesn't mean libssl is separate from openssl source-wise: both are
build from the same source package.

So the single source package dtc could be packaged as 2 binary packages: dtc
and libfdt.

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Network and Software Technology Center Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

Sony Network and Software Technology Center Europe
A division of Sony Service Centre (Europe) N.V.
Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium
VAT BE 0413.825.160 · RPR Brussels
Fortis Bank Zaventem · Swift GEBABEBB08A · IBAN BE39001382358619

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Josh Boyer]

On Fri, 29 Feb 2008 09:35:48 +0100 (CET)
Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com> wrote:

> On Thu, 28 Feb 2008, Josh Boyer wrote:
> > On Thu, 28 Feb 2008 10:30:44 -0600
> > Jerone Young <jyoung5@us.ibm.com> wrote:
> > > If it where broken out of dtc it would be easier to pickup and pull
> > > fixes from it. Even package it so programs can easily build it
> > > standalone.
> > 
> > That's akin to saying libcrypto should be broken out to be completely
> > standalone from openssl.  That doesn't make sense either.
> 
> Thanks, your openssl example triggered me posting this reply ;-)
> 
> I think people are confusing source and binary packages.
> 
> E.g. on Debian, the openssl source package is used to build 3 binary packages:
> openssl, libssl0.9.8, and libssl-dev. Hence to install applications that use
> libssl, you don't have to install all 3, just libssl0.9.8.
> 
> But this doesn't mean libssl is separate from openssl source-wise: both are
> build from the same source package.
> 
> So the single source package dtc could be packaged as 2 binary packages: dtc
> and libfdt.

Yes, that's certainly possible for the various distros.  For Fedora
we'd have to get an exception for a static library, or convert it to a
shared one.  Other distros might not have that restriction.

josh

Re: [dtc] breaking out libfdt from dtc so other progs can use it

[Fathi Boudra]

> I think people are confusing source and binary packages.
>
> E.g. on Debian, the openssl source package is used to build 3 binary
> packages: openssl, libssl0.9.8, and libssl-dev. Hence to install
> applications that use libssl, you don't have to install all 3, just
> libssl0.9.8.

That make sense.
Just ask downstream (distros) to ship libfdt binaries packages.

cheers,

Fathi