[Buildroot] Pre-RFC: clean integration of unclean packages

[Buildroot] Pre-RFC: clean integration of unclean packages

[Arnout Vandecappelle]

 Hoi all,

 [Please keep me in CC]

 I mentioned a couple of months ago that I was trying to integrate TI's SDK's for accessing the accelerator hardware on DaVinci and OMAP platforms.  I had given up on that but now I picked it up again.

 I did make some progress already.  The most important step is that I found tarballs or svn repositories for all the packages that I need, so we can avoid executing a binary installer.

 A second problem is that TI uses a horrible self-cooked build system. It makes assumptions about file locations that are difficult to satisfy.  In addition, some packages don't even have a Makefile.  And finally, they generate libraries with names like 'cmem.a470MV' instead of the usual 'libcmem.a'.  To overcome all this, I wrote a custom 'Makefile.ti' that is used by buildroot instead of the packages' Makefiles.  Does everybody agree that this is a good idea, or does someone have a better suggestion?

 A third problem is that some packages depend on 'xdctools'.  This is a system to ease integration of embedded components into Eclipse.  One small part of it is a header file that defines some standard types (basically stdint.h and stdbool.h but with different names).  The whole xdctools package is a source tree of about half a gig, but for buildroot we just need this one 'std.h' file out of it...  So I decided to copy that file directly into buildroot.  Does everybody agree that this is a good idea, or does someone have a better suggestion?

 Finally, a fourth problem is that some packages assume they can directly access other packages' source trees.  To be precise: in the source code, you'll see things like #include <ti/sdo/ce/osal/Memory.h>; the Makefile adds a -I<other-src-tree> to the CFLAGS.  Now, I consider it a Bad Thing to rely on the presence of another source directory, so as a solution I just install all the header files in the staging tree.  Does everybody agree that this is a good idea, or does someone have a better suggestion?


 Thank you for your feedback!  And hopefully you'll see a first RFC for the TI libraries in a week or two.

 Regards,
 Arnout


 PS I haven't had time to keep up with the BR mailling list lately (I'm 250 messages behind at the moment), so please keep me in CC.

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111205/8aa95db3/attachment.html>

[Buildroot] Pre-RFC: clean integration of unclean packages

[Thomas De Schampheleire]

Hi Arnout,

On Mon, Dec 5, 2011 at 10:46 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> Hoi all,
>
>
> [Please keep me in CC]
>
>
> I mentioned a couple of months ago that I was trying to integrate TI's SDK's
> for accessing the accelerator hardware on DaVinci and OMAP platforms. I had
> given up on that but now I picked it up again.
>
>
> I did make some progress already. The most important step is that I found
> tarballs or svn repositories for all the packages that I need, so we can
> avoid executing a binary installer.
>
>
> A second problem is that TI uses a horrible self-cooked build system. It
> makes assumptions about file locations that are difficult to satisfy. In
> addition, some packages don't even have a Makefile. And finally, they
> generate libraries with names like 'cmem.a470MV' instead of the usual
> 'libcmem.a'. To overcome all this, I wrote a custom 'Makefile.ti' that is
> used by buildroot instead of the packages' Makefiles. Does everybody agree
> that this is a good idea, or does someone have a better suggestion?

Where is the Makefile.ti located and how does it get there? Is it
created by patching the source tree, or is it directly present in the
buildroot sources?

>
>
> A third problem is that some packages depend on 'xdctools'. This is a system
> to ease integration of embedded components into Eclipse. One small part of
> it is a header file that defines some standard types (basically stdint.h and
> stdbool.h but with different names). The whole xdctools package is a source
> tree of about half a gig, but for buildroot we just need this one 'std.h'
> file out of it... So I decided to copy that file directly into buildroot.
> Does everybody agree that this is a good idea, or does someone have a better
> suggestion?

How is this achieved? Is there a dummy xdctools package that just
installs one file into staging? How is the file copied, is it directly
present in buildroot, or by patching?
Or did you add a patch adding that file to each of the packages that
depend on xdctools?

>
>
> Finally, a fourth problem is that some packages assume they can directly
> access other packages' source trees. To be precise: in the source code,
> you'll see things like #include <ti/sdo/ce/osal/Memory.h>; the Makefile adds
> a -I<other-src-tree> to the CFLAGS. Now, I consider it a Bad Thing to rely
> on the presence of another source directory, so as a solution I just install
> all the header files in the staging tree. Does everybody agree that this is
> a good idea, or does someone have a better suggestion?

This looks like a valid solution to me.

>
>
>
> Thank you for your feedback! And hopefully you'll see a first RFC for the TI
> libraries in a week or two.

Thanks for your contribution already,

Thomas

[Buildroot] Pre-RFC: clean integration of unclean packages

[Arnout Vandecappelle]

On Monday 05 December 2011 10:58:11 Thomas De Schampheleire wrote:
> On Mon, Dec 5, 2011 at 10:46 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
[snip]
> > A second problem is that TI uses a horrible self-cooked build system. It
> > makes assumptions about file locations that are difficult to satisfy. In
> > addition, some packages don't even have a Makefile. And finally, they
> > generate libraries with names like 'cmem.a470MV' instead of the usual
> > 'libcmem.a'. To overcome all this, I wrote a custom 'Makefile.ti' that is
> > used by buildroot instead of the packages' Makefiles. Does everybody agree
> > that this is a good idea, or does someone have a better suggestion?
> 
> Where is the Makefile.ti located and how does it get there? Is it
> created by patching the source tree, or is it directly present in the
> buildroot sources?

 It's in the packages/ti directory and is used by calling 
$(MAKE) -f packages/ti/Makefile.ti -C $(@D)

 I prefer that way to patching the actual source tree because of clarity.
Note that the same Makefile.ti is used for all ti packages.

> > A third problem is that some packages depend on 'xdctools'. This is a system
> > to ease integration of embedded components into Eclipse. One small part of
> > it is a header file that defines some standard types (basically stdint.h and
> > stdbool.h but with different names). The whole xdctools package is a source
> > tree of about half a gig, but for buildroot we just need this one 'std.h'
> > file out of it... So I decided to copy that file directly into buildroot.
> > Does everybody agree that this is a good idea, or does someone have a better
> > suggestion?
> 
> How is this achieved? Is there a dummy xdctools package that just
> installs one file into staging? How is the file copied, is it directly
> present in buildroot, or by patching?
> Or did you add a patch adding that file to each of the packages that
> depend on xdctools?

 packages/ti/xdctools/std.h which has the following .mk file:

XDCTOOLS_SOURCE =
XDCTOOLS_VERSION = 1

XDCTOOLS_INSTALL_STAGING = YES
XDCTOOLS_INSTALL_TARGET = NO

define XDCTOOLS_INSTALL_STAGING_CMDS
	$(INSTALL) -D -m 0644 package/ti/xdctools/std.h $(STAGING_DIR)/usr/include/xdc/std.h
endef

define XDCTOOLS_UNINSTALL_STAGING_CMDS
	$(RM) $(STAGING_DIR)/usr/include/xdc/std.h
endef

$(eval $(call GENTARGETS))

 It also doesn't have a Config.in, it can only be used as a dependency.


 Thanks,
 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111205/ded64422/attachment-0001.html>

[Buildroot] Pre-RFC: clean integration of unclean packages

[Thomas De Schampheleire]

On Mon, Dec 5, 2011 at 11:08 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On Monday 05 December 2011 10:58:11 Thomas De Schampheleire wrote:
>
>> On Mon, Dec 5, 2011 at 10:46 AM, Arnout Vandecappelle <arnout@mind.be>
>> wrote:
>
> [snip]
>
>> > A second problem is that TI uses a horrible self-cooked build system. It
>
>> > makes assumptions about file locations that are difficult to satisfy. In
>
>> > addition, some packages don't even have a Makefile. And finally, they
>
>> > generate libraries with names like 'cmem.a470MV' instead of the usual
>
>> > 'libcmem.a'. To overcome all this, I wrote a custom 'Makefile.ti' that
>> > is
>
>> > used by buildroot instead of the packages' Makefiles. Does everybody
>> > agree
>
>> > that this is a good idea, or does someone have a better suggestion?
>
>>
>
>> Where is the Makefile.ti located and how does it get there? Is it
>
>> created by patching the source tree, or is it directly present in the
>
>> buildroot sources?
>
>
> It's in the packages/ti directory and is used by calling
>
> $(MAKE) -f packages/ti/Makefile.ti -C $(@D)
>
>
> I prefer that way to patching the actual source tree because of clarity.
>
> Note that the same Makefile.ti is used for all ti packages.

Could you gives us an idea of the contents of this file?
Depending on its contents, this may be problematic: what if you update
only one of the ti packages and it became incompatible with the
existing Makefile.ti?
But, if this Makefile.ti file is sufficiently general, I wouldn't be
opposed to this principle.

>
>
>> > A third problem is that some packages depend on 'xdctools'. This is a
>> > system
>
>> > to ease integration of embedded components into Eclipse. One small part
>> > of
>
>> > it is a header file that defines some standard types (basically stdint.h
>> > and
>
>> > stdbool.h but with different names). The whole xdctools package is a
>> > source
>
>> > tree of about half a gig, but for buildroot we just need this one
>> > 'std.h'
>
>> > file out of it... So I decided to copy that file directly into
>> > buildroot.
>
>> > Does everybody agree that this is a good idea, or does someone have a
>> > better
>
>> > suggestion?
>
>>
>
>> How is this achieved? Is there a dummy xdctools package that just
>
>> installs one file into staging? How is the file copied, is it directly
>
>> present in buildroot, or by patching?
>
>> Or did you add a patch adding that file to each of the packages that
>
>> depend on xdctools?
>
>
> packages/ti/xdctools/std.h which has the following .mk file:
>
>
> XDCTOOLS_SOURCE =
>
> XDCTOOLS_VERSION = 1
>
>
> XDCTOOLS_INSTALL_STAGING = YES
>
> XDCTOOLS_INSTALL_TARGET = NO
>
>
> define XDCTOOLS_INSTALL_STAGING_CMDS
>
> $(INSTALL) -D -m 0644 package/ti/xdctools/std.h
> $(STAGING_DIR)/usr/include/xdc/std.h
>
> endef
>
>
> define XDCTOOLS_UNINSTALL_STAGING_CMDS
>
> $(RM) $(STAGING_DIR)/usr/include/xdc/std.h
>
> endef
>
>
> $(eval $(call GENTARGETS))
>
>
> It also doesn't have a Config.in, it can only be used as a dependency.

What if someone would really like the full xdctools for some reason?
I think it'd be better to rename your package to something more
specific, like xdctools-dep, or otherwise to have just one package but
with two targets (with a different name, of course).

Best regards,
Thomas

[Buildroot] Pre-RFC: clean integration of unclean packages

[Arnout Vandecappelle]

On Monday 05 December 2011 12:13:48 Thomas De Schampheleire wrote:
> > Note that the same Makefile.ti is used for all ti packages.
> 
> Could you gives us an idea of the contents of this file?
> Depending on its contents, this may be problematic: what if you update
> only one of the ti packages and it became incompatible with the
> existing Makefile.ti?
> But, if this Makefile.ti file is sufficiently general, I wouldn't be
> opposed to this principle.

 My Makefile.ti is currently still in flux.

 If you change or add a TI package, it may be necessary to modify Makefile.ti as well.  That's how I'm working now.  It should be possible to keep it compatible with the existing use cases.

> What if someone would really like the full xdctools for some reason?
> I think it'd be better to rename your package to something more
> specific, like xdctools-dep, or otherwise to have just one package but
> with two targets (with a different name, of course).

 Good point, I'll do that.

 Thank you for all your feedback!
 Regards,
 Arnout
-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

[Buildroot] Pre-RFC: clean integration of unclean packages

[Arnout Vandecappelle]

 Hoi all,

 [Please keep me in CC]

On Monday 05 December 2011 10:46:38 Arnout Vandecappelle wrote:
>  I mentioned a couple of months ago that I was trying to integrate TI's 
> SDK's for accessing the accelerator hardware on DaVinci and OMAP 
> platforms.  I had given up on that but now I picked it up again.
> 
>  I did make some progress already.  The most important step is that I 
> found tarballs or svn repositories for all the packages that I need, 
> so we can avoid executing a binary installer.

 I've ran into two more difficulties for which I could use some advise...

1. For some packages (notably the codec collection) I couldn't find a code
repository or tar file, they only seem to be available as a binary
installer.  I don't like running downloaded binaries...  In addition, the
binary checks that you are running a specific distro and have a specific
cross-compiler installed - which we don't want, of course.

 So as a work-around I'd like to create a .tar.gz for these packages
(which is fortunately allowed by the license, although the text of some
of the source files and PDF documents isn't so clear about it).  But
then we need a place to put it.

2. Some of the source files are generated by the XDCTool.  This tool
requires Java.  I guess that's a dependency we want to avoid in buildroot.
So I'll just add those generated files as patches.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43