[Buildroot] Hash checking infrastructure

[Buildroot] Hash checking infrastructure

[Mike]

A small nit about how the hash checking is
working (or in special cases, not working):

Use case:

When using a package source directory override,
the usual version number field is replaced with
the string 'custom'.

If the package being over-ridden has a hash file
in its Buildroot configuration directory,
the hash checker attempts to check the:
<package name>-custom-<archive type> in the 
download directory.

Reference section 8.11.6:
"... will no longer attempt to download, extract 
and patch the package."

I think the behavior should also include skipping
the hash check.

Mike

[Buildroot] Hash checking infrastructure

[Yann E. MORIN]

Mike, All,

On 2015-07-04 14:01 -0500, Mike spake thusly:
> A small nit about how the hash checking is
> working (or in special cases, not working):
> 
> Use case:
> 
> When using a package source directory override,
> the usual version number field is replaced with
> the string 'custom'.
> 
> If the package being over-ridden has a hash file
> in its Buildroot configuration directory,
> the hash checker attempts to check the:
> <package name>-custom-<archive type> in the 
> download directory.

I'm a bit surprised of this result. I just tried here with an override
srcdir or busybox, and I don't see anything related to checking a hash,
with this defconfig:

    BR2_x86_i686=y
    BR2_TOOLCHAIN_EXTERNAL=y

and this local.mk:

    BUSYBOX_OVERRIDE_SRCDIR=/home/ymorin/dev/buildroot/foo/busybox

and here's the output of 'make busybox':

    >>> busybox custom Syncing from source dir /home/ymorin/dev/buildroot/foo/busybox
    rsync -au --exclude .svn --exclude .git --exclude .hg --exclude .bzr
    --exclude CVS /home/ymorin/dev/buildroot/foo/busybox/
    /home/ymorin/dev/buildroot/O/build/busybox-custom

So, no, there's no message about checking any hash...

In fact, I can't even see how we could check a hash in such a case, as
checking the hash is directly done by our download wrapper, in
support/download/dl-wrapper, which does not get called for packages that
are overriden.

Can you provide a .config and a local.mk that exhibit this behaviour?

> Reference section 8.11.6:
> "... will no longer attempt to download, extract 
> and patch the package."
> 
> I think the behavior should also include skipping
> the hash check.

As far as I can see, that's already the case. Maybe we can add that in
the manual if that's what you're worried about. Care to send a patch?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Hash checking infrastructure

[Mike]

On Sat, 4 Jul 2015 23:47:34 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Mike, All,
> 
> On 2015-07-04 14:01 -0500, Mike spake thusly:
> > A small nit about how the hash checking is
> > working (or in special cases, not working):
> > 
> > Use case:
> > 
> > When using a package source directory override,
> > the usual version number field is replaced with
> > the string 'custom'.
> > 
> > If the package being over-ridden has a hash file
> > in its Buildroot configuration directory,
> > the hash checker attempts to check the:
> > <package name>-custom-<archive type> in the 
> > download directory.
> 
> I'm a bit surprised of this result. I just tried here with an override
> srcdir or busybox, and I don't see anything related to checking a
> hash, with this defconfig:
> 
>     BR2_x86_i686=y
>     BR2_TOOLCHAIN_EXTERNAL=y
> 
> and this local.mk:
> 
>     BUSYBOX_OVERRIDE_SRCDIR=/home/ymorin/dev/buildroot/foo/busybox
> 
> and here's the output of 'make busybox':
> 
>     >>> busybox custom Syncing from source
>     >>> dir /home/ymorin/dev/buildroot/foo/busybox
>     rsync -au --exclude .svn --exclude .git --exclude .hg
> --exclude .bzr --exclude CVS /home/ymorin/dev/buildroot/foo/busybox/
>     /home/ymorin/dev/buildroot/O/build/busybox-custom
> 
> So, no, there's no message about checking any hash...
> 

Sorry, I wasn't clear -
The build stops with a: "No hash for ...-custom-..." message.

To your test case above add, in a top-level local.mk:
BUSYBOX_OVERRIDE_SRCDIR = <some directory on your system>

Ah, point it at clone of busybox.git just to eliminate any
other strange side effects.

Note: The local.mk file needs to be in the top-level of your
output directory.
So if your doing an out-of-tree build (I am) with O=MyOut,
the local.mk needs to be in the MyOut directory.

(Which isn't mentioned clearly in the manual - that I can
probable send a patch for the description of where to put
the file.)

Mike


> In fact, I can't even see how we could check a hash in such a case, as
> checking the hash is directly done by our download wrapper, in
> support/download/dl-wrapper, which does not get called for packages
> that are overriden.
> 
> Can you provide a .config and a local.mk that exhibit this behaviour?
> 
> > Reference section 8.11.6:
> > "... will no longer attempt to download, extract 
> > and patch the package."
> > 
> > I think the behavior should also include skipping
> > the hash check.
> 
> As far as I can see, that's already the case. Maybe we can add that in
> the manual if that's what you're worried about. Care to send a patch?
> 
> Regards,
> Yann E. MORIN.
> 

[Buildroot] Hash checking infrastructure

[Mike]

On Sat, 4 Jul 2015 21:36:28 -0500
Mike <mikez@OpenPlayer.org> wrote:

> On Sat, 4 Jul 2015 23:47:34 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > Mike, All,
> > 
> > On 2015-07-04 14:01 -0500, Mike spake thusly:
> > > A small nit about how the hash checking is
> > > working (or in special cases, not working):
> > > 
> > > Use case:
> > > 
> > > When using a package source directory override,
> > > the usual version number field is replaced with
> > > the string 'custom'.
> > > 
> > > If the package being over-ridden has a hash file
> > > in its Buildroot configuration directory,
> > > the hash checker attempts to check the:
> > > <package name>-custom-<archive type> in the 
> > > download directory.
> > 
> > I'm a bit surprised of this result. I just tried here with an
> > override srcdir or busybox, and I don't see anything related to
> > checking a hash, with this defconfig:
> > 
> >     BR2_x86_i686=y
> >     BR2_TOOLCHAIN_EXTERNAL=y
> > 
> > and this local.mk:
> > 
> >     BUSYBOX_OVERRIDE_SRCDIR=/home/ymorin/dev/buildroot/foo/busybox
> > 
> > and here's the output of 'make busybox':
> > 
> >     >>> busybox custom Syncing from source
> >     >>> dir /home/ymorin/dev/buildroot/foo/busybox
> >     rsync -au --exclude .svn --exclude .git --exclude .hg
> > --exclude .bzr --exclude CVS /home/ymorin/dev/buildroot/foo/busybox/
> >     /home/ymorin/dev/buildroot/O/build/busybox-custom
> > 
> > So, no, there's no message about checking any hash...
> > 
> 

My bad -
I didn't try to duplicate your good result using Busybox.

I will do that tomorrow and post the result.

Meanwhile, the package I had specified as external was
LuaJIT.
And that package may be a **very special** case.

I can prevent the build from stopping by adding a
luajit-custom-tar.gz
to my download directory;
AND
adding an md5 hash for luajit-custom-tar.gz to the
package's hashfile.

I also have a lua module selected that requires the use of
the LuaRocks package infrastructure.
That may be complicating this use case also.

My reason for suspecting this is a very special case:
The Lua virtual package calls:
$(eval $(generic-package))
$(eval $(host-generic-package))

Also, the LuaJit makefile makes the same two calls.

And, I haven't tracked down the path that triggers a Lua
build when the LuaRocks package infrastructure is used.

- - - -

I fell into this strange behavior when trying a simple(tm)
LuaJIT version bump (from 2.0.3 -> 2.0.4).
And
I had a package source directory over-ride pointing at
my LuaJIT git clone (where I could work on a changed
patch-set for the new version).
And
I intentionally selected one of the Lua modules that
used the LuaRocks build infrastructure.

Next time - I'll pick something simpler to change.  ;)
Mike

> Sorry, I wasn't clear -
> The build stops with a: "No hash for ...-custom-..." message.
> 
> To your test case above add, in a top-level local.mk:
> BUSYBOX_OVERRIDE_SRCDIR = <some directory on your system>
> 
> Ah, point it at clone of busybox.git just to eliminate any
> other strange side effects.
> 
> Note: The local.mk file needs to be in the top-level of your
> output directory.
> So if your doing an out-of-tree build (I am) with O=MyOut,
> the local.mk needs to be in the MyOut directory.
> 
> (Which isn't mentioned clearly in the manual - that I can
> probable send a patch for the description of where to put
> the file.)
> 
> Mike
> 
> 
> > In fact, I can't even see how we could check a hash in such a case,
> > as checking the hash is directly done by our download wrapper, in
> > support/download/dl-wrapper, which does not get called for packages
> > that are overriden.
> > 
> > Can you provide a .config and a local.mk that exhibit this
> > behaviour?
> > 
> > > Reference section 8.11.6:
> > > "... will no longer attempt to download, extract 
> > > and patch the package."
> > > 
> > > I think the behavior should also include skipping
> > > the hash check.
> > 
> > As far as I can see, that's already the case. Maybe we can add that
> > in the manual if that's what you're worried about. Care to send a
> > patch?
> > 
> > Regards,
> > Yann E. MORIN.
> > 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] Hash checking infrastructure

[Thomas Petazzoni]

Dear Mike,

On Sun, 5 Jul 2015 00:00:50 -0500, Mike wrote:

> Meanwhile, the package I had specified as external was
> LuaJIT.
> And that package may be a **very special** case.

I've tried with LuaJIT and out of tree build, and it works just fine
for me:

 */ I have the LuaJIT Git repository cloned in
    $(HOME)/projets/luajit-2.0

 */ I have my Buildroot source code in $(HOME)/projets/buildroot

 */ I create my out of tree build directory in /tmp/luajit-build/

Here is what I do:

$ cat > .config
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2015.05.tar.bz2"
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_0=y
BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
BR2_TOOLCHAIN_EXTERNAL_INET_RPC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_LUAJIT=y
# BR2_TARGET_ROOTFS_TAR is not set
$ make olddefconfig
$ cat > local.mk
LUAJIT_OVERRIDE_SRCDIR = $(HOME)/projets/luajit-2.0/

And what happens:

$ make
[...]
>>> luajit custom Syncing from source dir /home/thomas/projets/luajit-2.0/
rsync -au --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS /home/thomas/projets/luajit-2.0// /tmp/luajit-build/build/luajit-custom
>>> luajit custom Configuring
>>> luajit custom Building
/usr/bin/make -j5 PREFIX="/usr" STATIC_CC="/tmp/luajit-build/host/usr/bin/arm-linux-gcc" DYNAMIC_CC="/tmp/luajit-build/host/usr/bin/arm-linux-gcc -fPIC" TARGET_LD="/tmp/luajit-build/host/usr/bin/arm-linux-gcc" TARGET_AR="/tmp/luajit-build/host/usr/bin/arm-linux-ar rcus" TARGET_STRIP=true TARGET_CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -Os " TARGET_LDFLAGS="" HOST_CC="/usr/bin/gcc -m32" HOST_CFLAGS="-O2 -I/tmp/luajit-build/host/usr/include" HOST_LDFLAGS="-L/tmp/luajit-build/host/lib -L/tmp/luajit-build/host/usr/lib -Wl,-rpath,/tmp/luajit-build/host/usr/lib" BUILDMODE=dynamic -C /tmp/luajit-build/build/luajit-custom amalg
Building LuaJIT 2.0.4
/usr/bin/make -C src amalg
+--------------------------------------------------------------------------+
| WARNING: Compiling the amalgamation needs a lot of virtual memory        |
| (around 300 MB with GCC 4.x)! If you don't have enough physical memory   |
| your machine will start swapping to disk and the compile will not finish |
| within a reasonable amount of time.                                      |
| So either compile on a bigger machine or use the non-amalgamated build.  |
+--------------------------------------------------------------------------+
/usr/bin/make all "LJCORE_O=ljamalg.o"
HOSTCC    host/minilua.o
HOSTCC    host/buildvm_asm.o
HOSTCC    host/buildvm_peobj.o
HOSTCC    host/buildvm_lib.o
[...]

I.e, it works perfectly fine.

So there is something else you are doing, or that is special in your
setup.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Hash checking infrastructure

[Yann E. MORIN]

Mike, Thomas, All,

On 2015-07-05 11:10 +0200, Thomas Petazzoni spake thusly:
> On Sun, 5 Jul 2015 00:00:50 -0500, Mike wrote:
> > Meanwhile, the package I had specified as external was
> > LuaJIT.
> > And that package may be a **very special** case.
> 
> I've tried with LuaJIT and out of tree build, and it works just fine
> for me:
[--SNIP--]
> $ make
> [...]
> >>> luajit custom Syncing from source dir /home/thomas/projets/luajit-2.0/
> rsync -au --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS /home/thomas/projets/luajit-2.0// /tmp/luajit-build/build/luajit-custom
> >>> luajit custom Configuring
> >>> luajit custom Building
> /usr/bin/make -j5 PREFIX="/usr" STATIC_CC="/tmp/luajit-build/host/usr/bin/arm-linux-gcc" DYNAMIC_CC="/tmp/luajit-build/host/usr/bin/arm-linux-gcc -fPIC" TARGET_LD="/tmp/luajit-build/host/usr/bin/arm-linux-gcc" TARGET_AR="/tmp/luajit-build/host/usr/bin/arm-linux-ar rcus" TARGET_STRIP=true TARGET_CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -Os " TARGET_LDFLAGS="" HOST_CC="/usr/bin/gcc -m32" HOST_CFLAGS="-O2 -I/tmp/luajit-build/host/usr/include" HOST_LDFLAGS="-L/tmp/luajit-build/host/lib -L/tmp/luajit-build/host/usr/lib -Wl,-rpath,/tmp/luajit-build/host/usr/lib" BUILDMODE=dynamic -C /tmp/luajit-build/build/luajit-custom amalg
> Building LuaJIT 2.0.4
> /usr/bin/make -C src amalg
> +--------------------------------------------------------------------------+
> | WARNING: Compiling the amalgamation needs a lot of virtual memory        |
> | (around 300 MB with GCC 4.x)! If you don't have enough physical memory   |
> | your machine will start swapping to disk and the compile will not finish |
> | within a reasonable amount of time.                                      |
> | So either compile on a bigger machine or use the non-amalgamated build.  |
> +--------------------------------------------------------------------------+
> /usr/bin/make all "LJCORE_O=ljamalg.o"
> HOSTCC    host/minilua.o
> HOSTCC    host/buildvm_asm.o
> HOSTCC    host/buildvm_peobj.o
> HOSTCC    host/buildvm_lib.o
> [...]
> 
> I.e, it works perfectly fine.

I concur with Thomas, it works as expected for me, too.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Hash checking infrastructure

[Mike]

On Sun, 5 Jul 2015 11:10:07 +0200
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:

> Dear Mike,
> 

- - - - -

> I.e, it works perfectly fine.
> 
> So there is something else you are doing, or that is special in your
> setup.
> 

This should allow duplication of the problem I am seeing:

Setup:

Brand new (2 hours ago) clone of buildroot.git and luajit.git
Both with heads left on master.

Brand new out-of-tree working directory.

Content of local.mk:

LUAJIT_OVERRIDE_SRCDIR = /var1/arm/luajit-2.0.git

The defconfig (from make savedefconfig):

BR2_PACKAGE_LUAJIT=y
BR2_PACKAGE_LUALOGGING=y
BR2_PACKAGE_LUAPOSIX=y

The (relevant) progress highlights, up to the failure:

>>> luajit custom Syncing from source dir /var1/arm/luajit-2.0.git
>>> luajit custom Configuring
>>> luajit custom Building
>>> luajit custom Installing to staging directory
>>> luajit custom Installing to target

The problem area:

>>> host-luajit custom Downloading
--2015-07-05 08:15:09--  http://luajit.org/download/LuaJIT-custom.tar.gz
Resolving luajit.org... 94.23.39.66
Connecting to luajit.org|94.23.39.66|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-07-05 08:15:10 ERROR 404: Not Found.

--2015-07-05 08:15:10--  http://sources.buildroot.net/LuaJIT-custom.tar.gz
Resolving sources.buildroot.net... 176.9.16.109
Connecting to sources.buildroot.net|176.9.16.109|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-07-05 08:15:10 ERROR 404: Not Found.

make: *** [/var1/arm/new-i386/build/host-luajit-custom/.stamp_downloaded] Error 1
make: Leaving directory `/var1/arm/buildroot.git'

- - - -

I was not imagining this problem.  ;)
And that does look like it is from the LuaRocks infrastructure.

I.E: the host-luajit (required to execute the lua-rocks install
in /host).

Mike

> Best regards,
> 
> Thomas

[Buildroot] Hash checking infrastructure

[Yann E. MORIN]

Mike, All,

On 2015-07-05 08:32 -0500, Mike spake thusly:
> On Sun, 5 Jul 2015 11:10:07 +0200
> Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:
[--SNIP--]
> > I.e, it works perfectly fine.
> > 
> > So there is something else you are doing, or that is special in your
> > setup.
[--SNIP--]
> The problem area:
> 
> >>> host-luajit custom Downloading
> --2015-07-05 08:15:09--  http://luajit.org/download/LuaJIT-custom.tar.gz
> Resolving luajit.org... 94.23.39.66
> Connecting to luajit.org|94.23.39.66|:80... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2015-07-05 08:15:10 ERROR 404: Not Found.
> 
> --2015-07-05 08:15:10--  http://sources.buildroot.net/LuaJIT-custom.tar.gz
> Resolving sources.buildroot.net... 176.9.16.109
> Connecting to sources.buildroot.net|176.9.16.109|:80... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2015-07-05 08:15:10 ERROR 404: Not Found.
> 
> make: *** [/var1/arm/new-i386/build/host-luajit-custom/.stamp_downloaded] Error 1
> make: Leaving directory `/var1/arm/buildroot.git'

Ah, but that is *not* a problem with the hash infra at all!

And infact, I already reproduced *that* error and already sent a patch:
    https://patchwork.ozlabs.org/patch/491307/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Hash checking infrastructure

[Yann E. MORIN]

Mike, all,

[Please, keep the list in Cc]

On 2015-07-05 09:54 -0500, Mike spake thusly:
> On Sun, 5 Jul 2015 16:27:27 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > And infact, I already reproduced *that* error and already sent a
> > patch: https://patchwork.ozlabs.org/patch/491307/
> 
> Tested, same set-up as previous, with this result:
> 
> ==== Successfully installed LuaJIT 2.0.4 to /usr ====
> make[1]: Leaving directory `/var1/arm/new-i386/build/host-luajit-custom'
> >>> host-luainterpreter virtual Extracting
> >>> host-luainterpreter virtual Patching
> >>> host-luainterpreter virtual Configuring
> >>> host-luainterpreter virtual Building
> >>> host-luainterpreter virtual Installing to host directory
> WARNING: no hash file for luarocks-2.2.2.tar.gz
> >>> host-luarocks 2.2.2 Extracting
> gzip -d -c /var1/arm/buildroot.git/dl/luarocks-2.2.2.tar.gz | tar --strip-components=1 -C /var1/arm/new-i386/build/host-luarocks-2.2.2  -xf -
> >>> host-luarocks 2.2.2 Patching
> >>> host-luarocks 2.2.2 Configuring
> cd /var1/arm/new-i386/build/host-luarocks-2.2.2 && ./configure --prefix=/var1/arm/new-i386/host/usr --sysconfdir=/var1/arm/new-i386/host/usr/etc/luarocks --with-lua=/var1/arm/new-i386/host/usr --lua-suffix=jit
> Checking Lua interpreter... luajit found in /var1/arm/new-i386/host/usr/bin
> Checking Lua includes... lua.h not found (looked in /var1/arm/new-i386/host/usr/include, /var1/arm/new-i386/host/usr/include/lua/5.1, /var1/arm/new-i386/host/usr/include/lua5.1)
> You may want to use the flag --with-lua or --with-lua-include. See --help.
> 
> configure failed.
> 
> make: *** [/var1/arm/new-i386/build/host-luarocks-2.2.2/.stamp_configured] Error 1
> make: Leaving directory `/var1/arm/buildroot.git'

This is expected behaviour, because the patches fronm Buildroot are
*not* applied when using an override-srcdir. This is done on-purpose, on
the assumption that the content of override-srcdir is entirely managed
by the user, and because there is *no* way for Buildroot to know if the
patches we have would still apply.

So, the new "error" you reported is totally expected.

Thanks for testing! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] host-luarocks include path does not match installation of host-luajit

[Mike]

On Sun, 5 Jul 2015 17:03:35 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> > >>> host-luarocks 2.2.2 Extracting  
> > gzip -d -c /var1/arm/buildroot.git/dl/luarocks-2.2.2.tar.gz | tar
> > --strip-components=1
> > -C /var1/arm/new-i386/build/host-luarocks-2.2.2  -xf -  
> > >>> host-luarocks 2.2.2 Patching
> > >>> host-luarocks 2.2.2 Configuring  

That is the luarocks package that is reporting that it can not
find 'lua.h'.

Only LuaJIT is in an external source directory, and it installed
just fine, with your proposed patch.

There is just a mis-match between where host-luajit was installed
and where host-luarocks is expecting them.

- - - - -

Translation: One problem fixed, now exposes another one.

Hah! 
And someone commented on how the title of this thread does
not strictly apply.
So I changed the title, but kept the same threading.

Mike

[Buildroot] host-luarocks include path does not match installation of host-luajit

[Yann E. MORIN]

Mike, All,

On 2015-07-05 11:56 -0500, Mike spake thusly:
> On Sun, 5 Jul 2015 17:03:35 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > > >>> host-luarocks 2.2.2 Extracting  
> > > gzip -d -c /var1/arm/buildroot.git/dl/luarocks-2.2.2.tar.gz | tar
> > > --strip-components=1
> > > -C /var1/arm/new-i386/build/host-luarocks-2.2.2  -xf -  
> > > >>> host-luarocks 2.2.2 Patching
> > > >>> host-luarocks 2.2.2 Configuring  
> 
> That is the luarocks package that is reporting that it can not
> find 'lua.h'.
> 
> Only LuaJIT is in an external source directory, and it installed
> just fine, with your proposed patch.
> 
> There is just a mis-match between where host-luajit was installed
> and where host-luarocks is expecting them.
> 
> - - - - -
> 
> Translation: One problem fixed, now exposes another one.

No, there is no problem.

As I stated in my previous reply: when one is using an override-srcdir,
Buildroot does *not* apply the bbundled patches, and we do have three
patches in Buildroot to fix LuaJIT:
    package/luajit/0001-no-bin-symlink.patch
    package/luajit/0002-install-inc.patch
    package/luajit/0003-install-so.patch    

and one of them (guess which!) is actually fixing the issue you are
seeing.

So, this is not a Buildroot issue, but a LuaJIT issue.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Hash checking infrastructure

[Yann E. MORIN]

Mike, All,

On 2015-07-05 00:00 -0500, Mike spake thusly:
[--SNIP--]
> I fell into this strange behavior when trying a simple(tm)
> LuaJIT version bump (from 2.0.3 -> 2.0.4).

Note that we already have luajit 2.0.4 in Buildroot.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Hash checking infrastructure

[Mike]

On Sun, 5 Jul 2015 11:44:52 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Mike, All,
> 
> On 2015-07-05 00:00 -0500, Mike spake thusly:
> [--SNIP--]
> > I fell into this strange behavior when trying a simple(tm)
> > LuaJIT version bump (from 2.0.3 -> 2.0.4).
> 
> Note that we already have luajit 2.0.4 in Buildroot.
> 
> Regards,
> Yann E. MORIN.
> 

@Thomas P & Yann -
Thanks for checking and reporting no problems.
I presume you where both using the current head.

I have been working on this for a few days now,
so my work was against the head of a few days ago.

I.E: Back in the "old days" (hours?) when it still
had only 2.0.3 in it.

- - - -

I don't see any signs of enabling a Lua module in
the .config snippet that Thomas P. sent.

Not just any Lua module, but one that uses the
LuaRocks infrastructure.

Why?
Because to install the host LuaRocks infrastructure
causes a host-Lua to be built.

At the moment, I am thinking that part is what falls
into the hash checking of a non-existent tar-ball.

The related part of my .config:

BR2_LUAROCKS_MIRROR="http://rocks.moonscript.org"
# BR2_PACKAGE_LUA is not set
BR2_PACKAGE_PROVIDES_LUAINTERPRETER="luajit"
BR2_PACKAGE_LUAINTERPRETER_ABI_VERSION="5.1"
BR2_PACKAGE_HAS_LUAINTERPRETER=y
BR2_PACKAGE_LUAJIT_ARCH_SUPPORTS=y
BR2_PACKAGE_LUAJIT=y

# Right here is the significant difference:
BR2_PACKAGE_LUALOGGING=y
BR2_PACKAGE_LUAPOSIX=y

- - - - -

Which is why my question about the auto-builder:

Once it had picked one of the Lua providers, did
it also go on to pick some (or all) of the Lua
modules.

From the lack of noise on the subject in the
auto-builder reports, I suspect not.

I.E: Some of the modules do not properly set
Lua-Include-Path (however named) and they just
are not going to build that way.  ;)

- - - - -

Come Monday morning, I'll do a pull to pick up the
recent changes made when the version was bump'd.

Then start over.

Thanks again,
Mike

[Buildroot] Hash checking infrastructure

[Mike]

On Sun, 5 Jul 2015 07:15:26 -0500
Mike <mikez@OpenPlayer.org> wrote:

> On Sun, 5 Jul 2015 11:44:52 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > Mike, All,
> > 
> > On 2015-07-05 00:00 -0500, Mike spake thusly:
> > [--SNIP--]
> > > I fell into this strange behavior when trying a simple(tm)
> > > LuaJIT version bump (from 2.0.3 -> 2.0.4).
> > 
> > Note that we already have luajit 2.0.4 in Buildroot.
> > 

Found it:
Subject: [Buildroot] [git commit] package/luajit: bump to version 2.0.4
Date: Thu, 21 May 2015 19:12:00 +0200

2 months ago? Must be something very wrong with my testing setup here.

Mike