[Buildroot] handling package souce in .zip file

[Buildroot] handling package souce in .zip file

[Paulius Zaleckas]

I want to add package, but its source is available only in zip format :(

I was thinking how to handle zip, but I don't have preference:
1. Not generic methods, since this may be the only package with zip:
a) add possibility to override the whole extraction procedure.
b) add post download hook, and repackage this zip as tar.
2. Generic method:
a) add very hacky INFLATE.zip which will repackage zip as tar.
b) add special handling for INFLATE.zip without involving tar.

Repackaging zip as tar is a headache, since it is not possible to
pipe unzip output to tar. So one has to extract zip to some dir and
then tar this dir and delete dir after taring.
Extracting zip without tar is also a little bit problematic since unzip
has no equivalent option for --strip-path/--strip-components
So maybe it is not possible to make generic support...

What do you guys think?

[Buildroot] handling package souce in .zip file

[Lionel Landwerlin]

What's about this :

unzip file.zip /tmp/plop
mv /tmp/plop/*/* output/build/myzippkg-0.1/

Regards,

--
Lionel Landwerlin

Le vendredi 26 novembre 2010 ? 20:37 +0200, Paulius Zaleckas a ?crit :
> I want to add package, but its source is available only in zip format :(
> 
> I was thinking how to handle zip, but I don't have preference:
> 1. Not generic methods, since this may be the only package with zip:
> a) add possibility to override the whole extraction procedure.
> b) add post download hook, and repackage this zip as tar.
> 2. Generic method:
> a) add very hacky INFLATE.zip which will repackage zip as tar.
> b) add special handling for INFLATE.zip without involving tar.
> 
> Repackaging zip as tar is a headache, since it is not possible to
> pipe unzip output to tar. So one has to extract zip to some dir and
> then tar this dir and delete dir after taring.
> Extracting zip without tar is also a little bit problematic since unzip
> has no equivalent option for --strip-path/--strip-components
> So maybe it is not possible to make generic support...
> 
> What do you guys think?
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] handling package souce in .zip file

[Paulius Zaleckas]

On Fri, Nov 26, 2010 at 9:35 PM, Lionel Landwerlin
<llandwerlin@gmail.com> wrote:
> What's about this :
>
> unzip file.zip /tmp/plop
> mv /tmp/plop/*/* output/build/myzippkg-0.1/

Yes, this is almost what I was thinking, but the problem is that there
is no way to override "standard" buildroot extraction step or at least
I don't know it (using GENTARGET) :)

> Regards,
>
> --
> Lionel Landwerlin
>
> Le vendredi 26 novembre 2010 ? 20:37 +0200, Paulius Zaleckas a ?crit :
>> I want to add package, but its source is available only in zip format :(
>>
>> I was thinking how to handle zip, but I don't have preference:
>> 1. Not generic methods, since this may be the only package with zip:
>> a) add possibility to override the whole extraction procedure.
>> b) add post download hook, and repackage this zip as tar.
>> 2. Generic method:
>> a) add very hacky INFLATE.zip which will repackage zip as tar.
>> b) add special handling for INFLATE.zip without involving tar.
>>
>> Repackaging zip as tar is a headache, since it is not possible to
>> pipe unzip output to tar. So one has to extract zip to some dir and
>> then tar this dir and delete dir after taring.
>> Extracting zip without tar is also a little bit problematic since unzip
>> has no equivalent option for --strip-path/--strip-components
>> So maybe it is not possible to make generic support...
>>
>> What do you guys think?
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
>
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] handling package souce in .zip file

[Lionel Landwerlin]

Le samedi 27 novembre 2010 ? 01:05 +0200, Paulius Zaleckas a ?crit :
> On Fri, Nov 26, 2010 at 9:35 PM, Lionel Landwerlin
> <llandwerlin@gmail.com> wrote:
> > What's about this :
> >
> > unzip file.zip /tmp/plop
> > mv /tmp/plop/*/* output/build/myzippkg-0.1/
> 
> Yes, this is almost what I was thinking, but the problem is that there
> is no way to override "standard" buildroot extraction step or at least
> I don't know it (using GENTARGET) :)
> 

You just need to define a macro per ".ext", just like what we've got for
the download macros.

INFLATE.zip would be the 2 lines above, and for the rest it would be the
"bzcat/zcat/cat | tar -C ...".

Regards,

--
Lionel Landwerlin

[Buildroot] handling package souce in .zip file

[Paulius Zaleckas]

On 11/27/2010 01:35 AM, Lionel Landwerlin wrote:
> Le samedi 27 novembre 2010 ? 01:05 +0200, Paulius Zaleckas a ?crit :
>> On Fri, Nov 26, 2010 at 9:35 PM, Lionel Landwerlin
>> <llandwerlin@gmail.com>  wrote:
>>> What's about this :
>>>
>>> unzip file.zip /tmp/plop
>>> mv /tmp/plop/*/* output/build/myzippkg-0.1/
>>
>> Yes, this is almost what I was thinking, but the problem is that there
>> is no way to override "standard" buildroot extraction step or at least
>> I don't know it (using GENTARGET) :)
>>
>
> You just need to define a macro per ".ext", just like what we've got for
> the download macros.
>
> INFLATE.zip would be the 2 lines above, and for the rest it would be the
> "bzcat/zcat/cat | tar -C ...".

Yes, but the problem is here:

# Unpack the archive
$(BUILD_DIR)/%/.stamp_extracted:
	@$(call MESSAGE,"Extracting")
	$(Q)mkdir -p $(@D)
	$(Q)$(INFLATE$(suffix $($(PKG)_SOURCE))) $(DL_DIR)/$($(PKG)_SOURCE) | \
	$(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) -
# some packages have messed up permissions inside
	$(Q)chmod -R ug+rw $(@D)
	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
	$(Q)touch $@

As you can see $(INFLATE$(suffix $($(PKG)_SOURCE))) $(DL_DIR)/$($(PKG)_SOURCE)
output is piped to tar...

[Buildroot] handling package souce in .zip file

[Thomas Petazzoni]

On Mon, 29 Nov 2010 22:05:29 +0200
Paulius Zaleckas <paulius.zaleckas@gmail.com> wrote:

> As you can see $(INFLATE$(suffix $($(PKG)_SOURCE))) $(DL_DIR)/$($(PKG)_SOURCE)
> output is piped to tar...

This can obviously be changed if needed.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] handling package souce in .zip file

[Lionel Landwerlin]

On Mon, Nov 29, 2010 at 9:05 PM, Paulius Zaleckas
<paulius.zaleckas@gmail.com> wrote:
> On 11/27/2010 01:35 AM, Lionel Landwerlin wrote:
>>
>> Le samedi 27 novembre 2010 ? 01:05 +0200, Paulius Zaleckas a ?crit :
>>>
>>> On Fri, Nov 26, 2010 at 9:35 PM, Lionel Landwerlin
>>> <llandwerlin@gmail.com> ?wrote:
>>>>
>>>> What's about this :
>>>>
>>>> unzip file.zip /tmp/plop
>>>> mv /tmp/plop/*/* output/build/myzippkg-0.1/
>>>
>>> Yes, this is almost what I was thinking, but the problem is that there
>>> is no way to override "standard" buildroot extraction step or at least
>>> I don't know it (using GENTARGET) :)
>>>
>>
>> You just need to define a macro per ".ext", just like what we've got for
>> the download macros.
>>
>> INFLATE.zip would be the 2 lines above, and for the rest it would be the
>> "bzcat/zcat/cat | tar -C ...".
>
> Yes, but the problem is here:
>
> # Unpack the archive
> $(BUILD_DIR)/%/.stamp_extracted:
> ? ? ? ?@$(call MESSAGE,"Extracting")
> ? ? ? ?$(Q)mkdir -p $(@D)
> ? ? ? ?$(Q)$(INFLATE$(suffix $($(PKG)_SOURCE))) $(DL_DIR)/$($(PKG)_SOURCE) |
> \
> ? ? ? ?$(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) -
> # some packages have messed up permissions inside
> ? ? ? ?$(Q)chmod -R ug+rw $(@D)
> ? ? ? ?$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
> ? ? ? ?$(Q)touch $@
>
> As you can see $(INFLATE$(suffix $($(PKG)_SOURCE)))
> $(DL_DIR)/$($(PKG)_SOURCE)
> output is piped to tar...
>

I still don't catch your problem.
You can embed this piped bzip/gzip + tar into one macro.
You would then have :

INFLATE_TAR.bz2 = bzcat $(1) | tar -C $(2) xf -
INFLATE_TAR.gz = zcat $(1) | tar -C $(2) xf -
INFLATE.zip = unzip $(1) /tmp/plop && mv /tmp/plop/*/* $(2)/

(altough it might be a little bit complex...)

Regards,

--
Lionel Landwerlin