[Buildroot] [PATCH] Restrict tar transform to regular files Without this

[Buildroot] [PATCH] Restrict tar transform to regular files Without this

[Timothy Pearson]

 restriction, symlinks are rewritten and corrupted.

Example without the restriction:

Input tree (valid):
package-githash/file1
package-githash/link -> ./file1

Output tree (broken):
package-githash/file1
package-githash/link -> package-githash/file1
---
 support/download/git | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/download/git b/support/download/git
index 17ca04eb98..1acb795c81 100755
--- a/support/download/git
+++ b/support/download/git
@@ -190,7 +190,7 @@ LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
 
 # Create GNU-format tarballs, since that's the format of the tarballs on
 # sources.buildroot.org and used in the *.hash files
-tar cf - --transform="s#^\./#${basename}/#" \
+tar cf - --transform="flags=r;s#^\./#${basename}/#" \
          --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
          -T "${output}.list.sorted" >"${output}.tar"
 gzip -6 -n <"${output}.tar" >"${output}"

[Buildroot] [PATCH] Restrict tar transform to regular files Without this

[Yann E. MORIN]

Timothy, All,

On 2019-04-26 11:49 -0500, Timothy Pearson spake thusly:
>  restriction, symlinks are rewritten and corrupted.

Something is amiss in your commit log. It shoul;d be:

    titile on a single line

    Some text on
    multiple lines.

    Signed-off-by: Your NAME <your@email>

> Example without the restriction:
> 
> Input tree (valid):
> package-githash/file1
> package-githash/link -> ./file1
> 
> Output tree (broken):
> package-githash/file1
> package-githash/link -> package-githash/file1
> ---
>  support/download/git | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/support/download/git b/support/download/git
> index 17ca04eb98..1acb795c81 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -190,7 +190,7 @@ LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
>  
>  # Create GNU-format tarballs, since that's the format of the tarballs on
>  # sources.buildroot.org and used in the *.hash files
> -tar cf - --transform="s#^\./#${basename}/#" \
> +tar cf - --transform="flags=r;s#^\./#${basename}/#" \

OK, so this is a nice trick and all, but I can't find any reference to
it in the man page. But I see it's in the git tree in doc/tar.texi...

Your proposal would exclude hardlinks too, which is not correct I think.
In fact, we would need to use 'S' (upper-case 'S') here, because we want
it to apply to anything but symlinks.

Also, the texi file states that "Default scope flags can also be changed
using flags= statement in the transform expression." However, they are
standard flags too, quoting:

    Supported flags are:
        g [...]
        i [...]
        x [...]
        <number> [...]

    [...]

    In addition, several transformation scope flags are supported, that
    control to what files transformations apply.  These are:
        r   Apply transformation to regular archive members.
        R   Do not apply transformation to regular archive members.
        s   Apply transformation to symbolic link targets.
        S   Do not apply transformation to symbolic link targets.
        h   Apply transformation to hard link targets.
        H   Do not apply transformation to hard link targets.

So I think we could simply write:

    --transform="s#^\./#${basename}/#S'

Regards,
Yann E. MORIN.

>           --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
>           -T "${output}.list.sorted" >"${output}.tar"
>  gzip -6 -n <"${output}.tar" >"${output}"
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] Restrict tar transform to regular files Without this

[Timothy Pearson]

> Timothy, All,
> 
> On 2019-04-26 11:49 -0500, Timothy Pearson spake thusly:
>>  restriction, symlinks are rewritten and corrupted.
> 
> Something is amiss in your commit log. It shoul;d be:
> 
>    titile on a single line
> 
>    Some text on
>    multiple lines.
> 
>    Signed-off-by: Your NAME <your@email>

Looks like the signed off line wasn't part of the commit...sorry about that.  I've re-uploaded it.  The rest of the formatting was generated by git format-patch and should be correct?

>> Example without the restriction:
>> 
>> Input tree (valid):
>> package-githash/file1
>> package-githash/link -> ./file1
>> 
>> Output tree (broken):
>> package-githash/file1
>> package-githash/link -> package-githash/file1
>> ---
>>  support/download/git | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/support/download/git b/support/download/git
>> index 17ca04eb98..1acb795c81 100755
>> --- a/support/download/git
>> +++ b/support/download/git
>> @@ -190,7 +190,7 @@ LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
>>  
>>  # Create GNU-format tarballs, since that's the format of the tarballs on
>>  # sources.buildroot.org and used in the *.hash files
>> -tar cf - --transform="s#^\./#${basename}/#" \
>> +tar cf - --transform="flags=r;s#^\./#${basename}/#" \
> 
> OK, so this is a nice trick and all, but I can't find any reference to
> it in the man page. But I see it's in the git tree in doc/tar.texi...
> 
> Your proposal would exclude hardlinks too, which is not correct I think.
> In fact, we would need to use 'S' (upper-case 'S') here, because we want
> it to apply to anything but symlinks.
> 
> Also, the texi file states that "Default scope flags can also be changed
> using flags= statement in the transform expression." However, they are
> standard flags too, quoting:
> 
>    Supported flags are:
>        g [...]
>        i [...]
>        x [...]
>        <number> [...]
> 
>    [...]
> 
>    In addition, several transformation scope flags are supported, that
>    control to what files transformations apply.  These are:
>        r   Apply transformation to regular archive members.
>        R   Do not apply transformation to regular archive members.
>        s   Apply transformation to symbolic link targets.
>        S   Do not apply transformation to symbolic link targets.
>        h   Apply transformation to hard link targets.
>        H   Do not apply transformation to hard link targets.
> 
> So I think we could simply write:
> 
>    --transform="s#^\./#${basename}/#S'

If that is functionally equivalent I'd be OK with it.  Let me run a quick test against the original failing downloads first.

> Regards,
> Yann E. MORIN.
> 
>>           --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
>>           -T "${output}.list.sorted" >"${output}.tar"
>>  gzip -6 -n <"${output}.tar" >"${output}"
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
> 
> --
> .-----------------.--------------------.------------------.--------------------.
>|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
>| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
>| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
>| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] Restrict tar transform to regular files Without this

[Yann E. MORIN]

Timothy, All,

[Please, wrap your messages at ~72 chars]

On 2019-04-27 03:49 -0500, Timothy Pearson spake thusly:
> > On 2019-04-26 11:49 -0500, Timothy Pearson spake thusly:
> >>  restriction, symlinks are rewritten and corrupted.
> > 
> > Something is amiss in your commit log. It shoul;d be:
> > 
> >    titile on a single line
> > 
> >    Some text on
> >    multiple lines.
> > 
> >    Signed-off-by: Your NAME <your@email>
> 
> Looks like the signed off line wasn't part of the commit...sorry about that.
> I've re-uploaded it. The rest of the formatting was generated by git
> format-patch and should be correct?

No it's not correct, see the subject of the mail which ends up in the
middle of a sentence, and see how patchwork stored it (even your v2):

    https://patchwork.ozlabs.org/patch/1091900/

By the way, why did you resend a v2 that does not address my comment,
which suggests putting the flag at the end of the expression?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'