Re: [meta-arago][master/kirkstone][PATCH 1/2] tisdk-sw-manifest: Add code to not fail on tar error

[Ryan Eatmon <reatmon@ti.com>]

On 11/7/2023 4:19 PM, Bin Liu wrote:
> Hi,
> 
> On Fri, Nov 03, 2023 at 05:17:45PM -0400, Denys Dmytriyenko wrote:
>> On Fri, Nov 03, 2023 at 11:18:31AM -0500, Ryan Eatmon via lists.yoctoproject.org wrote:
>>>
>>>
>>> On 11/2/2023 3:37 PM, Bin Liu wrote:
>>>> Hi,
>>>>
>>>> On Thu, Nov 02, 2023 at 05:18:21PM +0530, Chirag Shilwant wrote:
>>>>> Hi Ryan,
>>>>>
>>>>> On 31/10/23 21:29, Ryan Eatmon via lists.yoctoproject.org wrote:
>>>>>>
>>>>>>
>>>>>> On 10/31/2023 10:44 AM, Denys Dmytriyenko wrote:
>>>>>>> On Tue, Oct 31, 2023 at 09:04:32AM -0500, Ryan Eatmon via
>>>>>>> lists.yoctoproject.org wrote:
>>>>>>>> We want to add an image into the core bundle, but that image does not
>>>>>>>> have any opkg .control files.  tar apparently errors out if you ask it
>>>>>>>> to extract out files but the files are not in the archive, and the
>>>>>>>> recipes are setup that if any of the commands in the shell error out,
>>>>>>>> then the entire recipe fails.
>>>>>>>>
>>>>>>>> Simple fix, add an || (or) condition to the tar command to print a
>>>>>>>> message that there were not any control files instead of erroring out.
>>>>>>>
>>>>>>> So, simply bypassing tar error due to missing *.control files still
>>>>>>> won't
>>>>>>> enable you to properly generate the SW manifest. As those *.control
>>>>>>> files
>>>>>> >from individual packages are parsed to extract the license information.
>>>>>>> And they are missing because tiny image specifically disables "package
>>>>>>> management" to save on space. Therefore tiny image was not included in
>>>>>>> the bundle, since it doesn't generate the SW manifest.
>>>>>>
>>>>>> And this is why I sent the patch in rather than just taking it. This
>>>>>> patch was a response to Chirag running in the above error when trying to
>>>>>> add the tiny image into the core bundle in the processor sdk.
>>>>>>
>>>>>> So, why do we want to add tiny to the bundle?
>>>>>
>>>>>
>>>>> We got a requirement from Bin Liu to add tiny-image in
>>>>> /board-support/prebuilt-images/ of the SDK bundle.
>>>>> I'm not sure about the "why do we want it to the bundle" part. Adding Bin to
>>>>> provide more insights on the requirement part.
>>>>
>>>> The tiny image is very small and uses SysVInit. It has been used in many
>>>> use cases, initramfs is the most important one as far as I am aware.
>>>
>>>
>>> All of the images now use systemd.
>>
>> Correction - all of the images in a single build use the same init manager,
>> which by default is systemd.
> 
> systermd or sysvinit is not a concern, as long as there is a "tiny"
> image for initramfs use cases.
> 
>>
>> There used to be a hack that would force tiny image to use SysVinit, even
>> when other images use systemd. It was non-standard and problematic, so was
>> removed recently.
>>
>> There is still a way to control init manager for all Arago images by setting
>> ARAGO_SYSVINIT="1" or more directly INIT_MANAGER="sysvinit". But that would
>> affect all images, not just tiny.
>>
>> And if you really want tiny to use SysVinit and other images to use systemd,
>> you have to build them separately and you cannot have tiny in tisdk bundle.
>>
>> FS images in the bundle.
>>> The issue comes down to license compliance.  The core bundle has the
>>> requirement to produce a software license manifest for all of the
>>> items within the image.  That manifest is generated off of the opkg
>>> control files.  But the tiny image does not have opkg installed as
>>> it is not needed for the purpose of that image.  So we cannot gather
>>> the manifest information from the tiny image.
>>>
>>> So we have an impossible situation here.  We cannot include the tiny
>>> image without violation the manifest requirement.  And given that
>>> the tiny image is mainly meant for wakeup efforts, I don't see a
>>> need to include it in the bundle.  If someone wants the tiny image,
> 
> I guess by "wakeup efforts" you meant TI internal consumption? No,
> customers often used it for custom board bringup. The importance has no
> difference from the "tiny" image to other
> 
> -Bin.

The issue is not if customers find the image useful, the issue is 
license reporting compliance we have for including the prebuilt image as 
part of the core-bundle.  Since we cannot auto generate the license 
manifest since our code uses the opkg framework to extract that info, we 
cannot generate the license manifest for the tiny image.

A new system for correctly generating the license manifest would have to 
be devised to support this request.  At this time, we cannot include the 
tiny image in the core-bundle.  A must larger effort to revamp the 
manifest code would have be done to correctly fulfill this corporate 
requirement.

And there is nothing stopping a customer from setting up a yocto build 
and running:

MACHINE=????? bitbake tisdk-tiny-image

We just cannot, currently, do it for them and include it in the prebuilt 
bundle.



>>> they can always run bitbake and generate it.  It should not take
>>> that much time given that it contains almost no packages within it.
>>>
>>>
>>>> -Bin.
>>>>
>>>>>
>>>>> Regards,
>>>>> Chirag
>>>>>
>>>>>>    Since tiny should be a subset of the other images, is there anything
>>>>>> not covered in the software manifest that should prevent this patch?
>>>>>>
>>>>>>>
>>>>>>>> Signed-off-by: Ryan Eatmon <reatmon@ti.com>
>>>>>>>> ---
>>>>>>>>    meta-arago-distro/classes/tisdk-sw-manifest.bbclass | 4 ++--
>>>>>>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/meta-arago-distro/classes/tisdk-sw-manifest.bbclass
>>>>>>>> b/meta-arago-distro/classes/tisdk-sw-manifest.bbclass
>>>>>>>> index 14d14f08..b9c63610 100644
>>>>>>>> --- a/meta-arago-distro/classes/tisdk-sw-manifest.bbclass
>>>>>>>> +++ b/meta-arago-distro/classes/tisdk-sw-manifest.bbclass
>>>>>>>> @@ -405,10 +405,10 @@ sw_manifest_target() {
>>>>>>>>            # Only extract tar.gz or tar.bz2 types
>>>>>>>>            if [ -e
>>>>>>>> ${IMAGE_ROOTFS}/filesystem/${image}-${MACHINE}.tar.xz ]
>>>>>>>>            then
>>>>>>>> -            tar xJf
>>>>>>>> ${IMAGE_ROOTFS}/filesystem/${image}-${MACHINE}.tar.xz -C
>>>>>>>> ${IMAGE_ROOTFS}/filesystem --wildcards *.control
>>>>>>>> +            tar xJf
>>>>>>>> ${IMAGE_ROOTFS}/filesystem/${image}-${MACHINE}.tar.xz -C
>>>>>>>> ${IMAGE_ROOTFS}/filesystem --wildcards *.control || echo "No
>>>>>>>> control files found in ${image}"
>>>>>>>>            elif [ -e
>>>>>>>> ${IMAGE_ROOTFS}/filesystem/${image}-${MACHINE}.tar.gz ]
>>>>>>>>            then
>>>>>>>> -            tar xzf
>>>>>>>> ${IMAGE_ROOTFS}/filesystem/${image}-${MACHINE}.tar.gz -C
>>>>>>>> ${IMAGE_ROOTFS}/filesystem --wildcards *.control
>>>>>>>> +            tar xzf
>>>>>>>> ${IMAGE_ROOTFS}/filesystem/${image}-${MACHINE}.tar.gz -C
>>>>>>>> ${IMAGE_ROOTFS}/filesystem --wildcards *.control || echo "No
>>>>>>>> control files found in ${image}"
>>>>>>>>            fi
>>>>>>>>        done
>>>>>>>>    --
>>>>>>>> 2.17.1
>>>>>>

-- 
Ryan Eatmon                reatmon@ti.com
-----------------------------------------
Texas Instruments, Inc.  -  LCPD  -  MGTS