Using smart within an SDK

Using smart within an SDK

[Ash Charles]

Hi,

I'd like to be able install packages using smart within an SDK
environment (much like [1]; @Erik, did you find a solution?).

I stumbled on two questions in looking into this.

1. Why does the nativesdk version of smart get wrapped with these
environment variables, 'RPM_USRLIBRPM', 'RPM_ETCRPM', and
'RPM_LOCALEDIRRPM'?  As per [2], the RPM utilities themselves are
already wrapped.  It might be nice to specify the '--data-dir'
argument to smart with a create_cmdline_wrapper instead.

2. Why does the smart configuration data get removed during SDK
creation?  In particular, the self.target_pm.remove_packaging_data
line in meta/lib/oe/sdk.py.

[1] https://lists.yoctoproject.org/pipermail/yocto/2015-March/024152.html
[2] http://lists.openembedded.org/pipermail/openembedded-core/2012-December/071894.html

Thanks for any advice!
--Ash

Re: Using smart within an SDK

[ChenQi]

Hi Ash,

SDK has basically two parts, nativesdk part and the target part.
For the nativesdk part, we can use smart/rpm to manage packages. A long 
time ago, I tried this out and succeeded.
However, for the target part, we cannot do the same thing as the rpm 
database for the target part is not usable after installation. I once 
tried to solve this problem but failed.
You can have a little try and if you have some good news, let me know.

Best Regards,
Chen Qi

On 05/14/2015 08:51 AM, Ash Charles wrote:
> Hi,
>
> I'd like to be able install packages using smart within an SDK
> environment (much like [1]; @Erik, did you find a solution?).
>
> I stumbled on two questions in looking into this.
>
> 1. Why does the nativesdk version of smart get wrapped with these
> environment variables, 'RPM_USRLIBRPM', 'RPM_ETCRPM', and
> 'RPM_LOCALEDIRRPM'?  As per [2], the RPM utilities themselves are
> already wrapped.  It might be nice to specify the '--data-dir'
> argument to smart with a create_cmdline_wrapper instead.
>
> 2. Why does the smart configuration data get removed during SDK
> creation?  In particular, the self.target_pm.remove_packaging_data
> line in meta/lib/oe/sdk.py.
>
> [1] https://lists.yoctoproject.org/pipermail/yocto/2015-March/024152.html
> [2] http://lists.openembedded.org/pipermail/openembedded-core/2012-December/071894.html
>
> Thanks for any advice!
> --Ash

Re: Using smart within an SDK

[Ash Charles]

On Wed, May 13, 2015 at 7:26 PM, ChenQi <Qi.Chen@windriver.com> wrote:
> For the nativesdk part, we can use smart/rpm to manage packages. A long time
> ago, I tried this out and succeeded.
> However, for the target part, we cannot do the same thing as the rpm
> database for the target part is not usable after installation. I once tried
> to solve this problem but failed.
> You can have a little try and if you have some good news, let me know.
Thanks for your response.

I've not been able to use smart to install packages even on the
nativesdk part---could you explain your steps?

What are the barriers to using the RPM database for the target part?

--Ash

Re: Using smart within an SDK

[ChenQi]

On 05/15/2015 02:21 AM, Ash Charles wrote:
> On Wed, May 13, 2015 at 7:26 PM, ChenQi <Qi.Chen@windriver.com> wrote:
>> For the nativesdk part, we can use smart/rpm to manage packages. A long time
>> ago, I tried this out and succeeded.
>> However, for the target part, we cannot do the same thing as the rpm
>> database for the target part is not usable after installation. I once tried
>> to solve this problem but failed.
>> You can have a little try and if you have some good news, let me know.
> Thanks for your response.
>
> I've not been able to use smart to install packages even on the
> nativesdk part---could you explain your steps?

I can recall that I used something like `PSEUDO_xxx=xxx PSEUDO_xxx=xxx 
smart(or rpm?) ... --data-dir=xxx <command>'. But I don't remember 
whether I tweaked something else. Sorry :(
That's really a long time ago and I didn't write a document at the time.

> What are the barriers to using the RPM database for the target part?

The rpm database is not usable. You do a query to list files that a 
package installed and you will find all the paths are not correct.

We are now working a new kind of SDK in OE. We call it extensible SDK. 
By design, its components could be updated/installed/removed using 
sstate objects. The task is not completed yet, but we are working on it.

Regards,
Chen Qi

>
> --Ash
>
>

Re: Using smart within an SDK

[Ash Charles]

On Thu, May 14, 2015 at 7:54 PM, ChenQi <Qi.Chen@windriver.com> wrote:
> The rpm database is not usable. You do a query to list files that a package
> installed and you will find all the paths are not correct.
Ah okay---that makes sense.  Thanks.
>
> We are now working a new kind of SDK in OE. We call it extensible SDK. By
> design, its components could be updated/installed/removed using sstate
> objects. The task is not completed yet, but we are working on it.

I'd seen the extensible SDK and was thinking it looked pretty cool so
I'll give this a whirl.  From what I've read, it doesn't yet support
updating.  Is there a roadmap for this (happy to help out if I'm
able)?  I'd also need to maintain an online sstate cache to make this
SDK useful to others correct?

Re: Using smart within an SDK

[randy.e.witt]

> On Thu, May 14, 2015 at 7:54 PM, ChenQi <Qi.Chen@windriver.com> wrote:
>> The rpm database is not usable. You do a query to list files that a
>> package
>> installed and you will find all the paths are not correct.
> Ah okay---that makes sense.  Thanks.
>>
>> We are now working a new kind of SDK in OE. We call it extensible SDK.
>> By
>> design, its components could be updated/installed/removed using sstate
>> objects. The task is not completed yet, but we are working on it.
>
> I'd seen the extensible SDK and was thinking it looked pretty cool so
> I'll give this a whirl.  From what I've read, it doesn't yet support
> updating.  Is there a roadmap for this (happy to help out if I'm
> able)?  I'd also need to maintain an online sstate cache to make this
> SDK useful to others correct?
> --

The idea is that an sdk will comprise of a manifest(contains list of
sstate items in the sdk) and some location that contains the items in the
manifest. So to update the sdk you would run a command and give it the
location of the manifest and sstate, and it would make your sdk match what
is in the manifest.

So the sstate could be hosted on a webserver, nfs, locally, etc.

> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: Using smart within an SDK

[Ash Charles]

On Mon, May 18, 2015 at 10:13 PM,  <randy.e.witt@linux.intel.com> wrote:
> The idea is that an sdk will comprise of a manifest(contains list of
> sstate items in the sdk) and some location that contains the items in the
> manifest. So to update the sdk you would run a command and give it the
> location of the manifest and sstate, and it would make your sdk match what
> is in the manifest.
>
> So the sstate could be hosted on a webserver, nfs, locally, etc.
Thanks Randy.

I stepped through the process of making an extensible SDK based on my
image.  Is the conf/locked-sigs.inc file the manifest you mentioned?

Unfortunately, devtool is failing for me ("ImportError: No module
named site"...looks like the SDK is missing a python library?) but it
is great to at least be able to build it and get a sense of the
contents.

I'm uploading my sstate to a server so theoretically updating would be
a process of grabbing items from this server cache into the
'sstate-cache' directory within the SDK?  If I needed a new package
not originally included in the SDK, I would do something like devtool
get opencv-dev?

Thanks for any insights.  I realize this is work-in-progress so I'm
just looking to get a sense of how things might work.

--Ash

Re: Using smart within an SDK

[Randy Witt]

On 05/22/2015 03:24 PM, Ash Charles wrote:
> On Mon, May 18, 2015 at 10:13 PM,  <randy.e.witt@linux.intel.com> wrote:
>> The idea is that an sdk will comprise of a manifest(contains list of
>> sstate items in the sdk) and some location that contains the items in the
>> manifest. So to update the sdk you would run a command and give it the
>> location of the manifest and sstate, and it would make your sdk match what
>> is in the manifest.
>>
>> So the sstate could be hosted on a webserver, nfs, locally, etc.
> Thanks Randy.
>
> I stepped through the process of making an extensible SDK based on my
> image.  Is the conf/locked-sigs.inc file the manifest you mentioned?

Yes, that is the one.

> Unfortunately, devtool is failing for me ("ImportError: No module
> named site"...looks like the SDK is missing a python library?) but it
> is great to at least be able to build it and get a sense of the
> contents.

Did you source the environment-setup script? If so, what distro were you using?

>
> I'm uploading my sstate to a server so theoretically updating would be
> a process of grabbing items from this server cache into the
> 'sstate-cache' directory within the SDK?  If I needed a new package
> not originally included in the SDK, I would do something like devtool
> get opencv-dev?
>
We were thinking it wouldn't be so granular Basically it would end up matching 
everything in a manifest rather than asking for one particular package. So it 
would look more like "devtool publish-sdk location", followed by users being 
able to then update to whatever "sdk's" exist at that location.

> Thanks for any insights.  I realize this is work-in-progress so I'm
> just looking to get a sense of how things might work.
>
> --Ash
>

Re: Using smart within an SDK

[Ash Charles]

On Tue, May 26, 2015 at 3:18 PM, Randy Witt
<randy.e.witt@linux.intel.com> wrote:
> Did you source the environment-setup script? If so, what distro were you
> using?
Ubuntu 15.04 (Vivid-Vervet).  I used an SDK created based on the
gumstix-console-image rather than a mainstream image from meta-yocto
so perhaps there is a particular configuration etc. that messes up the
creation of the SDK?
<snip>
> We were thinking it wouldn't be so granular Basically it would end up
> matching everything in a manifest rather than asking for one particular
> package. So it would look more like "devtool publish-sdk location", followed
> by users being able to then update to whatever "sdk's" exist at that
> location.
Okay---If I understand correctly, that's a little more limiting than I
would like.  No matter how many different SDKs I provide, each
customer will need a different set of a software packages in their
sysroot.  Yocto makes it easy to build up a big sstate or package
repository and post this online---users can just grab a baseline SDK
and then pull in the pieces they need (which probably is pretty
comfortable for folks who are used to a 'sudo apt-get install
libboost-dev' etc.).

Based on this, should I be turning my attention back to using smart
install with a regular SDK environment or is this imagined workflow
just not a reasonable objective?

Thanks,
--Ash

Re: Using smart within an SDK

[Randy Witt]

On 05/26/2015 03:38 PM, Ash Charles wrote:
> On Tue, May 26, 2015 at 3:18 PM, Randy Witt
> <randy.e.witt@linux.intel.com> wrote:
>> Did you source the environment-setup script? If so, what distro were you
>> using?
> Ubuntu 15.04 (Vivid-Vervet).  I used an SDK created based on the
> gumstix-console-image rather than a mainstream image from meta-yocto
> so perhaps there is a particular configuration etc. that messes up the
> creation of the SDK?
> <snip>
>> We were thinking it wouldn't be so granular Basically it would end up
>> matching everything in a manifest rather than asking for one particular
>> package. So it would look more like "devtool publish-sdk location", followed
>> by users being able to then update to whatever "sdk's" exist at that
>> location.
> Okay---If I understand correctly, that's a little more limiting than I
> would like.  No matter how many different SDKs I provide, each
> customer will need a different set of a software packages in their
> sysroot.  Yocto makes it easy to build up a big sstate or package
> repository and post this online---users can just grab a baseline SDK
> and then pull in the pieces they need (which probably is pretty
> comfortable for folks who are used to a 'sudo apt-get install
> libboost-dev' etc.).
>
> Based on this, should I be turning my attention back to using smart
> install with a regular SDK environment or is this imagined workflow
> just not a reasonable objective?

It is a bit of a different workflow than we were initially looking at, but I 
don't see a reason we couldn't do it. The locked signatures file should be able 
to be a superset of items you would want, so theoretically if you only wanted 
items from your sstate mirror to be used, the manifest would contain all items 
on the mirror. This would prevent the user from building something locally 
rather than pulling from the mirror and potentially not matching.

The nativesdk items are not in the extensible sdk, and the native items are just 
part of the bitbake workspace. This is why it should be fairly easy to add what 
you want. Because in the end it would just be adding an sstate mirror and doing 
a bitbake "native-foo". And we could wrap that with devtool or some other command.

I do see the convenience and why you want it, so I'll talk it over with Paul 
tomorrow.

> Thanks,
> --Ash
>

Re: Using smart within an SDK

[Ash Charles]

On Tue, May 26, 2015 at 4:04 PM, Randy Witt
<randy.e.witt@linux.intel.com> wrote:
> It is a bit of a different workflow than we were initially looking at, but I
> don't see a reason we couldn't do it. The locked signatures file should be
> able to be a superset of items you would want, so theoretically if you only
> wanted items from your sstate mirror to be used, the manifest would contain
> all items on the mirror. This would prevent the user from building something
> locally rather than pulling from the mirror and potentially not matching.
So the user would download a minimal SDK, run a devtool update to get
the super manifest listing everything in the sstate mirror as which
point they could pull what they needed.
>
> The nativesdk items are not in the extensible sdk, and the native items are
> just part of the bitbake workspace. This is why it should be fairly easy to
> add what you want. Because in the end it would just be adding an sstate
> mirror and doing a bitbake "native-foo". And we could wrap that with devtool
> or some other command.
That would be great.
>
> I do see the convenience and why you want it, so I'll talk it over with Paul
> tomorrow.
Thanks.  And please let me know if you need me to do any testing
:)---I'm watching for extensible SDK related patches.

--Ash