Package management DB in old-style SDK

Package management DB in old-style SDK

[Denys Dmytriyenko]

Hi,

I'm still using the old-style SDK (meta-toolchain) for our products due to 
some code customizations in it and some of our use cases rely on the opkg 
package management database being present in sysroots, so packages can be 
manipulated in SDK with opkg command. It appears that /var/lib/opkg is now 
empty and the package database is no longer supplied with the SDK.

I know about the new-style populate_sdk and the even newer Extensible SDK, but 
not yet ready to convert our old use cases to that. Since there were lots of 
changes around the way SDKs work in OE-Core in the past couple releases, I'm 
failing to find the change that causes opkg DB to be missing from the old 
style SDK. I'll appreciate any pointers in helping me debug and/or fix this 
issue with my SDKs. Thanks.

-- 
Denys

Re: Package management DB in old-style SDK

[Paul Eggleton]

Hi Denys,

On Mon, 18 Apr 2016 18:14:55 Denys Dmytriyenko wrote:
> I'm still using the old-style SDK (meta-toolchain) for our products due to
> some code customizations in it and some of our use cases rely on the opkg
> package management database being present in sysroots, so packages can be
> manipulated in SDK with opkg command. It appears that /var/lib/opkg is now
> empty and the package database is no longer supplied with the SDK.
> 
> I know about the new-style populate_sdk and the even newer Extensible SDK,
> but not yet ready to convert our old use cases to that. Since there were
> lots of changes around the way SDKs work in OE-Core in the past couple
> releases, I'm failing to find the change that causes opkg DB to be missing
> from the old style SDK. I'll appreciate any pointers in helping me debug
> and/or fix this issue with my SDKs. Thanks.

As far as ipk is concerned I think it was this commit: 

  http://cgit.openembedded.org/openembedded-core/commit/?id=c8e0ec2da9ad4ce1c103966906a85f68c15400dd

Unfortunately it wasn't made optional.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Package management DB in old-style SDK

[Denys Dmytriyenko]

On Tue, Apr 19, 2016 at 10:35:29AM +1200, Paul Eggleton wrote:
> Hi Denys,
> 
> On Mon, 18 Apr 2016 18:14:55 Denys Dmytriyenko wrote:
> > I'm still using the old-style SDK (meta-toolchain) for our products due to
> > some code customizations in it and some of our use cases rely on the opkg
> > package management database being present in sysroots, so packages can be
> > manipulated in SDK with opkg command. It appears that /var/lib/opkg is now
> > empty and the package database is no longer supplied with the SDK.
> > 
> > I know about the new-style populate_sdk and the even newer Extensible SDK,
> > but not yet ready to convert our old use cases to that. Since there were
> > lots of changes around the way SDKs work in OE-Core in the past couple
> > releases, I'm failing to find the change that causes opkg DB to be missing
> > from the old style SDK. I'll appreciate any pointers in helping me debug
> > and/or fix this issue with my SDKs. Thanks.
> 
> As far as ipk is concerned I think it was this commit: 
> 
>   http://cgit.openembedded.org/openembedded-core/commit/?id=c8e0ec2da9ad4ce1c103966906a85f68c15400dd
> 
> Unfortunately it wasn't made optional.

Thanks, Paul!

Unfortunately, the change is in the library, which is quite difficult to 
override in another layer...

Will there be any objections for making that change optional with the default 
being like it is now, but with the ability to restore the old way with a 
variable?

-- 
Denys

Re: Package management DB in old-style SDK

[Paul Eggleton]

On Mon, 18 Apr 2016 21:01:27 Denys Dmytriyenko wrote:
> On Tue, Apr 19, 2016 at 10:35:29AM +1200, Paul Eggleton wrote:
> > Hi Denys,
> > On Mon, 18 Apr 2016 18:14:55 Denys Dmytriyenko wrote:
> > > I'm still using the old-style SDK (meta-toolchain) for our products due
> > > to some code customizations in it and some of our use cases rely on the
> > > opkg package management database being present in sysroots, so packages
> > > can be manipulated in SDK with opkg command. It appears that
> > > /var/lib/opkg is now empty and the package database is no longer
> > > supplied with the SDK.
> > > 
> > > I know about the new-style populate_sdk and the even newer Extensible
> > > SDK, but not yet ready to convert our old use cases to that. Since there
> > > were lots of changes around the way SDKs work in OE-Core in the past
> > > couple releases, I'm failing to find the change that causes opkg DB to
> > > be missing from the old style SDK. I'll appreciate any pointers in
> > > helping me debug and/or fix this issue with my SDKs. Thanks.
> > 
> > As far as ipk is concerned I think it was this commit:
> >   http://cgit.openembedded.org/openembedded-core/commit/?id=c8e0ec2da9ad4c
> >   e1c103966906a85f68c15400dd> 
> > Unfortunately it wasn't made optional.
> 
> Thanks, Paul!
> 
> Unfortunately, the change is in the library, which is quite difficult to
> override in another layer...
> 
> Will there be any objections for making that change optional with the
> default being like it is now, but with the ability to restore the old way
> with a variable?

As long as we make it have the same effect across the three different package 
managers, no objection from me at least. Thinking about it we could even make 
it switch based on package-management being in SDKIMAGE_FEATURES and then it 
would be consistent with how it works for images as well.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Package management DB in old-style SDK

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2088 bytes --]

On Mon, Apr 18, 2016 at 6:38 PM Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> On Mon, 18 Apr 2016 21:01:27 Denys Dmytriyenko wrote:
> > On Tue, Apr 19, 2016 at 10:35:29AM +1200, Paul Eggleton wrote:
> > > Hi Denys,
> > > On Mon, 18 Apr 2016 18:14:55 Denys Dmytriyenko wrote:
> > > > I'm still using the old-style SDK (meta-toolchain) for our products
> due
> > > > to some code customizations in it and some of our use cases rely on
> the
> > > > opkg package management database being present in sysroots, so
> packages
> > > > can be manipulated in SDK with opkg command. It appears that
> > > > /var/lib/opkg is now empty and the package database is no longer
> > > > supplied with the SDK.
> > > >
> > > > I know about the new-style populate_sdk and the even newer Extensible
> > > > SDK, but not yet ready to convert our old use cases to that. Since
> there
> > > > were lots of changes around the way SDKs work in OE-Core in the past
> > > > couple releases, I'm failing to find the change that causes opkg DB
> to
> > > > be missing from the old style SDK. I'll appreciate any pointers in
> > > > helping me debug and/or fix this issue with my SDKs. Thanks.
> > >
> > > As far as ipk is concerned I think it was this commit:
> > >
> http://cgit.openembedded.org/openembedded-core/commit/?id=c8e0ec2da9ad4c
> > >   e1c103966906a85f68c15400dd>
> > > Unfortunately it wasn't made optional.
> >
> > Thanks, Paul!
> >
> > Unfortunately, the change is in the library, which is quite difficult to
> > override in another layer...
>


Actually, overriding a library is trivial given the way we do our python
package. Copy meta/lib/oe/__init__.py into <your layer>/lib/oe/, copy
meta/lib/oe/sdk.py to <your layer>/lib/oe/ and modify, and make sure your
layer is prioritized correctly, and your forked version will be used in
preference.

It's not ideal, for the same reasons it's not ideal to override a class --
you're stuck keeping it in sync, so for that reason I'd not recommend it,
but it's not difficult to do the override :)

[-- Attachment #2: Type: text/html, Size: 2778 bytes --]

Re: Package management DB in old-style SDK

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2454 bytes --]

On Mon, Apr 18, 2016 at 7:19 PM Christopher Larson <clarson@kergoth.com>
wrote:

> On Mon, Apr 18, 2016 at 6:38 PM Paul Eggleton <
> paul.eggleton@linux.intel.com> wrote:
>
>> On Mon, 18 Apr 2016 21:01:27 Denys Dmytriyenko wrote:
>> > On Tue, Apr 19, 2016 at 10:35:29AM +1200, Paul Eggleton wrote:
>> > > Hi Denys,
>> > > On Mon, 18 Apr 2016 18:14:55 Denys Dmytriyenko wrote:
>> > > > I'm still using the old-style SDK (meta-toolchain) for our products
>> due
>> > > > to some code customizations in it and some of our use cases rely on
>> the
>> > > > opkg package management database being present in sysroots, so
>> packages
>> > > > can be manipulated in SDK with opkg command. It appears that
>> > > > /var/lib/opkg is now empty and the package database is no longer
>> > > > supplied with the SDK.
>> > > >
>> > > > I know about the new-style populate_sdk and the even newer
>> Extensible
>> > > > SDK, but not yet ready to convert our old use cases to that. Since
>> there
>> > > > were lots of changes around the way SDKs work in OE-Core in the past
>> > > > couple releases, I'm failing to find the change that causes opkg DB
>> to
>> > > > be missing from the old style SDK. I'll appreciate any pointers in
>> > > > helping me debug and/or fix this issue with my SDKs. Thanks.
>> > >
>> > > As far as ipk is concerned I think it was this commit:
>> > >
>> http://cgit.openembedded.org/openembedded-core/commit/?id=c8e0ec2da9ad4c
>> > >   e1c103966906a85f68c15400dd>
>> > > Unfortunately it wasn't made optional.
>> >
>> > Thanks, Paul!
>> >
>> > Unfortunately, the change is in the library, which is quite difficult to
>> > override in another layer...
>>
>
>
> Actually, overriding a library is trivial given the way we do our python
> package. Copy meta/lib/oe/__init__.py into <your layer>/lib/oe/, copy
> meta/lib/oe/sdk.py to <your layer>/lib/oe/ and modify, and make sure your
> layer is prioritized correctly, and your forked version will be used in
> preference.
>
> It's not ideal, for the same reasons it's not ideal to override a class --
> you're stuck keeping it in sync, so for that reason I'd not recommend it,
> but it's not difficult to do the override :)
>

Just as an FYI, we've done this on many occasions in meta-mentor-staging as
a temporary measure while waiting for the fix to propagate to upstream.
It's handy for cases like that, where the lifetime is limited.

[-- Attachment #2: Type: text/html, Size: 3410 bytes --]

Re: Package management DB in old-style SDK

[Denys Dmytriyenko]

On Tue, Apr 19, 2016 at 01:38:23PM +1200, Paul Eggleton wrote:
> On Mon, 18 Apr 2016 21:01:27 Denys Dmytriyenko wrote:
> > On Tue, Apr 19, 2016 at 10:35:29AM +1200, Paul Eggleton wrote:
> > > Hi Denys,
> > > On Mon, 18 Apr 2016 18:14:55 Denys Dmytriyenko wrote:
> > > > I'm still using the old-style SDK (meta-toolchain) for our products due
> > > > to some code customizations in it and some of our use cases rely on the
> > > > opkg package management database being present in sysroots, so packages
> > > > can be manipulated in SDK with opkg command. It appears that
> > > > /var/lib/opkg is now empty and the package database is no longer
> > > > supplied with the SDK.
> > > > 
> > > > I know about the new-style populate_sdk and the even newer Extensible
> > > > SDK, but not yet ready to convert our old use cases to that. Since there
> > > > were lots of changes around the way SDKs work in OE-Core in the past
> > > > couple releases, I'm failing to find the change that causes opkg DB to
> > > > be missing from the old style SDK. I'll appreciate any pointers in
> > > > helping me debug and/or fix this issue with my SDKs. Thanks.
> > > 
> > > As far as ipk is concerned I think it was this commit:
> > >   http://cgit.openembedded.org/openembedded-core/commit/?id=c8e0ec2da9ad4c
> > >   e1c103966906a85f68c15400dd> 
> > > Unfortunately it wasn't made optional.
> > 
> > Thanks, Paul!
> > 
> > Unfortunately, the change is in the library, which is quite difficult to
> > override in another layer...
> > 
> > Will there be any objections for making that change optional with the
> > default being like it is now, but with the ability to restore the old way
> > with a variable?
> 
> As long as we make it have the same effect across the three different package 
> managers, no objection from me at least. Thinking about it we could even make 
> it switch based on package-management being in SDKIMAGE_FEATURES and then it 
> would be consistent with how it works for images as well.

Hi, Paul,

I submitted a patch yesterday to make the discussed change - can you please 
review and provide feedback? Thanks.

-- 
Denys