[RFC] Solving ipkg-utils issues

[RFC] Solving ipkg-utils issues

[Paul Sokolovsky]

Hello openembedded-devel,

  For quite some time bugs in ipkg-utils package (mostly in
ipkg-make-index tool) plagued image creation process in OE. Many of
them are known to old-time developers, and being (manually) worked
around. I was working on resolving them for some time, and guess now
finally have both clear picture of the issues and their interaction
with each other, and fixes for them. So, I would like to push these
fixes to mainline, and am opening this thread for review of
them, as well as to collect any other concerns about ipkg-utils.

  In my list, there're 3 major issues:

1. Broken version comparison logic in ipkg-utils (note that this is
exactly ipkg-utils issues, nether bitbake nor ipkg is affected).

2. Inability of ipkg-make-index to detect packages which were updated,
while version stays the same.

3. Performance issues.


  Patch for p.1 is here: http://bugs.openembedded.org/show_bug.cgi?id=2061

  

-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com

Re: [RFC] Solving ipkg-utils issues

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Sokolovsky schreef:
> Hello openembedded-devel,
> 
>   For quite some time bugs in ipkg-utils package (mostly in
> ipkg-make-index tool) plagued image creation process in OE. Many of
> them are known to old-time developers, and being (manually) worked
> around. I was working on resolving them for some time, and guess now
> finally have both clear picture of the issues and their interaction
> with each other, and fixes for them. So, I would like to push these
> fixes to mainline, and am opening this thread for review of
> them, as well as to collect any other concerns about ipkg-utils.
> 
>   In my list, there're 3 major issues:
> 
> 1. Broken version comparison logic in ipkg-utils (note that this is
> exactly ipkg-utils issues, nether bitbake nor ipkg is affected).
> 
> 2. Inability of ipkg-make-index to detect packages which were updated,
> while version stays the same.

It's hard to distinguish between 'update' and 'data corruption' without knowing if someone
rebuilt the package.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFGDNIkMkyGM64RGpERAhhLAKCN5l2e+MELHVIBsMfLNujKTVM3hwCeJLzu
M5vId3I51zd6AaX98qwWAmI=
=/GV6
-----END PGP SIGNATURE-----

Re: [RFC] Solving ipkg-utils issues

[Paul Sokolovsky]

Hello Koen,

Friday, March 30, 2007, 12:02:28 PM, you wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> Paul Sokolovsky schreef:
>> Hello openembedded-devel,
>> 
>>   For quite some time bugs in ipkg-utils package (mostly in
>> ipkg-make-index tool) plagued image creation process in OE. Many of
>> them are known to old-time developers, and being (manually) worked
>> around. I was working on resolving them for some time, and guess now
>> finally have both clear picture of the issues and their interaction
>> with each other, and fixes for them. So, I would like to push these
>> fixes to mainline, and am opening this thread for review of
>> them, as well as to collect any other concerns about ipkg-utils.
>> 
>>   In my list, there're 3 major issues:
>> 
>> 1. Broken version comparison logic in ipkg-utils (note that this is
>> exactly ipkg-utils issues, nether bitbake nor ipkg is affected).
>> 
>> 2. Inability of ipkg-make-index to detect packages which were updated,
>> while version stays the same.

> It's hard to distinguish between 'update' and 'data corruption' without knowing if someone
> rebuilt the package.

  Yeah, so I followed suggestion to track ipk timestamps. But we
indeed have overloaded Packages files, and this timestamp is clearly
ipkg-make-index internal thing, so I did it a-la Packages.files,
introduced Packages.timestamps for that.

  But this will be second patch, and I'd like to have it step by step,
with interim testing, because interaction of issues can be indeed pretty bad.

> regards,

> Koen

-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] Solving ipkg-utils issues

[Richard Purdie]

Hi Paul,

Thanks for looking at this, its something that does need fixing.

On Fri, 2007-03-30 at 11:52 +0300, Paul Sokolovsky wrote:
>   For quite some time bugs in ipkg-utils package (mostly in
> ipkg-make-index tool) plagued image creation process in OE. Many of
> them are known to old-time developers, and being (manually) worked
> around. I was working on resolving them for some time, and guess now
> finally have both clear picture of the issues and their interaction
> with each other, and fixes for them. So, I would like to push these
> fixes to mainline, and am opening this thread for review of
> them, as well as to collect any other concerns about ipkg-utils.
> 
>   In my list, there're 3 major issues:
> 
> 1. Broken version comparison logic in ipkg-utils (note that this is
> exactly ipkg-utils issues, nether bitbake nor ipkg is affected).
> 
> 2. Inability of ipkg-make-index to detect packages which were updated,
> while version stays the same.
> 
> 3. Performance issues.
> 
> 
>   Patch for p.1 is here: http://bugs.openembedded.org/show_bug.cgi?id=2061

I agree with fixing all the above. I've only had a quick glance at the
actual patch but I'm fine with committing it and getting testing that
way. There is python version comparison code in bitbake which may or may
not be of use.

Does this correctly handle epochs btw? We're going to need that soon...

Cheers,

Richard

Re: [RFC] Solving ipkg-utils issues

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
> Hi Paul,
> 
> Thanks for looking at this, its something that does need fixing.
> 
> On Fri, 2007-03-30 at 11:52 +0300, Paul Sokolovsky wrote:
>>   For quite some time bugs in ipkg-utils package (mostly in
>> ipkg-make-index tool) plagued image creation process in OE. Many of
>> them are known to old-time developers, and being (manually) worked
>> around. I was working on resolving them for some time, and guess now
>> finally have both clear picture of the issues and their interaction
>> with each other, and fixes for them. So, I would like to push these
>> fixes to mainline, and am opening this thread for review of
>> them, as well as to collect any other concerns about ipkg-utils.
>>
>>   In my list, there're 3 major issues:
>>
>> 1. Broken version comparison logic in ipkg-utils (note that this is
>> exactly ipkg-utils issues, nether bitbake nor ipkg is affected).
>>
>> 2. Inability of ipkg-make-index to detect packages which were updated,
>> while version stays the same.
>>
>> 3. Performance issues.
>>
>>
>>   Patch for p.1 is here: http://bugs.openembedded.org/show_bug.cgi?id=2061
> 
> I agree with fixing all the above. I've only had a quick glance at the
> actual patch but I'm fine with committing it and getting testing that
> way. There is python version comparison code in bitbake which may or may
> not be of use.
> 
> Does this correctly handle epochs btw? We're going to need that soon...

It does, we debugged that yesterday :)

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFGDOEcMkyGM64RGpERAgXZAJ4iSSJLUw4yx6uFZN0nwxaVvXs0TgCeINN6
OiiojsD1JBQg1f0xW/LXdzA=
=7rTS
-----END PGP SIGNATURE-----

Re: [RFC] Solving ipkg-utils issues

[Paul Sokolovsky]

Hello Richard,

Friday, March 30, 2007, 12:50:00 PM, you wrote:

> Hi Paul,

> Thanks for looking at this, its something that does need fixing.

[]
>>
>> 1. Broken version comparison logic in ipkg-utils (note that this is
>> exactly ipkg-utils issues, nether bitbake nor ipkg is affected).
>> 
[]
>>
>>   Patch for p.1 is here: http://bugs.openembedded.org/show_bug.cgi?id=2061

> I agree with fixing all the above. I've only had a quick glance at the
> actual patch but I'm fine with committing it and getting testing that
> way. There is python version comparison code in bitbake which may or may
> not be of use.

  Ok, thanks. I'll commit (cleaned up version of) it on weekend then,
after more people get chance to read this thread and be in loop.
  
[]

> Cheers,

> Richard



-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] Solving ipkg-utils issues

[Paul Sokolovsky]

Hello,

Friday, March 30, 2007, 4:28:51 PM, you wrote:

> Hello Richard,

> Friday, March 30, 2007, 12:50:00 PM, you wrote:

>> Hi Paul,

>> Thanks for looking at this, its something that does need fixing.

> []
>>>
>>> 1. Broken version comparison logic in ipkg-utils (note that this is
>>> exactly ipkg-utils issues, nether bitbake nor ipkg is affected).
>>> 
> []
>>>
>>>   Patch for p.1 is here: http://bugs.openembedded.org/show_bug.cgi?id=2061

>> I agree with fixing all the above. I've only had a quick glance at the
>> actual patch but I'm fine with committing it and getting testing that
>> way. There is python version comparison code in bitbake which may or may
>> not be of use.

>   Ok, thanks. I'll commit (cleaned up version of) it on weekend then,
> after more people get chance to read this thread and be in loop.

  Ok, committed finally.

  Please reopen http://bugs.openembedded.org/show_bug.cgi?id=2061 if
you face any issues.

>   
> []

>> Cheers,

>> Richard






-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com