Question about rebuilding RPM package index for updated RPMs when bitbake completes

Question about rebuilding RPM package index for updated RPMs when bitbake completes

[Alex J Lennon]

Hi,

I'm trying to put in place a development workflow using the PR server,
RPM package feeds and smart update/install on a target.

I see that when I modify and rebuild my recipe, foo, the PR server
increments its count within the RPM filename, but the RPM feed data
doesn't seem to update.

e.g. in my deploy/rpm/cortexa9_vfp_neon folder I see a change (say) from

foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm

to

foo-1.7.11-r0.2.cortexa9_vfp_neon.rpm

However if I then run smart on the target (configured to use that rpm
package feed) I don't see new packages

smart update

Similarly if I remove the package and reinstall it I get an error

smart remove foo
smart install foo

Fetching packages...
-> http://pkgrepo.my.local/.../foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm
foo-1.7.11-r0.1...                                          [  0%]

error: Failed to download packages:
error:    
http://pkgrepo.my.local/cortexa9_vfp_neon/foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm:
File not found

I believe this is because the old package has been removed, and the new
package added to the folder,
but the repodata has not been rebuilt, leaving the packages out of sync
with the feed representation.

If then try running

bitbake package-index

When I now run an update on the target it picks up the new package and I
can fetch and install this

smart update

Updating cache...               ########################################
[100%]

Channels have 3 new packages:
    foo-1.7.11-r0.3@cortexa9_vfp_neon
...


So if I am understanding the above correctly, when I make a change to a
recipe and build it, PR automatically
updates, the old RPM is removed and the new RPM added to the feed
directory. However the package index
for the feed is not updated.

So if at that point I try to make use of the feed on a target I am
likely to find something is broken.

If that is true would it make sense to leave the old RPM in the feed
directory until package-index
is re-ran, or to run package-index automatically at the end of a build
when RPMs have changed?

Thanks,

Alex

Re: Question about rebuilding RPM package index for updated RPMs when bitbake completes

[Mark Hatle]

On 4/20/14, 7:15 AM, Alex J Lennon wrote:
> Hi,
>
> I'm trying to put in place a development workflow using the PR server,
> RPM package feeds and smart update/install on a target.
>
> I see that when I modify and rebuild my recipe, foo, the PR server
> increments its count within the RPM filename, but the RPM feed data
> doesn't seem to update.
>

...

> So if I am understanding the above correctly, when I make a change to a
> recipe and build it, PR automatically
> updates, the old RPM is removed and the new RPM added to the feed
> directory. However the package index
> for the feed is not updated.
>
> So if at that point I try to make use of the feed on a target I am
> likely to find something is broken.

The feed is normally indexed (createrepo) either when you manually run the 
package-index operation, or when you construct a filesystem.  Until you do that, 
the feed directories are transient.

> If that is true would it make sense to leave the old RPM in the feed
> directory until package-index
> is re-ran, or to run package-index automatically at the end of a build
> when RPMs have changed?

I -never- export the feed directories from the project directory.  Instead, I 
copy the packages from the feed directory to where I share them, and then run 
the indexer against the external repository.  This preserves the older versions 
and also makes the new ones available.

To run the indexer I have to configure and run it manually...

PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo 
--update -q <path to feed>

So for qemux86_64, I end up running the above three times.  all, x86_64 and 
qemux86_64.

Then on the target I just do:

smart update
smart upgrade -y

--Mark

> Thanks,
>
> Alex
>
>
>

Re: Question about rebuilding RPM package index for updated RPMs when bitbake completes

[Alex J Lennon]

[-- Attachment #1: Type: text/plain, Size: 4043 bytes --]


On 22/04/2014 02:36, Mark Hatle wrote:
> On 4/20/14, 7:15 AM, Alex J Lennon wrote:
>> Hi,
>>
>> I'm trying to put in place a development workflow using the PR server,
>> RPM package feeds and smart update/install on a target.
>>
>> I see that when I modify and rebuild my recipe, foo, the PR server
>> increments its count within the RPM filename, but the RPM feed data
>> doesn't seem to update.
>>
>
> ...
>
>> So if I am understanding the above correctly, when I make a change to a
>> recipe and build it, PR automatically
>> updates, the old RPM is removed and the new RPM added to the feed
>> directory. However the package index
>> for the feed is not updated.
>>
>> So if at that point I try to make use of the feed on a target I am
>> likely to find something is broken.
>
> The feed is normally indexed (createrepo) either when you manually run
> the package-index operation, or when you construct a filesystem. 
> Until you do that, the feed directories are transient.
>

Is there a reason for this? Surely having a package index that is out of
sync with the packages until a manual operation is performed isn't ideal?

>> If that is true would it make sense to leave the old RPM in the feed
>> directory until package-index
>> is re-ran, or to run package-index automatically at the end of a build
>> when RPMs have changed?
>
> I -never- export the feed directories from the project directory. 
> Instead, I copy the packages from the feed directory to where I share
> them, and then run the indexer against the external repository.  This
> preserves the older versions and also makes the new ones available.
>
> To run the indexer I have to configure and run it manually...
>
> PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
>
> /home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo
> --update -q <path to feed>
>
> So for qemux86_64, I end up running the above three times.  all,
> x86_64 and qemux86_64.
>
> Then on the target I just do:
>
> smart update
> smart upgrade -y
>

Thanks, that's useful

However I still believe the core question stands, which is why the
package index is out of sync, and why it's a good thing to have to run
bitbake package-index manually to bring the package index back into sync
with the packages?

I too export the feed directories to a server for 3rd party consumption
via compressed rsync over SSH. I don't want to expose more on the server
than a simple SSH endpoint, and rsync seems a sensible way to minimise
copying times, so either I need the package index to be correct prior to
the rsync, I need to export, say, an NFS filesystem which I don't want
to do, or I need some createrepo specifics on what I would prefer to be
a dumb web-server.
 
Best,

Alex

> --Mark
>
>> Thanks,
>>
>> Alex
>>
>>
>>
>

-- 

Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>

Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA

mobile: +44 (0)7956 668178

Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
<skype:alexjlennon?add>

This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended
recipient(s). Any unauthorized disclosure, dissemination, distribution,
copying or the taking of any action in reliance on the information
herein is prohibited. E-mails are not secure and cannot be guaranteed to
be error free as they can be intercepted, amended, or contain viruses.
Anyone who communicates with us by e-mail is deemed to have accepted
these risks. Company Name is not responsible for errors or omissions in
this message and denies any responsibility for any damage arising from
the use of e-mail. Any opinion and other statement contained in this
message and any attachment are solely those of the author and do not
necessarily represent those of the company.


[-- Attachment #2.1: Type: text/html, Size: 7530 bytes --]

[-- Attachment #2.2: ddlogo-4.png --]
[-- Type: image/png, Size: 3997 bytes --]

[-- Attachment #2.3: linkedin.png --]
[-- Type: image/png, Size: 631 bytes --]

[-- Attachment #2.4: skype.png --]
[-- Type: image/png, Size: 800 bytes --]

Re: Question about rebuilding RPM package index for updated RPMs when bitbake completes

[Mark Hatle]

On 4/22/14, 1:50 AM, Alex J Lennon wrote:
>
> On 22/04/2014 02:36, Mark Hatle wrote:
>> On 4/20/14, 7:15 AM, Alex J Lennon wrote:
>>> Hi,
>>>
>>> I'm trying to put in place a development workflow using the PR server,
>>> RPM package feeds and smart update/install on a target.
>>>
>>> I see that when I modify and rebuild my recipe, foo, the PR server
>>> increments its count within the RPM filename, but the RPM feed data
>>> doesn't seem to update.
>>>
>>
>> ...
>>
>>> So if I am understanding the above correctly, when I make a change to a
>>> recipe and build it, PR automatically
>>> updates, the old RPM is removed and the new RPM added to the feed
>>> directory. However the package index
>>> for the feed is not updated.
>>>
>>> So if at that point I try to make use of the feed on a target I am
>>> likely to find something is broken.
>>
>> The feed is normally indexed (createrepo) either when you manually run the
>> package-index operation, or when you construct a filesystem.  Until you do
>> that, the feed directories are transient.
>>
>
> Is there a reason for this? Surely having a package index that is out of sync
> with the packages until a manual operation is performed isn't ideal?

Yes, if you re-indexed the set of packages after each package, you will add 
multiple minutes to the construction of each package, and introduce locking into 
the system.  (You won't be able to generate multiple recipes in parallel due to 
this.)  (I don't consider it a package feed until all of the packages are built 
and the index is constructed.)

>>> If that is true would it make sense to leave the old RPM in the feed
>>> directory until package-index
>>> is re-ran, or to run package-index automatically at the end of a build
>>> when RPMs have changed?
>>
>> I -never- export the feed directories from the project directory. Instead, I
>> copy the packages from the feed directory to where I share them, and then run
>> the indexer against the external repository.  This preserves the older
>> versions and also makes the new ones available.
>>
>> To run the indexer I have to configure and run it manually...
>>
>> PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
>>
>> /home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo
>> --update -q <path to feed>
>>
>> So for qemux86_64, I end up running the above three times.  all, x86_64 and
>> qemux86_64.
>>
>> Then on the target I just do:
>>
>> smart update
>> smart upgrade -y
>>
>
> Thanks, that's useful
>
> However I still believe the core question stands, which is why the package index
> is out of sync, and why it's a good thing to have to run bitbake package-index

The other option is to remove the repo indexes when packages are written. 
Again, I don't consider these out of sync though, as the files have different 
purposes and constraints.  They are simply constructed to enable the rootfs 
generation.  Upgrades are external of the build system.

> manually to bring the package index back into sync with the packages?
>
> I too export the feed directories to a server for 3rd party consumption via
> compressed rsync over SSH. I don't want to expose more on the server than a
> simple SSH endpoint, and rsync seems a sensible way to minimise copying times,
> so either I need the package index to be correct prior to the rsync, I need to
> export, say, an NFS filesystem which I don't want to do, or I need some
> createrepo specifics on what I would prefer to be a dumb web-server.

For an rsync, I would either index on the server using a cron-job.. so when new 
packages appear they get added to the index, or have a local mirror of the 
directory including the updated index... you just want to make sure the index 
files are synced last somehow.

--Mark

> Best,
>
> Alex
>
>> --Mark
>>
>>> Thanks,
>>>
>>> Alex
>>>
>>>
>>>
>>
>
> --
>
> Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>
>
> Alex J Lennon / Director
> 1 Queensway, Liverpool L22 4RA
>
> mobile: +44 (0)7956 668178
>
> Linkedin <http://www.linkedin.com/in/alexjlennon> Skype <skype:alexjlennon?add>
>
> This e-mail message may contain confidential or legally privileged information
> and is intended only for the use of the intended recipient(s). Any unauthorized
> disclosure, dissemination, distribution, copying or the taking of any action in
> reliance on the information herein is prohibited. E-mails are not secure and
> cannot be guaranteed to be error free as they can be intercepted, amended, or
> contain viruses. Anyone who communicates with us by e-mail is deemed to have
> accepted these risks. Company Name is not responsible for errors or omissions in
> this message and denies any responsibility for any damage arising from the use
> of e-mail. Any opinion and other statement contained in this message and any
> attachment are solely those of the author and do not necessarily represent those
> of the company.
>

Re: Question about rebuilding RPM package index for updated RPMs when bitbake completes

[Alex J Lennon]

[-- Attachment #1: Type: text/plain, Size: 8400 bytes --]


On 22/04/2014 19:14, Mark Hatle wrote:
> On 4/22/14, 1:50 AM, Alex J Lennon wrote:
>>
>> On 22/04/2014 02:36, Mark Hatle wrote:
>>> On 4/20/14, 7:15 AM, Alex J Lennon wrote:
>>>> Hi,
>>>>
>>>> I'm trying to put in place a development workflow using the PR server,
>>>> RPM package feeds and smart update/install on a target.
>>>>
>>>> I see that when I modify and rebuild my recipe, foo, the PR server
>>>> increments its count within the RPM filename, but the RPM feed data
>>>> doesn't seem to update.
>>>>
>>>
>>> ...
>>>
>>>> So if I am understanding the above correctly, when I make a change
>>>> to a
>>>> recipe and build it, PR automatically
>>>> updates, the old RPM is removed and the new RPM added to the feed
>>>> directory. However the package index
>>>> for the feed is not updated.
>>>>
>>>> So if at that point I try to make use of the feed on a target I am
>>>> likely to find something is broken.
>>>
>>> The feed is normally indexed (createrepo) either when you manually
>>> run the
>>> package-index operation, or when you construct a filesystem.  Until
>>> you do
>>> that, the feed directories are transient.
>>>
>>
>> Is there a reason for this? Surely having a package index that is out
>> of sync
>> with the packages until a manual operation is performed isn't ideal?
>
> Yes, if you re-indexed the set of packages after each package, you
> will add multiple minutes to the construction of each package, and
> introduce locking into the system.  (You won't be able to generate
> multiple recipes in parallel due to this.)  (I don't consider it a
> package feed until all of the packages are built and the index is
> constructed.)
>

I wasn't suggesting that the index is built mid-run . Not knowing the
internals of the bitbake dependency chain, presumably the package index
would/could be rebuilt at the end of the build process and so would not
block parallelisation (which I agree would be a time-sink)

>>>> If that is true would it make sense to leave the old RPM in the feed
>>>> directory until package-index
>>>> is re-ran, or to run package-index automatically at the end of a build
>>>> when RPMs have changed?
>>>
>>> I -never- export the feed directories from the project directory.
>>> Instead, I
>>> copy the packages from the feed directory to where I share them, and
>>> then run
>>> the indexer against the external repository.  This preserves the older
>>> versions and also makes the new ones available.
>>>
>>> To run the indexer I have to configure and run it manually...
>>>
>>> PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
>>>
>>>
>>> /home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo
>>>
>>> --update -q <path to feed>
>>>
>>> So for qemux86_64, I end up running the above three times.  all,
>>> x86_64 and
>>> qemux86_64.
>>>
>>> Then on the target I just do:
>>>
>>> smart update
>>> smart upgrade -y
>>>
>>
>> Thanks, that's useful
>>
>> However I still believe the core question stands, which is why the
>> package index
>> is out of sync, and why it's a good thing to have to run bitbake
>> package-index
>
> The other option is to remove the repo indexes when packages are
> written. Again, I don't consider these out of sync though, as the
> files have different purposes and constraints.  They are simply
> constructed to enable the rootfs generation.  Upgrades are external of
> the build system.
>

Fair comment. I take the view that these feeds are available, could
easily be relied upon not to be transient, and I'd like to use them
locally to speed up my development workflow. I think other users would too.

Much of the value of the build environment is the automation. If bitbake
can do a task that means I don't have to remember to do it, and I don't
have to explain to somebody else that they need to do it, then that
seems like a win to me (even if that is an option I can turn on).

>> manually to bring the package index back into sync with the packages?
>>
>> I too export the feed directories to a server for 3rd party
>> consumption via
>> compressed rsync over SSH. I don't want to expose more on the server
>> than a
>> simple SSH endpoint, and rsync seems a sensible way to minimise
>> copying times,
>> so either I need the package index to be correct prior to the rsync,
>> I need to
>> export, say, an NFS filesystem which I don't want to do, or I need some
>> createrepo specifics on what I would prefer to be a dumb web-server.
>
> For an rsync, I would either index on the server using a cron-job.. so
> when new packages appear they get added to the index, or have a local
> mirror of the directory including the updated index... you just want
> to make sure the index files are synced last somehow.
>

To me the server side customisation brings in in extra work that
shouldn't be needed in all cases, (although will undoubtely be needed in
some cases).

I would much prefer not to have to worry about it.

If the package feed could be relied upon to be in sync with the contents
of the tmp/deploy/rpm/* tree at the end of a bitbake build then the
problem simply goes away, potentially at the cost of a little lost time
during the build.

Understanding that other use-cases will be different from mine I have
been wondering today,

- Can I add something into local.conf that would add a dependency to
cause the package-index task to run at the end of the bitbake build
  (preferably only if the contents of the package directories have changed?

- Does the build environment have the concept of post-build triggers?
i.e. Understanding that I could script something myself, is there a
recommended
  method for defining a post-build trigger script for bitbake to run on
certain conditions e.g to do the the package reindex and feed upload in
this case?

Thanks,

Alex


> --Mark
>
>> Best,
>>
>> Alex
>>
>>> --Mark
>>>
>>>> Thanks,
>>>>
>>>> Alex
>>>>
>>>>
>>>>
>>>
>>
>> -- 
>>
>> Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>
>>
>> Alex J Lennon / Director
>> 1 Queensway, Liverpool L22 4RA
>>
>> mobile: +44 (0)7956 668178
>>
>> Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
>> <skype:alexjlennon?add>
>>
>> This e-mail message may contain confidential or legally privileged
>> information
>> and is intended only for the use of the intended recipient(s). Any
>> unauthorized
>> disclosure, dissemination, distribution, copying or the taking of any
>> action in
>> reliance on the information herein is prohibited. E-mails are not
>> secure and
>> cannot be guaranteed to be error free as they can be intercepted,
>> amended, or
>> contain viruses. Anyone who communicates with us by e-mail is deemed
>> to have
>> accepted these risks. Company Name is not responsible for errors or
>> omissions in
>> this message and denies any responsibility for any damage arising
>> from the use
>> of e-mail. Any opinion and other statement contained in this message
>> and any
>> attachment are solely those of the author and do not necessarily
>> represent those
>> of the company.
>>
>
>
>

-- 

Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>

Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA

mobile: +44 (0)7956 668178

Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
<skype:alexjlennon?add>

This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended
recipient(s). Any unauthorized disclosure, dissemination, distribution,
copying or the taking of any action in reliance on the information
herein is prohibited. E-mails are not secure and cannot be guaranteed to
be error free as they can be intercepted, amended, or contain viruses.
Anyone who communicates with us by e-mail is deemed to have accepted
these risks. Company Name is not responsible for errors or omissions in
this message and denies any responsibility for any damage arising from
the use of e-mail. Any opinion and other statement contained in this
message and any attachment are solely those of the author and do not
necessarily represent those of the company.


[-- Attachment #2.1: Type: text/html, Size: 14258 bytes --]

[-- Attachment #2.2: ddlogo-4.png --]
[-- Type: image/png, Size: 3997 bytes --]

[-- Attachment #2.3: linkedin.png --]
[-- Type: image/png, Size: 631 bytes --]

[-- Attachment #2.4: skype.png --]
[-- Type: image/png, Size: 800 bytes --]