[PATCH 0/1] [v2] Automatically generate package repos for rpm and deb [bug #1024]

[PATCH 0/1] [v2] Automatically generate package repos for rpm and deb [bug #1024]

[Dexuan Cui]

From: Dexuan Cui <dexuan.cui@intel.com>

This was made to address http://bugzilla.yoctoproject.org/show_bug.cgi?id=1024.
Please comment.

This version moved the package_update_index_* into package_*.bbclass.

As to RPM, both package_update_index_rpm(creating solvedb) and
createrepo(creating rpm-metadata) seems necessary.
Mark, please comment on this.

As to DEB, I found currently in target, some important config files, like
/etc/apt/sources.list, for deb/apt are missing. As a result, we can't
install/remove package in target at all.
I'm trying to resolve this by add "deb http://192.168.7.1/deb/all/ ./" into
the file, but later "apt-get update" failes -- it tries to find a file named
Packages.gz, but the direcory only has Packages.bz2. Not sure what's wrong. Investigating.


Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: dcui/package-index
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/package-index

Thanks,
    Dexuan Cui <dexuan.cui@intel.com>
---


Dexuan Cui (1):
  package-index.bb: add support for deb and rpm.

 meta/classes/package_deb.bbclass        |    9 +++++++++
 meta/classes/package_ipk.bbclass        |   12 ++++++++++++
 meta/classes/package_rpm.bbclass        |    7 +++++++
 meta/recipes-core/meta/package-index.bb |    8 --------
 4 files changed, 28 insertions(+), 8 deletions(-)

-- 
1.7.2

[RFC][v2] [PATCH 1/1] package-index.bb: add support for deb and rpm.

[Dexuan Cui]

From: Dexuan Cui <dexuan.cui@intel.com>

[YOCTO #1024]
Currently package-index.bb only supports ipk. This commit addes the support
for rpm and deb, too.

------------------------------
How to generate and use repos:

1) run "bitbake package-index" after building some target,
e.g., core-image-sato-sdk;

2) export ${DEPLOY_DIR_RPM}, ${DEPLOY_DIR_IPK} and ${DEPLOY_DIR_DEB} by a
webserver on the host, assuming the host IP is 192.168.7.1, at
http://192.168.7.1/rpm
http://192.168.7.1/ipk
http://192.168.7.1/deb

3) inside the target, according to the packaging system (rpm, ipk or deb) used
when we generate the target image, we can use different ways to manage
packages:

3.1) RPM
    run "zypper addrepo http://192.168.7.1/rpm main; zypper refresh"
    to retrieve info about the repo; next, we can use "zypper install/remove"
    to manage packages.

3.2) IPK
    add the repo info into opkg config file, i.e., in
    /etc/opkg/arch.conf, we can add something like
    "src i586 http://192.168.7.1/ipk/i586", and next, we run "opkg update" to
    make opkg update the list of available packages. And later, we can use
    "opkg install/remove" to manage packages.

3.3) DEB
    Currently in target, some important config files, like
/etc/apt/sources.list, for deb/apt are missing. So we can't install/remove package
in target at present.

Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>
---
 meta/classes/package_deb.bbclass        |    9 +++++++++
 meta/classes/package_ipk.bbclass        |   12 ++++++++++++
 meta/classes/package_rpm.bbclass        |    7 +++++++
 meta/recipes-core/meta/package-index.bb |    8 --------
 4 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 4faeb4a..4cc0b69 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -431,3 +431,12 @@ python do_package_write_deb () {
 do_package_write_deb[dirs] = "${PKGWRITEDIRDEB}"
 addtask package_write_deb before do_package_write after do_package
 
+
+do_package_index[depends] += "dpkg-native:do_populate_sysroot apt-native:do_populate_sysroot"
+do_package_index[recrdeptask] += "package_update_index_deb"
+
+do_package_index() {
+   set -ex
+   package_update_index_deb
+   set +ex
+}
diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 3c2472b..2a2d9c3 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -386,3 +386,15 @@ python do_package_write_ipk () {
 }
 do_package_write_ipk[dirs] = "${PKGWRITEDIRIPK}"
 addtask package_write_ipk before do_package_write after do_package
+
+
+do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
+do_package_index[depends] += "opkg-native:do_populate_sysroot"
+do_package_index[recrdeptask] += "package_update_index_ipk"
+
+do_package_index() {
+   set -ex
+   package_update_index_ipk
+   set +ex
+}
+
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 70170d1..d9470d6 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -814,3 +814,10 @@ python do_package_write_rpm () {
 do_package_write_rpm[dirs] = "${PKGWRITEDIRRPM}"
 addtask package_write_rpm before do_package_write after do_package
 
+do_package_index[depends] += "createrepo-native:do_populate_sysroot"
+do_package_index() {
+    set -ex
+    package_update_index_rpm
+    createrepo "${DEPLOY_DIR_RPM}"
+    set +ex
+}
diff --git a/meta/recipes-core/meta/package-index.bb b/meta/recipes-core/meta/package-index.bb
index 3c642cb..3f1caea 100644
--- a/meta/recipes-core/meta/package-index.bb
+++ b/meta/recipes-core/meta/package-index.bb
@@ -19,14 +19,6 @@ do_package_write_deb[noexec] = "1"
 do_populate_sysroot[noexec] = "1"
 
 do_package_index[nostamp] = "1"
-do_package_index[dirs] = "${DEPLOY_DIR_IPK}"
-do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
-do_package_index[depends] += "opkg-native:do_populate_sysroot"
 
-do_package_index() {
-	set -ex
-	package_update_index_ipk
-	set +ex
-}
 addtask do_package_index before do_build
 EXCLUDE_FROM_WORLD = "1"
-- 
1.7.2

Re: [RFC][v2] [PATCH 1/1] package-index.bb: add support for deb and rpm.

[Saul Wold]

On 05/13/2011 02:53 AM, Dexuan Cui wrote:
> From: Dexuan Cui<dexuan.cui@intel.com>
>
> [YOCTO #1024]
> Currently package-index.bb only supports ipk. This commit addes the support
> for rpm and deb, too.
>
> ------------------------------
> How to generate and use repos:
>
> 1) run "bitbake package-index" after building some target,
> e.g., core-image-sato-sdk;
>
> 2) export ${DEPLOY_DIR_RPM}, ${DEPLOY_DIR_IPK} and ${DEPLOY_DIR_DEB} by a
> webserver on the host, assuming the host IP is 192.168.7.1, at
> http://192.168.7.1/rpm
> http://192.168.7.1/ipk
> http://192.168.7.1/deb
>
> 3) inside the target, according to the packaging system (rpm, ipk or deb) used
> when we generate the target image, we can use different ways to manage
> packages:
>
> 3.1) RPM
>      run "zypper addrepo http://192.168.7.1/rpm main; zypper refresh"
>      to retrieve info about the repo; next, we can use "zypper install/remove"
>      to manage packages.
>
> 3.2) IPK
>      add the repo info into opkg config file, i.e., in
>      /etc/opkg/arch.conf, we can add something like
>      "src i586 http://192.168.7.1/ipk/i586", and next, we run "opkg update" to
>      make opkg update the list of available packages. And later, we can use
>      "opkg install/remove" to manage packages.
>
> 3.3) DEB
>      Currently in target, some important config files, like
> /etc/apt/sources.list, for deb/apt are missing. So we can't install/remove package
> in target at present.
>
> Signed-off-by: Dexuan Cui<dexuan.cui@intel.com>
> ---
>   meta/classes/package_deb.bbclass        |    9 +++++++++
>   meta/classes/package_ipk.bbclass        |   12 ++++++++++++
>   meta/classes/package_rpm.bbclass        |    7 +++++++
>   meta/recipes-core/meta/package-index.bb |    8 --------
>   4 files changed, 28 insertions(+), 8 deletions(-)
>
> diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
> index 4faeb4a..4cc0b69 100644
> --- a/meta/classes/package_deb.bbclass
> +++ b/meta/classes/package_deb.bbclass
> @@ -431,3 +431,12 @@ python do_package_write_deb () {
>   do_package_write_deb[dirs] = "${PKGWRITEDIRDEB}"
>   addtask package_write_deb before do_package_write after do_package
>
> +
> +do_package_index[depends] += "dpkg-native:do_populate_sysroot apt-native:do_populate_sysroot"
> +do_package_index[recrdeptask] += "package_update_index_deb"
> +
> +do_package_index() {
> +   set -ex
> +   package_update_index_deb
> +   set +ex
> +}

Why do you continue to include the do_package_index here, it's no longer 
needed with the setting of do_package_index[recrdeptask], this is true 
below also.

> diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
> index 3c2472b..2a2d9c3 100644
> --- a/meta/classes/package_ipk.bbclass
> +++ b/meta/classes/package_ipk.bbclass
> @@ -386,3 +386,15 @@ python do_package_write_ipk () {
>   }
>   do_package_write_ipk[dirs] = "${PKGWRITEDIRIPK}"
>   addtask package_write_ipk before do_package_write after do_package
> +
> +
> +do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
> +do_package_index[depends] += "opkg-native:do_populate_sysroot"
> +do_package_index[recrdeptask] += "package_update_index_ipk"
> +
> +do_package_index() {
> +   set -ex
> +   package_update_index_ipk
> +   set +ex
> +}

The do_package_index() function is not needed any longer here

> +
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index 70170d1..d9470d6 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -814,3 +814,10 @@ python do_package_write_rpm () {
>   do_package_write_rpm[dirs] = "${PKGWRITEDIRRPM}"
>   addtask package_write_rpm before do_package_write after do_package
>
> +do_package_index[depends] += "createrepo-native:do_populate_sysroot"
> +do_package_index() {
> +    set -ex
> +    package_update_index_rpm
> +    createrepo "${DEPLOY_DIR_RPM}"
> +    set +ex
> +}

Nor this do_package_index() function, why no 
do_package_index[recrdeptask] setting in RPM?  If you need a special 
function here them call it do_package_index_rpm() and set the 
do_package_index[recrdeptask] to that function.



> diff --git a/meta/recipes-core/meta/package-index.bb b/meta/recipes-core/meta/package-index.bb
> index 3c642cb..3f1caea 100644
> --- a/meta/recipes-core/meta/package-index.bb
> +++ b/meta/recipes-core/meta/package-index.bb
> @@ -19,14 +19,6 @@ do_package_write_deb[noexec] = "1"
>   do_populate_sysroot[noexec] = "1"
>
>   do_package_index[nostamp] = "1"
> -do_package_index[dirs] = "${DEPLOY_DIR_IPK}"
Does the dirs need to be saved or moved to the packagers?

> -do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
> -do_package_index[depends] += "opkg-native:do_populate_sysroot"
>
> -do_package_index() {
> -	set -ex
> -	package_update_index_ipk
> -	set +ex
> -}
>   addtask do_package_index before do_build
>   EXCLUDE_FROM_WORLD = "1"

Re: [RFC][v2] [PATCH 1/1] package-index.bb: add support for deb and rpm.

[Cui, Dexuan]

Saul Wold wrote:
> On 05/13/2011 02:53 AM, Dexuan Cui wrote:
>> From: Dexuan Cui<dexuan.cui@intel.com>
>> 
>> diff --git a/meta/classes/package_deb.bbclass
>> b/meta/classes/package_deb.bbclass 
>> index 4faeb4a..4cc0b69 100644
>> --- a/meta/classes/package_deb.bbclass
>> +++ b/meta/classes/package_deb.bbclass
>> @@ -431,3 +431,12 @@ python do_package_write_deb () {
>>   do_package_write_deb[dirs] = "${PKGWRITEDIRDEB}"
>>   addtask package_write_deb before do_package_write after do_package
>> 
>> +
>> +do_package_index[depends] += "dpkg-native:do_populate_sysroot
>> apt-native:do_populate_sysroot" +do_package_index[recrdeptask] +=
>> "package_update_index_deb" + +do_package_index() {
>> +   set -ex
>> +   package_update_index_deb
>> +   set +ex
>> +}
> 
> Why do you continue to include the do_package_index here, it's no
> longer needed with the setting of do_package_index[recrdeptask], this
> is true below also.
Sorry, I didn't use the recrdeptask flag and I guess I still don't quite understand it.

Here removing the do_package_index() I will get such an ERROR:
ERROR: Task do_package_index from /distro/dcui/pc1/meta/recipes-core/meta/package-index.bb seems to be empty?!##########################                   | ETA:  00:00:00
ERROR: Error parsing /distro/dcui/pc1/meta/recipes-core/meta/package-index.bb: md5() argument 1 must be string or read-only buffer, not None               | ETA:  00:00:00

About the recrdeptask  flag: "These are specified with the 'recrdeptask' flag and is used signify the task(s) of each RDEPENDS which must have completed before that task can be executed. It applies recursively so also, the RDEPENDS of each item in the original RDEPENDS must be met and so on. It also runs all DEPENDS first too".
I even added RDEPENDS += " package-index" in package-index.bb, but no change.

I'm trying to figure this out. Please point out my issue once this is obvious to you. :-)

BTW: using the recrdeptask method, we have no chance to do "set -ex" and as a result we lose some debug info in the log file. Is this ok?

>> +
>> diff --git a/meta/classes/package_rpm.bbclass
>> b/meta/classes/package_rpm.bbclass 
>> index 70170d1..d9470d6 100644
>> --- a/meta/classes/package_rpm.bbclass
>> +++ b/meta/classes/package_rpm.bbclass
>> @@ -814,3 +814,10 @@ python do_package_write_rpm () {
>>   do_package_write_rpm[dirs] = "${PKGWRITEDIRRPM}"
>>   addtask package_write_rpm before do_package_write after do_package
>> 
>> +do_package_index[depends] +=
>> "createrepo-native:do_populate_sysroot" +do_package_index() { +   
>> set -ex +    package_update_index_rpm
>> +    createrepo "${DEPLOY_DIR_RPM}"
>> +    set +ex
>> +}
> 
> Nor this do_package_index() function, why no
> do_package_index[recrdeptask] setting in RPM?  If you need a special
> function here them call it do_package_index_rpm() and set the
> do_package_index[recrdeptask] to that function.
Yes, I should have added a new function including package_update_index_rpm and  createrepo "${DEPLOY_DIR_RPM}". and set do_package_index[recrdeptask] to it. 

Thanks,
-- Dexuan

Re: [RFC][v2] [PATCH 1/1] package-index.bb: add support for deb and rpm.

[Richard Purdie]

Hi Dexuan,

When I read Saul's original emails I thought this approach would work
although it wasn't the first one that came to my mind.

Looking at the code you have, the problem is you have nowhere you can
call the "addtask" for the individual tasks and we don't want to the
overhead system wide for every recipe as it makes no sense.

I'd therefore like to suggest we tweak this approach a little. Each
package_xxx.bbclass defines a function which updates the index for that
packaging backend:

package_ipk.bbclass:

PACKAGEINDEXES += "package_index_ipk();"
PACKAGEINDEXDEPS += "opkg-utils-native:do_populate_sysroot"
PACKAGEINDEXDEPS += "opkg-native:do_populate_sysroot"
package_index_ipk[dirs] = "${DEPLOY_DIR_IPK}"

package_index_ipk() {
       set -ex
       package_update_index_ipk
       set +ex
}

The key is the addition to the PACKAGEINDEXES variable and its
dependencies in PACKAGEINDEXDEPS.

In package-index.bb we can then just have:

do_package_index[depends] += "${PACKAGEINDEXDEPS}"
 
do_package_index() {
	${PACKAGEINDEXES}
}

and these variables should always have the things we need in them
depending on the package backends available.

Ideally we should cover the case there is no package backend enabled and
just return too rather than error (or bbwarn "No package backends
enabled, doing nothing").

Cheers,

Richard

Re: [RFC][v2] [PATCH 1/1] package-index.bb: add support for deb and rpm.

[Cui, Dexuan]

Richard Purdie wrote:
> Hi Dexuan,
> 
> When I read Saul's original emails I thought this approach would work
> although it wasn't the first one that came to my mind.
> 
> Looking at the code you have, the problem is you have nowhere you can
> call the "addtask" for the individual tasks and we don't want to the
> overhead system wide for every recipe as it makes no sense.
Yes, I realized this later.

> I'd therefore like to suggest we tweak this approach a little. Each
> package_xxx.bbclass defines a function which updates the index for
> that packaging backend:
I agree. I've actually made a patch in branch dcui/package-index, but the below implementation you suggested is obviously much better! :-)
Thank you very much for the detailed suggestion!

> package_ipk.bbclass:
> 
> PACKAGEINDEXES += "package_index_ipk();"
> PACKAGEINDEXDEPS += "opkg-utils-native:do_populate_sysroot"
> PACKAGEINDEXDEPS += "opkg-native:do_populate_sysroot"
> package_index_ipk[dirs] = "${DEPLOY_DIR_IPK}"
> 
> package_index_ipk() {
>        set -ex
>        package_update_index_ipk
>        set +ex
> }
> 
> The key is the addition to the PACKAGEINDEXES variable and its
> dependencies in PACKAGEINDEXDEPS.
> 
> In package-index.bb we can then just have:
> 
> do_package_index[depends] += "${PACKAGEINDEXDEPS}"
> 
> do_package_index() {
> 	${PACKAGEINDEXES}
> }
> 
> and these variables should always have the things we need in them
> depending on the package backends available.
> 
> Ideally we should cover the case there is no package backend enabled
> and just return too rather than error (or bbwarn "No package backends
> enabled, doing nothing").
Oh, I didn't realize this... I'll remember to handle this, too.

Thanks,
-- Dexuan