[PATCH 0/1] Automatically generate package repos for rpm and deb [bug #1024]

[PATCH 0/1] Automatically generate package repos for rpm and deb [bug #1024]

[Dexuan Cui]

From: Dexuan Cui <dexuan.cui@intel.com>

This was made to address http://bugzilla.yoctoproject.org/show_bug.cgi?id=1024.
Please comment.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: dcui/master
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master

Thanks,
    Dexuan Cui <dexuan.cui@intel.com>
---


Dexuan Cui (1):
  package-index-{ipk, rpm, deb}.bb: add the package-index for rpm and
    deb

 meta/recipes-core/meta/package-index-deb.bb        |   10 ++++++++++
 meta/recipes-core/meta/package-index-ipk.bb        |   11 +++++++++++
 meta/recipes-core/meta/package-index-rpm.bb        |   10 ++++++++++
 .../meta/{package-index.bb => package-index.inc}   |    8 --------
 4 files changed, 31 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-core/meta/package-index-deb.bb
 create mode 100644 meta/recipes-core/meta/package-index-ipk.bb
 create mode 100644 meta/recipes-core/meta/package-index-rpm.bb
 rename meta/recipes-core/meta/{package-index.bb => package-index.inc} (69%)

-- 
1.7.2

[PATCH 1/1] [RFC]package-index-{ipk, rpm, deb}.bb: add the package-index for rpm and deb

[Dexuan Cui]

From: Dexuan Cui <dexuan.cui@intel.com>

[YOCTO #1024]

The commit renamed package-index.bb to package-index.inc and added
3 bb files for ipk, rpm and deb.

-------------------------------

How to generate and use repo:

RPM:
    1) run "bitbake package-index-rpm" after building some target;
    2) export the ${DEPLOY_DIR_RPM} by a webserver on the host (let's assume
the IP is 192.168.7.1) at http://192.168.7.1/rpm;
    3) inside the target, run "zypper addrepo http://192.168.7.1/rpm main; zypper refresh"
to retrieve info about the repo; next, we can use zypper install/remove to
manage packages.

IPK:
    1) run "bitbake package-index-ipk" after building some target;
    2) export the ${DEPLOY_DIR_IPK} by a webserver on the host (let's assume
the IP is 192.168.7.1) at http://192.168.7.1/ipk;
    3) inside the target, add the repo info into opkg config file, i.e.,
in /etc/opkg/arch.conf, we can add something like
src i586 http://192.168.7.1/ipk/i586, and next we run "opkg update" to make
opkg update the list of available packages, and next, we can use
opkg install/remove to manage packages.

DEB: To be added.

Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>
---
 meta/recipes-core/meta/package-index-deb.bb        |   10 ++++++++++
 meta/recipes-core/meta/package-index-ipk.bb        |   11 +++++++++++
 meta/recipes-core/meta/package-index-rpm.bb        |   10 ++++++++++
 .../meta/{package-index.bb => package-index.inc}   |    8 --------
 4 files changed, 31 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-core/meta/package-index-deb.bb
 create mode 100644 meta/recipes-core/meta/package-index-ipk.bb
 create mode 100644 meta/recipes-core/meta/package-index-rpm.bb
 rename meta/recipes-core/meta/{package-index.bb => package-index.inc} (69%)

diff --git a/meta/recipes-core/meta/package-index-deb.bb b/meta/recipes-core/meta/package-index-deb.bb
new file mode 100644
index 0000000..db52e2a
--- /dev/null
+++ b/meta/recipes-core/meta/package-index-deb.bb
@@ -0,0 +1,10 @@
+require package-index.inc
+
+do_package_index[depends] += "dpkg-native:do_populate_sysroot apt-native:do_populate_sysroot"
+
+do_package_index() {
+    set -ex
+    package_update_index_deb
+    set +ex
+}
+
diff --git a/meta/recipes-core/meta/package-index-ipk.bb b/meta/recipes-core/meta/package-index-ipk.bb
new file mode 100644
index 0000000..5eddea1
--- /dev/null
+++ b/meta/recipes-core/meta/package-index-ipk.bb
@@ -0,0 +1,11 @@
+require package-index.inc
+
+do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
+do_package_index[depends] += "opkg-native:do_populate_sysroot"
+
+do_package_index() {
+    set -ex
+    package_update_index_ipk
+    set +ex
+}
+
diff --git a/meta/recipes-core/meta/package-index-rpm.bb b/meta/recipes-core/meta/package-index-rpm.bb
new file mode 100644
index 0000000..605a638
--- /dev/null
+++ b/meta/recipes-core/meta/package-index-rpm.bb
@@ -0,0 +1,10 @@
+require package-index.inc
+
+do_package_index[depends] += "createrepo-native:do_populate_sysroot"
+
+do_package_index() {
+    set -ex
+    createrepo "${DEPLOY_DIR_RPM}"
+    set +ex
+}
+
diff --git a/meta/recipes-core/meta/package-index.bb b/meta/recipes-core/meta/package-index.inc
similarity index 69%
rename from meta/recipes-core/meta/package-index.bb
rename to meta/recipes-core/meta/package-index.inc
index 3c642cb..3f1caea 100644
--- a/meta/recipes-core/meta/package-index.bb
+++ b/meta/recipes-core/meta/package-index.inc
@@ -19,14 +19,6 @@ do_package_write_deb[noexec] = "1"
 do_populate_sysroot[noexec] = "1"
 
 do_package_index[nostamp] = "1"
-do_package_index[dirs] = "${DEPLOY_DIR_IPK}"
-do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
-do_package_index[depends] += "opkg-native:do_populate_sysroot"
 
-do_package_index() {
-	set -ex
-	package_update_index_ipk
-	set +ex
-}
 addtask do_package_index before do_build
 EXCLUDE_FROM_WORLD = "1"
-- 
1.7.2

Re: [PATCH 0/1] Automatically generate package repos for rpm and deb [bug #1024]

[Cui, Dexuan]

Dexuan Cui wrote:
> From: Dexuan Cui <dexuan.cui@intel.com>
> 
> This was made to address
> http://bugzilla.yoctoproject.org/show_bug.cgi?id=1024. Please comment.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: dcui/master
>   Browse:
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master 

This is an improved and simple version that keeps 1  package-index recipe and updates the 3 kinds of packages at the same time:
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=dcui/package-index&id=fb26e81dcb6e27e2908294e406a3011baed0120d
This avoids forcing the user to remember which package type they enabled when running "bitbake package-index".
(I also past the new version at the end of this mail for easy reviewing)

Thanks!
-- Dexuan

commit fb26e81dcb6e27e2908294e406a3011baed0120d
Author: Dexuan Cui <dexuan.cui@intel.com>
Date:   Thu May 12 19:15:59 2011 +0800

    package-index.bb: also add the support for rpm/deb

    [YOCTO #1024]

    -------------------------------
    (I'll make a new patch to add the below description to the manual).

    How to generate and use repos:

    1) run "bitbake package-index" after building some target;
    2) export ${DEPLOY_DIR_RPM} and ${DEPLOY_DIR_IPK} by a webserver on the host
    (let's assume the host IP is 192.168.7.1) at
    http://192.168.7.1/rpm
    http://192.168.7.1/ipk
    3) inside the target, according to the packaging system (ipk, rpm, deb) used
    when we generate the target image, we can use different ways to manage
    packages:
        3.1) RPM:  "zypper addrepo http://192.168.7.1/rpm main; zypper refresh"
    to retrieve info about the repo; next, we can use zypper install/remove to
    manage packages.
        3.2) IPK:  add the repo info into opkg config file, i.e., in
    /etc/opkg/arch.conf, we can add something like
    src i586 http://192.168.7.1/ipk/i586, and next we run "opkg update" to make
    opkg update the list of available packages, and next we can use
    opkg install/remove to manage packages.
        3.3) DEB:(To be added)

    Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>

diff --git a/meta/recipes-core/meta/package-index.bb b/meta/recipes-core/meta/package-index.bb
index 3c642cb..969430b 100644
--- a/meta/recipes-core/meta/package-index.bb
+++ b/meta/recipes-core/meta/package-index.bb
@@ -22,10 +22,14 @@ do_package_index[nostamp] = "1"
 do_package_index[dirs] = "${DEPLOY_DIR_IPK}"
 do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
 do_package_index[depends] += "opkg-native:do_populate_sysroot"
+do_package_index[depends] += "createrepo-native:do_populate_sysroot"
+do_package_index[depends] += "dpkg-native:do_populate_sysroot apt-native:do_populate_sysroot"

 do_package_index() {
    set -ex
    package_update_index_ipk
+   createrepo "${DEPLOY_DIR_RPM}"
+   package_update_index_deb
    set +ex
 }
 addtask do_package_index before do_build

Re: [PATCH 0/1] Automatically generate package repos for rpm and deb [bug #1024]

[Saul Wold]

On 05/12/2011 04:38 AM, Cui, Dexuan wrote:
> Dexuan Cui wrote:
>> From: Dexuan Cui<dexuan.cui@intel.com>
>>
>> This was made to address
>> http://bugzilla.yoctoproject.org/show_bug.cgi?id=1024. Please comment.
>>
>> Pull URL: git://git.pokylinux.org/poky-contrib.git
>>    Branch: dcui/master
>>    Browse:
>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master
>
> This is an improved and simple version that keeps 1  package-index recipe and updates the 3 kinds of packages at the same time:
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=dcui/package-index&id=fb26e81dcb6e27e2908294e406a3011baed0120d
> This avoids forcing the user to remember which package type they enabled when running "bitbake package-index".
> (I also past the new version at the end of this mail for easy reviewing)
>
Dexuan,

I am not sure that this is the correct approach.  I think that you 
should look at how the packages get generated based on which package 
classes are included via PACKAGE_CLASSES, then in each of those classes 
set it up so that when the package-index task is called you know which 
indexer to run. This would be moving what you have in the 
package-index-<packager>.bb files to their respective 
package_<packager>.bbclass files and use

do_package_index[recrdeptask] += package_index_<packager>

in the case of ipk it would be in package_ipk.bbclass

do_package_index[recrdeptask] += "package_update_index_ipk"

You will also need to move the do_package_index[depends]

Another thing to verify is the current package_update_index_* correct 
for the different package types? Looking at package_update_index_rpm, I 
am not sure since it does not call createrepo, there might be a reason 
for this that Mark H or Qing can comment on.

Hope that made sense, I am sure Richard will correct me if I got 
something wrong!

Sau!

> Thanks!
> -- Dexuan
>
> commit fb26e81dcb6e27e2908294e406a3011baed0120d
> Author: Dexuan Cui<dexuan.cui@intel.com>
> Date:   Thu May 12 19:15:59 2011 +0800
>
>      package-index.bb: also add the support for rpm/deb
>
>      [YOCTO #1024]
>
>      -------------------------------
>      (I'll make a new patch to add the below description to the manual).
>
>      How to generate and use repos:
>
>      1) run "bitbake package-index" after building some target;
>      2) export ${DEPLOY_DIR_RPM} and ${DEPLOY_DIR_IPK} by a webserver on the host
>      (let's assume the host IP is 192.168.7.1) at
>      http://192.168.7.1/rpm
>      http://192.168.7.1/ipk
>      3) inside the target, according to the packaging system (ipk, rpm, deb) used
>      when we generate the target image, we can use different ways to manage
>      packages:
>          3.1) RPM:  "zypper addrepo http://192.168.7.1/rpm main; zypper refresh"
>      to retrieve info about the repo; next, we can use zypper install/remove to
>      manage packages.
>          3.2) IPK:  add the repo info into opkg config file, i.e., in
>      /etc/opkg/arch.conf, we can add something like
>      src i586 http://192.168.7.1/ipk/i586, and next we run "opkg update" to make
>      opkg update the list of available packages, and next we can use
>      opkg install/remove to manage packages.
>          3.3) DEB:(To be added)
>
>      Signed-off-by: Dexuan Cui<dexuan.cui@intel.com>
>
> diff --git a/meta/recipes-core/meta/package-index.bb b/meta/recipes-core/meta/package-index.bb
> index 3c642cb..969430b 100644
> --- a/meta/recipes-core/meta/package-index.bb
> +++ b/meta/recipes-core/meta/package-index.bb
> @@ -22,10 +22,14 @@ do_package_index[nostamp] = "1"
>   do_package_index[dirs] = "${DEPLOY_DIR_IPK}"
>   do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
>   do_package_index[depends] += "opkg-native:do_populate_sysroot"
> +do_package_index[depends] += "createrepo-native:do_populate_sysroot"
> +do_package_index[depends] += "dpkg-native:do_populate_sysroot apt-native:do_populate_sysroot"
>
>   do_package_index() {
>      set -ex
>      package_update_index_ipk
> +   createrepo "${DEPLOY_DIR_RPM}"
> +   package_update_index_deb
>      set +ex
>   }
>   addtask do_package_index before do_build
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH 0/1] Automatically generate package repos for rpm and deb [bug #1024]

[Cui, Dexuan]

Wold, Saul wrote:
> On 05/12/2011 04:38 AM, Cui, Dexuan wrote:
>> Dexuan Cui wrote:
>>> From: Dexuan Cui<dexuan.cui@intel.com>
>>> 
>>> This was made to address
>>> http://bugzilla.yoctoproject.org/show_bug.cgi?id=1024. Please
>>> comment. 
>>> 
>>> Pull URL: git://git.pokylinux.org/poky-contrib.git
>>>    Branch: dcui/master
>>>    Browse:
>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master
>> 
>> This is an improved and simple version that keeps 1  package-index
>> recipe and updates the 3 kinds of packages at the same time:
>> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=dcui/package-index&id=fb26e81dcb6e27e2908294e406a3011baed0120d
>> This avoids forcing the user to remember which package type they
>> enabled when running "bitbake package-index". (I also past the new
>> version at the end of this mail for easy reviewing)  
>> 
> Dexuan,
> 
> I am not sure that this is the correct approach.  I think that you
> should look at how the packages get generated based on which package
> classes are included via PACKAGE_CLASSES, then in each of those
> classes set it up so that when the package-index task is called you
> know which indexer to run. This would be moving what you have in the
> package-index-<packager>.bb files to their respective
> package_<packager>.bbclass files and use
> 
> do_package_index[recrdeptask] += package_index_<packager>
> 
> in the case of ipk it would be in package_ipk.bbclass
> 
> do_package_index[recrdeptask] += "package_update_index_ipk"
> 
> You will also need to move the do_package_index[depends]
OK, thanks very much for the detailed suggestion! I'll change to this better method.

> Another thing to verify is the current package_update_index_* correct
> for the different package types? Looking at package_update_index_rpm,
> I am not sure since it does not call createrepo, there might be a
> reason for this that Mark H or Qing can comment on.
My understanding about package_update_index_rpm vs. createrepo is:
both can be used to generate rpm repo;
package_update_index_rpm generates the metadata about the packages in solvedb to have rpm install the packages into target rootfs. This is the current method used in do_rootfs for rpm. This is complex and the generated repo is not suitable to be exported (e.g., via http) for general use;
createrepo generated the metadata in .xml files and the generated repo can be easily exported via http and can be used easily by standard zypper (and yum) tools.
So in package-index.bb, I adopt createrepo for rpm.

Let me Cc Qing and Mark for comments about this.

Thanks!
-- Dexuan