[PATCH] package_manager.py: check the dest before create hard link

[PATCH] package_manager.py: check the dest before create hard link

[mingli.yu]

From: Mingli Yu <Mingli.Yu@windriver.com>

After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon

When add below line into conf/local.conf
INHERIT += "packagefeed-stability"

There comes below error when do_rootfs
Exception: FileExistsError: [Errno 17] File exists: '/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' -> '/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm'

Add the check before create hard link to
avoid the above error.

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
---
 meta/lib/oe/package_manager.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 2cc1c752b3..5d4739d66d 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -703,7 +703,8 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
                     bb.utils.mkdirhier(destdir)
                     seendirs.add(destdir)
                 try:
-                    os.link(l, dest)
+                    if not os.path.exists(dest):
+                        os.link(l, dest)
                 except OSError as err:
                     if err.errno == errno.EXDEV:
                         bb.utils.copyfile(l, dest)
-- 
2.18.0

[PATCH v2] package_manager.py: correct the deploydir when packagefeed-stability inherited

[mingli.yu]

From: Mingli Yu <Mingli.Yu@windriver.com>

After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon

When add below line into conf/local.conf
INHERIT += "packagefeed-stability"

There comes below error when do_rootfs
Exception: FileExistsError: [Errno 17] File exists: '/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' -> '/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm'

In create_packages_dir function, there is a logic
as bb.utils.remove(subrepo_dir, recurse=True) to
clean subrepo_dir which is actually as example is
/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm.

But currently when inherit packagefeed-stability class,
the parameter deploydir passed to create_packages_dir is
still /$Prj/tmp/deploy/rpm as example, it should be
/$Prj/tmp/deploy/rpm-diff. Otherwise, there is a logic
os.link(l, dest) in create_packages_dir function will
result in below logic:
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm")

Actually doesn't clean /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff
, need to update the deploydir to make the
logic changed as below to fix the do_rootfs failure
s.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm")

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
---
 meta/lib/oe/package_manager.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 2cc1c752b3..032d0d811b 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -733,8 +733,10 @@ class RpmPM(PackageManager):
             self.primary_arch = self.d.getVar('MACHINE_ARCH')
 
         self.rpm_repo_dir = oe.path.join(self.d.getVar('WORKDIR'), rpm_repo_workdir)
-        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), d.getVar("DEPLOY_DIR_RPM"), "package_write_rpm", filterbydependencies)
-
+        deploy_prepath = d.getVar("DEPLOY_DIR_RPM")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), deploy_prepath, "package_write_rpm", filterbydependencies)
         self.saved_packaging_data = self.d.expand('${T}/saved_packaging_data/%s' % self.task_name)
         if not os.path.exists(self.d.expand('${T}/saved_packaging_data')):
             bb.utils.mkdirhier(self.d.expand('${T}/saved_packaging_data'))
@@ -1147,9 +1149,12 @@ class OpkgPM(OpkgDpkgPM):
         self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg")
         self.opkg_args = "--volatile-cache -f %s -t %s -o %s " % (self.config_file, self.d.expand('${T}/ipktemp/') ,target_rootfs)
         self.opkg_args += self.d.getVar("OPKG_ARGS")
+        deploy_prepath = d.getVar("DEPLOY_DIR_IPK")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
 
         if prepare_index:
-            create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies)
+            create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_ipk", filterbydependencies)
 
         opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
         if opkg_lib_dir[0] == "/":
@@ -1526,7 +1531,10 @@ class DpkgPM(OpkgDpkgPM):
         super(DpkgPM, self).__init__(d, target_rootfs)
         self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), deb_repo_workdir)
 
-        create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_DEB"), "package_write_deb", filterbydependencies)
+        deploy_prepath = d.getVar("DEPLOY_DIR_DEB")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_deb", filterbydependencies)
 
         if apt_conf_dir is None:
             self.apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt")
-- 
2.18.0

Re: [PATCH v2] package_manager.py: correct the deploydir when packagefeed-stability inherited

[Yu, Mingli]

On 2018年10月24日 18:35, mingli.yu@windriver.com wrote:
> From: Mingli Yu <Mingli.Yu@windriver.com>
>
> After create_packages_dir added in below commit:
> 85e72e1 package_manager: Filter to only rpms we depend upon
>
> When add below line into conf/local.conf
> INHERIT += "packagefeed-stability"
>
> There comes below error when do_rootfs
> Exception: FileExistsError: [Errno 17] File exists: '/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' -> '/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm'
>
> In create_packages_dir function, there is a logic
> as bb.utils.remove(subrepo_dir, recurse=True) to
> clean subrepo_dir which is actually as example is
> /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm.
>
> But currently when inherit packagefeed-stability class,
> the parameter deploydir passed to create_packages_dir is
> still /$Prj/tmp/deploy/rpm as example, it should be
> /$Prj/tmp/deploy/rpm-diff. Otherwise, there is a logic
> os.link(l, dest) in create_packages_dir function will
> result in below logic:
> os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm")
>
> Actually doesn't clean /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff
> , need to update the deploydir to make the
> logic changed as below to fix the do_rootfs failure
> s.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm")

A typo here, resend out V3.

Thanks,

>
> Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
> ---
>   meta/lib/oe/package_manager.py | 16 ++++++++++++----
>   1 file changed, 12 insertions(+), 4 deletions(-)
>
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index 2cc1c752b3..032d0d811b 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -733,8 +733,10 @@ class RpmPM(PackageManager):
>               self.primary_arch = self.d.getVar('MACHINE_ARCH')
>
>           self.rpm_repo_dir = oe.path.join(self.d.getVar('WORKDIR'), rpm_repo_workdir)
> -        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), d.getVar("DEPLOY_DIR_RPM"), "package_write_rpm", filterbydependencies)
> -
> +        deploy_prepath = d.getVar("DEPLOY_DIR_RPM")
> +        if bb.data.inherits_class('packagefeed-stability', d):
> +            deploy_prepath = deploy_prepath + "-prediff"
> +        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), deploy_prepath, "package_write_rpm", filterbydependencies)
>           self.saved_packaging_data = self.d.expand('${T}/saved_packaging_data/%s' % self.task_name)
>           if not os.path.exists(self.d.expand('${T}/saved_packaging_data')):
>               bb.utils.mkdirhier(self.d.expand('${T}/saved_packaging_data'))
> @@ -1147,9 +1149,12 @@ class OpkgPM(OpkgDpkgPM):
>           self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg")
>           self.opkg_args = "--volatile-cache -f %s -t %s -o %s " % (self.config_file, self.d.expand('${T}/ipktemp/') ,target_rootfs)
>           self.opkg_args += self.d.getVar("OPKG_ARGS")
> +        deploy_prepath = d.getVar("DEPLOY_DIR_IPK")
> +        if bb.data.inherits_class('packagefeed-stability', d):
> +            deploy_prepath = deploy_prepath + "-prediff"
>
>           if prepare_index:
> -            create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies)
> +            create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_ipk", filterbydependencies)
>
>           opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
>           if opkg_lib_dir[0] == "/":
> @@ -1526,7 +1531,10 @@ class DpkgPM(OpkgDpkgPM):
>           super(DpkgPM, self).__init__(d, target_rootfs)
>           self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), deb_repo_workdir)
>
> -        create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_DEB"), "package_write_deb", filterbydependencies)
> +        deploy_prepath = d.getVar("DEPLOY_DIR_DEB")
> +        if bb.data.inherits_class('packagefeed-stability', d):
> +            deploy_prepath = deploy_prepath + "-prediff"
> +        create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_deb", filterbydependencies)
>
>           if apt_conf_dir is None:
>               self.apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt")
>

[PATCH v3] package_manager.py: correct the deploydir when packagefeed-stability inherited

[mingli.yu]

From: Mingli Yu <Mingli.Yu@windriver.com>

After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon

When add below line into conf/local.conf
INHERIT += "packagefeed-stability"

There comes below error when do_rootfs
Exception: FileExistsError: [Errno 17] File exists: '/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' -> '/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm'

In create_packages_dir function, there is a logic
as bb.utils.remove(subrepo_dir, recurse=True) to
clean subrepo_dir which is actually as example is
/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm.

But currently when inherit packagefeed-stability class,
the parameter deploydir passed to create_packages_dir is
still /$Prj/tmp/deploy/rpm as example, it should be
/$Prj/tmp/deploy/rpm-diff. Otherwise, there is a logic
os.link(l, dest) in create_packages_dir function will
result in below logic:
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm")

Actually doesn't clean /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff
, need to update the deploydir to make the
logic changed as below to fix the do_rootfs failure
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm")

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
---
 meta/lib/oe/package_manager.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 2cc1c752b3..032d0d811b 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -733,8 +733,10 @@ class RpmPM(PackageManager):
             self.primary_arch = self.d.getVar('MACHINE_ARCH')
 
         self.rpm_repo_dir = oe.path.join(self.d.getVar('WORKDIR'), rpm_repo_workdir)
-        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), d.getVar("DEPLOY_DIR_RPM"), "package_write_rpm", filterbydependencies)
-
+        deploy_prepath = d.getVar("DEPLOY_DIR_RPM")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), deploy_prepath, "package_write_rpm", filterbydependencies)
         self.saved_packaging_data = self.d.expand('${T}/saved_packaging_data/%s' % self.task_name)
         if not os.path.exists(self.d.expand('${T}/saved_packaging_data')):
             bb.utils.mkdirhier(self.d.expand('${T}/saved_packaging_data'))
@@ -1147,9 +1149,12 @@ class OpkgPM(OpkgDpkgPM):
         self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg")
         self.opkg_args = "--volatile-cache -f %s -t %s -o %s " % (self.config_file, self.d.expand('${T}/ipktemp/') ,target_rootfs)
         self.opkg_args += self.d.getVar("OPKG_ARGS")
+        deploy_prepath = d.getVar("DEPLOY_DIR_IPK")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
 
         if prepare_index:
-            create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies)
+            create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_ipk", filterbydependencies)
 
         opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
         if opkg_lib_dir[0] == "/":
@@ -1526,7 +1531,10 @@ class DpkgPM(OpkgDpkgPM):
         super(DpkgPM, self).__init__(d, target_rootfs)
         self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), deb_repo_workdir)
 
-        create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_DEB"), "package_write_deb", filterbydependencies)
+        deploy_prepath = d.getVar("DEPLOY_DIR_DEB")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_deb", filterbydependencies)
 
         if apt_conf_dir is None:
             self.apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt")
-- 
2.18.0

[PATCH v4] package_manager.py: correct the deploydir when packagefeed-stability inherited

[mingli.yu]

From: Mingli Yu <Mingli.Yu@windriver.com>

After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon

When add below line into conf/local.conf
INHERIT += "packagefeed-stability"

There comes below error when do_rootfs
Exception: FileExistsError: [Errno 17] File exists: '/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' -> '/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm'

def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies):
  [snip]
  bb.utils.remove(subrepo_dir, recurse=True)
  [snip]

In create_packages_dir function, there is a logic
as bb.utils.remove(subrepo_dir, recurse=True) to
clean subrepo_dir which is actually as example is
/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm.

But currently when inherit packagefeed-stability class,
the deploydir should be /$Prj/tmp/deploy/rpm-prediff,
not the default /$Prj/tmp/deploy/rpm.

If use /$Prj/tmp/deploy/rpm, then result in the
logic as below:
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm")

Update to the actual deploydir to guarantee the logic
as below:
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm")

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
---
 meta/lib/oe/package_manager.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 2cc1c752b3..882e7c429f 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -690,7 +690,10 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
             for l in f:
                 l = l.strip()
                 deploydir = os.path.normpath(deploydir)
-                dest = l.replace(deploydir, "")
+                if bb.data.inherits_class('packagefeed-stability', d):
+                    dest = l.replace(deploydir + "-prediff", "")
+                else:
+                    dest = l.replace(deploydir, "")
                 dest = subrepo_dir + dest
                 if l.endswith("/"):
                     if dest not in seendirs:
-- 
2.18.0