[PATCH 0/3][RFC] Poky disk space size optimization

[PATCH 0/3][RFC] Poky disk space size optimization

[Dongxiao Xu]

This RFC aims to reduce the poky disk space size.

The optimization point is the duplicated file copies in WORKDIR and
sysroot directories. When installing, files will be copied from source
code package into the image folder; Then some files will be copied
from image folder to sysroot-destdir, and some other files will be
copied from image folder to package folder. While doing package split,
some files will be copied from package folder to packages-split folder,
etc.

The thought is to use hard link to replace the direct copy.
The three commits do the work of:

a) package.bbclass: hard link "image-->sysroot-destdir"
b) sstate.bbclass:  hard link "package-->packages-split"
c) staging.bbclass: hard link "sysroot-destdir-->sysroots".

After testing with minimal/sato/sdk build, this approach could save
about 20% disk space.
(minimal: 23G -> 19G, sato: 44G -> 35G, sdk: 54G -> 44G).

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: dxu4/perf
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/perf

Thanks,
    Dongxiao Xu <dongxiao.xu@intel.com>
---


Dongxiao Xu (3):
  package.bbclass: Use hard link for package split instead of copy
  sstate.bbclass: Use hard link when staging files into sysroots
  staging.bbclass: use hardlink when staging files to sysroot-destdir

 meta/classes/package.bbclass |    8 +++++---
 meta/classes/sstate.bbclass  |    2 +-
 meta/classes/staging.bbclass |   27 ++++++++++++++++++++++++++-
 meta/lib/oe/path.py          |   28 ++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+), 5 deletions(-)

[PATCH 1/3] package.bbclass: Use hard link for package split instead of copy

[Dongxiao Xu]

When doing package split, we use hard link instead of copy, which can
save about 10% disk space when building poky-image-minimal.

If fail, it will fall back to the copyfile function.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 meta/classes/package.bbclass |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index d39c694..2b4747b 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -406,9 +406,11 @@ python populate_packages () {
 			fpath = os.path.join(root,file)
 			dpath = os.path.dirname(fpath)
 			bb.mkdirhier(dpath)
-			ret = bb.copyfile(file, fpath)
-			if ret is False or ret == 0:
-				raise bb.build.FuncFailed("File population failed")
+			ret = os.system("ln -f %s %s" % (file, fpath))
+			if ret is False:
+				ret = bb.copyfile(file, fpath)
+				if ret is False or ret == 0:
+					raise bb.build.FuncFailed("File population failed")
 		del localdata
 	os.chdir(workdir)
 
-- 
1.6.3.3

[PATCH 2/3] sstate.bbclass: Use hard link when staging files into sysroots

[Dongxiao Xu]

Use hard link instead of copy when staging files from sysroot-destdir
into sysroots directory. This can save about 10% disk space for
poky-image-minimal.

If fail, it will fall back to copy function

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 meta/classes/sstate.bbclass |    2 +-
 meta/lib/oe/path.py         |   28 ++++++++++++++++++++++++++++
 2 files changed, 29 insertions(+), 1 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 3696a8c..b6c477b 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -115,7 +115,7 @@ def sstate_install(ss, d):
         locks.append(bb.utils.lockfile(lock))
 
     for state in ss['dirs']:
-        oe.path.copytree(state[1], state[2])
+        oe.path.hardlink(state[1], state[2])
         for walkroot, dirs, files in os.walk(state[1]):
             for file in files:
                 srcpath = os.path.join(walkroot, file)
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index f42faea..589b524 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -106,3 +106,31 @@ def symlink(source, destination, force=False):
     except OSError, e:
         if e.errno != errno.EEXIST or os.readlink(destination) != source:
             raise
+
+def hardlink(src, dst):
+    names = os.listdir(src)
+
+    bb.mkdirhier(dst)
+
+    errors = []
+    for name in names:
+        srcname = os.path.join(src, name)
+        dstname = os.path.join(dst, name)
+        try:
+            if os.path.islink(srcname):
+                linkto = os.readlink(srcname)
+                if os.path.lexists(dstname):
+                    os.unlink(dstname)
+                os.symlink(linkto, dstname)
+            elif os.path.isdir(srcname):
+                hardlink(srcname, dstname)
+            else:
+                ret = os.system("ln -f %s %s" % (srcname, dstname))
+                if ret is False:
+                    bb.utils.copyfile(srcname, dstname)
+        except (IOError, os.error), why:
+            errors.append((srcname, dstname, str(why)))
+        except Error, err:
+            errors.extend(err.args[0])
+    if errors:
+        raise Error, errors
-- 
1.6.3.3

[PATCH 3/3] staging.bbclass: use hardlink when staging files to sysroot-destdir

[Dongxiao Xu]

Use hardlink to replace copy when staging files to sysroot-destdir.
If fail, it will fall back to copy function.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 meta/classes/staging.bbclass |   27 ++++++++++++++++++++++++++-
 1 files changed, 26 insertions(+), 1 deletions(-)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index e82db83..159562d 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -3,6 +3,31 @@ packagedstaging_fastpath () {
 	:
 }
 
+hardlink() {
+	local src="$1"
+	local dest="$2"
+	local name names srcname destname
+
+	if [ ! -d "$dest" ]; then
+		mkdir -p "$dest"
+	fi
+
+	names=$(ls $src)
+	for name in $names
+	do
+		srcname=$src"/"$name
+		destname=$dest"/"$name
+		if [ -d "$srcname" ]; then
+			hardlink $srcname $destname
+		else
+			ln -f $srcname $destname || true
+			if [ ! -e $destname ]; then
+				cp -fpPR $srcname $destname
+			fi
+		fi
+	done
+}
+
 sysroot_stage_dir() {
 	src="$1"
 	dest="$2"
@@ -17,7 +42,7 @@ sysroot_stage_dir() {
 	# However we always want to stage a $src itself, even if it's empty
 	mkdir -p "$dest"
 	if [ -d "$src" ]; then
-		cp -fpPR "$src"/* "$dest"
+		hardlink "$src" "$dest"
 	fi
 }
 
-- 
1.6.3.3

Re: [PATCH 0/3][RFC] Poky disk space size optimization

[Saul Wold]

On 12/06/2010 04:50 PM, Dongxiao Xu wrote:
> This RFC aims to reduce the poky disk space size.
>
> The optimization point is the duplicated file copies in WORKDIR and
> sysroot directories. When installing, files will be copied from source
> code package into the image folder; Then some files will be copied
> from image folder to sysroot-destdir, and some other files will be
> copied from image folder to package folder. While doing package split,
> some files will be copied from package folder to packages-split folder,
> etc.
>
There are a couple of concerns with this method, which may lead to 
problems, first is that using links and modifying permissions, it's 
possible that depending on the usage the permissions might be set 
differently.  This will could lead to a second problem: psuedo. It's 
unclear at this point how psuedo will handle this.

I am sure you have been doing testing, just not sure of corner cases.

I know we accepted the first change for the package split link to the 
deploy area, that one made sense as it was package level, this is file 
level and may have hidden problems.

I am sure that Richard will correct me and add his comments as he 
recovers from his travels.

Sau!

> The thought is to use hard link to replace the direct copy.
> The three commits do the work of:
>
> a) package.bbclass: hard link "image-->sysroot-destdir"
> b) sstate.bbclass:  hard link "package-->packages-split"
> c) staging.bbclass: hard link "sysroot-destdir-->sysroots".
>
> After testing with minimal/sato/sdk build, this approach could save
> about 20% disk space.
> (minimal: 23G ->  19G, sato: 44G ->  35G, sdk: 54G ->  44G).
>
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>    Branch: dxu4/perf
>    Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/perf
>
> Thanks,
>      Dongxiao Xu<dongxiao.xu@intel.com>
> ---
>
>
> Dongxiao Xu (3):
>    package.bbclass: Use hard link for package split instead of copy
>    sstate.bbclass: Use hard link when staging files into sysroots
>    staging.bbclass: use hardlink when staging files to sysroot-destdir
>
>   meta/classes/package.bbclass |    8 +++++---
>   meta/classes/sstate.bbclass  |    2 +-
>   meta/classes/staging.bbclass |   27 ++++++++++++++++++++++++++-
>   meta/lib/oe/path.py          |   28 ++++++++++++++++++++++++++++
>   4 files changed, 60 insertions(+), 5 deletions(-)
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>

Re: [PATCH 0/3][RFC] Poky disk space size optimization

[Tian, Kevin]

>From: Saul Wold
>Sent: Tuesday, December 07, 2010 10:47 AM
>
>On 12/06/2010 04:50 PM, Dongxiao Xu wrote:
>> This RFC aims to reduce the poky disk space size.
>>
>> The optimization point is the duplicated file copies in WORKDIR and
>> sysroot directories. When installing, files will be copied from source
>> code package into the image folder; Then some files will be copied
>> from image folder to sysroot-destdir, and some other files will be
>> copied from image folder to package folder. While doing package split,
>> some files will be copied from package folder to packages-split folder,
>> etc.
>>
>There are a couple of concerns with this method, which may lead to
>problems, first is that using links and modifying permissions, it's
>possible that depending on the usage the permissions might be set
>differently.  This will could lead to a second problem: psuedo. It's
>unclear at this point how psuedo will handle this.
>
>I am sure you have been doing testing, just not sure of corner cases.
>
>I know we accepted the first change for the package split link to the
>deploy area, that one made sense as it was package level, this is file
>level and may have hidden problems.

I also agree with Saul here. There do have some difference among those directories
for their own purpose, which finally result in some difference. For example, there're 
different tweaks on libtool and pkgconfig files between sysroot and packaging. That's
why 'image' is copied to package and then sysroot-destdir, and then do necessary
tweaks independently. Here if you want to hard link image to sysroot-destdir, you
need make sure it done after the copy from image to package. Or else you'll end up
to have problem there.

But if you can make sure that given a copy path you're modifying, the
source and the destination are exactly same, and no other path refers to
the source directory, perhaps it's fine to do hard link.

Thanks
Kevin

>
>I am sure that Richard will correct me and add his comments as he
>recovers from his travels.
>
>Sau!
>
>> The thought is to use hard link to replace the direct copy.
>> The three commits do the work of:
>>
>> a) package.bbclass: hard link "image-->sysroot-destdir"
>> b) sstate.bbclass:  hard link "package-->packages-split"
>> c) staging.bbclass: hard link "sysroot-destdir-->sysroots".
>>
>> After testing with minimal/sato/sdk build, this approach could save
>> about 20% disk space.
>> (minimal: 23G ->  19G, sato: 44G ->  35G, sdk: 54G ->  44G).
>>
>> Pull URL: git://git.pokylinux.org/poky-contrib.git
>>    Branch: dxu4/perf
>>    Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/perf
>>
>> Thanks,
>>      Dongxiao Xu<dongxiao.xu@intel.com>
>> ---
>>
>>
>> Dongxiao Xu (3):
>>    package.bbclass: Use hard link for package split instead of copy
>>    sstate.bbclass: Use hard link when staging files into sysroots
>>    staging.bbclass: use hardlink when staging files to sysroot-destdir
>>
>>   meta/classes/package.bbclass |    8 +++++---
>>   meta/classes/sstate.bbclass  |    2 +-
>>   meta/classes/staging.bbclass |   27 ++++++++++++++++++++++++++-
>>   meta/lib/oe/path.py          |   28 ++++++++++++++++++++++++++++
>>   4 files changed, 60 insertions(+), 5 deletions(-)
>>
>> _______________________________________________
>> poky mailing list
>> poky@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/poky
>>
>
>_______________________________________________
>poky mailing list
>poky@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/poky

Re: [PATCH 0/3][RFC] Poky disk space size optimization

[Xu, Dongxiao]

Thanks for your review.
Some comments from me, see inline.

Tian, Kevin wrote:
>> From: Saul Wold
>> Sent: Tuesday, December 07, 2010 10:47 AM
>> 
>> On 12/06/2010 04:50 PM, Dongxiao Xu wrote:
>>> This RFC aims to reduce the poky disk space size.
>>> 
>>> The optimization point is the duplicated file copies in WORKDIR and
>>> sysroot directories. When installing, files will be copied from
>>> source code package into the image folder; Then some files will be
>>> copied from image folder to sysroot-destdir, and some other files
>>> will be copied from image folder to package folder. While doing
>>> package split, some files will be copied from package folder to
>>> packages-split folder, etc. 
>>> 
>> There are a couple of concerns with this method, which may lead to
>> problems, first is that using links and modifying permissions, it's
>> possible that depending on the usage the permissions might be set
>> differently.  This will could lead to a second problem: psuedo. It's
>> unclear at this point how psuedo will handle this.

For normal files, the file name is a hard link to the disk storage. Therefore if we use hard link instead of copy, what we present to pseudo doesn't change.

For file permissions, could you give some example on permission change?
Per my understanding, file permissions are determined while do_install, which is beyond the following process. (package.bbclass will also do some file permission operations, but it is done before the hard link.)

hard link "image-->sysroot-destdir"
hard link "package-->packages-split"
hard link "sysroot-destdir-->sysroots".

After these hardlink is done, I am not aware there is file permission change. (Postinst is an exception, which is run in target system and not related with our build process).

>> 
>> I am sure you have been doing testing, just not sure of corner cases.
>> 
>> I know we accepted the first change for the package split link to the
>> deploy area, that one made sense as it was package level, this is
>> file level and may have hidden problems.

Actually this is also file level link. But it doesn't matter since package-->packages-split doesn't change any file.

> 
> I also agree with Saul here. There do have some difference among
> those directories for their own purpose, which finally result in some
> difference. For example, there're different tweaks on libtool and
> pkgconfig files between sysroot and packaging. That's why 'image' is
> copied to package and then sysroot-destdir, and then do necessary
> tweaks independently. Here if you want to hard link image to
> sysroot-destdir, you need make sure it done after the copy from image
> to package. Or else you'll end up to have problem there.       

For those ".la" files,
In "image-->sysroot-destdir" process, dependency_libs will be modified in image folder before hard link.
In "package-->packages-split", dependency_libs are the same and it is modified after "image-->package" copies is done.

No matter which runs first, they will modify dependency_libs for their own copy, since files within "sysroot-destdir" and "package" are NOT hardlink. They maintain separate copies.

I think there should be no problem.

Anyway we hope Richard could give us a comment in case we missed something.

Thanks,
Dongxiao

> 
> But if you can make sure that given a copy path you're modifying, the
> source and the destination are exactly same, and no other path refers
> to the source directory, perhaps it's fine to do hard link.  
> 
> Thanks
> Kevin
> 
>> 
>> I am sure that Richard will correct me and add his comments as he
>> recovers from his travels. 
>> 
>> Sau!
>> 
>>> The thought is to use hard link to replace the direct copy.
>>> The three commits do the work of:
>>> 
>>> a) package.bbclass: hard link "image-->sysroot-destdir"
>>> b) sstate.bbclass:  hard link "package-->packages-split"
>>> c) staging.bbclass: hard link "sysroot-destdir-->sysroots".
>>> 
>>> After testing with minimal/sato/sdk build, this approach could save
>>> about 20% disk space. (minimal: 23G ->  19G, sato: 44G ->  35G,
>>> sdk: 54G ->  44G). 
>>> 
>>> Pull URL: git://git.pokylinux.org/poky-contrib.git
>>>    Branch: dxu4/perf
>>>    Browse:
>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/perf
>>> 
>>> Thanks,
>>>      Dongxiao Xu<dongxiao.xu@intel.com>
>>> ---
>>> 
>>> 
>>> Dongxiao Xu (3):
>>>    package.bbclass: Use hard link for package split instead of copy
>>>    sstate.bbclass: Use hard link when staging files into sysroots
>>>    staging.bbclass: use hardlink when staging files to
>>> sysroot-destdir 
>>> 
>>>   meta/classes/package.bbclass |    8 +++++---
>>>   meta/classes/sstate.bbclass  |    2 +-
>>>   meta/classes/staging.bbclass |   27 ++++++++++++++++++++++++++-
>>>   meta/lib/oe/path.py          |   28 ++++++++++++++++++++++++++++
>>>   4 files changed, 60 insertions(+), 5 deletions(-)
>>> 
>>> _______________________________________________
>>> poky mailing list
>>> poky@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/poky
>>> 
>> 
>> _______________________________________________
>> poky mailing list
>> poky@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/poky