* [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed @ 2013-11-08 15:18 Richard Purdie 2013-11-08 15:41 ` Bruce Ashfield 0 siblings, 1 reply; 13+ messages in thread From: Richard Purdie @ 2013-11-08 15:18 UTC (permalink / raw) To: openembedded-core; +Cc: Hart, Darren The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python. This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around. For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass index 0b6360f..e793ccb 100644 --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -308,8 +308,8 @@ kernelscripts_sstate_postinst () { fi } -sysroot_stage_all_append() { - sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} +python sysroot_stage_all () { + oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"), d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}")) } kernel_do_configure() { ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-08 15:18 [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed Richard Purdie @ 2013-11-08 15:41 ` Bruce Ashfield 2013-11-08 15:54 ` Hans Beckérus 2013-11-08 15:55 ` Richard Purdie 0 siblings, 2 replies; 13+ messages in thread From: Bruce Ashfield @ 2013-11-08 15:41 UTC (permalink / raw) To: Richard Purdie, openembedded-core; +Cc: Hart, Darren On 13-11-08 10:18 AM, Richard Purdie wrote: > The kernel tree is large and doesn't need to be copied. Override > the default sysroot handling function to use a hardlink copying > function in python. > > This commit also drops the copying of the /lib directory which > just contains the kernel modules. We never use those in the sysroot > so there is little point in carrying those around. > > For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found! Bruce > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass > index 0b6360f..e793ccb 100644 > --- a/meta/classes/kernel.bbclass > +++ b/meta/classes/kernel.bbclass > @@ -308,8 +308,8 @@ kernelscripts_sstate_postinst () { > fi > } > > -sysroot_stage_all_append() { > - sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} > +python sysroot_stage_all () { > + oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"), d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}")) > } > > kernel_do_configure() { > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-08 15:41 ` Bruce Ashfield @ 2013-11-08 15:54 ` Hans Beckérus 2013-11-08 15:55 ` Richard Purdie 1 sibling, 0 replies; 13+ messages in thread From: Hans Beckérus @ 2013-11-08 15:54 UTC (permalink / raw) To: Bruce Ashfield; +Cc: Hart, Darren, openembedded-core On Fri, Nov 8, 2013 at 4:41 PM, Bruce Ashfield <bruce.ashfield@windriver.com> wrote: > On 13-11-08 10:18 AM, Richard Purdie wrote: >> >> The kernel tree is large and doesn't need to be copied. Override >> the default sysroot handling function to use a hardlink copying >> function in python. >> >> This commit also drops the copying of the /lib directory which >> just contains the kernel modules. We never use those in the sysroot >> so there is little point in carrying those around. >> >> For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. > > > Fantastic. One less thing for me to dig into later. I thought this > was already in place, so I'm pleasantly surprised that there was a > time savings to be found! > +1 Looks really promising. Trying them out now in my Yocto dora branch. Btw, would it make sense to group these into a patch set and then number them appropriately? I realized this when I was trying to apply them since I had to take some care not mixing them up. Thanks. Hans > Bruce > > >> >> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> >> --- >> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass >> index 0b6360f..e793ccb 100644 >> --- a/meta/classes/kernel.bbclass >> +++ b/meta/classes/kernel.bbclass >> @@ -308,8 +308,8 @@ kernelscripts_sstate_postinst () { >> fi >> } >> >> -sysroot_stage_all_append() { >> - sysroot_stage_dir ${D}${KERNEL_SRC_PATH} >> ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} >> +python sysroot_stage_all () { >> + oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"), >> d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}")) >> } >> >> kernel_do_configure() { >> >> > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-08 15:41 ` Bruce Ashfield 2013-11-08 15:54 ` Hans Beckérus @ 2013-11-08 15:55 ` Richard Purdie 2013-11-08 15:59 ` Bruce Ashfield 1 sibling, 1 reply; 13+ messages in thread From: Richard Purdie @ 2013-11-08 15:55 UTC (permalink / raw) To: Bruce Ashfield; +Cc: Hart, Darren, openembedded-core On Fri, 2013-11-08 at 10:41 -0500, Bruce Ashfield wrote: > On 13-11-08 10:18 AM, Richard Purdie wrote: > > The kernel tree is large and doesn't need to be copied. Override > > the default sysroot handling function to use a hardlink copying > > function in python. > > > > This commit also drops the copying of the /lib directory which > > just contains the kernel modules. We never use those in the sysroot > > so there is little point in carrying those around. > > > > For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. > > Fantastic. One less thing for me to dig into later. I thought this > was already in place, so I'm pleasantly surprised that there was a > time savings to be found! I was somewhat surprised too. We still need to optimise what we install in do_install since that is where significant gains can still be made. Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-08 15:55 ` Richard Purdie @ 2013-11-08 15:59 ` Bruce Ashfield 2013-11-08 16:50 ` Richard Purdie 0 siblings, 1 reply; 13+ messages in thread From: Bruce Ashfield @ 2013-11-08 15:59 UTC (permalink / raw) To: Richard Purdie; +Cc: Hart, Darren, openembedded-core On 13-11-08 10:55 AM, Richard Purdie wrote: > On Fri, 2013-11-08 at 10:41 -0500, Bruce Ashfield wrote: >> On 13-11-08 10:18 AM, Richard Purdie wrote: >>> The kernel tree is large and doesn't need to be copied. Override >>> the default sysroot handling function to use a hardlink copying >>> function in python. >>> >>> This commit also drops the copying of the /lib directory which >>> just contains the kernel modules. We never use those in the sysroot >>> so there is little point in carrying those around. >>> >>> For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. >> >> Fantastic. One less thing for me to dig into later. I thought this >> was already in place, so I'm pleasantly surprised that there was a >> time savings to be found! > > I was somewhat surprised too. > > We still need to optimise what we install in do_install since that is > where significant gains can still be made. Agreed. I started some changes in that area right after ELC-e, I'll try and get them out sooner rather than later. Bruce > > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-08 15:59 ` Bruce Ashfield @ 2013-11-08 16:50 ` Richard Purdie 2013-11-08 17:23 ` Hart, Darren 0 siblings, 1 reply; 13+ messages in thread From: Richard Purdie @ 2013-11-08 16:50 UTC (permalink / raw) To: Bruce Ashfield; +Cc: Hart, Darren, openembedded-core On Fri, 2013-11-08 at 10:59 -0500, Bruce Ashfield wrote: > On 13-11-08 10:55 AM, Richard Purdie wrote: > > On Fri, 2013-11-08 at 10:41 -0500, Bruce Ashfield wrote: > >> On 13-11-08 10:18 AM, Richard Purdie wrote: > >>> The kernel tree is large and doesn't need to be copied. Override > >>> the default sysroot handling function to use a hardlink copying > >>> function in python. > >>> > >>> This commit also drops the copying of the /lib directory which > >>> just contains the kernel modules. We never use those in the sysroot > >>> so there is little point in carrying those around. > >>> > >>> For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. > >> > >> Fantastic. One less thing for me to dig into later. I thought this > >> was already in place, so I'm pleasantly surprised that there was a > >> time savings to be found! > > > > I was somewhat surprised too. > > > > We still need to optimise what we install in do_install since that is > > where significant gains can still be made. > > Agreed. I started some changes in that area right after ELC-e, I'll > try and get them out sooner rather than later. I thought I'd share this for people's interest: http://dan.rpsys.net/kernelbuildissue.png Its the output from pybootchart of a bitbake core-image-sato from scratch. I've zoomed out to put some bars in particular into perspective. The pink colour is linux-yocto:do_install, the cyan is linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm. So the final thing to build is the kernel by quite some margin, its holding the rest of the build up. Hopefully these patches start to improve that a bit! Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-08 16:50 ` Richard Purdie @ 2013-11-08 17:23 ` Hart, Darren 2013-11-09 20:38 ` Andrea Adami 0 siblings, 1 reply; 13+ messages in thread From: Hart, Darren @ 2013-11-08 17:23 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Fri, 2013-11-08 at 16:50 +0000, Richard Purdie wrote: > On Fri, 2013-11-08 at 10:59 -0500, Bruce Ashfield wrote: > > On 13-11-08 10:55 AM, Richard Purdie wrote: > > > On Fri, 2013-11-08 at 10:41 -0500, Bruce Ashfield wrote: > > >> On 13-11-08 10:18 AM, Richard Purdie wrote: > > >>> The kernel tree is large and doesn't need to be copied. Override > > >>> the default sysroot handling function to use a hardlink copying > > >>> function in python. > > >>> > > >>> This commit also drops the copying of the /lib directory which > > >>> just contains the kernel modules. We never use those in the sysroot > > >>> so there is little point in carrying those around. > > >>> > > >>> For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. > > >> > > >> Fantastic. One less thing for me to dig into later. I thought this > > >> was already in place, so I'm pleasantly surprised that there was a > > >> time savings to be found! > > > > > > I was somewhat surprised too. > > > > > > We still need to optimise what we install in do_install since that is > > > where significant gains can still be made. > > > > Agreed. I started some changes in that area right after ELC-e, I'll > > try and get them out sooner rather than later. > > I thought I'd share this for people's interest: > > http://dan.rpsys.net/kernelbuildissue.png > > Its the output from pybootchart of a bitbake core-image-sato from > scratch. I've zoomed out to put some bars in particular into > perspective. > > The pink colour is linux-yocto:do_install, the cyan is > linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. > The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm. > > So the final thing to build is the kernel by quite some margin, its > holding the rest of the build up. > > Hopefully these patches start to improve that a bit! > Yup, linux-yocto do_package is the biggest build-time hindrance for me and something we've wanted to look into for a long time. Great to see some progress. -- Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-08 17:23 ` Hart, Darren @ 2013-11-09 20:38 ` Andrea Adami 2013-11-09 22:58 ` Richard Purdie 0 siblings, 1 reply; 13+ messages in thread From: Andrea Adami @ 2013-11-09 20:38 UTC (permalink / raw) To: Hart, Darren; +Cc: openembedded-core On Fri, Nov 8, 2013 at 6:23 PM, Hart, Darren <darren.hart@intel.com> wrote: > On Fri, 2013-11-08 at 16:50 +0000, Richard Purdie wrote: >> On Fri, 2013-11-08 at 10:59 -0500, Bruce Ashfield wrote: >> > On 13-11-08 10:55 AM, Richard Purdie wrote: >> > > On Fri, 2013-11-08 at 10:41 -0500, Bruce Ashfield wrote: >> > >> On 13-11-08 10:18 AM, Richard Purdie wrote: >> > >>> The kernel tree is large and doesn't need to be copied. Override >> > >>> the default sysroot handling function to use a hardlink copying >> > >>> function in python. >> > >>> >> > >>> This commit also drops the copying of the /lib directory which >> > >>> just contains the kernel modules. We never use those in the sysroot >> > >>> so there is little point in carrying those around. >> > >>> >> > >>> For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. >> > >> >> > >> Fantastic. One less thing for me to dig into later. I thought this >> > >> was already in place, so I'm pleasantly surprised that there was a >> > >> time savings to be found! >> > > >> > > I was somewhat surprised too. >> > > >> > > We still need to optimise what we install in do_install since that is >> > > where significant gains can still be made. >> > >> > Agreed. I started some changes in that area right after ELC-e, I'll >> > try and get them out sooner rather than later. >> >> I thought I'd share this for people's interest: >> >> http://dan.rpsys.net/kernelbuildissue.png >> >> Its the output from pybootchart of a bitbake core-image-sato from >> scratch. I've zoomed out to put some bars in particular into >> perspective. >> >> The pink colour is linux-yocto:do_install, the cyan is >> linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. >> The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm. >> >> So the final thing to build is the kernel by quite some margin, its >> holding the rest of the build up. >> >> Hopefully these patches start to improve that a bit! >> > > Yup, linux-yocto do_package is the biggest build-time hindrance for me > and something we've wanted to look into for a long time. Great to see > some progress. > > -- > Darren Hart > Intel Open Source Technology Center > Yocto Project - Linux Kernel > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core On my old quad core: Before: 95.24 seconds After: 83.25 seconds Thanks Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-09 20:38 ` Andrea Adami @ 2013-11-09 22:58 ` Richard Purdie 2013-11-11 8:06 ` Hans Beckérus 0 siblings, 1 reply; 13+ messages in thread From: Richard Purdie @ 2013-11-09 22:58 UTC (permalink / raw) To: Andrea Adami; +Cc: Hart, Darren, openembedded-core On Sat, 2013-11-09 at 21:38 +0100, Andrea Adami wrote: > On Fri, Nov 8, 2013 at 6:23 PM, Hart, Darren <darren.hart@intel.com> wrote: > > On Fri, 2013-11-08 at 16:50 +0000, Richard Purdie wrote: > >> On Fri, 2013-11-08 at 10:59 -0500, Bruce Ashfield wrote: > >> > On 13-11-08 10:55 AM, Richard Purdie wrote: > >> > > On Fri, 2013-11-08 at 10:41 -0500, Bruce Ashfield wrote: > >> > >> On 13-11-08 10:18 AM, Richard Purdie wrote: > >> > >>> The kernel tree is large and doesn't need to be copied. Override > >> > >>> the default sysroot handling function to use a hardlink copying > >> > >>> function in python. > >> > >>> > >> > >>> This commit also drops the copying of the /lib directory which > >> > >>> just contains the kernel modules. We never use those in the sysroot > >> > >>> so there is little point in carrying those around. > >> > >>> > >> > >>> For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. > >> > >> > >> > >> Fantastic. One less thing for me to dig into later. I thought this > >> > >> was already in place, so I'm pleasantly surprised that there was a > >> > >> time savings to be found! > >> > > > >> > > I was somewhat surprised too. > >> > > > >> > > We still need to optimise what we install in do_install since that is > >> > > where significant gains can still be made. > >> > > >> > Agreed. I started some changes in that area right after ELC-e, I'll > >> > try and get them out sooner rather than later. > >> > >> I thought I'd share this for people's interest: > >> > >> http://dan.rpsys.net/kernelbuildissue.png > >> > >> Its the output from pybootchart of a bitbake core-image-sato from > >> scratch. I've zoomed out to put some bars in particular into > >> perspective. > >> > >> The pink colour is linux-yocto:do_install, the cyan is > >> linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. > >> The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm. > >> > >> So the final thing to build is the kernel by quite some margin, its > >> holding the rest of the build up. > >> > >> Hopefully these patches start to improve that a bit! > >> > > > > Yup, linux-yocto do_package is the biggest build-time hindrance for me > > and something we've wanted to look into for a long time. Great to see > > some progress. > > > > -- > > Darren Hart > > Intel Open Source Technology Center > > Yocto Project - Linux Kernel > > > > _______________________________________________ > > Openembedded-core mailing list > > Openembedded-core@lists.openembedded.org > > http://lists.openembedded.org/mailman/listinfo/openembedded-core > > On my old quad core: > > Before: 95.24 seconds > After: 83.25 seconds We have some "standard" tests we use for comparison purposes as detailed on: https://wiki.yoctoproject.org/wiki/Performance_Test Some numbers from master and a test branch containing the recent speedups: fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:37,12:05.50,1:17:53,4:02.68,0:31.71,0:15.70,0:01.61,26347744,5035816 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:59,12:09.26,1:17:29,4:06.75,0:31.82,0:15.69,0:01.70,26348556,5035876 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:21,12:07.23,1:18:21,4:07.44,0:31.79,0:15.63,0:01.62,26351088,5035908 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:23:16,12:15.33,1:18:00,4:02.50,0:31.87,0:15.74,0:01.63,26348132,5035848 fedora19,rpurdie/timing:9d8c0ef3349936b3bd0bbf485b50cf81f4feaf80,poky-10.0.0.final-373-g9d8c0ef,1:19:42,10:28.53,1:15:30,4:12.24,0:31.30,0:15.69,0:01.66,26137344,5055628 fedora19,rpurdie/timing2:eb2221d6247c12322ac38b9ac6f24c9de5877e53,poky-10.0.0.final-401-geb2221d,1:18:51,10:44.26,1:16:32,4:13.47,0:31.27,0:15.69,0:01.65,26137828,5055916 So we have 1:23m -> 1:19m for overall buildtime and 12.0m -> 10.5m for "bitbake virtual/kernel" from cleansstate. Not too bad :) Thanks to Stefan for running the tests! Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-09 22:58 ` Richard Purdie @ 2013-11-11 8:06 ` Hans Beckérus 2013-11-11 9:35 ` Richard Purdie 2013-11-11 9:45 ` Richard Purdie 0 siblings, 2 replies; 13+ messages in thread From: Hans Beckérus @ 2013-11-11 8:06 UTC (permalink / raw) To: Richard Purdie; +Cc: Hart, Darren, openembedded-core On Sat, Nov 9, 2013 at 11:58 PM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > On Sat, 2013-11-09 at 21:38 +0100, Andrea Adami wrote: >> On Fri, Nov 8, 2013 at 6:23 PM, Hart, Darren <darren.hart@intel.com> wrote: >> > On Fri, 2013-11-08 at 16:50 +0000, Richard Purdie wrote: >> >> On Fri, 2013-11-08 at 10:59 -0500, Bruce Ashfield wrote: >> >> > On 13-11-08 10:55 AM, Richard Purdie wrote: >> >> > > On Fri, 2013-11-08 at 10:41 -0500, Bruce Ashfield wrote: >> >> > >> On 13-11-08 10:18 AM, Richard Purdie wrote: >> >> > >>> The kernel tree is large and doesn't need to be copied. Override >> >> > >>> the default sysroot handling function to use a hardlink copying >> >> > >>> function in python. >> >> > >>> >> >> > >>> This commit also drops the copying of the /lib directory which >> >> > >>> just contains the kernel modules. We never use those in the sysroot >> >> > >>> so there is little point in carrying those around. >> >> > >>> >> >> > >>> For linux-yocto this takes the do_populate_sysroot time 24s -> 14s. >> >> > >> >> >> > >> Fantastic. One less thing for me to dig into later. I thought this >> >> > >> was already in place, so I'm pleasantly surprised that there was a >> >> > >> time savings to be found! >> >> > > >> >> > > I was somewhat surprised too. >> >> > > >> >> > > We still need to optimise what we install in do_install since that is >> >> > > where significant gains can still be made. >> >> > >> >> > Agreed. I started some changes in that area right after ELC-e, I'll >> >> > try and get them out sooner rather than later. >> >> >> >> I thought I'd share this for people's interest: >> >> >> >> http://dan.rpsys.net/kernelbuildissue.png >> >> >> >> Its the output from pybootchart of a bitbake core-image-sato from >> >> scratch. I've zoomed out to put some bars in particular into >> >> perspective. >> >> >> >> The pink colour is linux-yocto:do_install, the cyan is >> >> linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. >> >> The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm. >> >> >> >> So the final thing to build is the kernel by quite some margin, its >> >> holding the rest of the build up. >> >> >> >> Hopefully these patches start to improve that a bit! >> >> >> > >> > Yup, linux-yocto do_package is the biggest build-time hindrance for me >> > and something we've wanted to look into for a long time. Great to see >> > some progress. >> > >> > -- >> > Darren Hart >> > Intel Open Source Technology Center >> > Yocto Project - Linux Kernel >> > >> > _______________________________________________ >> > Openembedded-core mailing list >> > Openembedded-core@lists.openembedded.org >> > http://lists.openembedded.org/mailman/listinfo/openembedded-core >> >> On my old quad core: >> >> Before: 95.24 seconds >> After: 83.25 seconds > > We have some "standard" tests we use for comparison purposes as detailed > on: > > https://wiki.yoctoproject.org/wiki/Performance_Test > > Some numbers from master and a test branch containing the recent > speedups: > > fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:37,12:05.50,1:17:53,4:02.68,0:31.71,0:15.70,0:01.61,26347744,5035816 > fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:59,12:09.26,1:17:29,4:06.75,0:31.82,0:15.69,0:01.70,26348556,5035876 > fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:21,12:07.23,1:18:21,4:07.44,0:31.79,0:15.63,0:01.62,26351088,5035908 > fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:23:16,12:15.33,1:18:00,4:02.50,0:31.87,0:15.74,0:01.63,26348132,5035848 > fedora19,rpurdie/timing:9d8c0ef3349936b3bd0bbf485b50cf81f4feaf80,poky-10.0.0.final-373-g9d8c0ef,1:19:42,10:28.53,1:15:30,4:12.24,0:31.30,0:15.69,0:01.66,26137344,5055628 > fedora19,rpurdie/timing2:eb2221d6247c12322ac38b9ac6f24c9de5877e53,poky-10.0.0.final-401-geb2221d,1:18:51,10:44.26,1:16:32,4:13.47,0:31.27,0:15.69,0:01.65,26137828,5055916 > > So we have 1:23m -> 1:19m for overall buildtime and 12.0m -> 10.5m for > "bitbake virtual/kernel" from cleansstate. Not too bad :) > > Thanks to Stefan for running the tests! > I am having some trouble with these patches :( The new approach is using the '-n' flag to the 'cp' command. That is not supported on our SuSE11 based system. To be honest, I do not know how portable '-n' is? Some system have it, others seems to provide '-u' instead. I guess by removing '-n' part of the performance gain is lost? Would it be possible to test for error from the 'cp' command and if it fails try '-u' instead (and cache the result)? Or maybe even better, make this configurable in local.conf (or the distro) for the build platforms that does not support '-n' but can instead fall-back to using '-u'. From what I can tell most systems support one or the other, but never both. Thanks. Hans > Cheers, > > Richard > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-11 8:06 ` Hans Beckérus @ 2013-11-11 9:35 ` Richard Purdie 2013-11-11 9:45 ` Richard Purdie 1 sibling, 0 replies; 13+ messages in thread From: Richard Purdie @ 2013-11-11 9:35 UTC (permalink / raw) To: Hans Beckérus; +Cc: Hart, Darren, openembedded-core On Mon, 2013-11-11 at 09:06 +0100, Hans Beckérus wrote: > I am having some trouble with these patches :( The new approach is > using the '-n' flag to the 'cp' command. > That is not supported on our SuSE11 based system. To be honest, I do > not know how portable '-n' is? > Some system have it, others seems to provide '-u' instead. I guess by > removing '-n' part of the performance gain is lost? > Would it be possible to test for error from the 'cp' command and if it > fails try '-u' instead (and cache the result)? Or maybe > even better, make this configurable in local.conf (or the distro) for > the build platforms that does not support '-n' but can > instead fall-back to using '-u'. From what I can tell most systems > support one or the other, but never both. Hmm, its been in coreutils since 2009 which I guess isn't that long in the scheme of things: http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/cp.c?id=d01338eb3d30e5634f1b4d4179c229f54eea0b44 Just to double check, is cp on your system provided by coreutils or something else? Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-11 8:06 ` Hans Beckérus 2013-11-11 9:35 ` Richard Purdie @ 2013-11-11 9:45 ` Richard Purdie 2013-11-11 9:47 ` Hans Beckérus 1 sibling, 1 reply; 13+ messages in thread From: Richard Purdie @ 2013-11-11 9:45 UTC (permalink / raw) To: Hans Beckérus; +Cc: Hart, Darren, openembedded-core On Mon, 2013-11-11 at 09:06 +0100, Hans Beckérus wrote: > I am having some trouble with these patches :( The new approach is > using the '-n' flag to the 'cp' command. > That is not supported on our SuSE11 based system. To be honest, I do > not know how portable '-n' is? > Some system have it, others seems to provide '-u' instead. I guess by > removing '-n' part of the performance gain is lost? > Would it be possible to test for error from the 'cp' command and if it > fails try '-u' instead (and cache the result)? Or maybe > even better, make this configurable in local.conf (or the distro) for > the build platforms that does not support '-n' but can > instead fall-back to using '-u'. From what I can tell most systems > support one or the other, but never both. http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2&id=276b9df3588ecd438c3e2f502d56cd6b9fb24676 Thankfully the above works just as well. I used -n to get rid of an error, then added -l for efficiency. We can get away just with -l. Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed 2013-11-11 9:45 ` Richard Purdie @ 2013-11-11 9:47 ` Hans Beckérus 0 siblings, 0 replies; 13+ messages in thread From: Hans Beckérus @ 2013-11-11 9:47 UTC (permalink / raw) To: Richard Purdie; +Cc: Hart, Darren, openembedded-core On Mon, Nov 11, 2013 at 10:45 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > On Mon, 2013-11-11 at 09:06 +0100, Hans Beckérus wrote: >> I am having some trouble with these patches :( The new approach is >> using the '-n' flag to the 'cp' command. >> That is not supported on our SuSE11 based system. To be honest, I do >> not know how portable '-n' is? >> Some system have it, others seems to provide '-u' instead. I guess by >> removing '-n' part of the performance gain is lost? >> Would it be possible to test for error from the 'cp' command and if it >> fails try '-u' instead (and cache the result)? Or maybe >> even better, make this configurable in local.conf (or the distro) for >> the build platforms that does not support '-n' but can >> instead fall-back to using '-u'. From what I can tell most systems >> support one or the other, but never both. > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2&id=276b9df3588ecd438c3e2f502d56cd6b9fb24676 > > Thankfully the above works just as well. I used -n to get rid of an > error, then added -l for efficiency. We can get away just with -l. > Great! Will try it asap. Btw, cp was from coreutils back in 2008. cp (GNU coreutils) 6.12 Copyright (C) 2008 Free Software Foundation, Inc. Thanks. Hans > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2013-11-11 9:47 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-11-08 15:18 [PATCH] kernel: Use hardlinks for do_populate_sysroot for speed Richard Purdie 2013-11-08 15:41 ` Bruce Ashfield 2013-11-08 15:54 ` Hans Beckérus 2013-11-08 15:55 ` Richard Purdie 2013-11-08 15:59 ` Bruce Ashfield 2013-11-08 16:50 ` Richard Purdie 2013-11-08 17:23 ` Hart, Darren 2013-11-09 20:38 ` Andrea Adami 2013-11-09 22:58 ` Richard Purdie 2013-11-11 8:06 ` Hans Beckérus 2013-11-11 9:35 ` Richard Purdie 2013-11-11 9:45 ` Richard Purdie 2013-11-11 9:47 ` Hans Beckérus
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.
