[PATCH 0/1] testsdk.bbclass: set PREMIRRORS for kernel to speed up test

[PATCH 0/1] testsdk.bbclass: set PREMIRRORS for kernel to speed up test

[Chen Qi]

The following changes since commit b6bc5b2840246ac0f0e5d8cebcdffc1a0ca2518f:

  bitbake: toaster: add 'Sumo' to release selection (2018-04-26 11:43:26 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib ChenQi/testsdkext-speed
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/testsdkext-spped

Chen Qi (1):
  testsdk.bbclass: set PREMIRRORS for kernel to speed up test

 meta/classes/testsdk.bbclass | 1 +
 1 file changed, 1 insertion(+)

-- 
1.9.1

[PATCH 1/1] testsdk.bbclass: set PREMIRRORS for kernel to speed up test

[Chen Qi]

Currently if we do a testsdkext task for the image, it would take
very long time to finish. The time is mostly spent fetching kernel
source via network.

We have done some configuration in auto.conf, trying to make use
of own-mirrors.bbclass to avoid fetching kernel via network.

However, the solution normally does not work. Below is some log
from log.do_fetch.

  DEBUG: Fetcher failure: Unable to find file \
  file:///path/to/downloads/git2_git.yoctoproject.org.linux-yocto.git.tar.gz \
  anywhere.

The tar.gz file is not available. It is generated only if
BB_GENERATE_MIRROR_TARBALLS is set to "1". The default value of
BB_GENERATE_MIRROR_TARBALLS is "0", and according to the manual, users
need choose to set it to "1" only if they are trying to make a source
mirror. So generally, this var's value is "0".

Anyway, we do need to avoid fetching kernel source from network when
doing testsdkext. So set PREMIRRORS in auto.conf to achieve this.

After this change, the time reduces from 4209.131s to 1399.436s on
my local machine.

[YOCTO #12729]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/testsdk.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 2e43343..853c6fd 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -157,6 +157,7 @@ def testsdkext_main(d):
             f.write('SSTATE_MIRRORS += " \\n file://.* file://%s/PATH"\n' % test_data.get('SSTATE_DIR'))
             f.write('SOURCE_MIRROR_URL = "file://%s"\n' % test_data.get('DL_DIR'))
             f.write('INHERIT += "own-mirrors"\n')
+            f.write('PREMIRRORS_prepend = " git://git.yoctoproject.org/.* git://%s/git2/git.yoctoproject.org.BASENAME;protocal=file \\n "\n' % test_data.get('DL_DIR'))
 
         # We need to do this in case we have a minimal SDK
         subprocess.check_output(". %s > /dev/null; devtool sdk-install meta-extsdk-toolchain" % \
-- 
1.9.1

Re: [PATCH 1/1] testsdk.bbclass: set PREMIRRORS for kernel to speed up test

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

On 3 May 2018 at 09:52, Chen Qi <Qi.Chen@windriver.com> wrote:

> +            f.write('PREMIRRORS_prepend = " git://git.yoctoproject.org/.*
> git://%s/git2/git.yoctoproject.org.BASENAME;protocal=file \\n "\n' %
> test_data.get('DL_DIR'))
>

protocal?

We could turn on BB_GENERATE_SOURCE_MIRRORS by default.  What's the
overhead?

Ross

[-- Attachment #2: Type: text/html, Size: 821 bytes --]

Re: [PATCH 1/1] testsdk.bbclass: set PREMIRRORS for kernel to speed up test

[ChenQi]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]

On 05/03/2018 10:38 PM, Burton, Ross wrote:
> On 3 May 2018 at 09:52, Chen Qi <Qi.Chen@windriver.com 
> <mailto:Qi.Chen@windriver.com>> wrote:
>
>     +     f.write('PREMIRRORS_prepend = "
>     git://git.yoctoproject.org/.* <http://git.yoctoproject.org/.*>
>     git://%s/git2/git.yoctoproject.org.BASENAME;protocal=file \\n "\n'
>     % test_data.get('DL_DIR'))
>
>
> protocal?
>
> We could turn on BB_GENERATE_SOURCE_MIRRORS by default.  What's the 
> overhead?
>
> Ross

Hi Ross,

I didn't test the overhead.

The reference manual says that this variable is only useful when users 
want to generate a source mirror.

Some related info from bitbake/Changelog
"""
         - Add BB_GENERATE_MIRROR_TARBALLS option, set to 0 to make git 
fetches
           faster at the expense of not creating mirror tarballs.
"""

Besides, testsdkext is a task to verify correctness. It does not affect 
the generated eSDK that the users actually use. I'm not sure if I should 
change some default value merely to speed it up. I'd rather do some 
tweaks to this task instead of doing some change which could affect the 
building results.

That's why I chose not to change its default value.

I've removed the 'protocol=file' and sent out V2.

Best Regards,
Chen Qi



[-- Attachment #2: Type: text/html, Size: 2705 bytes --]