Kernel development using externalsrc

Kernel development using externalsrc

[CHMIELARZ Radoslaw]

[-- Attachment #1: Type: text/plain, Size: 1348 bytes --]

Hi all,

The company I work for uses yocto to build an image with a custom linux kernel. The documentation (https://www.yoctoproject.org/docs/2.5.1/kernel-dev/kernel-dev.html) suggests a setup where the kernel is under git and the kernel recipe specifies this git repository hence the workflow is the following:

  1.  Make changes in local kernel directory
  2.  Submit changes to git (and presumably push to shared space)
  3.  Run yocto build which fetches the changes

However since I don't want to push the changes to git server before I test locally I have added the kernel sources using externalsrc. Unfortunately this has the drawback that linux-libc-headers runs do_configure and do_install phase which invalidates glibc-initial and requires a recompilation of a couple of hundreds of packages. I would like to avoid it.

Thus my question is: how can I retain this setup where I only make local changes to the kernel without yocto rebuilding glibc and dependent packages?

To add more background we are using morty release and the changes in kernel will be in configuration and devices (probably as modules but some may be included with the kernel binary). Glibc is also an externalsrc project at the moment since I'm trying to upgrade to newer yocto release but this will be taken from git in the long run.

Cheers,
Radek

[-- Attachment #2: Type: text/html, Size: 6102 bytes --]

Re: Kernel development using externalsrc

[Khem Raj]

On Fri, Nov 2, 2018 at 8:47 AM CHMIELARZ Radoslaw
<radoslaw.chmielarz@leica-geosystems.com> wrote:
>
> Hi all,
>
>
>
> The company I work for uses yocto to build an image with a custom linux kernel. The documentation (https://www.yoctoproject.org/docs/2.5.1/kernel-dev/kernel-dev.html) suggests a setup where the kernel is under git and the kernel recipe specifies this git repository hence the workflow is the following:
>
> Make changes in local kernel directory
> Submit changes to git (and presumably push to shared space)
> Run yocto build which fetches the changes
>
>
>
> However since I don’t want to push the changes to git server before I test locally I have added the kernel sources using externalsrc. Unfortunately this has the drawback that linux-libc-headers runs do_configure and do_install phase which invalidates glibc-initial and requires a recompilation of a couple of hundreds of packages. I would like to avoid it.
>
>
>
> Thus my question is: how can I retain this setup where I only make local changes to the kernel without yocto rebuilding glibc and dependent packages?
>
>
>
> To add more background we are using morty release and the changes in kernel will be in configuration and devices (probably as modules but some may be included with the kernel binary). Glibc is also an externalsrc project at the moment since I’m trying to upgrade to newer yocto release but this will be taken from git in the long run.
>

may be you want to take a look at devtool workflow
https://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#using-devtool-in-your-workflow
>
>
> Cheers,
>
> Radek
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: Kernel development using externalsrc

[Uwe Geuder]

On Fri, Nov 2, 2018 at 5:47 PM CHMIELARZ Radoslaw radoslaw.chmielarz-at-leica-geosystems.com wrote:

> The company I work for uses yocto to build an image with a custom
> linux kernel. The documentation
> (https://www.yoctoproject.org/docs/2.5.1/kernel-dev/kernel-dev.html)
> suggests a setup where the kernel is under git and the kernel recipe
> specifies this git repository hence the workflow is the following:
>
> Make changes in local kernel directory
> Submit changes to git (and presumably push to shared space)
> Run yocto build which fetches the changes
>
> However since I don’t want to push the changes to git server before I
> test locally I have added the kernel sources using
> externalsrc. 

Devtool is of course what they recommend, but can always change the git
repo to a local one in a .bbappend, so you save the push step and Yocto
build saves fetching over the network. (Making a git commit for each
build attempt is very little overhead and I would recommend it anyway as
a good working practice. You can clean up using "git rebase -i" once you
are happy with your code and before you publish it.)

In one of my development branches I find

SRC_URI = "\
  git:///home/myname/projects/yoctobuild/pkgsrc/somerepo/;protocol=file;branch=debug/#675-v2.16.0 \
  file://some.patch \
  file://other.patch \
"

SRCREV = "${AUTOREV}"

I have not done it for the kernel, but I see no reason why it would not
work there.

(Obviously the absolute path is nothing you would be able to share
unmodified with others. Maybe it can be made a bit more portable using
${THISDIR}, but I have not had any need to try it.) 

> Unfortunately this has the drawback that
> linux-libc-headers runs do_configure and do_install phase which
> invalidates glibc-initial and requires a recompilation of a couple of
> hundreds of packages. I would like to avoid it.
>

How editing the kernel source would cause rebuilds trailing
linux-libc-haeaders I cannot see in my build here

$ bitbake-diffsigs -t linux-libc-headers configure
NOTE: Starting bitbake server...
ERROR: Only one matching sigdata file found for the specified task (linux-libc-headers do_configure)

Well, maybe I have never changed the kernel in this build area, so let's
check what it depends on:
 
$ bitbake-dumpsig -t linux-libc-headers configure
NOTE: Starting bitbake server...
ERROR: Metadata does not support finding signature data files

Hmm, no idea what that means. Until someone tells us what's wrong let's
use stamp files instead

$ bitbake-dumpsig tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_configure.sigdata...
This task depends on the checksums of files: []
Hash for dependent task
/home/geuder/projects/yoctobuild/meta-nel/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb.do_prepare_recipe_sysroot is 5ac6adcdd9edfe2705fd461a42942613

$ bitbake-dumpsig tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_prepare_recipe_sysroot.sigdata...
This task depends on the checksums of files: []
Hash for dependent task /home/geuder/projects/yoctobuild/meta-nel/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb.do_fetch is 362b694aea2202ce72ad259579d001ad

$ bitbake-dumpsig tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_fetch.sigdata...
Variable SRC_URI value is ${KERNELORG_MIRROR}/linux/kernel/v${HEADER_FETCH_VER}/linux-${PV}.tar.${KORG_ARCHIVE_COMPRESSION}
Tasks this task depends on: []
This task depends on the checksums of files: []

So my build gets kernel headers from kernel.org (mirror), it does not
use the kernel source I use to build my kernel. Maybe that's different
for you?

Of course the output of the bitbake-dumpsig commands is massively
shortened, I just picked those lines that seemed likely candidates to
cause rebuilds.  To get the whole truth in your build call...

$ bitbake-diffsigs -t linux-libc-headers configure

... after you made a small kernel edit and rebuilt.

Regards,

Uwe

Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
uwe.gexder at neuroeventlabs.com (Bot check: fix one obvious typo)

Re: Kernel development using externalsrc

[Dimitris Tassopoulos]

[-- Attachment #1: Type: text/plain, Size: 4984 bytes --]

Another thing you could do is to work with git branches and have a bbappend
file to set the branch and commit hash. You will still need to push the
changes to the remote git repo, but you don't have to merge to your master
branch.
Then if you don't like just drop the branch or cherry-pick only the commits
that are valuable to the master.

Regards,
Dimitris

On Sat, 3 Nov 2018, 15:25 Uwe Geuder <jrswdnan22@snkmail.com wrote:

> On Fri, Nov 2, 2018 at 5:47 PM CHMIELARZ Radoslaw
> radoslaw.chmielarz-at-leica-geosystems.com wrote:
>
> > The company I work for uses yocto to build an image with a custom
> > linux kernel. The documentation
> > (https://www.yoctoproject.org/docs/2.5.1/kernel-dev/kernel-dev.html)
> > suggests a setup where the kernel is under git and the kernel recipe
> > specifies this git repository hence the workflow is the following:
> >
> > Make changes in local kernel directory
> > Submit changes to git (and presumably push to shared space)
> > Run yocto build which fetches the changes
> >
> > However since I don’t want to push the changes to git server before I
> > test locally I have added the kernel sources using
> > externalsrc.
>
> Devtool is of course what they recommend, but can always change the git
> repo to a local one in a .bbappend, so you save the push step and Yocto
> build saves fetching over the network. (Making a git commit for each
> build attempt is very little overhead and I would recommend it anyway as
> a good working practice. You can clean up using "git rebase -i" once you
> are happy with your code and before you publish it.)
>
> In one of my development branches I find
>
> SRC_URI = "\
>
> git:///home/myname/projects/yoctobuild/pkgsrc/somerepo/;protocol=file;branch=debug/#675-v2.16.0
> \
>   file://some.patch \
>   file://other.patch \
> "
>
> SRCREV = "${AUTOREV}"
>
> I have not done it for the kernel, but I see no reason why it would not
> work there.
>
> (Obviously the absolute path is nothing you would be able to share
> unmodified with others. Maybe it can be made a bit more portable using
> ${THISDIR}, but I have not had any need to try it.)
>
> > Unfortunately this has the drawback that
> > linux-libc-headers runs do_configure and do_install phase which
> > invalidates glibc-initial and requires a recompilation of a couple of
> > hundreds of packages. I would like to avoid it.
> >
>
> How editing the kernel source would cause rebuilds trailing
> linux-libc-haeaders I cannot see in my build here
>
> $ bitbake-diffsigs -t linux-libc-headers configure
> NOTE: Starting bitbake server...
> ERROR: Only one matching sigdata file found for the specified task
> (linux-libc-headers do_configure)
>
> Well, maybe I have never changed the kernel in this build area, so let's
> check what it depends on:
>
> $ bitbake-dumpsig -t linux-libc-headers configure
> NOTE: Starting bitbake server...
> ERROR: Metadata does not support finding signature data files
>
> Hmm, no idea what that means. Until someone tells us what's wrong let's
> use stamp files instead
>
> $ bitbake-dumpsig
> tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_configure.sigdata...
> This task depends on the checksums of files: []
> Hash for dependent task
> /home/geuder/projects/yoctobuild/meta-nel/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb.do_prepare_recipe_sysroot
> is 5ac6adcdd9edfe2705fd461a42942613
>
> $ bitbake-dumpsig
> tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_prepare_recipe_sysroot.sigdata...
> This task depends on the checksums of files: []
> Hash for dependent task
> /home/geuder/projects/yoctobuild/meta-nel/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb.do_fetch
> is 362b694aea2202ce72ad259579d001ad
>
> $ bitbake-dumpsig
> tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_fetch.sigdata...
> Variable SRC_URI value is
> ${KERNELORG_MIRROR}/linux/kernel/v${HEADER_FETCH_VER}/linux-${PV}.tar.${KORG_ARCHIVE_COMPRESSION}
> Tasks this task depends on: []
> This task depends on the checksums of files: []
>
> So my build gets kernel headers from kernel.org (mirror), it does not
> use the kernel source I use to build my kernel. Maybe that's different
> for you?
>
> Of course the output of the bitbake-dumpsig commands is massively
> shortened, I just picked those lines that seemed likely candidates to
> cause rebuilds.  To get the whole truth in your build call...
>
> $ bitbake-diffsigs -t linux-libc-headers configure
>
> ... after you made a small kernel edit and rebuilt.
>
> Regards,
>
> Uwe
>
> Uwe Geuder
> Neuro Event Labs Oy
> Tampere, Finland
> uwe.gexder at neuroeventlabs.com (Bot check: fix one obvious typo)
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 6188 bytes --]