Recompiles in Yocto/Poky

Recompiles in Yocto/Poky

[Keskinarkaus, Teemu]

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

Hi,

I have Yocto based system for iMX6. First Yocto 1.7 was used and system implemented with that. Later Yocto (and meta-layers) were updated to 2.1.

We have several libraries/applications/modules that use externalsrc ie. sources sources are not fetched from git repository, but they are locally available.

I noticed with Yocto 2.1 that if I recreate image without changing anything, all those externalsrc recipes get recompiled/created. This didn't happened with Yocto 1.7. I checked the recipes and they are the same.

Why are those recipes that use externalsrc - feature recompiled every time? In Yocto 1.7 if you try to re-create image without changing anything, nothing happen, but in Yocto 2.1 you do the same all those externalsrc recipes get recompiled and image file also recreated.

Teemu Keskinarkaus
Software system engineer
Cell: +358 400 330047
Fax: +358 207 669199
www.maximatecc.com<http://www.maximatecc.com/>
maximatecc
making machines smart, safe and productive


________________________________

Actuant Corporation Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.

[-- Attachment #2: Type: text/html, Size: 6219 bytes --]

Re: Recompiles in Yocto/Poky

[Peter Kjellerstedt]

[-- Attachment #1: Type: text/plain, Size: 2673 bytes --]

This is a feature introduced to complement the main use of externalsrc, which is together with devtool. When you use "devtool modify -x <recipe>", it fetches the sources as specified in the recipe and makes them available locally. At the same time it sets up a bbappend in the workspace layer which uses externalsrc to use those sources when building the recipe. This is intended to make it easy to use BitBake to actually work on and develop the sources. However, for this to work in practice, BitBake must rebuild the package if there are any changes in the sources, but since BitBake normally only looks at the meta data to determine if a recipe task needs to run, there is nothing telling it that the sources have changed. Therefore changes were made so that code specified using externalsrc is always rebuilt.

//Peter

From: poky-bounces@yoctoproject.org [mailto:poky-bounces@yoctoproject.org] On Behalf Of Keskinarkaus, Teemu
Sent: den 2 juni 2016 06:28
To: poky@yoctoproject.org
Subject: [poky] Recompiles in Yocto/Poky

Hi,

I have Yocto based system for iMX6. First Yocto 1.7 was used and system implemented with that. Later Yocto (and meta-layers) were updated to 2.1.

We have several libraries/applications/modules that use externalsrc ie. sources sources are not fetched from git repository, but they are locally available.

I noticed with Yocto 2.1 that if I recreate image without changing anything, all those externalsrc recipes get recompiled/created. This didn't happened with Yocto 1.7. I checked the recipes and they are the same.

Why are those recipes that use externalsrc - feature recompiled every time? In Yocto 1.7 if you try to re-create image without changing anything, nothing happen, but in Yocto 2.1 you do the same all those externalsrc recipes get recompiled and image file also recreated.

Teemu Keskinarkaus
Software system engineer
Cell: +358 400 330047
Fax: +358 207 669199
www.maximatecc.com<http://www.maximatecc.com/>
maximatecc
making machines smart, safe and productive


________________________________

Actuant Corporation Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.

[-- Attachment #2: Type: text/html, Size: 8255 bytes --]

Re: Recompiles in Yocto/Poky

[Keskinarkaus, Teemu]

[-- Attachment #1: Type: text/plain, Size: 3763 bytes --]

Thanks for the clarification. Is there an option that can disable this feature or is it 'hardcoded' into the system? Not so much of a critical issue but annoyance.

Teemu Keskinarkaus
Software system engineer
maximatecc
making machines smart, safe and productive

From: Peter Kjellerstedt [mailto:peter.kjellerstedt@axis.com]
Sent: 3. kesäkuuta 2016 10:00
To: Keskinarkaus, Teemu
Cc: poky@yoctoproject.org
Subject: RE: Recompiles in Yocto/Poky

This is a feature introduced to complement the main use of externalsrc, which is together with devtool. When you use "devtool modify -x <recipe>", it fetches the sources as specified in the recipe and makes them available locally. At the same time it sets up a bbappend in the workspace layer which uses externalsrc to use those sources when building the recipe. This is intended to make it easy to use BitBake to actually work on and develop the sources. However, for this to work in practice, BitBake must rebuild the package if there are any changes in the sources, but since BitBake normally only looks at the meta data to determine if a recipe task needs to run, there is nothing telling it that the sources have changed. Therefore changes were made so that code specified using externalsrc is always rebuilt.

//Peter

From: poky-bounces@yoctoproject.org [mailto:poky-bounces@yoctoproject.org] On Behalf Of Keskinarkaus, Teemu
Sent: den 2 juni 2016 06:28
To: poky@yoctoproject.org
Subject: [poky] Recompiles in Yocto/Poky

Hi,

I have Yocto based system for iMX6. First Yocto 1.7 was used and system implemented with that. Later Yocto (and meta-layers) were updated to 2.1.

We have several libraries/applications/modules that use externalsrc ie. sources sources are not fetched from git repository, but they are locally available.

I noticed with Yocto 2.1 that if I recreate image without changing anything, all those externalsrc recipes get recompiled/created. This didn't happened with Yocto 1.7. I checked the recipes and they are the same.

Why are those recipes that use externalsrc - feature recompiled every time? In Yocto 1.7 if you try to re-create image without changing anything, nothing happen, but in Yocto 2.1 you do the same all those externalsrc recipes get recompiled and image file also recreated.

Teemu Keskinarkaus
Software system engineer
Cell: +358 400 330047
Fax: +358 207 669199
www.maximatecc.com<http://www.maximatecc.com/>
maximatecc
making machines smart, safe and productive


________________________________

Actuant Corporation Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.

________________________________

Actuant Corporation Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.

[-- Attachment #2: Type: text/html, Size: 11603 bytes --]

Re: Recompiles in Yocto/Poky

[Paul Eggleton]

On Fri, 03 Jun 2016 07:00:14 Peter Kjellerstedt wrote:
> This is a feature introduced to complement the main use of externalsrc,
> which is together with devtool. When you use "devtool modify -x <recipe>",
> it fetches the sources as specified in the recipe and makes them available
> locally. At the same time it sets up a bbappend in the workspace layer
> which uses externalsrc to use those sources when building the recipe. This
> is intended to make it easy to use BitBake to actually work on and develop
> the sources. However, for this to work in practice, BitBake must rebuild
> the package if there are any changes in the sources, but since BitBake
> normally only looks at the meta data to determine if a recipe task needs to
> run, there is nothing telling it that the sources have changed. Therefore
> changes were made so that code specified using externalsrc is always
> rebuilt.

In 2.0 that's how it behaved - what it's supposed to do in 2.1 is determine if 
the sources have changed by either relying on the git metadata (if the source 
tree is tracked in git) or by hashing all files in the source tree if it 
isn't. It could be false triggering if you have files in the source tree that 
get regenerated as part of the compilation process that aren't hidden. Teemu, 
is that the case for your source tree?

Unfortunately there is no way at present to turn off this functionality; we 
could add one, however before doing that I would be more interested in looking 
at making the code more intelligent so it did automatically rebuild in 
response to source changes for this case and not when there were no changes, 
assuming that's practical.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Recompiles in Yocto/Poky

[Keskinarkaus, Teemu]

Hi,

I think there are some files that are being generated at compilation time so most likely those triggers the recompilation every time.

The next question is that how to I 'hide' files so that they don't trigger the recompilation? Is that something that I need to do on my application side or can I list files in Yocto to ignore when hashing files? This is most likely the proper way to deal with this issue rather than disabling whole feature.

Teemu Keskinarkaus
Software system engineer
maximatecc
making machines smart, safe and productive


-----Original Message-----
From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
Sent: 8. kesäkuuta 2016 4:52
To: Peter Kjellerstedt; Keskinarkaus, Teemu
Cc: poky@yoctoproject.org
Subject: Re: [poky] Recompiles in Yocto/Poky

On Fri, 03 Jun 2016 07:00:14 Peter Kjellerstedt wrote:
> This is a feature introduced to complement the main use of
> externalsrc, which is together with devtool. When you use "devtool
> modify -x <recipe>", it fetches the sources as specified in the recipe
> and makes them available locally. At the same time it sets up a
> bbappend in the workspace layer which uses externalsrc to use those
> sources when building the recipe. This is intended to make it easy to
> use BitBake to actually work on and develop the sources. However, for
> this to work in practice, BitBake must rebuild the package if there
> are any changes in the sources, but since BitBake normally only looks
> at the meta data to determine if a recipe task needs to run, there is
> nothing telling it that the sources have changed. Therefore changes
> were made so that code specified using externalsrc is always rebuilt.

In 2.0 that's how it behaved - what it's supposed to do in 2.1 is determine if the sources have changed by either relying on the git metadata (if the source tree is tracked in git) or by hashing all files in the source tree if it isn't. It could be false triggering if you have files in the source tree that get regenerated as part of the compilation process that aren't hidden. Teemu, is that the case for your source tree?

Unfortunately there is no way at present to turn off this functionality; we could add one, however before doing that I would be more interested in looking at making the code more intelligent so it did automatically rebuild in response to source changes for this case and not when there were no changes, assuming that's practical.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre

________________________________

Actuant Corporation Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.

Re: Recompiles in Yocto/Poky

[Paul Eggleton]

There's no mechanism to exclude files at present, but it sounds like we should 
add one. One workaround right now would be to track your source tree in a git 
repository and have .gitignore cover the generated files; I appreciate though 
that if you aren't using git for version control this is hardly ideal so it's 
not a substitute for a proper fix.

Cheers,
Paul

On Wed, 08 Jun 2016 04:50:15 Keskinarkaus, Teemu wrote:
> I think there are some files that are being generated at compilation time so
> most likely those triggers the recompilation every time.
> 
> The next question is that how to I 'hide' files so that they don't trigger
> the recompilation? Is that something that I need to do on my application
> side or can I list files in Yocto to ignore when hashing files? This is
> most likely the proper way to deal with this issue rather than disabling
> whole feature.
> 
> Teemu Keskinarkaus
> Software system engineer
> maximatecc
> making machines smart, safe and productive
> 
> 
> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: 8. kesäkuuta 2016 4:52
> To: Peter Kjellerstedt; Keskinarkaus, Teemu
> Cc: poky@yoctoproject.org
> Subject: Re: [poky] Recompiles in Yocto/Poky
> 
> On Fri, 03 Jun 2016 07:00:14 Peter Kjellerstedt wrote:
> > This is a feature introduced to complement the main use of
> > externalsrc, which is together with devtool. When you use "devtool
> > modify -x <recipe>", it fetches the sources as specified in the recipe
> > and makes them available locally. At the same time it sets up a
> > bbappend in the workspace layer which uses externalsrc to use those
> > sources when building the recipe. This is intended to make it easy to
> > use BitBake to actually work on and develop the sources. However, for
> > this to work in practice, BitBake must rebuild the package if there
> > are any changes in the sources, but since BitBake normally only looks
> > at the meta data to determine if a recipe task needs to run, there is
> > nothing telling it that the sources have changed. Therefore changes
> > were made so that code specified using externalsrc is always rebuilt.
> 
> In 2.0 that's how it behaved - what it's supposed to do in 2.1 is determine
> if the sources have changed by either relying on the git metadata (if the
> source tree is tracked in git) or by hashing all files in the source tree
> if it isn't. It could be false triggering if you have files in the source
> tree that get regenerated as part of the compilation process that aren't
> hidden. Teemu, is that the case for your source tree?
> 
> Unfortunately there is no way at present to turn off this functionality; we
> could add one, however before doing that I would be more interested in
> looking at making the code more intelligent so it did automatically rebuild
> in response to source changes for this case and not when there were no
> changes, assuming that's practical.
> 
> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre
> 
> ________________________________
> 
> Actuant Corporation Email Notice
> 
> This message is intended only for the use of the Addressee and may contain
> information that is PRIVILEGED and/or CONFIDENTIAL. This email is intended
> only for the personal and confidential use of the recipient(s) named above.
> If the reader of this email is not an intended recipient, you have received
> this email in error and any review, dissemination, distribution or copying
> is strictly prohibited. If you have received this email in error, please
> notify the sender immediately by return mail and permanently delete the
> copy you received.
> 
> Thank you.

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Recompiles in Yocto/Poky

[Keskinarkaus, Teemu]

Unfortunately we are not using git but svn as version control.

So as it stands at the moment there isn't way to prevent the recompilation(in my case) expect modifying Yocto sources which I rather not do.

Teemu Keskinarkaus
Software system engineer
maximatecc
making machines smart, safe and productive


-----Original Message-----
From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
Sent: 8. kesäkuuta 2016 7:56
To: Keskinarkaus, Teemu
Cc: Peter Kjellerstedt; poky@yoctoproject.org; Markus Lehtonen
Subject: Re: [poky] Recompiles in Yocto/Poky

There's no mechanism to exclude files at present, but it sounds like we should add one. One workaround right now would be to track your source tree in a git repository and have .gitignore cover the generated files; I appreciate though that if you aren't using git for version control this is hardly ideal so it's not a substitute for a proper fix.

Cheers,
Paul

On Wed, 08 Jun 2016 04:50:15 Keskinarkaus, Teemu wrote:
> I think there are some files that are being generated at compilation
> time so most likely those triggers the recompilation every time.
>
> The next question is that how to I 'hide' files so that they don't
> trigger the recompilation? Is that something that I need to do on my
> application side or can I list files in Yocto to ignore when hashing
> files? This is most likely the proper way to deal with this issue
> rather than disabling whole feature.
>
> Teemu Keskinarkaus
> Software system engineer
> maximatecc
> making machines smart, safe and productive
>
>
> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: 8. kesäkuuta 2016 4:52
> To: Peter Kjellerstedt; Keskinarkaus, Teemu
> Cc: poky@yoctoproject.org
> Subject: Re: [poky] Recompiles in Yocto/Poky
>
> On Fri, 03 Jun 2016 07:00:14 Peter Kjellerstedt wrote:
> > This is a feature introduced to complement the main use of
> > externalsrc, which is together with devtool. When you use "devtool
> > modify -x <recipe>", it fetches the sources as specified in the
> > recipe and makes them available locally. At the same time it sets up
> > a bbappend in the workspace layer which uses externalsrc to use
> > those sources when building the recipe. This is intended to make it
> > easy to use BitBake to actually work on and develop the sources.
> > However, for this to work in practice, BitBake must rebuild the
> > package if there are any changes in the sources, but since BitBake
> > normally only looks at the meta data to determine if a recipe task
> > needs to run, there is nothing telling it that the sources have
> > changed. Therefore changes were made so that code specified using externalsrc is always rebuilt.
>
> In 2.0 that's how it behaved - what it's supposed to do in 2.1 is
> determine if the sources have changed by either relying on the git
> metadata (if the source tree is tracked in git) or by hashing all
> files in the source tree if it isn't. It could be false triggering if
> you have files in the source tree that get regenerated as part of the
> compilation process that aren't hidden. Teemu, is that the case for your source tree?
>
> Unfortunately there is no way at present to turn off this
> functionality; we could add one, however before doing that I would be
> more interested in looking at making the code more intelligent so it
> did automatically rebuild in response to source changes for this case
> and not when there were no changes, assuming that's practical.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
>
> ________________________________
>
> Actuant Corporation Email Notice
>
> This message is intended only for the use of the Addressee and may
> contain information that is PRIVILEGED and/or CONFIDENTIAL. This email
> is intended only for the personal and confidential use of the recipient(s) named above.
> If the reader of this email is not an intended recipient, you have
> received this email in error and any review, dissemination,
> distribution or copying is strictly prohibited. If you have received
> this email in error, please notify the sender immediately by return
> mail and permanently delete the copy you received.
>
> Thank you.

--

Paul Eggleton
Intel Open Source Technology Centre

________________________________

Actuant Corporation Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.

Re: Recompiles in Yocto/Poky

[Markus Lehtonen]

Hi,

There would be at least two options,that I can think of to "hide" the
files. First, if possible, make the recipe build in a separate
directory so that the build-time generated files won't pollute your
source tree. Second, make your external source tree a Git tree and add
the files in question to .gitignore. 

Thanks,
  Markus

On Wed, 2016-06-08 at 04:50 +0000, Keskinarkaus, Teemu wrote:
> Hi,
> 
> I think there are some files that are being generated at compilation
> time so most likely those triggers the recompilation every time.
> 
> The next question is that how to I 'hide' files so that they don't
> trigger the recompilation? Is that something that I need to do on my
> application side or can I list files in Yocto to ignore when hashing
> files? This is most likely the proper way to deal with this issue
> rather than disabling whole feature.
> 
> Teemu Keskinarkaus
> Software system engineer
> maximatecc
> making machines smart, safe and productive
> 
> 
> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: 8. kesäkuuta 2016 4:52
> To: Peter Kjellerstedt; Keskinarkaus, Teemu
> Cc: poky@yoctoproject.org
> Subject: Re: [poky] Recompiles in Yocto/Poky
> 
> On Fri, 03 Jun 2016 07:00:14 Peter Kjellerstedt wrote:
> > This is a feature introduced to complement the main use of
> > externalsrc, which is together with devtool. When you use "devtool
> > modify -x <recipe>", it fetches the sources as specified in the
> > recipe
> > and makes them available locally. At the same time it sets up a
> > bbappend in the workspace layer which uses externalsrc to use those
> > sources when building the recipe. This is intended to make it easy
> > to
> > use BitBake to actually work on and develop the sources. However,
> > for
> > this to work in practice, BitBake must rebuild the package if there
> > are any changes in the sources, but since BitBake normally only
> > looks
> > at the meta data to determine if a recipe task needs to run, there
> > is
> > nothing telling it that the sources have changed. Therefore changes
> > were made so that code specified using externalsrc is always
> > rebuilt.
> 
> In 2.0 that's how it behaved - what it's supposed to do in 2.1 is
> determine if the sources have changed by either relying on the git
> metadata (if the source tree is tracked in git) or by hashing all
> files in the source tree if it isn't. It could be false triggering if
> you have files in the source tree that get regenerated as part of the
> compilation process that aren't hidden. Teemu, is that the case for
> your source tree?
> 
> Unfortunately there is no way at present to turn off this
> functionality; we could add one, however before doing that I would be
> more interested in looking at making the code more intelligent so it
> did automatically rebuild in response to source changes for this case
> and not when there were no changes, assuming that's practical.
> 
> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre
> 
> ________________________________
> 
> Actuant Corporation Email Notice
> 
> This message is intended only for the use of the Addressee and may
> contain information that is PRIVILEGED and/or CONFIDENTIAL.
> This email is intended only for the personal and confidential use of
> the recipient(s) named above. If the reader of this email is not an
> intended recipient, you have received this email in error and any
> review, dissemination, distribution or copying is strictly
> prohibited.
> If you have received this email in error, please notify the sender
> immediately by return mail and permanently delete the copy you
> received.
> 
> Thank you.