[PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[Naveen Saini]

This provides git large file storage (lfs) extension.

Include git-lfs conditionally. If git-lfs is present on host and repo
has lfs pointers, then git-lfs will be used. If git-lfs is not present
on host, it will be ignored.

[YOCTO #13198]

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
---
 meta/conf/bitbake.conf | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7f8b043cc4..d35a6f64d9 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -514,6 +514,9 @@ HOSTTOOLS_NONFATAL += "bzr"
 # Used by ssh fetcher
 HOSTTOOLS_NONFATAL += "scp"
 
+# Link to git-lfs if present
+HOSTTOOLS_NONFATAL += "git-lfs"
+
 CCACHE ??= ""
 
 TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"
-- 
2.17.0

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[Richard Purdie]

On Tue, 2019-04-16 at 13:59 +0800, Naveen Saini wrote:
> This provides git large file storage (lfs) extension.
> 
> Include git-lfs conditionally. If git-lfs is present on host and repo
> has lfs pointers, then git-lfs will be used. If git-lfs is not
> present
> on host, it will be ignored.
> 
> [YOCTO #13198]
> 
> Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
> ---
>  meta/conf/bitbake.conf | 3 +++
>  1 file changed, 3 insertions(+)

This sounds like we swap the current situation where things are
deterministic for a situation where the situation is not deterministic?

I'm not sure that is an improvement.

Cheers,

Richard

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[Saini, Naveen Kumar]

On Selasa 16 Apr 2019 06:23 , Richard Purdie wrote:
> On Tue, 2019-04-16 at 13:59 +0800, Naveen Saini wrote:
>> This provides git large file storage (lfs) extension.
>>
>> Include git-lfs conditionally. If git-lfs is present on host and repo
>> has lfs pointers, then git-lfs will be used. If git-lfs is not
>> present
>> on host, it will be ignored.
>>
>> [YOCTO #13198]
>>
>> Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
>> ---
>>   meta/conf/bitbake.conf | 3 +++
>>   1 file changed, 3 insertions(+)
> 
> This sounds like we swap the current situation where things are
> deterministic for a situation where the situation is not deterministic?
> 
> I'm not sure that is an improvement.

Currently, if host machine has git-lfs installed, build always fails
and if host doesn't have git-lfs installed, build passes but
does not download files pointed by lfs. So, the current behaviour
depends on whether or not git-lfs is installed.

With this patch, build will pass in both the cases and files stored
using lfs will be fetched when git-lfs is present. So this is more
deterministic than before? If git-lfs isn't present, then files won't
be fetched as before.

Should this be added to HOSTTOOLS instead so lfs files are always
fetched for packages that use it?

> 
> Cheers,
> 
> Richard
> 

Regards,
Naveen

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[richard.purdie]

On Wed, 2019-04-17 at 10:20 +0800, Saini, Naveen Kumar wrote:
> 
> On Selasa 16 Apr 2019 06:23 , Richard Purdie wrote:
> > On Tue, 2019-04-16 at 13:59 +0800, Naveen Saini wrote:
> > > This provides git large file storage (lfs) extension.
> > > 
> > > Include git-lfs conditionally. If git-lfs is present on host and
> > > repo
> > > has lfs pointers, then git-lfs will be used. If git-lfs is not
> > > present
> > > on host, it will be ignored.
> > > 
> > > [YOCTO #13198]
> > > 
> > > Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
> > > ---
> > >   meta/conf/bitbake.conf | 3 +++
> > >   1 file changed, 3 insertions(+)
> > 
> > This sounds like we swap the current situation where things are
> > deterministic for a situation where the situation is not
> > deterministic?
> > 
> > I'm not sure that is an improvement.
> 
> Currently, if host machine has git-lfs installed, build always fails
> and if host doesn't have git-lfs installed, build passes but
> does not download files pointed by lfs. So, the current behaviour
> depends on whether or not git-lfs is installed.

That is clearly a problem as the output is not deterministic (a given
sstate hash input does not guarantee a given output).

> With this patch, build will pass in both the cases and files stored
> using lfs will be fetched when git-lfs is present. So this is more
> deterministic than before? If git-lfs isn't present, then files won't
> be fetched as before.

So we swap one problem for a different one and its still not
deterministic.

> Should this be added to HOSTTOOLS instead so lfs files are always
> fetched for packages that use it?

No, that would mean everyone has to install it.

I'm ok with adding it to HOSTTOOLS_NONFATAL as soon as we have some
mechanism in the fetcher or in recipes which detects lfs usage and hard
errors if its present but git-lfs is not installed.

That way its deterministic - it builds with git-lfs installed and
errors if it is not.

Cheers,

Richard

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[Adrian Bunk]

On Wed, Apr 17, 2019 at 09:46:25AM +0100, richard.purdie@linuxfoundation.org wrote:
> On Wed, 2019-04-17 at 10:20 +0800, Saini, Naveen Kumar wrote:
>...
> > Should this be added to HOSTTOOLS instead so lfs files are always
> > fetched for packages that use it?
> 
> No, that would mean everyone has to install it.
> 
> I'm ok with adding it to HOSTTOOLS_NONFATAL as soon as we have some
> mechanism in the fetcher or in recipes which detects lfs usage and hard
> errors if its present but git-lfs is not installed.
> 
> That way its deterministic - it builds with git-lfs installed and
> errors if it is not.

Is it OK that some packages are not buildable at all on supported
distributions?

Your "everyone has to install it" worry misses the much bigger problem 
that git-lfs is a relatively new tool and not available in distributions 
like Debian 9 or Ubuntu 16.04.

> Cheers,
> 
> Richard

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[richard.purdie]

On Wed, 2019-04-17 at 12:35 +0300, Adrian Bunk wrote:
> On Wed, Apr 17, 2019 at 09:46:25AM +0100, 
> richard.purdie@linuxfoundation.org wrote:
> > On Wed, 2019-04-17 at 10:20 +0800, Saini, Naveen Kumar wrote:
> > ...
> > > Should this be added to HOSTTOOLS instead so lfs files are always
> > > fetched for packages that use it?
> > 
> > No, that would mean everyone has to install it.
> > 
> > I'm ok with adding it to HOSTTOOLS_NONFATAL as soon as we have some
> > mechanism in the fetcher or in recipes which detects lfs usage and
> > hard
> > errors if its present but git-lfs is not installed.
> > 
> > That way its deterministic - it builds with git-lfs installed and
> > errors if it is not.
> 
> Is it OK that some packages are not buildable at all on supported
> distributions?

We're talking about a non-core layer and that would be something for
that layer's maintainers to decide.

> Your "everyone has to install it" worry misses the much bigger
> problem 
> that git-lfs is a relatively new tool and not available in
> distributions 
> like Debian 9 or Ubuntu 16.04.

Someone could certainly write a native recipe for lfs and add that as a
dependency which would also solve this problem. I'd be quite happy to
see that and it would neatly solve the problems.

Cheers,

Richard

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[Adrian Bunk]

On Wed, Apr 17, 2019 at 10:37:45AM +0100, richard.purdie@linuxfoundation.org wrote:
> On Wed, 2019-04-17 at 12:35 +0300, Adrian Bunk wrote:
>...
> > Your "everyone has to install it" worry misses the much bigger
> > problem 
> > that git-lfs is a relatively new tool and not available in
> > distributions 
> > like Debian 9 or Ubuntu 16.04.
> 
> Someone could certainly write a native recipe for lfs and add that as a
> dependency which would also solve this problem. I'd be quite happy to
> see that and it would neatly solve the problems.

This was also my first thought.

It stops being neat once you realize that it requires packaging
a 3 digit number of Go libraries.

> Cheers,
> 
> Richard

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[Bach, Pascal]

> > > Your "everyone has to install it" worry misses the much bigger
> > > problem that git-lfs is a relatively new tool and not available in
> > > distributions like Debian 9 or Ubuntu 16.04.
> >
> > Someone could certainly write a native recipe for lfs and add that as
> > a dependency which would also solve this problem. I'd be quite happy
> > to see that and it would neatly solve the problems.
> 
> This was also my first thought.
> 
> It stops being neat once you realize that it requires packaging a 3 digit
> number of Go libraries.

I don't think that's how go packaging works. Go usually statically links everything at compile time and bundles the sources with modules.
So from a yocto point of view you only need to create one recipe with everything inside. Have a look at the dependency of the Debian package of git-lfs [1] they are very minimal.

I havent' tried but maybe devtool might produce something useful very quickly.

Pascal

[1] https://packages.debian.org/sid/git-lfs

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[Adrian Bunk]

On Wed, Apr 17, 2019 at 09:57:50AM +0000, Bach, Pascal wrote:
> > > > Your "everyone has to install it" worry misses the much bigger
> > > > problem that git-lfs is a relatively new tool and not available in
> > > > distributions like Debian 9 or Ubuntu 16.04.
> > >
> > > Someone could certainly write a native recipe for lfs and add that as
> > > a dependency which would also solve this problem. I'd be quite happy
> > > to see that and it would neatly solve the problems.
> > 
> > This was also my first thought.
> > 
> > It stops being neat once you realize that it requires packaging a 3 digit
> > number of Go libraries.
> 
> I don't think that's how go packaging works. Go usually statically links everything at compile time and bundles the sources with modules.

It is not really static linking, it is more like the "libraries" being 
source files instead of compiled code.

> So from a yocto point of view you only need to create one recipe with everything inside.

Fetching from 100 different git repositories would be plenty of
work to setup and to maintain, no matter how many recipes you use.

Certainly not impossible, but I wouldn't call this neat.

>...
> Pascal
>...

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] bitbake.conf: add git-lfs to HOSTTOOLS_NONFATAL

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2583 bytes --]

Would it make sense to at least as temporary work around add a flag saying
that git-lfs does or doesn't exist in HOSTTOOLS after it being added
to HOSTTOOLS_NONFATAL and then let those few recipes which already use
git-lfs to check this flag and fail early with meaningful error message
about git-lfs? It can be in a bbclass so that all recipes can easily share
it and it can be later changed to add dependency on git-lfs-native (if
there is one later).

Only issue is that people will forget to inherit this bbclass in their
recipes if it just happen to work.

Adding it to some subdirectory inside HOSTTOOLS which is added to PATH only
when the bbclass is inherited might be a bit too much for just a work
around :/.

Regards,

On Wed, Apr 17, 2019 at 12:25 PM Adrian Bunk <bunk@stusta.de> wrote:

> On Wed, Apr 17, 2019 at 09:57:50AM +0000, Bach, Pascal wrote:
> > > > > Your "everyone has to install it" worry misses the much bigger
> > > > > problem that git-lfs is a relatively new tool and not available in
> > > > > distributions like Debian 9 or Ubuntu 16.04.
> > > >
> > > > Someone could certainly write a native recipe for lfs and add that as
> > > > a dependency which would also solve this problem. I'd be quite happy
> > > > to see that and it would neatly solve the problems.
> > >
> > > This was also my first thought.
> > >
> > > It stops being neat once you realize that it requires packaging a 3
> digit
> > > number of Go libraries.
> >
> > I don't think that's how go packaging works. Go usually statically links
> everything at compile time and bundles the sources with modules.
>
> It is not really static linking, it is more like the "libraries" being
> source files instead of compiled code.
>
> > So from a yocto point of view you only need to create one recipe with
> everything inside.
>
> Fetching from 100 different git repositories would be plenty of
> work to setup and to maintain, no matter how many recipes you use.
>
> Certainly not impossible, but I wouldn't call this neat.
>
> >...
> > Pascal
> >...
>
> cu
> Adrian
>
> --
>
>        "Is there not promise of rain?" Ling Tan asked suddenly out
>         of the darkness. There had been need of rain for many days.
>        "Only a promise," Lao Er said.
>                                        Pearl S. Buck - Dragon Seed
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3437 bytes --]