FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Andrey Zhizhikin]

Hello Andre,

I've just pulled the master and experienced a build failure during
fetching of updated recipe's source tarballs.

The reason for this being that defaults for FETCHCMD has been dropped
with your patch b259bd31eb from the series. Once defaults are removed
and appends are used - the FETCHCMD gets defined to the value listed
in append, which normally does not contain a command itself rather
than necessary additional parameters (like user/passwd if working with
local pre-mirror servers).

This patch would also break several other recipes which are using
appends to FETCHCMD, for example for FETCHCMD_wget the libedit would
fail since it appends the wget to use different User-Agent.

I've copied Raj here since he introduced this recipe in the form it is
and would definitely break.

Can you please have a look at this and advise on how one can continue
to use the FETCHCMD appends for the future?

For now, I've defined the FETCHCMD_wget in my local.conf but I do not
believe that this is the general way everyone should follow if they
would need to append fetcher commands...

Thanks a lot!

-- 
Regards,
Andrey.

Re: FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Andre McCurdy]

On Fri, Aug 30, 2019 at 3:08 AM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
>
> Hello Andre,
>
> I've just pulled the master and experienced a build failure during
> fetching of updated recipe's source tarballs.
>
> The reason for this being that defaults for FETCHCMD has been dropped
> with your patch b259bd31eb from the series. Once defaults are removed
> and appends are used - the FETCHCMD gets defined to the value listed
> in append, which normally does not contain a command itself rather
> than necessary additional parameters (like user/passwd if working with
> local pre-mirror servers).

I think for the specific case of usernames and passwords the advice
would be to put them in .netrc etc rather than trying to append to the
fetcher command lines (but mainly for security reasons rather than
this issue).

> This patch would also break several other recipes which are using
> appends to FETCHCMD, for example for FETCHCMD_wget the libedit would
> fail since it appends the wget to use different User-Agent.
>
> I've copied Raj here since he introduced this recipe in the form it is
> and would definitely break.
>
> Can you please have a look at this and advise on how one can continue
> to use the FETCHCMD appends for the future?

One answer could be that modifications of the fetcher command lines
should be done by completely defining them rather than appending. I'm
not sure how reasonable that is though.

In the end the approach to fixing this depends on whether appending to
the default fetcher commands is considered valid usage or not... and I
don't know the answer to that.

> For now, I've defined the FETCHCMD_wget in my local.conf but I do not
> believe that this is the general way everyone should follow if they
> would need to append fetcher commands...
>
> Thanks a lot!
>
> --
> Regards,
> Andrey.

Re: FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Andre McCurdy
> Sent: den 30 augusti 2019 16:41
> To: Andrey Zhizhikin <andrey.z@gmail.com>
> Cc: OE Core mailing list <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] FETCHCMD drop breaks build when append is used
> (from patch b259bd31eb)
> 
> On Fri, Aug 30, 2019 at 3:08 AM Andrey Zhizhikin <andrey.z@gmail.com>
> wrote:
> >
> > Hello Andre,
> >
> > I've just pulled the master and experienced a build failure during
> > fetching of updated recipe's source tarballs.
> >
> > The reason for this being that defaults for FETCHCMD has been dropped
> > with your patch b259bd31eb from the series. Once defaults are removed
> > and appends are used - the FETCHCMD gets defined to the value listed
> > in append, which normally does not contain a command itself rather
> > than necessary additional parameters (like user/passwd if working
> with
> > local pre-mirror servers).
> 
> I think for the specific case of usernames and passwords the advice
> would be to put them in .netrc etc rather than trying to append to the
> fetcher command lines (but mainly for security reasons rather than
> this issue).
> 
> > This patch would also break several other recipes which are using
> > appends to FETCHCMD, for example for FETCHCMD_wget the libedit would
> > fail since it appends the wget to use different User-Agent.
> >
> > I've copied Raj here since he introduced this recipe in the form it
> is
> > and would definitely break.
> >
> > Can you please have a look at this and advise on how one can continue
> > to use the FETCHCMD appends for the future?
> 
> One answer could be that modifications of the fetcher command lines
> should be done by completely defining them rather than appending. I'm
> not sure how reasonable that is though.
> 
> In the end the approach to fixing this depends on whether appending to
> the default fetcher commands is considered valid usage or not... and I
> don't know the answer to that.
> 
> > For now, I've defined the FETCHCMD_wget in my local.conf but I do not
> > believe that this is the general way everyone should follow if they
> > would need to append fetcher commands...
> >
> > Thanks a lot!
> >
> > --
> > Regards,
> > Andrey.

Given that the libedit recipe in OE-Core does:

FETCHCMD_wget += "-U bitbake"

someone will probably have to do something about this...

One can also note that wget.py and npm.py will now use different arguments 
to wget after the default was removed from bitbake.conf (the difference 
being the added use of -nv in npm.py so nothing major, but still...)

//Peter

Re: FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Adrian Bunk]

On Fri, Aug 30, 2019 at 07:40:51AM -0700, Andre McCurdy wrote:
> On Fri, Aug 30, 2019 at 3:08 AM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
>...
> > This patch would also break several other recipes which are using
> > appends to FETCHCMD, for example for FETCHCMD_wget the libedit would
> > fail since it appends the wget to use different User-Agent.
> >
> > I've copied Raj here since he introduced this recipe in the form it is
> > and would definitely break.
> >
> > Can you please have a look at this and advise on how one can continue
> > to use the FETCHCMD appends for the future?
> 
> One answer could be that modifications of the fetcher command lines
> should be done by completely defining them rather than appending. I'm
> not sure how reasonable that is though.
> 
> In the end the approach to fixing this depends on whether appending to
> the default fetcher commands is considered valid usage or not... and I
> don't know the answer to that.
>...

Disallowing appends could cause huge problems for a user or layer that 
has to append local options (e.g. proxy) building a recipe like libedit 
that has to change the User-Agent.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Andrey Zhizhikin]

On Fri, Aug 30, 2019 at 4:41 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>
> I think for the specific case of usernames and passwords the advice
> would be to put them in .netrc etc rather than trying to append to the
> fetcher command lines (but mainly for security reasons rather than
> this issue).

This I do agree, but it is only related to the username/passwd
combination for a local user to perform a local fetch. Sometimes
however this append might be needed for authentication tokens for
CI/CD systems running in containers. In this case it would be quite
tricky to do it via .netrc (IMHO).

>
>
> One answer could be that modifications of the fetcher command lines
> should be done by completely defining them rather than appending. I'm
> not sure how reasonable that is though.

I believe in this case we would fall back to the original solution,
where defaults were defined in bitbake.conf file.

>
> In the end the approach to fixing this depends on whether appending to
> the default fetcher commands is considered valid usage or not... and I
> don't know the answer to that.

I guess the best course of action should be to check whether appends
are introduced in the fetcher class and use them if they are. This
need to be confirmer with BItbak people though.

-- 
Regards,
Andrey.

Re: FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Andrey Zhizhikin]

On Sat, Aug 31, 2019 at 2:18 PM Adrian Bunk <bunk@stusta.de> wrote:
>
> Disallowing appends could cause huge problems for a user or layer that
> has to append local options (e.g. proxy) building a recipe like libedit
> that has to change the User-Agent.

My point exactly! I belive that even though this is currently the only
recipe to have this append, this might not be the last one. And in
this case, that prospective recipe would require user to setup his
local build system with additional .netrc file, just to have it built.

>
> cu
> Adrian
>

-- 
Regards,
Andrey.

Re: FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Andre McCurdy]

On Sat, Aug 31, 2019 at 11:07 AM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
> On Sat, Aug 31, 2019 at 2:18 PM Adrian Bunk <bunk@stusta.de> wrote:
> > Disallowing appends could cause huge problems for a user or layer that
> > has to append local options (e.g. proxy) building a recipe like libedit
> > that has to change the User-Agent.

Clearly we need to have a solution for libedit. I don't see it's
related to user or layer specific issues of setting a password or a
proxy though.

There seem to be three independent cases where appending to the
fetcher commands in bitbake.conf has been used:

1) Setting usernames/passwords for private mirrors, sstate servers,
etc. For that I believe the recommended solution is .netrc. Not only
is putting passwords on the command line a bad idea in general, but it
also won't work consistently for fetchers where download() and
checkstatus() are implemented differently - e.g. for wget, download()
calls wget (and therefore respects FETCHCMD_wget) but checkstatus() is
implemented directly in python (and ignores FETCHCMD_wget completely).
To see the effects, try setting up a password protected http or https
sstate server... it will work if you put your credentials in .netrc
but not if you add them to FETCHCMD_wget.

2) Configuring a proxy server for users behind a firewall. For that I
believe the recommended solution is the various *_proxy environment
variables?

3) Forcing a custom User-Agent. I don't know what the best solution is
here but it feels like the wget fetcher should either handle this
internally (e.g. set a more compatible User-Agent by default?
Automatic retries with various User-Agents?) or provide a documented
API which specifically sets the User-Agent in cases where the default
really does need to be over-ridden.

In the end the bitbake fetchers are abstractions and if users need to
force their own options directly into the final command line (ie
bypassing the abstraction) then it suggests the abstraction is
incomplete. It feels like a slippery slope if we start to encourage or
rely on doing that.

Re: FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Andre McCurdy]

On Sat, Aug 31, 2019 at 1:24 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>
> On Sat, Aug 31, 2019 at 11:07 AM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
> > On Sat, Aug 31, 2019 at 2:18 PM Adrian Bunk <bunk@stusta.de> wrote:
> > > Disallowing appends could cause huge problems for a user or layer that
> > > has to append local options (e.g. proxy) building a recipe like libedit
> > > that has to change the User-Agent.
>
> Clearly we need to have a solution for libedit. I don't see it's
> related to user or layer specific issues of setting a password or a
> proxy though.
>
> There seem to be three independent cases where appending to the
> fetcher commands in bitbake.conf has been used:
>
> 1) Setting usernames/passwords for private mirrors, sstate servers,
> etc. For that I believe the recommended solution is .netrc. Not only
> is putting passwords on the command line a bad idea in general, but it
> also won't work consistently for fetchers where download() and
> checkstatus() are implemented differently - e.g. for wget, download()
> calls wget (and therefore respects FETCHCMD_wget) but checkstatus() is
> implemented directly in python (and ignores FETCHCMD_wget completely).
> To see the effects, try setting up a password protected http or https
> sstate server... it will work if you put your credentials in .netrc
> but not if you add them to FETCHCMD_wget.
>
> 2) Configuring a proxy server for users behind a firewall. For that I
> believe the recommended solution is the various *_proxy environment
> variables?
>
> 3) Forcing a custom User-Agent. I don't know what the best solution is
> here but it feels like the wget fetcher should either handle this
> internally (e.g. set a more compatible User-Agent by default?
> Automatic retries with various User-Agents?) or provide a documented
> API which specifically sets the User-Agent in cases where the default
> really does need to be over-ridden.
>
> In the end the bitbake fetchers are abstractions and if users need to
> force their own options directly into the final command line (ie
> bypassing the abstraction) then it suggests the abstraction is
> incomplete. It feels like a slippery slope if we start to encourage or
> rely on doing that.

It looks like the User-Agent workaround for libedit might not be
needed any more anyway...

  wget http://thrysoee.dk/editline/libedit-20190324-3.1.tar.gz

works fine for me ( User-Agent: Wget/1.17.1 (linux-gnu) )

Re: FETCHCMD drop breaks build when append is used (from patch b259bd31eb)

[Ross Burton]

On 31/08/2019 21:45, Andre McCurdy wrote:
> It looks like the User-Agent workaround for libedit might not be
> needed any more anyway...
> 
>    wget http://thrysoee.dk/editline/libedit-20190324-3.1.tar.gz
> 
> works fine for me ( User-Agent: Wget/1.17.1 (linux-gnu) )

Verified here, patch sent.

Ross