[PATCH 0/3] Setup for bitbake perforce fetcher changes

[PATCH 0/3] Setup for bitbake perforce fetcher changes

[Andrew Bradford]

From: Andrew Bradford <andrew.bradford@kodakalaris.com>

The perforce fetcher does not yet support using SRCREV="${AUTOREV}" or
the use of the P4CONFIG variable [1].  These patches setup a few things
in oe-core so that changes in bitbake can take advantage of them to fix
these two issues as well as simplify and better document the use of the
perforce fetcher.

[1]: https://bugzilla.yoctoproject.org/show_bug.cgi?id=6303

Thanks,
Andrew

Andrew Bradford (3):
  oe-buildenv-internal: Add P4CONFIG to BB_ENV_EXTRAWHITE_OE
  bitbake.conf: Add P4DIR and FETCHCMD_p4
  base.bbclass: p4 fetcher supports srcrev

 meta/classes/base.bbclass    | 4 ++++
 meta/conf/bitbake.conf       | 2 ++
 scripts/oe-buildenv-internal | 2 +-
 3 files changed, 7 insertions(+), 1 deletion(-)

-- 
2.8.1

[PATCH 1/3] oe-buildenv-internal: Add P4CONFIG to BB_ENV_EXTRAWHITE_OE

[Andrew Bradford]

From: Andrew Bradford <andrew.bradford@kodakalaris.com>

So that the user's P4CONFIG variable can be used by recipes which
utilize the perforce fetcher.

References [YOCTO #6303]

Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
---
 scripts/oe-buildenv-internal | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
index 81ee784..e51b950 100755
--- a/scripts/oe-buildenv-internal
+++ b/scripts/oe-buildenv-internal
@@ -114,7 +114,7 @@ BB_ENV_EXTRAWHITE_OE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy \
 HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY \
 all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \
 SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND \
-SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA"
+SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA P4CONFIG"
 
 BB_ENV_EXTRAWHITE="$(echo $BB_ENV_EXTRAWHITE $BB_ENV_EXTRAWHITE_OE | tr ' ' '\n' | LC_ALL=C sort --unique | tr '\n' ' ')"
 
-- 
2.8.1

[PATCH 2/3] bitbake.conf: Add P4DIR and FETCHCMD_p4

[Andrew Bradford]

From: Andrew Bradford <andrew.bradford@kodakalaris.com>

So that perforce fetching can work more like the other source control
system fetchers.

Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
---
 meta/conf/bitbake.conf | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index acef30b..1aa1d37 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -339,6 +339,7 @@ SVNDIR = "${CO_DIR}/svn"
 GITDIR = "${CO_DIR}/git2"
 BZRDIR = "${CO_DIR}/bzr"
 HGDIR = "${CO_DIR}/hg"
+P4DIR = "${CO_DIR}/p4"
 
 STAMPS_DIR ?= "${TMPDIR}/stamps"
 STAMP = "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}"
@@ -618,6 +619,7 @@ FETCHCMD_cvs = "/usr/bin/env cvs"
 FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate"
 FETCHCMD_bzr = "/usr/bin/env bzr"
 FETCHCMD_hg = "/usr/bin/env hg"
+FETCHCMD_p4 = "/usr/bin/env p4"
 
 SRCDATE = "${DATE}"
 SRCREV ??= "INVALID"
-- 
2.8.1

[PATCH 3/3] base.bbclass: p4 fetcher supports srcrev

[Andrew Bradford]

From: Andrew Bradford <andrew.bradford@kodakalaris.com>

Enable the perforce fetcher to call bb.fetch2.get_srcrev() as it can use
'SRCREV = "${AUTOREV}"'.

Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
---
 meta/classes/base.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 4be0a7e..9f3945c 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -569,6 +569,10 @@ python () {
             needsrcrev = True
             d.appendVarFlag('do_fetch', 'depends', ' mercurial-native:do_populate_sysroot')
 
+        # Perforce packages support SRCREV = "${AUTOREV}"
+        elif scheme == "p4":
+            needsrcrev = True
+
         # OSC packages should DEPEND on osc-native
         elif scheme == "osc":
             d.appendVarFlag('do_fetch', 'depends', ' osc-native:do_populate_sysroot')
-- 
2.8.1

Re: [PATCH 0/3] Setup for bitbake perforce fetcher changes

[Andrew Bradford]

On 05/25 15:46, Andrew Bradford wrote:
> From: Andrew Bradford <andrew.bradford@kodakalaris.com>
> 
> The perforce fetcher does not yet support using SRCREV="${AUTOREV}" or
> the use of the P4CONFIG variable [1].  These patches setup a few things
> in oe-core so that changes in bitbake can take advantage of them to fix
> these two issues as well as simplify and better document the use of the
> perforce fetcher.
> 
> [1]: https://bugzilla.yoctoproject.org/show_bug.cgi?id=6303
> 
> Thanks,
> Andrew
> 
> Andrew Bradford (3):
>   oe-buildenv-internal: Add P4CONFIG to BB_ENV_EXTRAWHITE_OE
>   bitbake.conf: Add P4DIR and FETCHCMD_p4
>   base.bbclass: p4 fetcher supports srcrev
> 
>  meta/classes/base.bbclass    | 4 ++++
>  meta/conf/bitbake.conf       | 2 ++
>  scripts/oe-buildenv-internal | 2 +-
>  3 files changed, 7 insertions(+), 1 deletion(-)
> 
> -- 
> 2.8.1
> 
> -- 

Ping on this series?

Thanks,
Andrew

Re: [PATCH 1/3] oe-buildenv-internal: Add P4CONFIG to BB_ENV_EXTRAWHITE_OE

[Richard Purdie]

On Wed, 2016-05-25 at 15:46 -0400, Andrew Bradford wrote:
> From: Andrew Bradford <andrew.bradford@kodakalaris.com>
> 
> So that the user's P4CONFIG variable can be used by recipes which
> utilize the perforce fetcher.
> 
> References [YOCTO #6303]
> 
> Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
> ---
>  scripts/oe-buildenv-internal | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv
> -internal
> index 81ee784..e51b950 100755
> --- a/scripts/oe-buildenv-internal
> +++ b/scripts/oe-buildenv-internal
> @@ -114,7 +114,7 @@ BB_ENV_EXTRAWHITE_OE="MACHINE DISTRO TCMODE
> TCLIBC HTTP_PROXY http_proxy \
>  HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy
> ALL_PROXY \
>  all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK
> BB_SRCREV_POLICY \
>  SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE
> GIT_PROXY_COMMAND \
> -SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA"
> +SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA
> P4CONFIG"
>  
>  BB_ENV_EXTRAWHITE="$(echo $BB_ENV_EXTRAWHITE $BB_ENV_EXTRAWHITE_OE |
> tr ' ' '\n' | LC_ALL=C sort --unique | tr '\n' ' ')"

With the recent changes to the bitbake fetcher, its able to pull
environment variables out of BB_ORIGENV. This should mean this patch
isn't necessary?

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=f9c623ddbfc140c0c8911f06aa2824edef8a5f4f

Cheers,

Richard

Re: [PATCH 2/3] bitbake.conf: Add P4DIR and FETCHCMD_p4

[Richard Purdie]

On Wed, 2016-05-25 at 15:46 -0400, Andrew Bradford wrote:
> From: Andrew Bradford <andrew.bradford@kodakalaris.com>
> 
> So that perforce fetching can work more like the other source control
> system fetchers.
> 
> Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
> ---
>  meta/conf/bitbake.conf | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index acef30b..1aa1d37 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -339,6 +339,7 @@ SVNDIR = "${CO_DIR}/svn"
>  GITDIR = "${CO_DIR}/git2"
>  BZRDIR = "${CO_DIR}/bzr"
>  HGDIR = "${CO_DIR}/hg"
> +P4DIR = "${CO_DIR}/p4"
>  
>  STAMPS_DIR ?= "${TMPDIR}/stamps"
>  STAMP =
> "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}"
> @@ -618,6 +619,7 @@ FETCHCMD_cvs = "/usr/bin/env cvs"
>  FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no
> -check-certificate"
>  FETCHCMD_bzr = "/usr/bin/env bzr"
>  FETCHCMD_hg = "/usr/bin/env hg"
> +FETCHCMD_p4 = "/usr/bin/env p4"

I've kind of been hoping we could move to a model where the fetcher has
sane defaults and we don't need to change anything in bitbake.conf
unless the user has some need to customise.

The FETCHCMD is easy to do that for, the P4DIR is slightly harder since
it means teaching the fetcher about CO_DIR which bitbake currently
knows nothing about. It may be worth dropping the notion of CO_DIR and
making the default ${DL_DIR}/p4 which bitbake does know about and
making the other fetchers match which such defaults?

Cheers,

Richard

Re: [PATCH 2/3] bitbake.conf: Add P4DIR and FETCHCMD_p4

[Andrew Bradford]

Hi Richard,

On 06/02 16:56, Richard Purdie wrote:
> On Wed, 2016-05-25 at 15:46 -0400, Andrew Bradford wrote:
> > From: Andrew Bradford <andrew.bradford@kodakalaris.com>
> > 
> > So that perforce fetching can work more like the other source control
> > system fetchers.
> > 
> > Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
> > ---
> >  meta/conf/bitbake.conf | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index acef30b..1aa1d37 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -339,6 +339,7 @@ SVNDIR = "${CO_DIR}/svn"
> >  GITDIR = "${CO_DIR}/git2"
> >  BZRDIR = "${CO_DIR}/bzr"
> >  HGDIR = "${CO_DIR}/hg"
> > +P4DIR = "${CO_DIR}/p4"
> >  
> >  STAMPS_DIR ?= "${TMPDIR}/stamps"
> >  STAMP =
> > "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}"
> > @@ -618,6 +619,7 @@ FETCHCMD_cvs = "/usr/bin/env cvs"
> >  FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no
> > -check-certificate"
> >  FETCHCMD_bzr = "/usr/bin/env bzr"
> >  FETCHCMD_hg = "/usr/bin/env hg"
> > +FETCHCMD_p4 = "/usr/bin/env p4"
> 
> I've kind of been hoping we could move to a model where the fetcher has
> sane defaults and we don't need to change anything in bitbake.conf
> unless the user has some need to customise.
> 
> The FETCHCMD is easy to do that for, the P4DIR is slightly harder since
> it means teaching the fetcher about CO_DIR which bitbake currently
> knows nothing about. It may be worth dropping the notion of CO_DIR and
> making the default ${DL_DIR}/p4 which bitbake does know about and
> making the other fetchers match which such defaults?

OK, I'll take a look at that.  It should be easy to have a sane default
for P4DIR and even FETCHCMD_p4 in the fetcher itself and not need
changes to bitbake.conf.

Thanks for the feedback! :)
-Andrew

Re: [PATCH 3/3] base.bbclass: p4 fetcher supports srcrev

[Andrew Bradford]

Hi,

On 05/25 15:46, Andrew Bradford wrote:
> From: Andrew Bradford <andrew.bradford@kodakalaris.com>
> 
> Enable the perforce fetcher to call bb.fetch2.get_srcrev() as it can use
> 'SRCREV = "${AUTOREV}"'.
> 
> Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
> ---
>  meta/classes/base.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 4be0a7e..9f3945c 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -569,6 +569,10 @@ python () {
>              needsrcrev = True
>              d.appendVarFlag('do_fetch', 'depends', ' mercurial-native:do_populate_sysroot')
>  
> +        # Perforce packages support SRCREV = "${AUTOREV}"
> +        elif scheme == "p4":
> +            needsrcrev = True
> +
>          # OSC packages should DEPEND on osc-native
>          elif scheme == "osc":
>              d.appendVarFlag('do_fetch', 'depends', ' osc-native:do_populate_sysroot')

I noticed that this patch has not been applied to oe-core, but it is a
dependency for the bitbake perforce fetcher change to support srcrev
which has been applied to bitbake's master branch [1].

[1]:http://git.openembedded.org/bitbake/commit/?id=6298696bb94a127cdec7964315f6891ba92cd026

Any chance that this patch could be applied to oe-core?
The other two patches in this series are not needed, just this one.
Sorry if my bitbake patch was not super clear about this dependency.

Thanks!
-Andrew

Re: [PATCH 3/3] base.bbclass: p4 fetcher supports srcrev

[Richard Purdie]

On Fri, 2016-06-17 at 08:52 -0400, Andrew Bradford wrote:
> On 05/25 15:46, Andrew Bradford wrote:
> I noticed that this patch has not been applied to oe-core, but it is
> a
> dependency for the bitbake perforce fetcher change to support srcrev
> which has been applied to bitbake's master branch [1].
> 
> [1]:http://git.openembedded.org/bitbake/commit/?id=6298696bb94a127cde
> c7964315f6891ba92cd026
> 
> Any chance that this patch could be applied to oe-core?
> The other two patches in this series are not needed, just this one.
> Sorry if my bitbake patch was not super clear about this dependency.

Normally patches land in bitbake first, then the code which uses them
goes into OE-Core and we bump the minimum bitbake version. In this case
I doubt we need the version bump as p4 isn't widely used.

I've merged the patch, thanks for the reminder we need that one.

Cheers,

Richard