git do_unpack

git do_unpack

[John Klug]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

Recently it has come to my attention that in morty, and maybe later, do_unpack for git does not use the downloads directory, and attempts to contact the git server.

This is unfortunate, as we often do bitbake -c fetchall and give tarballs of our workspace to customers, thinking it would save something, but it appears the savings are limited, and now I have a customer complaint about this, since I have one recipe that uses a private repository.

What is the logic of not using the downloads directory for the source of the unpack?

I recently fixed a recipe by writing an unpack that does not use networking and does a clone of the downloads directory to get around this.  Why is this not done by openembedded-core?

Thanks.

[-- Attachment #2: Type: text/html, Size: 781 bytes --]

Re: [OE-core] git do_unpack

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

It would help if you can:

a) provide log.do_unpack that shows that network access is indeed happening;

b) demonstrate the behavior with oe-core master, as morty is soon eligible
for retro computing museum, and hardly anyone wants to look into that :)

Alex

On Sun, 28 Jun 2020 at 19:27, John Klug <john.klug@multitech.com> wrote:

> Recently it has come to my attention that in morty, and maybe later,
> do_unpack for git does not use the downloads directory, and attempts to
> contact the git server.
>
> This is unfortunate, as we often do bitbake -c fetchall and give tarballs
> of our workspace to customers, thinking it would save something, but it
> appears the savings are limited, and now I have a customer complaint about
> this, since I have one recipe that uses a private repository.
>
> What is the logic of not using the downloads directory for the source of
> the unpack?
>
> I recently fixed a recipe by writing an unpack that does not use
> networking and does a clone of the downloads directory to get around this.
> Why is this not done by openembedded-core?
>
> Thanks.
> 
>

[-- Attachment #2: Type: text/html, Size: 1469 bytes --]

Re: [OE-core] git do_unpack

[John Klug]

a:

(fetch was already run successfully while connected with a VPN).

VPN then shut down.

  $  bitbake -c unpack dhq-client

     ...
  NOTE: Executing RunQueue Tasks
  ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command export  DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1"; export SSH_AGENT_PID="2444"; export SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git  failed with exit code 128, output:
  ssh: connect to host gitlab.multitech.net port 22: Connection timed out
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.

  ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
  ERROR: Logfile of failure stored in: /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
  ERROR: Task (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
  NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.

  Summary: 1 task failed:
    /home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack
  Summary: There was 1 WARNING message shown.
  Summary: There were 2 ERROR messages shown, returning a non-zero exit code.


Line 3 of the do_unpack.log:

  DEBUG: Fetcher accessed the network with the command git -c core.fsyncobjectfiles=0 ls-remote ssh://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git

Testing the download of the git (download is verifiable):
  klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$ git fsck
  Checking object directories: 100% (256/256), done.
  Checking objects: 100% (485/485), done.
  klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$



b.  Too busy writing my own git commands in my own do_unpack to fix this.  Maybe someday.

Re: [OE-core] git do_unpack

[Quentin Schulz]

Hi John,

On Tue, Jun 30, 2020 at 04:45:32AM +0000, John Klug wrote:
> 
> a:
> 
> (fetch was already run successfully while connected with a VPN).
> 
> VPN then shut down.
> 
>   $  bitbake -c unpack dhq-client
> 
>      ...
>   NOTE: Executing RunQueue Tasks
>   ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command export  DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1"; export SSH_AGENT_PID="2444"; export SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git  failed with exit code 128, output:
>   ssh: connect to host gitlab.multitech.net port 22: Connection timed out
>   fatal: Could not read from remote repository.
> 
>   Please make sure you have the correct access rights
>   and the repository exists.
> 
>   ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
>   ERROR: Logfile of failure stored in: /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
>   ERROR: Task (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
>   NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.
> 
>   Summary: 1 task failed:
>     /home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack
>   Summary: There was 1 WARNING message shown.
>   Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
> 
> 
> Line 3 of the do_unpack.log:
> 
>   DEBUG: Fetcher accessed the network with the command git -c core.fsyncobjectfiles=0 ls-remote ssh://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git
> 

Any chance you're using a tag in your SRC_URI?

https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#git-fetcher

or AUTOREV? Which I guess needs network access to know if there's
something to pull or not.

Quentin

Re: [OE-core] git do_unpack

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3483 bytes --]

This happens if the recipe does not specify the exact revision hash to use
in SRCREV.  If it says "use latest revision" or "use this tag", then
bitbake will do ls-remote on the original repo to resolve those to
up-to-date revisions, which is not unreasonable.

What does the recipe look like?

Alex

On Tue, 30 Jun 2020 at 06:45, John Klug <John.Klug@multitech.com> wrote:

>
> a:
>
> (fetch was already run successfully while connected with a VPN).
>
> VPN then shut down.
>
>   $  bitbake -c unpack dhq-client
>
>      ...
>   NOTE: Executing RunQueue Tasks
>   ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command
> export  DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1";
> export SSH_AGENT_PID="2444"; export
> SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export
> PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin";
> export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://
> git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git  failed with exit
> code 128, output:
>   ssh: connect to host gitlab.multitech.net port 22: Connection timed out
>   fatal: Could not read from remote repository.
>
>   Please make sure you have the correct access rights
>   and the repository exists.
>
>   ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
>   ERROR: Logfile of failure stored in:
> /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
>   ERROR: Task
> (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack)
> failed with exit code '1'
>   NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be
> rerun and 1 failed.
>
>   Summary: 1 task failed:
>
> /home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:
> do_unpack
>   Summary: There was 1 WARNING message shown.
>   Summary: There were 2 ERROR messages shown, returning a non-zero exit
> code.
>
>
> Line 3 of the do_unpack.log:
>
>   DEBUG: Fetcher accessed the network with the command git -c
> core.fsyncobjectfiles=0 ls-remote ssh://
> git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git
>
> Testing the download of the git (download is verifiable):
>   klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$
> git fsck
>   Checking object directories: 100% (256/256), done.
>   Checking objects: 100% (485/485), done.
>   klugja@peru
> :~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$
>
>
>
> b.  Too busy writing my own git commands in my own do_unpack to fix this.
> Maybe someday.
>

[-- Attachment #2: Type: text/html, Size: 4265 bytes --]

Re: [OE-core] git do_unpack

[John Klug]

We typically use tags to match PV and name our recipes NAME_TAG.bb

Recipe name:

dhq-client_0.39.bb 

from bitbake -e dhq-client output:

# pre-expansion value:
#   "${PN}-${PV}"
P="dhq-client-0.39"

In recipe:

SRCREV = "${PV}"
SRC_URI = "git://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git;protocol=ssh;nobranch=1 \            file://config.json.sample \            file://dhqclient.init \            file://dhqclient \            file://dhq-rootCA.pem \ "

Sanity check:

jklug@trustytahrdm1:~$ git clone git@gitlab.multitech.net:devicehq/mAPI-dhq-client
Cloning into 'mAPI-dhq-client'...
remote: Counting objects: 485, done.
remote: Compressing objects: 100% (199/199), done.
remote: Total 485 (delta 285), reused 450 (delta 265)
Receiving objects: 100% (485/485), 84.73 KiB | 14.12 MiB/s, done.
Resolving deltas: 100% (285/285), done.
jklug@trustytahrdm1:~$ cd mAPI-dhq-client/
jklug@trustytahrdm1:~/mAPI-dhq-client$ git log -1 --format=%ai 0.39 | head
2020-05-05 09:47:03 -0500
jklug@trustytahrdm1:~/mAPI-dhq-client$ git checkout 0.39
Note: checking out '0.39'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d21aab3 Merge branch 'vp/fix_logs' into 'master'

Re: [OE-core] git do_unpack

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1897 bytes --]

SRCREV should be the actual commit hash that matches the tag, then
do_unpack will not do git ls-remote. Tags can, and do move around.

Alex

On Tue, 30 Jun 2020 at 19:24, John Klug <John.Klug@multitech.com> wrote:

> We typically use tags to match PV and name our recipes NAME_TAG.bb
>
> Recipe name:
>
> dhq-client_0.39.bb
>
> from bitbake -e dhq-client output:
>
> # pre-expansion value:
> #   "${PN}-${PV}"
> P="dhq-client-0.39"
>
> In recipe:
>
> SRCREV = "${PV}"
> SRC_URI = "git://
> git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git;protocol=ssh;nobranch=1
> \            file://config.json.sample \            file://dhqclient.init
> \            file://dhqclient \            file://dhq-rootCA.pem \ "
>
> Sanity check:
>
> jklug@trustytahrdm1:~$ git clone git@gitlab.multitech.net:
> devicehq/mAPI-dhq-client
> Cloning into 'mAPI-dhq-client'...
> remote: Counting objects: 485, done.
> remote: Compressing objects: 100% (199/199), done.
> remote: Total 485 (delta 285), reused 450 (delta 265)
> Receiving objects: 100% (485/485), 84.73 KiB | 14.12 MiB/s, done.
> Resolving deltas: 100% (285/285), done.
> jklug@trustytahrdm1:~$ cd mAPI-dhq-client/
> jklug@trustytahrdm1:~/mAPI-dhq-client$ git log -1 --format=%ai 0.39 | head
> 2020-05-05 09:47:03 -0500
> jklug@trustytahrdm1:~/mAPI-dhq-client$ git checkout 0.39
> Note: checking out '0.39'.
>
> You are in 'detached HEAD' state. You can look around, make experimental
> changes and commit them, and you can discard any commits you make in this
> state without impacting any branches by performing another checkout.
>
> If you want to create a new branch to retain commits you create, you may
> do so (now or later) by using -b with the checkout command again. Example:
>
>   git checkout -b <new-branch-name>
>
> HEAD is now at d21aab3 Merge branch 'vp/fix_logs' into 'master'
>

[-- Attachment #2: Type: text/html, Size: 2614 bytes --]