SSTATE_MIRRORS with ssh/sftp fetchers #yocto

SSTATE_MIRRORS with ssh/sftp fetchers #yocto

[angeal1105]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

Hi everyone,

Recently, I tried to use the SSTATE_MIRRORS variable to get back sstate cache from a build machine shared amongst developers.
When I tried with the simple FILE fetcher, it worked great and resumed the work where the build machine stopped.
But when I tried to replace it with SSH and/or SFTP fetchers (setup to connect on my own machine), I had no errors or warnings, but the steps started from scratch, ignoring the sstate cache mirror.

The thing is that we can't use unauthenticated HTTP/FTP fetchers for corporate security reasons.
SSH and SFTP are convenients as we use crops and mount ssh keys inside to authenticate ourselves on corporate git servers.

Environment:

I'm using Yocto 2.6 (Thud) and CROPS.
I've set up my machine as SSH/SFTP server, and I tried it manually, it works: the connection is established (even in CROPS).

I defined the SSTATE_MIRRORS variable in the build/conf/local.conf file.
*Working* : SSTATE_MIRRORS = "file://.* file://{COMMON_PATH}/sstate-cache/PATH"
*Not working* : SSTATE_MIRRORS = "file://.* ssh://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"
*Not* *working* : SSTATE_MIRRORS = "file://.* sftp://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"

Do you have any idea on how to use these fetchers (or other SSH keys-based secure fetcher) to share sstate cache ?

Don't hesitate to ask for more information/tests from me.
Regards,
Anthony

[-- Attachment #2: Type: text/html, Size: 1667 bytes --]

Re: [yocto] SSTATE_MIRRORS with ssh/sftp fetchers #yocto

[Richard Purdie]

On Thu, 2019-12-19 at 02:37 -0800, angeal1105@gmail.com wrote:
> Hi everyone,
> 
> Recently, I tried to use the SSTATE_MIRRORS variable to get back
> sstate cache from a build machine shared amongst developers.
> When I tried with the simple FILE fetcher, it worked great and
> resumed the work where the build machine stopped.
> But when I tried to replace it with SSH and/or SFTP fetchers (setup
> to connect on my own machine), I had no errors or warnings, but the
> steps started from scratch, ignoring the sstate cache mirror.
> 
> The thing is that we can't use unauthenticated HTTP/FTP fetchers for
> corporate security reasons. 
> SSH and SFTP are convenients as we use crops and mount ssh keys
> inside to authenticate ourselves on corporate git servers.
> 
> Environment:
> 
> I'm using Yocto 2.6 (Thud) and CROPS.
> I've set up my machine as SSH/SFTP server, and I tried it manually,
> it works: the connection is established (even in CROPS).
> 
> I defined the SSTATE_MIRRORS variable in the build/conf/local.conf
> file.
> Working:        SSTATE_MIRRORS = "file://.*
> file://{COMMON_PATH}/sstate-cache/PATH"
> Not working:  SSTATE_MIRRORS = "file://.* 
> ssh://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"
> Notworking:  SSTATE_MIRRORS = "file://.* 
> sftp://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"
> 
> Do you have any idea on how to use these fetchers (or other SSH keys-
> based secure fetcher) to share sstate cache ?

I think this is a "not implemented" problem.

If you look at bitbake/lib/bb/fetch2/ssh.py or sftp.py, they don't have
a checkstatus() method that local.py or wget.py has.

I think the sstate code relies on the checkstatus method working to
know know if an sstate artefact is present.

Performance of that function is key and creating a new ssh connection
to check for each sstate artefact may make the build prohibitively
slow. There are some connection caching mechanisms present which may
help with that.

Probably not the answer you wanted, sorry but at least it might explain
why its not working.

Cheers,

Richard

Re: [yocto] SSTATE_MIRRORS with ssh/sftp fetchers #yocto

[Ross Burton]

On 19/12/2019 11:16, Richard Purdie wrote:
>> Do you have any idea on how to use these fetchers (or other SSH keys-
>> based secure fetcher) to share sstate cache ?
> 
> I think this is a "not implemented" problem.
> 
> If you look at bitbake/lib/bb/fetch2/ssh.py or sftp.py, they don't have
> a checkstatus() method that local.py or wget.py has.
> 
> I think the sstate code relies on the checkstatus method working to
> know know if an sstate artefact is present.
> 
> Performance of that function is key and creating a new ssh connection
> to check for each sstate artefact may make the build prohibitively
> slow. There are some connection caching mechanisms present which may
> help with that.

I actually started implementing this a while ago:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=ross/mut2&id=42bdcd8181d2826a5a5dff6803958a900c0c4a2d

I *think* that worked but I can't be sure, it was a while ago and 
typically I forgot to put notes to my future self in the commit message.

As RP says, you'll want to use connection sharing: this is configured 
using ControlMaster and friends in the ssh configuration.  That should 
make the performance reasonable.

Ross

Re: SSTATE_MIRRORS with ssh/sftp fetchers #yocto

[angeal1105]

[-- Attachment #1: Type: text/plain, Size: 262 bytes --]

Thank you both very much for your quick answers !

I will definitely give a shot at Ross' contrib. I also thought about mounting a sshFS when the container starts and trick Yocto by using the FILE protocol.
I will compare the performances.

Thanks again !

[-- Attachment #2: Type: text/html, Size: 289 bytes --]