Multiple conncetion to svn (svn+ssh)

Multiple conncetion to svn (svn+ssh)

[Marcin Krzemiński]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

Hello again,

I have 12 recipes that download code from same svn repository (using
svn+ssh)protocol.
Recipes are grouped in packagegroup.
When I want to bitbake packagegroup fetcher fail, but when I run recipe
alone all is ok.
I think it is because I want to open 8 connection to one svn repository.How
can I fix this, for example by allowing only eg. 2 recipes from
packagegroup to be executed in paralell.

| DEBUG: Python function base_do_fetch finished
| DEBUG: Python function do_fetch finished
| ERROR: Function failed: Fetcher failure: Fetch command failed with exit
code 1, output:
| svn: E210002: Unable to connect to a repository at URL 'svn+ssh://SVN_TAG'
| svn: E210002: To better debug SSH connection problems, remove the -q
option from 'ssh' in the [tunnels] section of your Subversion configuration
file.
| svn: E210002: Network connection closed unexpectedl


My Yocto version is 1.7.2

Regards,
Marcin

[-- Attachment #2: Type: text/html, Size: 1183 bytes --]

Re: Multiple conncetion to svn (svn+ssh)

[Bryan Evenson]

Marcin,

> From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Marcin Krzeminski
> Sent: Friday, July 03, 2015 7:55 AM
> To: yocto@yoctoproject.org
> Subject: [yocto] Multiple conncetion to svn (svn+ssh)
>
> Hello again,
>
> I have 12 recipes that download code from same svn repository (using svn+ssh)protocol.
> Recipes are grouped in packagegroup.
> When I want to bitbake packagegroup fetcher fail, but when I run recipe alone all is ok.
> I think it is because I want to open 8 connection to one svn repository.How can I fix this, for example by allowing only eg. 2 recipes from packagegroup to be executed in paralell.
>

There is no fetcher-specific variable that I know of, but I think you can set PARALLEL_MAKE="2" in your recipes to reduce the number of fetches on your repository at a time.  The
downside is your build will go slower when it is building your recipes as it won't be doing as many things in parallel.

Another option may be to change your Subversion server settings, your build machine's SSH settings, or both.  If you have control over your Subversion server, you may be able to
change the number of allowed simultaneous connections.  See the documentation for your server's Subversion/web hosting/SSH tools to see what your options are.  If you don't
have that kind of control over the Subversion server, then maybe you can change your SSH settings on your build machine to extend the login timeout.  Again, see your specific
tool's documentation to see how to do this for your build machine.

> | DEBUG: Python function base_do_fetch finished
> | DEBUG: Python function do_fetch finished
> | ERROR: Function failed: Fetcher failure: Fetch command failed with exit code 1, output:
> | svn: E210002: Unable to connect to a repository at URL 'svn+ssh://SVN_TAG'
> | svn: E210002: To better debug SSH connection problems, remove the -q option from 'ssh' in the [tunnels] section of your Subversion configuration file.
> | svn: E210002: Network connection closed unexpectedl
>
>
> My Yocto version is 1.7.2
>
> Regards,
> Marcin

Regards,
Bryan

Re: Multiple conncetion to svn (svn+ssh)

[Paul Eggleton]

On Monday 06 July 2015 12:57:39 Bryan Evenson wrote:
> Marcin,
> 
> > From: yocto-bounces@yoctoproject.org
> > [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Marcin Krzeminski
> > Sent: Friday, July 03, 2015 7:55 AM
> > To: yocto@yoctoproject.org
> > Subject: [yocto] Multiple conncetion to svn (svn+ssh)
> > 
> > Hello again,
> > 
> > I have 12 recipes that download code from same svn repository (using
> > svn+ssh)protocol. Recipes are grouped in packagegroup.
> > When I want to bitbake packagegroup fetcher fail, but when I run recipe
> > alone all is ok. I think it is because I want to open 8 connection to one
> > svn repository.How can I fix this, for example by allowing only eg. 2
> > recipes from packagegroup to be executed in paralell.
> There is no fetcher-specific variable that I know of, but I think you can
> set PARALLEL_MAKE="2" in your recipes to reduce the number of fetches on
> your repository at a time.  The downside is your build will go slower when
> it is building your recipes as it won't be doing as many things in
> parallel.

That's not going to help. The parallelism we are talking about here is across 
recipes - PARALLEL_MAKE is just passed through to make within one task in one 
recipe, and make isn't even involved at the fetch stage.

Something that might work would be to set a lockfile on the do_fetch task such 
that only one of the recipes could fetch at once. That could not allow two 
executing at once, but at least it would solve the problem. e.g. you could add 
this to all of the recipes:

do_fetch[lockfiles] += "${TMPDIR}/mysvnlock.lock"

(The file name isn't critical, it just needs to be the same for all of the 
recipes you wish to participate in the exclusive fetching.)

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Multiple conncetion to svn (svn+ssh)

[Bryan Evenson]

Paul,

> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: Monday, July 06, 2015 10:08 AM
> To: Marcin Krzemiński
> Cc: Bryan Evenson; yocto@yoctoproject.org
> Subject: Re: [yocto] Multiple conncetion to svn (svn+ssh)
> 
> On Monday 06 July 2015 12:57:39 Bryan Evenson wrote:
> > Marcin,
> >
> > > From: yocto-bounces@yoctoproject.org
> > > [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Marcin
> > > Krzeminski
> > > Sent: Friday, July 03, 2015 7:55 AM
> > > To: yocto@yoctoproject.org
> > > Subject: [yocto] Multiple conncetion to svn (svn+ssh)
> > >
> > > Hello again,
> > >
> > > I have 12 recipes that download code from same svn repository (using
> > > svn+ssh)protocol. Recipes are grouped in packagegroup.
> > > When I want to bitbake packagegroup fetcher fail, but when I run
> > > recipe alone all is ok. I think it is because I want to open 8
> > > connection to one svn repository.How can I fix this, for example by
> > > allowing only eg. 2 recipes from packagegroup to be executed in paralell.
> > There is no fetcher-specific variable that I know of, but I think you
> > can set PARALLEL_MAKE="2" in your recipes to reduce the number of
> > fetches on your repository at a time.  The downside is your build will
> > go slower when it is building your recipes as it won't be doing as
> > many things in parallel.
> 
> That's not going to help. The parallelism we are talking about here is across
> recipes - PARALLEL_MAKE is just passed through to make within one task in
> one recipe, and make isn't even involved at the fetch stage.

Sorry, I grabbed the wrong variable.  I meant BB_NUMBER_THREADS, not PARALLEL_MAKE.

> 
> Something that might work would be to set a lockfile on the do_fetch task
> such that only one of the recipes could fetch at once. That could not allow
> two executing at once, but at least it would solve the problem. e.g. you could
> add this to all of the recipes:
> 
> do_fetch[lockfiles] += "${TMPDIR}/mysvnlock.lock"
> 
> (The file name isn't critical, it just needs to be the same for all of the recipes
> you wish to participate in the exclusive fetching.)

I had no idea that we could do this.  I don't see any documentation on lockfiles anywhere.
If you use a lockfile, do the all recipes with the same lockfile wait until the lockfile is available
before continuing on with that step?  Is there a timeout for waiting for the lockfile or does
the recipe wait indefinitely?

Regards,
Bryan

> 
> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre

Re: Multiple conncetion to svn (svn+ssh)

[Paul Eggleton]

On Monday 06 July 2015 15:04:57 Bryan Evenson wrote:
> Paul,
> 
> > -----Original Message-----
> > From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> > Sent: Monday, July 06, 2015 10:08 AM
> > To: Marcin Krzemiński
> > Cc: Bryan Evenson; yocto@yoctoproject.org
> > Subject: Re: [yocto] Multiple conncetion to svn (svn+ssh)
> > 
> > On Monday 06 July 2015 12:57:39 Bryan Evenson wrote:
> > > Marcin,
> > > 
> > > > From: yocto-bounces@yoctoproject.org
> > > > [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Marcin
> > > > Krzeminski
> > > > Sent: Friday, July 03, 2015 7:55 AM
> > > > To: yocto@yoctoproject.org
> > > > Subject: [yocto] Multiple conncetion to svn (svn+ssh)
> > > > 
> > > > Hello again,
> > > > 
> > > > I have 12 recipes that download code from same svn repository (using
> > > > svn+ssh)protocol. Recipes are grouped in packagegroup.
> > > > When I want to bitbake packagegroup fetcher fail, but when I run
> > > > recipe alone all is ok. I think it is because I want to open 8
> > > > connection to one svn repository.How can I fix this, for example by
> > > > allowing only eg. 2 recipes from packagegroup to be executed in
> > > > paralell.
> > > 
> > > There is no fetcher-specific variable that I know of, but I think you
> > > can set PARALLEL_MAKE="2" in your recipes to reduce the number of
> > > fetches on your repository at a time.  The downside is your build will
> > > go slower when it is building your recipes as it won't be doing as
> > > many things in parallel.
> > 
> > That's not going to help. The parallelism we are talking about here is
> > across recipes - PARALLEL_MAKE is just passed through to make within one
> > task in one recipe, and make isn't even involved at the fetch stage.
> 
> Sorry, I grabbed the wrong variable.  I meant BB_NUMBER_THREADS, not
> PARALLEL_MAKE.

Sure, but that's not going to work either from within a recipe. It would work at the configuration level but that'll quite severely impact build performance.

> > Something that might work would be to set a lockfile on the do_fetch task
> > such that only one of the recipes could fetch at once. That could not
> > allow
> > two executing at once, but at least it would solve the problem. e.g. you
> > could add this to all of the recipes:
> > 
> > do_fetch[lockfiles] += "${TMPDIR}/mysvnlock.lock"
> > 
> > (The file name isn't critical, it just needs to be the same for all of the
> > recipes you wish to participate in the exclusive fetching.)
> 
> I had no idea that we could do this.  I don't see any documentation on
> lockfiles anywhere. If you use a lockfile, do the all recipes with the same
> lockfile wait until the lockfile is available before continuing on with
> that step?  Is there a timeout for waiting for the lockfile or does the
> recipe wait indefinitely?

Yes, it's just a lock on the specified file - whoever gets there first can
continue, everyone else blocks, and it's an indefinite wait as far as
I'm aware.

It's a little obscure perhaps, but it is in the BitBake manual:

http://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#variable-flags

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Multiple conncetion to svn (svn+ssh)

[mar.krzeminski]

Hi Brian, Paul,

Thanks for response.

W dniu 06.07.2015 o 17:11, Paul Eggleton pisze:
> On Monday 06 July 2015 15:04:57 Bryan Evenson wrote:
>> Paul,
>>
>>> -----Original Message-----
>>> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
>>> Sent: Monday, July 06, 2015 10:08 AM
>>> To: Marcin Krzemiński
>>> Cc: Bryan Evenson; yocto@yoctoproject.org
>>> Subject: Re: [yocto] Multiple conncetion to svn (svn+ssh)
>>>
>>> On Monday 06 July 2015 12:57:39 Bryan Evenson wrote:
>>>> Marcin,
>>>>
>>>>> From: yocto-bounces@yoctoproject.org
>>>>> [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Marcin
>>>>> Krzeminski
>>>>> Sent: Friday, July 03, 2015 7:55 AM
>>>>> To: yocto@yoctoproject.org
>>>>> Subject: [yocto] Multiple conncetion to svn (svn+ssh)
>>>>>
>>>>> Hello again,
>>>>>
>>>>> I have 12 recipes that download code from same svn repository (using
>>>>> svn+ssh)protocol. Recipes are grouped in packagegroup.
>>>>> When I want to bitbake packagegroup fetcher fail, but when I run
>>>>> recipe alone all is ok. I think it is because I want to open 8
>>>>> connection to one svn repository.How can I fix this, for example by
>>>>> allowing only eg. 2 recipes from packagegroup to be executed in
>>>>> paralell.
>>>> There is no fetcher-specific variable that I know of, but I think you
>>>> can set PARALLEL_MAKE="2" in your recipes to reduce the number of
>>>> fetches on your repository at a time.  The downside is your build will
>>>> go slower when it is building your recipes as it won't be doing as
>>>> many things in parallel.
>>> That's not going to help. The parallelism we are talking about here is
>>> across recipes - PARALLEL_MAKE is just passed through to make within one
>>> task in one recipe, and make isn't even involved at the fetch stage.
>> Sorry, I grabbed the wrong variable.  I meant BB_NUMBER_THREADS, not
>> PARALLEL_MAKE.
> Sure, but that's not going to work either from within a recipe. It would work at the configuration level but that'll quite severely impact build performance.
This was my first idea.
>
>>> Something that might work would be to set a lockfile on the do_fetch task
>>> such that only one of the recipes could fetch at once. That could not
>>> allow
>>> two executing at once, but at least it would solve the problem. e.g. you
>>> could add this to all of the recipes:
>>>
>>> do_fetch[lockfiles] += "${TMPDIR}/mysvnlock.lock"
>>>
>>> (The file name isn't critical, it just needs to be the same for all of the
>>> recipes you wish to participate in the exclusive fetching.)
This works as you wrote, it is not a perfect solution for my problem,
but at least builds doesn't fail. Thanks!
>> I had no idea that we could do this.  I don't see any documentation on
>> lockfiles anywhere. If you use a lockfile, do the all recipes with the same
>> lockfile wait until the lockfile is available before continuing on with
>> that step?  Is there a timeout for waiting for the lockfile or does the
>> recipe wait indefinitely?
> Yes, it's just a lock on the specified file - whoever gets there first can
> continue, everyone else blocks, and it's an indefinite wait as far as
> I'm aware.
>
> It's a little obscure perhaps, but it is in the BitBake manual:
>
> http://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#variable-flags
>
> Cheers,
> Paul
>
Regards,
Marcin