From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dan.rpsys.net (5751f4a1.skybroadband.com [87.81.244.161]) by mail.openembedded.org (Postfix) with ESMTP id CFC6A605BF for ; Thu, 22 Sep 2016 12:16:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by dan.rpsys.net (8.14.4/8.14.4/Debian-4.1ubuntu1) with ESMTP id u8MCGXah012027; Thu, 22 Sep 2016 13:16:33 +0100 Received: from dan.rpsys.net ([127.0.0.1]) by localhost (dan.rpsys.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id IulRI80v_0Wx; Thu, 22 Sep 2016 13:16:33 +0100 (BST) Received: from hex ([192.168.3.34]) (authenticated bits=0) by dan.rpsys.net (8.14.4/8.14.4/Debian-4.1ubuntu1) with ESMTP id u8MCGRM2011966 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 22 Sep 2016 13:16:29 +0100 Message-ID: <1474546587.7207.361.camel@linuxfoundation.org> From: Richard Purdie To: Pascal Bach , Christopher Larson Date: Thu, 22 Sep 2016 13:16:27 +0100 In-Reply-To: <1474539467.7207.357.camel@linuxfoundation.org> References: <1474016293-26538-1-git-send-email-pascal.bach@siemens.com> <508fb644-1ed9-8c92-e6ed-e09f52c03e00@siemens.com> <64d81891-801e-292e-e26c-1c44bd38a8c5@siemens.com> <02946d54-8e3d-78ce-a094-e06365273cf3@siemens.com> <1474539467.7207.357.camel@linuxfoundation.org> X-Mailer: Evolution 3.18.5.2-0ubuntu3 Mime-Version: 1.0 Cc: "bitbake-devel@lists.openembedded.org" Subject: Re: [PATCH] fetch2/git: always use premirror first if update is required X-BeenThere: bitbake-devel@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussion that advance bitbake development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2016 12:16:38 -0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit On Thu, 2016-09-22 at 11:17 +0100, Richard Purdie wrote: > On Thu, 2016-09-22 at 09:45 +0200, Pascal Bach wrote: > > > > > > > > > > > > > > > > > > > >     The tarball should only be fetched if the local ud.clonedir > > > > is out of date, this means the revision that is requested via > > > > SRCREV is not in the local repository. > > > >     In the case the revision is present the tarball should not > > > > be > > > > fetched again, at least that was the intention and my local > > > > tests > > > > didn't indicate otherwise. > > > > > > > > > > > > Yes, but it’s better to run ‘git fetch’ in an out of date clone > > > > than to re-download a 1gb mirror tarball which may well be out > > > > of > > > > date too. > > > Agreed, but this doesn't work in the case where the machine > > > doesn't > > > have access to the upstream git repository. For example in the > > > case > > > where BB_ALLOWED_NETWORKS = "*.my.company". The fetcher would > > > still > > > got to to github.com which is not allowed, in that case it should > > > fall back to fetch the tarball from "mirror.my.company". > > > > > > I'm open for suggestions how to make this more efficient. > > Does anyone have a suggestion how this should behave in the ideal > > case? In my opinion if there is an internal mirror defined it > > should > > always have precedence even if it might be less efficient. > "less efficient" in this case translates a few seconds of git fetch > operation on something like linux-yocto into a hundreds of megabytes > download. > > Much as you might not like it, I really don't think we can change the > code as your patch does as it would badly affect standard usage for > many users. > > We need to find an alternative which allows your use case to work but > I'm not sure what that is. Much as I hate it, we may just have to > have > a setting which you can set which changes the behaviour. If we do > that, > I will want to see test cases added for it though as the fetcher code > is a mass of different configurations and its very very difficult to > maintain as it is without yet more different code paths :(. FWIW there is an open bug which I believe is related to this: https://bugzilla.yoctoproject.org/show_bug.cgi?id=9061 It might be worth making the code aware of git aware premirror sources compared to other things like mirror tarballs as a start at fixing this. Cheers, Richard