From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC0A8C52D6F for ; Wed, 21 Aug 2024 17:41:28 +0000 (UTC) Subject: Re: [PATCH] fetch2: Ensure GCP fetcher checks if file exists before download. To: bitbake-devel@lists.openembedded.org From: "Charlie Johnston" X-Originating-Location: US (4.42.15.78) X-Originating-Platform: Windows Chrome 127 User-Agent: GROUPS.IO Web Poster MIME-Version: 1.0 Date: Wed, 21 Aug 2024 10:41:21 -0700 References: In-Reply-To: Message-ID: <28323.1724262081445932896@lists.openembedded.org> Content-Type: multipart/alternative; boundary="nPkbQ8ujgvdPs9tNdzU2" List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 21 Aug 2024 17:41:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16511 --nPkbQ8ujgvdPs9tNdzU2 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, I actually ran into the slowness of gsutil stat recently and have been test= ing a patch locally that does something similar to what you describe. The d= ifference is that I modified it to drop the runfetchcomd and raise a FetchE= rror if self.gcp_client.bucket(ud.host).blob(path).exists() is false. Is that approach acceptable, or is it better to bundle it into the download= _to_filename logic as you describe? Regards, Charlie --nPkbQ8ujgvdPs9tNdzU2 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
Hello,

I actually ran into the slowness of gsutil stat rece= ntly and have been testing a patch locally that does something similar to w= hat you describe. The difference is that I modified it to drop the runfetch= comd and raise a FetchError if self.gcp_client.bucket(ud.host).blob= (path).exists()  is false.
Is that approach acceptable, or is it better to bundle it into the download_to_filename logic as you describe?
 
Regards,
Charlie
--nPkbQ8ujgvdPs9tNdzU2--