grokmirror from kernel.googlesource.com

grokmirror from kernel.googlesource.com

[Vitaly Chikunov]

Hi,

Is it OK to mirror git.kernel.org from it's mirror
kernel.googlesource.com using grokmirror?

I am not sure the manifest is updated after it's already mirrored there
and it doesn't get racy.

(Recently we getting 502 Bad Gateway from git fetch from git.kernel.org
directly.)

Thanks,

Re: grokmirror from kernel.googlesource.com

[Konstantin Ryabitsev]

On Sat, Mar 30, 2024 at 05:24:39PM +0300, Vitaly Chikunov wrote:
> Is it OK to mirror git.kernel.org from it's mirror
> kernel.googlesource.com using grokmirror?

I don't recommend it because kernel.googlesource.com doesn't publish its own
manifest, so you are always going to be in a race between our manifest and
their mirror.

> (Recently we getting 502 Bad Gateway from git fetch from git.kernel.org
> directly.)

Well, at least grokmirror should recover from such conditions. It doesn't
happen very frequently, but every now and again a CI farm will try to use
git.kernel.org directly and that tends to cause problems.

-K

Re: grokmirror from kernel.googlesource.com

[Vitaly Chikunov]

Konstantin,

On Wed, Apr 03, 2024 at 05:02:07PM -0400, Konstantin Ryabitsev wrote:
> On Sat, Mar 30, 2024 at 05:24:39PM +0300, Vitaly Chikunov wrote:
> > Is it OK to mirror git.kernel.org from it's mirror
> > kernel.googlesource.com using grokmirror?
> 
> I don't recommend it because kernel.googlesource.com doesn't publish its own
> manifest, so you are always going to be in a race between our manifest and
> their mirror.

Ah, good to know!

> 
> > (Recently we getting 502 Bad Gateway from git fetch from git.kernel.org
> > directly.)
> 
> Well, at least grokmirror should recover from such conditions. It doesn't
> happen very frequently, but every now and again a CI farm will try to use
> git.kernel.org directly and that tends to cause problems.

Good to know either. I noticed if I press ^C while grok-pull run then on the
next run it thinks work is already done and it looked racy.

  $ grok-pull -c grokmirror-korg.conf -v
   manifest: fetching https://git.kernel.org/manifest.js.gz
   manifest: 2 relevant entries
     search: finding all repos in /home/kernelbot/mirror
     worker: started (1 running)
      fetch: /pub/scm/linux/kernel/git/stable/linux.git
  ^CExiting on signal 2
  Exiting on signal 2
  $ grok-pull -c grokmirror-korg.conf -v
   manifest: fetching https://git.kernel.org/manifest.js.gz
   manifest: unchanged

Perhaps, with frequent runs it will self-recover. But this brings
another question - what is recommended minimal cron interval for
grok-pull for git.kernel.org (I only want to mirror torvalds and stable
repos)? I have seen in some articles and old READMEs example of every
minute (*) and once per 5 minutes (*/5).

Thanks,

> 
> -K

Re: grokmirror from kernel.googlesource.com

[Konstantin Ryabitsev]

On Thu, Apr 04, 2024 at 05:29:34PM +0300, Vitaly Chikunov wrote:
> Perhaps, with frequent runs it will self-recover. But this brings
> another question - what is recommended minimal cron interval for
> grok-pull for git.kernel.org (I only want to mirror torvalds and stable
> repos)? I have seen in some articles and old READMEs example of every
> minute (*) and once per 5 minutes (*/5).

If you are running grokmirror 2.x (and you should be, hopefully), then you can
just set "refresh" and run it as a service. You can continue running it as a
cronjob, though -- it can be once a minute if you like. It just downloads a
static manifest file, and only if it's changed.

-K

Re: grokmirror from kernel.googlesource.com

[Vitaly Chikunov]

On Thu, Apr 04, 2024 at 10:49:02AM -0400, Konstantin Ryabitsev wrote:
> On Thu, Apr 04, 2024 at 05:29:34PM +0300, Vitaly Chikunov wrote:
> > Perhaps, with frequent runs it will self-recover. But this brings
> > another question - what is recommended minimal cron interval for
> > grok-pull for git.kernel.org (I only want to mirror torvalds and stable
> > repos)? I have seen in some articles and old READMEs example of every
> > minute (*) and once per 5 minutes (*/5).
> 
> If you are running grokmirror 2.x (and you should be, hopefully), then you can
> just set "refresh" and run it as a service. You can continue running it as a
> cronjob, though -- it can be once a minute if you like. It just downloads a
> static manifest file, and only if it's changed.

That's good!

Thanks,

> 
> -K