Prefetch maintenance might lead to thundering herd issues

Prefetch maintenance might lead to thundering herd issues

[Shubham Kanodia]

The current git-maintenance prefetch command allows daily, weekly, and
hourly schedules.
However, in repositories with many active developers, timing `fetch`
to happen at the same wall clock time can lead to a thundering herd
problem for the backend server.

Ideally, these fetch requests would be spaced out by a small amount,
so as not to lead to a co-ordinated load on the server.

I don't see a way to configure this at the moment in a cross-platform
way at the moment. Perhaps it might be sensible to offset the cron
schedules slightly for users so as to not cause such an issue?

This might come at a small cost of predictability of the run, but that
should be okay as predictability isn't the primary goal for
maintenance.

Re: Prefetch maintenance might lead to thundering herd issues

[Derrick Stolee]

On 9/20/24 11:15 AM, Shubham Kanodia wrote:
> The current git-maintenance prefetch command allows daily, weekly, and
> hourly schedules.
> However, in repositories with many active developers, timing `fetch`
> to happen at the same wall clock time can lead to a thundering herd
> problem for the backend server.
> 
> Ideally, these fetch requests would be spaced out by a small amount,
> so as not to lead to a co-ordinated load on the server.
> 
> I don't see a way to configure this at the moment in a cross-platform
> way at the moment. Perhaps it might be sensible to offset the cron
> schedules slightly for users so as to not cause such an issue?

This is already handled by using a random minute of the hour, as
implemented in 9b43399057 (maintenance: use random minute in cron
scheduler, 2023-08-10), for example. There are similar uses for
systemd, launchctl, and schtasks schedulers (look for uses of
the get_random_minute() method).

Are you noticing that this isn't working as expected?

Thanks,
-Stolee

Re: Prefetch maintenance might lead to thundering herd issues

[Shubham Kanodia]

On Sat, Sep 21, 2024 at 1:20 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 9/20/24 11:15 AM, Shubham Kanodia wrote:
> > The current git-maintenance prefetch command allows daily, weekly, and
> > hourly schedules.
> > However, in repositories with many active developers, timing `fetch`
> > to happen at the same wall clock time can lead to a thundering herd
> > problem for the backend server.
> >
> > Ideally, these fetch requests would be spaced out by a small amount,
> > so as not to lead to a co-ordinated load on the server.
> >
> > I don't see a way to configure this at the moment in a cross-platform
> > way at the moment. Perhaps it might be sensible to offset the cron
> > schedules slightly for users so as to not cause such an issue?
>
> This is already handled by using a random minute of the hour, as
> implemented in 9b43399057 (maintenance: use random minute in cron
> scheduler, 2023-08-10), for example. There are similar uses for
> systemd, launchctl, and schtasks schedulers (look for uses of
> the get_random_minute() method).
>
> Are you noticing that this isn't working as expected?
>
> Thanks,
> -Stolee
>

Ah, thanks for pointing that out. It wasn't really clear from the
multiple examples and cron expressions on the git
maintenance documentation page (which have examples such as `0 1-23 *
* *` for hourly). I should've looked
more deeply into the implementation.

Perhaps adding a line to the existing doc might be of help? What do
you think about adding a line to the
section on scheduling —

diff --git a/Documentation/git-maintenance.txt
b/Documentation/git-maintenance.txt
index 51d0f7e94b..34828cdfe6 100644
--- a/Documentation/git-maintenance.txt
+++ b/Documentation/git-maintenance.txt
@@ -217,11 +217,13 @@ on an hourly basis. Each run executes the
"hourly" tasks. At midnight,
that process also executes the "daily" tasks. At midnight on the first day
of the week, that process also executes the "weekly" tasks. A single
process iterates over each registered repository, performing the scheduled
-tasks for that frequency. Depending on the number of registered
-repositories and their sizes, this process may take longer than an hour.
-In this case, multiple `git maintenance run` commands may run on the same
-repository at the same time, colliding on the object database lock. This
-results in one of the two tasks not running.
+tasks for that frequency. The processes are scheduled to a random minute of
+the hour per client to spread out load from multiple clients (e.g. from
+prefetching). Depending on the number of registered repositories and their
+sizes, this process may take longer than an hour. In this case, multiple
+`git maintenance run` commands may run on the same repository at the same
+time, colliding on the object database lock. This results in one of the two
+tasks not running.
If you find that some maintenance windows are taking longer than one hour
to complete, then consider reducing the complexity of your maintenance

--

Re: Prefetch maintenance might lead to thundering herd issues

[Derrick Stolee]

On 9/22/24 2:40 PM, Shubham Kanodia wrote:
> On Sat, Sep 21, 2024 at 1:20 AM Derrick Stolee <stolee@gmail.com> wrote:

>> This is already handled by using a random minute of the hour, as
>> implemented in 9b43399057 (maintenance: use random minute in cron
>> scheduler, 2023-08-10), for example. There are similar uses for
>> systemd, launchctl, and schtasks schedulers (look for uses of
>> the get_random_minute() method).

> Ah, thanks for pointing that out. It wasn't really clear from the
> multiple examples and cron expressions on the git
> maintenance documentation page (which have examples such as `0 1-23 *
> * *` for hourly). I should've looked
> more deeply into the implementation.
> 
> Perhaps adding a line to the existing doc might be of help? What do
> you think about adding a line to the
> section on scheduling —

I do think a doc change would help. Please submit it as a patch for
review!

Thanks,
-Stolee