cloning the linxu kernel repo at a VPS with small RAM

cloning the linxu kernel repo at a VPS with small RAM

[Toralf Förster]

Situation:

The command
     git clone
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
fails at a virtual server with about 2 GiB RAM under a recent Debian
bookworm with git 2.39.2. What works for me:
       git clone --depth=1
https://git.kernel.org/pub/scm/linux/kernel/git/stable
       cd ./linux
       git config gc.auto 0
       git config pack.threads 1
       git fetch --tags
as seen in [1].

Q:
I do wonder if Git could automatically try to deal with only 1.5 GiB
available RAM?

TIA

[1]
https://github.com/toralf/tor-relays/blob/main/playbooks/roles/setup/tasks/kernel-git.yaml#L22
--
Toralf

Re: cloning the linxu kernel repo at a VPS with small RAM

[Dragan Simic]

Hello Toralf,

On 2024-02-08 18:32, Toralf Förster wrote:
> Situation:
> 
> The command
>     git clone
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> fails at a virtual server with about 2 GiB RAM under a recent Debian
> bookworm with git 2.39.2. What works for me:
>       git clone --depth=1
> https://git.kernel.org/pub/scm/linux/kernel/git/stable
>       cd ./linux
>       git config gc.auto 0
>       git config pack.threads 1
>       git fetch --tags
> as seen in [1].
> 
> Q:
> I do wonder if Git could automatically try to deal with only 1.5 GiB
> available RAM?

Here's an excerpt from my ~/.bashrc, which sets the things up
on my Pinebook Pro laptop with an RK3399 SoC and 4 GB of RAM,
which is also thermally constrained:

# Missing nproc(1) is handled properly
REASONABLE_THREADS=$(nproc 2> /dev/null || echo 1)
REASONABLE_THREADS=$((${REASONABLE_THREADS} / 2))
((${REASONABLE_THREADS} == 0)) && REASONABLE_THREADS=1

export GIT_CONFIG_COUNT=3
export GIT_CONFIG_KEY_0='grep.threads'
export GIT_CONFIG_VALUE_0=${REASONABLE_THREADS}
export GIT_CONFIG_KEY_1='index.threads'
export GIT_CONFIG_VALUE_1=${REASONABLE_THREADS}
export GIT_CONFIG_KEY_2='pack.threads'
export GIT_CONFIG_VALUE_2=${REASONABLE_THREADS}

export ZSTD_NBTHREADS=${REASONABLE_THREADS}
unset REASONABLE_THREADS

Obviously, this does a bit more than configuring git only.

Perhaps modifying this to additionally take the amount of RAM
into the calculation of REASONABLE_THREADS could be a solution
for your use case?

Re: cloning the linxu kernel repo at a VPS with small RAM

[Dragan Simic]

On 2024-02-08 21:16, Dragan Simic wrote:
> On 2024-02-08 18:32, Toralf Förster wrote:
>> Situation:
>> 
>> The command
>>     git clone
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>> fails at a virtual server with about 2 GiB RAM under a recent Debian
>> bookworm with git 2.39.2. What works for me:
>>       git clone --depth=1
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable
>>       cd ./linux
>>       git config gc.auto 0
>>       git config pack.threads 1
>>       git fetch --tags
>> as seen in [1].
>> 
>> Q:
>> I do wonder if Git could automatically try to deal with only 1.5 GiB
>> available RAM?
> 
> Here's an excerpt from my ~/.bashrc, which sets the things up
> on my Pinebook Pro laptop with an RK3399 SoC and 4 GB of RAM,
> which is also thermally constrained:
> 
> # Missing nproc(1) is handled properly
> REASONABLE_THREADS=$(nproc 2> /dev/null || echo 1)
> REASONABLE_THREADS=$((${REASONABLE_THREADS} / 2))
> ((${REASONABLE_THREADS} == 0)) && REASONABLE_THREADS=1
> 
> export GIT_CONFIG_COUNT=3
> export GIT_CONFIG_KEY_0='grep.threads'
> export GIT_CONFIG_VALUE_0=${REASONABLE_THREADS}
> export GIT_CONFIG_KEY_1='index.threads'
> export GIT_CONFIG_VALUE_1=${REASONABLE_THREADS}
> export GIT_CONFIG_KEY_2='pack.threads'
> export GIT_CONFIG_VALUE_2=${REASONABLE_THREADS}
> 
> export ZSTD_NBTHREADS=${REASONABLE_THREADS}
> unset REASONABLE_THREADS
> 
> Obviously, this does a bit more than configuring git only.
> 
> Perhaps modifying this to additionally take the amount of RAM
> into the calculation of REASONABLE_THREADS could be a solution
> for your use case?

Actually, perhaps we could provide the following files as part
of the git project, which would take the above-described approach,
and let the distributions package them:

  - /etc/profile.d/git.sh
  - /etc/profile.d/git.csh

If everyone agrees, I'd we willing to take a crack on these.
Of course, all suggestions and thoughts are more than welcome.

Re: cloning the linxu kernel repo at a VPS with small RAM

[Eric Wong]

Toralf Förster <toralf.foerster@gmx.de> wrote:
> I do wonder if Git could automatically try to deal with only 1.5 GiB
> available RAM?

I have to use pack.threads=1 myself on SMP machines, too.

_SC_AVPHYS_PAGES should probably be accounted for in addition
to _SC_NPROCESSORS_CONF when calculating the default pack.threads value.
Calculating good defaults is hard, though...

I've also been meaning to try the newer tombstone-free khashl
to replace khash in our tree for memory savings:

https://attractivechaos.wordpress.com/2019/12/28/deletion-from-hash-tables-without-tombstones/

Perhaps you (or someone else) can take the lead, there.

Re: cloning the linxu kernel repo at a VPS with small RAM

[Toralf Förster]

On 2/8/24 18:32, Toralf Förster wrote:
>
> Q:
> I do wonder if Git could automatically try to deal with only 1.5 GiB
> available RAM?

Given that git fails for the linux kernel repo - where Git was used
first in the wild (?) - it should definitely works for that repo even
with a small RAM, right?

--
Toralf