Performance Degradation with rsync in container or cgroupv2 with MEM limit

Performance Degradation with rsync in container or cgroupv2 with MEM limit

[Alexandru Lazarev]

Hi everyone,
I'm experiencing a significant performance degradation when using
"rsync" to copy files over the network from within a container or
cgroup with memory limits on Oracle Linux 9.2. The issue occurs with
the Red Hat Compatible Kernel (RHCK) 5.14.0-284.11.1.el9_2.x86_64 but
not with the Unbreakable Enterprise Kernel (UEK)
5.15.0-101.103.2.1.el9uek.x86_64.

Details:
Setup: Oracle Linux 9.2 with containers/cgroups having memory limits.
Issue: Network file copying speed drops drastically when memory limits
are hit, specifically when the page cache (inactive files) fills up.
Tests:
- Using "rsync" from within a container or a cgroup to copy data from
remote source.
- Using "pg_basebackup" PostgreSQL data replication between two PG
Containers (Leader vs Replica).
Results:
- Initial high speeds (~100MBps) drop significantly (to ~1MBps) once
memory limits are reached.

Commands to Reproduce:
1. Create cgroup with memory limit and run rsync:
sudo systemd-run --scope --property=MemoryMax=1G rsync -av --progress
rsync://<source_ip>/files /destination_path

2. Test with drop_caches on hosting OS during slow rsync:
free && sync && echo 3 > /proc/sys/vm/drop_caches && free
After cache is dropped, rsync is again fast until MEM limit is reached again

Observations:
- When the container's memory limit is reached, the page cache
(inactive files) fills up, leading to network bandwidth degradation.
- This affects, for example, PostgreSQL replication, causing lag and
potential data loss.

Has anyone else encountered this issue? Any insights or suggestions on
how to address correctly (or maybe workarounds) this would be greatly
appreciated!

Thanks in advance!

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Performance Degradation with rsync in container or cgroupv2 with MEM limit

[Greg KH]

On Wed, Mar 26, 2025 at 02:11:41PM +0200, Alexandru Lazarev wrote:
> Hi everyone,
> I'm experiencing a significant performance degradation when using
> "rsync" to copy files over the network from within a container or
> cgroup with memory limits on Oracle Linux 9.2. The issue occurs with
> the Red Hat Compatible Kernel (RHCK) 5.14.0-284.11.1.el9_2.x86_64 but
> not with the Unbreakable Enterprise Kernel (UEK)
> 5.15.0-101.103.2.1.el9uek.x86_64.

<snip>

As you are paying for Oracle support for these kernels, please contact
them to take advantage of that support.  They are the only ones that can
help you out here as no one in the community has any idea of what is in
these kernels at all, sorry.

good luck!

greg k-h

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Performance Degradation with rsync in container or cgroupv2 with MEM limit

[Alexandru Lazarev]

hi Greg,
Thx for Your feedback - I would appreciate any feedback.

I'am not paying for it - I am doing some tests on a given platform.
Kernel is RHCK.
I posted this question-issue here because maybe it is some generic
kernel issue (I saw several of unanswered threads on various platforms
of rsync degradation in containers) amnd someone can suggest to me
right directions, OR..
I know here in mail list are some people from Red Hat and Oracle which
usually provide their valuable answered, so I am still keep my hopness
on finding some ideas on this issue

On Wed, Mar 26, 2025 at 4:00 PM Greg KH <greg@kroah.com> wrote:
>
> On Wed, Mar 26, 2025 at 02:11:41PM +0200, Alexandru Lazarev wrote:
> > Hi everyone,
> > I'm experiencing a significant performance degradation when using
> > "rsync" to copy files over the network from within a container or
> > cgroup with memory limits on Oracle Linux 9.2. The issue occurs with
> > the Red Hat Compatible Kernel (RHCK) 5.14.0-284.11.1.el9_2.x86_64 but
> > not with the Unbreakable Enterprise Kernel (UEK)
> > 5.15.0-101.103.2.1.el9uek.x86_64.
>
> <snip>
>
> As you are paying for Oracle support for these kernels, please contact
> them to take advantage of that support.  They are the only ones that can
> help you out here as no one in the community has any idea of what is in
> these kernels at all, sorry.
>
> good luck!
>
> greg k-h

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Performance Degradation with rsync in container or cgroupv2 with MEM limit

[Alexandru Lazarev]

I'll answer my own question here:
Looks like it is a bug in Oracle Linux (OL) 9.2 RHCK Kernel
"kernel-5.14.0-284.11.1" (see table:
https://docs.oracle.com/en/operating-systems/oracle-linux/9/boot/oracle_linux9_kernel_version_matrix.html)

Switching to UEK Kernel of the same OL version (9.2) fixed the issue,
but UEK is not supported by some SW (e.g. Vertica DB), so that I've
tried RHCK Kernels update: only Kernel and dependencies from OL9.4
(kernel-5.14.0-427) and OL9.5 (kernel-5.14.0-503.11.1) - both fixed
the issue, with remark: RHCK update using one from OL9.5 is suspected
to have some HW compatibility issues (under investigation).
Some more details discussed here:
https://www.linuxquestions.org/questions/linux-containers-122/performance-degradation-with-rsync-in-container-or-cgroupv2-with-mem-limit-on-oracle-linux-9-2-rhck-5-14-vs-uek-5-15-a-4175749063/

On Wed, Mar 26, 2025 at 4:10 PM Alexandru Lazarev
<alexandru.lazarev@gmail.com> wrote:
>
> hi Greg,
> Thx for Your feedback - I would appreciate any feedback.
>
> I'am not paying for it - I am doing some tests on a given platform.
> Kernel is RHCK.
> I posted this question-issue here because maybe it is some generic
> kernel issue (I saw several of unanswered threads on various platforms
> of rsync degradation in containers) amnd someone can suggest to me
> right directions, OR..
> I know here in mail list are some people from Red Hat and Oracle which
> usually provide their valuable answered, so I am still keep my hopness
> on finding some ideas on this issue
>
> On Wed, Mar 26, 2025 at 4:00 PM Greg KH <greg@kroah.com> wrote:
> >
> > On Wed, Mar 26, 2025 at 02:11:41PM +0200, Alexandru Lazarev wrote:
> > > Hi everyone,
> > > I'm experiencing a significant performance degradation when using
> > > "rsync" to copy files over the network from within a container or
> > > cgroup with memory limits on Oracle Linux 9.2. The issue occurs with
> > > the Red Hat Compatible Kernel (RHCK) 5.14.0-284.11.1.el9_2.x86_64 but
> > > not with the Unbreakable Enterprise Kernel (UEK)
> > > 5.15.0-101.103.2.1.el9uek.x86_64.
> >
> > <snip>
> >
> > As you are paying for Oracle support for these kernels, please contact
> > them to take advantage of that support.  They are the only ones that can
> > help you out here as no one in the community has any idea of what is in
> > these kernels at all, sorry.
> >
> > good luck!
> >
> > greg k-h

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies