Disk write deterioration in 5.x kernel

Disk write deterioration in 5.x kernel

[Gowtham]

Hi

We use BTRFS as a root filesystem and have recently upgraded our
servers from Ubuntu 16.04 to 20.04 (from 4.15.0-36 linux kernel to
5.4.0-137). Post that we see a deterioration in the disk write speeds

# dd if=/dev/zero of=/var/nvOS/log/dd.img bs=1G count=2 oflag=dsync;date
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.72458 s, 788 MB/s

vs

# dd if=/dev/zero of=/var/nvOS/log/dd.img bs=1G count=2 oflag=dsync;date
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.11866 s, 689 MB/s

# btrfs --version
btrfs-progs v5.4.1

BTRFS options
rw,noatime,compress=lzo,ssd,flushoncommit,space_cache,subvolid=269,subvol=

On average we see 10-15% slower speeds on 20.04 when compared with
16.04. And this data has been recorded from different systems and
different types of SSD.

Are there any known enhancements in BTRFS that can affect the write speeds?

Regards,
Gowtham

Re: Disk write deterioration in 5.x kernel

[Gowtham]

Hi

Is there anything I can collect to debug what is the problem in the new kernel?

Regards,
Gowtham

On Tue, Jan 16, 2024 at 11:21 AM Gowtham <trgowtham123@gmail.com> wrote:
>
> Hi
>
> We use BTRFS as a root filesystem and have recently upgraded our
> servers from Ubuntu 16.04 to 20.04 (from 4.15.0-36 linux kernel to
> 5.4.0-137). Post that we see a deterioration in the disk write speeds
>
> # dd if=/dev/zero of=/var/nvOS/log/dd.img bs=1G count=2 oflag=dsync;date
> 2+0 records in
> 2+0 records out
> 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.72458 s, 788 MB/s
>
> vs
>
> # dd if=/dev/zero of=/var/nvOS/log/dd.img bs=1G count=2 oflag=dsync;date
> 2+0 records in
> 2+0 records out
> 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.11866 s, 689 MB/s
>
> # btrfs --version
> btrfs-progs v5.4.1
>
> BTRFS options
> rw,noatime,compress=lzo,ssd,flushoncommit,space_cache,subvolid=269,subvol=
>
> On average we see 10-15% slower speeds on 20.04 when compared with
> 16.04. And this data has been recorded from different systems and
> different types of SSD.
>
> Are there any known enhancements in BTRFS that can affect the write speeds?
>
> Regards,
> Gowtham

Re: Disk write deterioration in 5.x kernel

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

[also Cc: btrfs maintainers]

On Fri, Jan 19, 2024 at 09:37:50AM +0530, Gowtham wrote:
> Hi
> 
> Is there anything I can collect to debug what is the problem in the new kernel?
> 

Please don't top-post, reply inline with appropriate context instead.

Since you have regression w.r.t. old LTS kernel (v5.4.y), can you check
current mainline (v6.7) to confirm or deny the regression? If the
regression still happens there, can you perform bisection (see
Documentation/admin-guide/bug-bisect.rst in the kernel sources for details)?

Also, it is also helpful to list SSD devices that you've tested.

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Disk write deterioration in 5.x kernel

[Linux regression tracking (Thorsten Leemhuis)]

On 19.01.24 07:58, Bagas Sanjaya wrote:
> [also Cc: btrfs maintainers]
> 
> On Fri, Jan 19, 2024 at 09:37:50AM +0530, Gowtham wrote:
>>
>> Is there anything I can collect to debug what is the problem in the new kernel?

From the version numbers you provided it seems you are using vendor
kernels containing patches. You thus might want to ask the vendor for
support. Most upstream developers are unlikely to help and some even
complete ignore reports using such kernels.

You also need to try latest mainline and bisect, as Bagas already
pointed out, as that problem might be solved already and might have
nothing to do with Btrfs at all.

> Please don't top-post, reply inline with appropriate context instead.

Bagas, FWIW, I think telling users this is not helpful at all and maybe
counter productive; please consider to stop doing this.

Yes, kernel development uses inline replies -- hence it's a good idea to
point that out *to developers* that submit patches et. al.

But most people in the world uses top-posting; you and I might not like
that, but that's how it is. Telling non-developers to adjust their
behavior to our habits will often just come over as rude. It might
nevertheless be worth it. But it's best to leave that decision to the
developers that handle the report, as they have to interact way more
with the reporter that you or I will have to.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: Disk write deterioration in 5.x kernel

[Bagas Sanjaya]

On 1/19/24 15:55, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 19.01.24 07:58, Bagas Sanjaya wrote:
>> [also Cc: btrfs maintainers]
>>
>> On Fri, Jan 19, 2024 at 09:37:50AM +0530, Gowtham wrote:
>>>
>>> Is there anything I can collect to debug what is the problem in the new kernel?
> 
> From the version numbers you provided it seems you are using vendor
> kernels containing patches. You thus might want to ask the vendor for
> support. Most upstream developers are unlikely to help and some even
> complete ignore reports using such kernels.
> 
> You also need to try latest mainline and bisect, as Bagas already
> pointed out, as that problem might be solved already and might have
> nothing to do with Btrfs at all.
> 
>> Please don't top-post, reply inline with appropriate context instead.
> 
> Bagas, FWIW, I think telling users this is not helpful at all and maybe
> counter productive; please consider to stop doing this.
> 

I was writing the reminder like broonie did.

> Yes, kernel development uses inline replies -- hence it's a good idea to
> point that out *to developers* that submit patches et. al.
> 
> But most people in the world uses top-posting; you and I might not like
> that, but that's how it is. Telling non-developers to adjust their
> behavior to our habits will often just come over as rude. It might
> nevertheless be worth it. But it's best to leave that decision to the
> developers that handle the report, as they have to interact way more
> with the reporter that you or I will have to.
> 

OK, thanks!

-- 
An old man doll... just what I always wanted! - Clara