Re: [linus:master] [iov_iter] c9eec08bac: vm-scalability.throughput -16.9% regression

Linus Torvalds
2023-11-15T17:39:37Z

On Wed, 15 Nov 2023 at 11:53, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> I wonder if gcc somehow decided to inline "memcpy()" in
> memcpy_from_iter() as a "rep movsb" because of other inlining changes?
>
> [ Goes out to look ]
>
> Yup, I think that's exactly what happened. Gcc seems to decide that it
> might be a small memcpy(), and seems to do at least part of it
> directly.
>
> So I *think* this all is mainly an artifact of gcc having changed code
> generation due to the code re-organization.

The gcc code generation here is *really* odd. I've never seen this
before, so it may be new to newer versions of gcc. I see code like
this:

# lib/iov_iter.c:73: memcpy(to + progress, iter_from, len);
cmpl $8, %edx #, _88
jb .L400 #,
movq (%rsi), %rax #, tmp288
movq %rax, (%rcx) # tmp288,
movl %edx, %eax # _88, _88
movq -8(%rsi,%rax), %rdi #, tmp295
movq %rdi, -8(%rcx,%rax) # tmp295,
leaq 8(%rcx), %rdi #, tmp296
andq $-8, %rdi #, tmp296
subq %rdi, %rcx # tmp296, tmp268
subq %rcx, %rsi # tmp268, tmp269
addl %edx, %ecx # _88, _88
shrl $3, %ecx #,
rep movsq
jmp .L392 #

.L398:
# lib/iov_iter.c:73: memcpy(to + progress, iter_from, len);
movl (%rsi), %eax #, tmp271
movl %eax, (%rcx) # tmp271,
movl %edx, %eax # _88, _88
movl -4(%rsi,%rax), %esi #, tmp278
movl %esi, -4(%rcx,%rax) # tmp278,
movl 8(%r9), %edi # p_72->bv_len, p_72->bv_len
jmp .L330 #
...

.L400:
# lib/iov_iter.c:73: memcpy(to + progress, iter_from, len);
testb $4, %dl #, _88
jne .L398 #,
testl %edx, %edx # _88
je .L330 #,
movzbl (%rsi), %eax #, tmp279
movb %al, (%rcx) # tmp279,
testb $2, %dl #, _88
jne .L390 #,
...

which makes *zero* sense. It first checks that the the length is at
least 8 bytes, then it moves *one* word by hand, then it aligns the
code to 8 bytes remaining, and does the remaining (possibly
overlapping at the beginning) words as one "rep movsq",

And L398 is the "I have 4..7 bytes to copy" target.

And L400 seems to be "I have 0..7 bytes to copy".

This is literally insane. And it seems to be all just gcc having for
some reason decided to do this instead of "rep movsb" or calling an
out-of-line function.

I get the feeling that this is related to how your patches made that
function be an inline function that is inlined through a function
pointer. I suspect that what happens is that gcc expands the memcpy()
first into that inlined function (without caller context), and then
inserts the crazily expanded inline later into the context of that
function pointer.

I dunno. I really only say that because I haven't seen gcc make this
kind of mess before, and that "inlined through a function pointer" is
the main unusual thing here.

How very annoying.

Linus