ERROR: modpost: "kmsan_handle_dma" [drivers/virtio/virtio_ring.ko] undefined!

ERROR: modpost: "kmsan_handle_dma" [drivers/virtio/virtio_ring.ko] undefined!

[kernel test robot]

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
head:   04f41cbf03ec7221ab0b179e336f4c805ee55520
commit: 35772d627b55cc7fb4f33bae57c564a25b3121a9 sched: Enable PREEMPT_DYNAMIC for PREEMPT_RT
date:   3 months ago
config: x86_64-buildonly-randconfig-005-20250215 (https://download.01.org/0day-ci/archive/20250215/202502150634.qjxwSeJR-lkp@intel.com/config)
compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250215/202502150634.qjxwSeJR-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202502150634.qjxwSeJR-lkp@intel.com/

All error/warnings (new ones prefixed by >>, old ones prefixed by <<):

In file included from arch/x86/kvm/cpuid.c:13:
In file included from include/linux/kvm_host.h:16:
In file included from include/linux/mm.h:2213:
include/linux/vmstat.h:518:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
518 |         return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
|                               ~~~~~~~~~~~ ^ ~~~
1 warning generated.
>> arch/x86/kvm/cpuid.o: warning: objtool: do_cpuid_func+0x2428: undefined stack state
--
>> ERROR: modpost: "kmsan_handle_dma" [drivers/virtio/virtio_ring.ko] undefined!

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

[PATCH] dma: kmsan: Export kmsan_handle_dma() for modules.

[Sebastian Andrzej Siewior]

kmsan_handle_dma() is used by virtio_ring() which can be built as a
module. kmsan_handle_dma() needs to be exported otherwise building the
virtio_ring fails.

Export kmsan_handle_dma for modules.

Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202502150634.qjxwSeJR-lkp@intel.com/
Fixes: 7ade4f10779cb ("dma: kmsan: unpoison DMA mappings")
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 mm/kmsan/hooks.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c
index 3ea50f09311fd..3df45c25c1f62 100644
--- a/mm/kmsan/hooks.c
+++ b/mm/kmsan/hooks.c
@@ -357,6 +357,7 @@ void kmsan_handle_dma(struct page *page, size_t offset, size_t size,
 		size -= to_go;
 	}
 }
+EXPORT_SYMBOL_GPL(kmsan_handle_dma);
 
 void kmsan_handle_dma_sg(struct scatterlist *sg, int nents,
 			 enum dma_data_direction dir)
-- 
2.47.2

Re: [PATCH] dma: kmsan: Export kmsan_handle_dma() for modules.

[Alexander Potapenko]

On Tue, Feb 18, 2025 at 10:14 AM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> kmsan_handle_dma() is used by virtio_ring() which can be built as a
> module. kmsan_handle_dma() needs to be exported otherwise building the
> virtio_ring fails.
>
> Export kmsan_handle_dma for modules.
>
> Reported-by: kernel test robot <lkp@intel.com>
> Closes: https://lore.kernel.org/oe-kbuild-all/202502150634.qjxwSeJR-lkp@intel.com/
> Fixes: 7ade4f10779cb ("dma: kmsan: unpoison DMA mappings")
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: Alexander Potapenko <glider@google.com>

Re: [PATCH] dma: kmsan: Export kmsan_handle_dma() for modules.

[Andrew Morton]

On Tue, 18 Feb 2025 10:14:11 +0100 Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote:

> kmsan_handle_dma() is used by virtio_ring() which can be built as a
> module. kmsan_handle_dma() needs to be exported otherwise building the
> virtio_ring fails.
> 
> Export kmsan_handle_dma for modules.
> 
> Reported-by: kernel test robot <lkp@intel.com>
> Closes: https://lore.kernel.org/oe-kbuild-all/202502150634.qjxwSeJR-lkp@intel.com/
> Fixes: 7ade4f10779cb ("dma: kmsan: unpoison DMA mappings")

It's strange that this took a few years to be noticed.

Thanks, I added cc:stable to this.

Re: ERROR: modpost: "kmsan_handle_dma" [drivers/virtio/virtio_ring.ko] undefined!

[Sebastian Andrzej Siewior]

On 2025-02-15 06:42:36 [+0800], kernel test robot wrote:
> >> arch/x86/kvm/cpuid.o: warning: objtool: do_cpuid_func+0x2428: undefined stack state

I have been staring at this and have no idea what to do. I get here:
| arch/x86/kvm/cpuid.o: warning: objtool: do_cpuid_func+0x21e1: undefined stack state

The assembly at this spot looks as from the .o:
|   110e4:       e8 00 00 00 00          call   110e9 <do_cpuid_func+0x21d9>
|                        110e5: R_X86_64_PLT32   __msan_metadata_ptr_for_load_4-0x4
|   110e9:       44 8b 38                mov    (%rax),%r15d
|   110ec:       48 85 db                test   %rbx,%rbx
|   110ef:       8b 1a                   mov    (%rdx),%ebx
|   110f1:       4c 8b 64 24 18          mov    0x18(%rsp),%r12
^^^ do_cpuid_func+0x21e1

|   110f6:       0f 85 72 01 00 00       jne    1126e <do_cpuid_func+0x235e>
|   110fc:       41 83 e7 fe             and    $0xfffffffe,%r15d
|   11100:       48 89 ef                mov    %rbp,%rdi
|   11103:       e8 00 00 00 00          call   11108 <do_cpuid_func+0x21f8>
|                        11104: R_X86_64_PLT32   __msan_metadata_ptr_for_store_4-0x4

I've been looking at .s file but this does not realy match the .o. The
registers are different, the offsets, too.
Anyway.

Looking at the opcodes it was processing "recently":
base: 4, offset 1109d Dest: 0 Src: 5 lea    0x10(%r14),%rax
base: 4, offset 110a1 Dest: 1 Src: 0 mov    %rax,0x20(%rsp)
base: 4, offset 110ae Dest: 0 Src: 0 mov    %rsp,%r8
base: 4, offset 1124b Dest: 0 Src: 1 mov    0x20(%rsp),%rsi
base: 4, offset 11253 Dest: 0 Src: 1 mov    0x8(%rsp),%rcx
base: 4, offset 11258 Dest: 0 Src: 0 mov    %r15,%rsp
-> set UNDEF
base: -1, offset 110f1 -> mov    0x18(%rsp),%r12
arch/x86/kvm/cpuid.o: warning: objtool: do_cpuid_func+0x21e1: undefined stack state 110f1

From the assembly it seems to make sense:
|   110ae:       49 89 e0                mov    %rsp,%r8
stash for later
|   110b1:       48 85 db                test   %rbx,%rbx
|   110b4:       c7 00 00 00 00 00       movl   $0x0,(%rax)
|   110ba:       45 89 7e 14             mov    %r15d,0x14(%r14)
|   110be:       0f 85 40 01 00 00       jne    11204 <do_cpuid_func+0x22f4>
…
|   11204:       44 8b 74 24 38          mov    0x38(%rsp),%r14d
|   11209:       44 89 f7                mov    %r14d,%edi
|   1120c:       4d 89 c7                mov    %r8,%r15

mov rsp to r15

|   1120f:       e8 00 00 00 00          call   11214 <do_cpuid_func+0x2304>
|                        11210: R_X86_64_PLT32   __msan_chain_origin-0x4
|   11214:       89 c7                   mov    %eax,%edi
|   11216:       e8 00 00 00 00          call   1121b <do_cpuid_func+0x230b>
|                        11217: R_X86_64_PLT32   __msan_warning-0x4
|   1121b:       44 89 f7                mov    %r14d,%edi
|   1121e:       e8 00 00 00 00          call   11223 <do_cpuid_func+0x2313>
|                        1121f: R_X86_64_PLT32   __msan_chain_origin-0x4
|   11223:       89 c7                   mov    %eax,%edi
|   11225:       e8 00 00 00 00          call   1122a <do_cpuid_func+0x231a>
|                        11226: R_X86_64_PLT32   __msan_warning-0x4
|   1122a:       44 89 f7                mov    %r14d,%edi
|   1122d:       e8 00 00 00 00          call   11232 <do_cpuid_func+0x2322>
|                        1122e: R_X86_64_PLT32   __msan_chain_origin-0x4
|   11232:       89 c7                   mov    %eax,%edi
|   11234:       e8 00 00 00 00          call   11239 <do_cpuid_func+0x2329>
|                        11235: R_X86_64_PLT32   __msan_warning-0x4
|   11239:       44 89 f7                mov    %r14d,%edi
|   1123c:       e8 00 00 00 00          call   11241 <do_cpuid_func+0x2331>
|                        1123d: R_X86_64_PLT32   __msan_chain_origin-0x4
|   11241:       89 c7                   mov    %eax,%edi
|   11243:       e8 00 00 00 00          call   11248 <do_cpuid_func+0x2338>
|                        11244: R_X86_64_PLT32   __msan_warning-0x4
|   11248:       4c 89 ef                mov    %r13,%rdi
|   1124b:       48 8b 74 24 20          mov    0x20(%rsp),%rsi
|   11250:       4c 89 e2                mov    %r12,%rdx
|   11253:       48 8b 4c 24 08          mov    0x8(%rsp),%rcx
|   11258:       4c 89 fc                mov    %r15,%rsp

restore rsp. I just don't see how rsp is destroyed but this could be
related to paravirt's xxl clobbing in__cpuid().

I miss 1120c in my output. I don't understand how it jumps from 110ae to
1124b. It misses the assignments in between but this might not be goal
here…

gcc does not cause objtool to produce the warning but then gcc does
shuffle rsp as much as llvm does.

Sebastian

Re: ERROR: modpost: "kmsan_handle_dma" [drivers/virtio/virtio_ring.ko] undefined!

[Peter Zijlstra]

On Tue, Feb 18, 2025 at 12:48:57PM +0100, Sebastian Andrzej Siewior wrote:
> On 2025-02-15 06:42:36 [+0800], kernel test robot wrote:
> > >> arch/x86/kvm/cpuid.o: warning: objtool: do_cpuid_func+0x2428: undefined stack state
> 

> From the assembly it seems to make sense:
> |   110ae:       49 89 e0                mov    %rsp,%r8
> stash for later
> |   110b1:       48 85 db                test   %rbx,%rbx
> |   110b4:       c7 00 00 00 00 00       movl   $0x0,(%rax)
> |   110ba:       45 89 7e 14             mov    %r15d,0x14(%r14)
> |   110be:       0f 85 40 01 00 00       jne    11204 <do_cpuid_func+0x22f4>
> …
> |   11204:       44 8b 74 24 38          mov    0x38(%rsp),%r14d
> |   11209:       44 89 f7                mov    %r14d,%edi
> |   1120c:       4d 89 c7                mov    %r8,%r15
> 
> mov rsp to r15

This, objtool doesn't track this one. It only does:

 mov %rsp, reg

 mov reg, %rsp

I'm not entirely sure how painful it would be to teach objtool about
this case. Horrible code it is :/

> |   1120f:       e8 00 00 00 00          call   11214 <do_cpuid_func+0x2304>
> |                        11210: R_X86_64_PLT32   __msan_chain_origin-0x4
> |   11214:       89 c7                   mov    %eax,%edi
> |   11216:       e8 00 00 00 00          call   1121b <do_cpuid_func+0x230b>
> |                        11217: R_X86_64_PLT32   __msan_warning-0x4
> |   1121b:       44 89 f7                mov    %r14d,%edi
> |   1121e:       e8 00 00 00 00          call   11223 <do_cpuid_func+0x2313>
> |                        1121f: R_X86_64_PLT32   __msan_chain_origin-0x4
> |   11223:       89 c7                   mov    %eax,%edi
> |   11225:       e8 00 00 00 00          call   1122a <do_cpuid_func+0x231a>
> |                        11226: R_X86_64_PLT32   __msan_warning-0x4
> |   1122a:       44 89 f7                mov    %r14d,%edi
> |   1122d:       e8 00 00 00 00          call   11232 <do_cpuid_func+0x2322>
> |                        1122e: R_X86_64_PLT32   __msan_chain_origin-0x4
> |   11232:       89 c7                   mov    %eax,%edi
> |   11234:       e8 00 00 00 00          call   11239 <do_cpuid_func+0x2329>
> |                        11235: R_X86_64_PLT32   __msan_warning-0x4
> |   11239:       44 89 f7                mov    %r14d,%edi
> |   1123c:       e8 00 00 00 00          call   11241 <do_cpuid_func+0x2331>
> |                        1123d: R_X86_64_PLT32   __msan_chain_origin-0x4
> |   11241:       89 c7                   mov    %eax,%edi
> |   11243:       e8 00 00 00 00          call   11248 <do_cpuid_func+0x2338>
> |                        11244: R_X86_64_PLT32   __msan_warning-0x4
> |   11248:       4c 89 ef                mov    %r13,%rdi
> |   1124b:       48 8b 74 24 20          mov    0x20(%rsp),%rsi
> |   11250:       4c 89 e2                mov    %r12,%rdx
> |   11253:       48 8b 4c 24 08          mov    0x8(%rsp),%rcx
> |   11258:       4c 89 fc                mov    %r15,%rsp
> 
> restore rsp. I just don't see how rsp is destroyed but this could be
> related to paravirt's xxl clobbing in__cpuid().
> 
> I miss 1120c in my output. I don't understand how it jumps from 110ae to
> 1124b. It misses the assignments in between but this might not be goal
> here…
> 
> gcc does not cause objtool to produce the warning but then gcc does
> shuffle rsp as much as llvm does.
> 
> Sebastian

Re: ERROR: modpost: "kmsan_handle_dma" [drivers/virtio/virtio_ring.ko] undefined!

[Josh Poimboeuf]

On Tue, Feb 18, 2025 at 02:28:55PM +0100, Peter Zijlstra wrote:
> On Tue, Feb 18, 2025 at 12:48:57PM +0100, Sebastian Andrzej Siewior wrote:
> > On 2025-02-15 06:42:36 [+0800], kernel test robot wrote:
> > > >> arch/x86/kvm/cpuid.o: warning: objtool: do_cpuid_func+0x2428: undefined stack state
> > 
> 
> > From the assembly it seems to make sense:
> > |   110ae:       49 89 e0                mov    %rsp,%r8
> > stash for later
> > |   110b1:       48 85 db                test   %rbx,%rbx
> > |   110b4:       c7 00 00 00 00 00       movl   $0x0,(%rax)
> > |   110ba:       45 89 7e 14             mov    %r15d,0x14(%r14)
> > |   110be:       0f 85 40 01 00 00       jne    11204 <do_cpuid_func+0x22f4>
> > …
> > |   11204:       44 8b 74 24 38          mov    0x38(%rsp),%r14d
> > |   11209:       44 89 f7                mov    %r14d,%edi
> > |   1120c:       4d 89 c7                mov    %r8,%r15
> > 
> > mov rsp to r15
> 
> This, objtool doesn't track this one. It only does:
> 
>  mov %rsp, reg
> 
>  mov reg, %rsp
> 
> I'm not entirely sure how painful it would be to teach objtool about
> this case. Horrible code it is :/
> 
> > restore rsp. I just don't see how rsp is destroyed but this could be
> > related to paravirt's xxl clobbing in__cpuid().
> > 
> > I miss 1120c in my output. I don't understand how it jumps from 110ae to
> > 1124b. It misses the assignments in between but this might not be goal
> > here…

FYI, you can add OBJTOOL_ARGS="--backtrace" or OBJTOOL_VERBOSE=1 to the
cmdline to see how objtool got there.  On my build there were some
ALTERNATIVE()s involved which makes it trickier to follow.

There's definitely some weirdness going on, I'm also seeing the .s not
match the .o.  Maybe it's the __msan_*() stuff.  Still looking...

-- 
Josh

Re: ERROR: modpost: "kmsan_handle_dma" [drivers/virtio/virtio_ring.ko] undefined!

[Josh Poimboeuf]

On Tue, Feb 18, 2025 at 08:47:26AM -0800, Josh Poimboeuf wrote:
> On Tue, Feb 18, 2025 at 02:28:55PM +0100, Peter Zijlstra wrote:
> > On Tue, Feb 18, 2025 at 12:48:57PM +0100, Sebastian Andrzej Siewior wrote:
> > > On 2025-02-15 06:42:36 [+0800], kernel test robot wrote:
> > > > >> arch/x86/kvm/cpuid.o: warning: objtool: do_cpuid_func+0x2428: undefined stack state
> > > 
> > 
> > > From the assembly it seems to make sense:
> > > |   110ae:       49 89 e0                mov    %rsp,%r8
> > > stash for later
> > > |   110b1:       48 85 db                test   %rbx,%rbx
> > > |   110b4:       c7 00 00 00 00 00       movl   $0x0,(%rax)
> > > |   110ba:       45 89 7e 14             mov    %r15d,0x14(%r14)
> > > |   110be:       0f 85 40 01 00 00       jne    11204 <do_cpuid_func+0x22f4>
> > > …
> > > |   11204:       44 8b 74 24 38          mov    0x38(%rsp),%r14d
> > > |   11209:       44 89 f7                mov    %r14d,%edi
> > > |   1120c:       4d 89 c7                mov    %r8,%r15

[ Adding Masahiro for #1 ]

1)

Masahiro, one problem we've seen is that "make LLVM=1
arch/x86/kvm/cpuid.s" doesn't match the final compiled .o binary.  Is it
a problem with cmd_cc_s_c?  The original .config is here:

  https://lore.kernel.org/202502150634.qjxwSeJR-lkp@intel.com

If I manually run the original gcc command with -S, cpuid.s looks right.

2)

Peter, Sebastian, the objtool warning is caused by Clang interpreting
ASM_CALL_CONSTRAINT a little too literally.  It's stashing the stack
pointer and restoring it later to comply with the "+r" rsp constraint.
Which makes perfect sense, but is also 100% unexpected.

I "fixed" it with the below, but it's not a proper fix as
ASM_CALL_CONSTRAINT is still needed for CONFIG_UNWINDER_FRAME_POINTER.

So I think the options are:

  - Make objtool smarter

  - Improve ASM_CALL_CONSTRAINT somehow.  /me wonders if "memory"
    clobber has the same effect?

  - Drop support for x86-64 frame pointers altogether, along with
    ASM_CALL_CONSTRAINT.  Is there any reason to keep frame pointers
    around anymore for 64 bit?  AFAICT objtool is 100% mandatory for all
    configs, and ORC has proven to be solid.

diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
index fea56b04f436..7414f0c7cc0e 100644
--- a/arch/x86/include/asm/paravirt_types.h
+++ b/arch/x86/include/asm/paravirt_types.h
@@ -394,7 +394,7 @@ int paravirt_disable_iospace(void);
 		PVOP_TEST_NULL(op);					\
 		asm volatile(ALTERNATIVE(PARAVIRT_CALL, ALT_CALL_INSTR,	\
 				ALT_CALL_ALWAYS)			\
-			     : call_clbr, ASM_CALL_CONSTRAINT		\
+			     : call_clbr				\
 			     : paravirt_ptr(op),			\
 			       ##__VA_ARGS__				\
 			     : "memory", "cc" extra_clbr);		\
@@ -409,7 +409,7 @@ int paravirt_disable_iospace(void);
 		asm volatile(ALTERNATIVE_2(PARAVIRT_CALL,		\
 				 ALT_CALL_INSTR, ALT_CALL_ALWAYS,	\
 				 alt, cond)				\
-			     : call_clbr, ASM_CALL_CONSTRAINT		\
+			     : call_clbr				\
 			     : paravirt_ptr(op),			\
 			       ##__VA_ARGS__				\
 			     : "memory", "cc" extra_clbr);		\

-- 
Josh