Unable to load modules from 9p filesystem with kmod 16

Unable to load modules from 9p filesystem with kmod 16

[Wakko Warner]

Kernel 3.12.7 from kernel.org
With kmod-16, I'm unable to load any modules on my guest kvm machines.
The vm is booted via direct kernel boot.  The modules are located on the
host and is passed to the guest via the fsdev.

I have a mountpoint on the guest filesystem located at /kernel.  It is
mounted like this:
kernel /kernel 9p rw,sync,dirsync,relatime,trans=virtio,version=9p2000.L 0 0

/boot, /lib/modules, and /lib/firmware are tmpfs filesystems like this:
kboot /boot tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
kfirmware /lib/firmware tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
kmodules /lib/modules tmpfs rw,relatime,size=0k,nr_inodes=8 0 0

/lib/modules/$(uname -r) is a symlink to /kernel/lib/modules/$(uname -r)

When trying to load any module, I get 
Error: could not insert module /lib/modules/3.12.7/kernel/crypto/af_alg.ko:
Invalid module format

This module was just one I picked at random, all modules fail the same way.
Strace shows this:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
munmap(0x7f199aebd000, 13822)           = 0
close(3)                                = 0

If I use kmod-9, it works w/o problems.  An strace of kmod-9 shows:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f658ba41000
init_module(0x7f658ba41000, 13822, "")  = 0
munmap(0x7f658ba41000, 13822)           = 0
close(3)                                = 0

If I copy the module to /tmp, kmod-16 loads it just fine.  It appears to be
related to the 9p filesystem.

I won't rule out a bug in kmod-16, but I was unable to figure out what the
cause of the error was.  I tried to search through kmod's source and the
kernel source, but I gave up because I don't understand what's going on.

-- 
 Microsoft has beaten Volkswagen's world record.  Volkswagen only created 22
 million bugs.

Re: Unable to load modules from 9p filesystem with kmod 16

[Wakko Warner]

Wakko Warner wrote:
> Kernel 3.12.7 from kernel.org
> With kmod-16, I'm unable to load any modules on my guest kvm machines.
> The vm is booted via direct kernel boot.  The modules are located on the
> host and is passed to the guest via the fsdev.
> 
> I have a mountpoint on the guest filesystem located at /kernel.  It is
> mounted like this:
> kernel /kernel 9p rw,sync,dirsync,relatime,trans=virtio,version=9p2000.L 0 0
> 
> /boot, /lib/modules, and /lib/firmware are tmpfs filesystems like this:
> kboot /boot tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
> kfirmware /lib/firmware tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
> kmodules /lib/modules tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
> 
> /lib/modules/$(uname -r) is a symlink to /kernel/lib/modules/$(uname -r)
> 
> When trying to load any module, I get 
> Error: could not insert module /lib/modules/3.12.7/kernel/crypto/af_alg.ko:
> Invalid module format
> 
> This module was just one I picked at random, all modules fail the same way.
> Strace shows this:
> open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
> mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
> syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
> munmap(0x7f199aebd000, 13822)           = 0
> close(3)                                = 0
> 
> If I use kmod-9, it works w/o problems.  An strace of kmod-9 shows:
> open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
> mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f658ba41000
> init_module(0x7f658ba41000, 13822, "")  = 0
> munmap(0x7f658ba41000, 13822)           = 0
> close(3)                                = 0
> 
> If I copy the module to /tmp, kmod-16 loads it just fine.  It appears to be
> related to the 9p filesystem.
> 
> I won't rule out a bug in kmod-16, but I was unable to figure out what the
> cause of the error was.  I tried to search through kmod's source and the
> kernel source, but I gave up because I don't understand what's going on.

I did some testing with the various kmod versions between kmod-9 and 16. 
The problem was introduced in 13.  The change that broke this was the
addition of finit_module().  Appears that the kernel cannot handle modules
that is on the 9p filesystem.  On the line that checked err == 0 (around
line libkmod-module.c:823), I added a test for ENOEXEC.  The finit_module
still fails,but it uses the old method and works.

I would say from what I've seen, the problem is in the kernel for
finit_module().  I don't know enough about the kernel to go any further.

Re: Unable to load modules from 9p filesystem with kmod 16

[Randy Dunlap]

[adding Cc:s]

On 01/10/2014 03:03 PM, Wakko Warner wrote:
> Wakko Warner wrote:
>> Kernel 3.12.7 from kernel.org
>> With kmod-16, I'm unable to load any modules on my guest kvm machines.
>> The vm is booted via direct kernel boot.  The modules are located on the
>> host and is passed to the guest via the fsdev.
>>
>> I have a mountpoint on the guest filesystem located at /kernel.  It is
>> mounted like this:
>> kernel /kernel 9p rw,sync,dirsync,relatime,trans=virtio,version=9p2000.L 0 0
>>
>> /boot, /lib/modules, and /lib/firmware are tmpfs filesystems like this:
>> kboot /boot tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
>> kfirmware /lib/firmware tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
>> kmodules /lib/modules tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
>>
>> /lib/modules/$(uname -r) is a symlink to /kernel/lib/modules/$(uname -r)
>>
>> When trying to load any module, I get 
>> Error: could not insert module /lib/modules/3.12.7/kernel/crypto/af_alg.ko:
>> Invalid module format
>>
>> This module was just one I picked at random, all modules fail the same way.
>> Strace shows this:
>> open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
>> fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
>> mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
>> syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
>> munmap(0x7f199aebd000, 13822)           = 0
>> close(3)                                = 0
>>
>> If I use kmod-9, it works w/o problems.  An strace of kmod-9 shows:
>> open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
>> fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
>> mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f658ba41000
>> init_module(0x7f658ba41000, 13822, "")  = 0
>> munmap(0x7f658ba41000, 13822)           = 0
>> close(3)                                = 0
>>
>> If I copy the module to /tmp, kmod-16 loads it just fine.  It appears to be
>> related to the 9p filesystem.
>>
>> I won't rule out a bug in kmod-16, but I was unable to figure out what the
>> cause of the error was.  I tried to search through kmod's source and the
>> kernel source, but I gave up because I don't understand what's going on.
> 
> I did some testing with the various kmod versions between kmod-9 and 16. 
> The problem was introduced in 13.  The change that broke this was the
> addition of finit_module().  Appears that the kernel cannot handle modules
> that is on the 9p filesystem.  On the line that checked err == 0 (around
> line libkmod-module.c:823), I added a test for ENOEXEC.  The finit_module
> still fails,but it uses the old method and works.
> 
> I would say from what I've seen, the problem is in the kernel for
> finit_module().  I don't know enough about the kernel to go any further.
> --


-- 
~Randy

Re: [V9fs-developer] Unable to load modules from 9p filesystem with kmod 16

[Dominique Martinet]

Hi,

Randy Dunlap wrote on Fri, Jan 10, 2014 :
> [adding Cc:s]

Thanks for the extra Cc.

> On 01/10/2014 03:03 PM, Wakko Warner wrote:
> > Wakko Warner wrote:
> >> Kernel 3.12.7 from kernel.org
> >> With kmod-16, I'm unable to load any modules on my guest kvm machines.
> >> The vm is booted via direct kernel boot.  The modules are located on the
> >> host and is passed to the guest via the fsdev.
> >>
> >> I have a mountpoint on the guest filesystem located at /kernel.  It is
> >> mounted like this:
> >> kernel /kernel 9p rw,sync,dirsync,relatime,trans=virtio,version=9p2000.L 0 0
> >>
> >> /boot, /lib/modules, and /lib/firmware are tmpfs filesystems like this:
> >> kboot /boot tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
> >> kfirmware /lib/firmware tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
> >> kmodules /lib/modules tmpfs rw,relatime,size=0k,nr_inodes=8 0 0
> >>
> >> /lib/modules/$(uname -r) is a symlink to /kernel/lib/modules/$(uname -r)
> >>
> >> When trying to load any module, I get 
> >> Error: could not insert module /lib/modules/3.12.7/kernel/crypto/af_alg.ko:
> >> Invalid module format
> >>
> >> This module was just one I picked at random, all modules fail the same way.
> >> Strace shows this:
> >> open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
> >> fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
> >> mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
> >> syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
> >> munmap(0x7f199aebd000, 13822)           = 0
> >> close(3)                                = 0

I didn't look at the patch very closely because the mail apparently
never reached v9fs-developer (although it looks in copy), but I got
pointed out this patch recently:
http://www.spinics.net/lists/linux-virtualization/msg21716.html

Which quotes "This patch enables 9p-virtio to correctly handle this
case. This not only enables us to load Linux kernel modules off virtfs,"

Perhaps would it be what you need to support this syscall_313?

Good luck,
-- 
Dominique Martinet

Re: [V9fs-developer] Unable to load modules from 9p filesystem with kmod 16

[Wakko Warner]

Dominique Martinet wrote:
> Hi,

Hi.

The following was trimmed.

> > On 01/10/2014 03:03 PM, Wakko Warner wrote:
> > > Wakko Warner wrote:
> > >> Kernel 3.12.7 from kernel.org
> > >> With kmod-16, I'm unable to load any modules on my guest kvm machines.

> > >> open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
> > >> fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
> > >> mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
> > >> syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
> > >> munmap(0x7f199aebd000, 13822)           = 0
> > >> close(3)                                = 0
> 
> I didn't look at the patch very closely because the mail apparently
> never reached v9fs-developer (although it looks in copy), but I got
> pointed out this patch recently:
> http://www.spinics.net/lists/linux-virtualization/msg21716.html

I applied the patch on that page.  Here's my result:

# df -T /lib/modules/3.12.7/kernel              
Filesystem     Type 1K-blocks   Used Available Use% Mounted on
kernel         9p     2015056 721356   1172460  39% /kernel
# uname -r
3.12.7
# modprobe -V
kmod version 16
# lsmod
Module                  Size  Used by
nfsd                  217278  2 
auth_rpcgss            41307  1 nfsd
oid_registry            2179  1 auth_rpcgss
exportfs                3499  1 nfsd
nfs                   111109  0 
...

> Which quotes "This patch enables 9p-virtio to correctly handle this
> case. This not only enables us to load Linux kernel modules off virtfs,"
> 
> Perhaps would it be what you need to support this syscall_313?

Looks like that's the fix.

-- 
 Microsoft has beaten Volkswagen's world record.  Volkswagen only created 22
 million bugs.