udev causes kernel oops when migrating from 2.6.30.4 to 2.6.31.2

udev causes kernel oops when migrating from 2.6.30.4 to 2.6.31.2

[hoefle marco]

Hello,
on your arm board (pxa270) we need to migrate to Kernel 2.6.31.x due to
block device driver api changes.
When compiling the new kernel with the same .config as with 2.6.30.4 we
get a kernel oops (attached later).
The udevd version is 125 which is that one used in debian. This version
works well for kernel 2.6.30.1, 2.6.30.4 but not for 2.6.31.0, 2.6.31.1
and 2.6.31.2.
Do we need to change something in the kernel config?
Has the according userspace API changed?
Any ideas on how to debug this?

Thanks,
Marco 


Synthesizing the initial hotplug events...Unable to handle kernel paging
request at virtual address 03a02001
pgd = c39e8000
[03a02001] *pgd\0000000
Internal error: Oops: f5 [#1]
Modules linked in:
CPU: 0    Not tainted  (2.6.31.1 #3)
PC is at strlen+0x18/0x2c
LR is at kobject_get_path+0x2c/0xac
pc : [<c013efac>]    lr : [<c013b910>]    psr: 20000013
sp : c39cbe48  ip : c39cbe58  fp : c39cbe54
r10: c02e2c9c  r9 : 00000000  r8 : c3802c60
r7 : c0020260  r6 : 000000d0  r5 : 00000007  r4 : 0000c328
r3 : 00000000  r2 : 03a02001  r1 : 000000d0  r0 : 03a02001
Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Control: 0000397f  Table: a39e8000  DAC: 00000015
Process udevadm (pid: 232, stack limit = 0xc39ca270)
Stack: (0xc39cbe48 to 0xc39cc000)
be40:                   c39cbe78 c39cbe58 c013b910 c013efa0 c0020258
00000003 
be60: c3932e40 c391d000 c3802c60 c39cbe9c c39cbe7c c015b814 c013b8f0
00000000 
be80: 00000000 00000003 c391d000 c3932e48 c39cbee8 c39cbea0 c013be28
c015b63c 
bea0: 00000000 00000000 c399f3e0 c028bc9e c0291f72 c39cbed4 00000000
00000000 
bec0: c3932e40 00000003 c39638e0 c3932e48 c02e2cdc c39638f4 c39cbf80
c39cbef8 
bee0: c39cbeec c013c068 c013bc54 c39cbf14 c39cbefc c015b8bc c013c060
00000000 
bf00: c39137c8 00000003 c39cbf24 c39cbf18 c015b0a0 c015b84c c39cbf50
c39cbf28 
bf20: c00cb83c c015b084 c38dee60 00016d7c c39cbf80 00000004 00000003
c39ca000 
bf40: 00000000 c39cbf7c c39cbf54 c008a804 c00cb73c c39cbf94 c39cbf64
00000000 
bf60: 00000000 c38dee60 00000004 c0024064 c39cbfa4 c39cbf80 c008a984
c008a75c 
bf80: 00000000 00000000 00000000 00000003 00016d7c bea5b354 00000000
c39cbfa8 
bfa0: c0023ec0 c008a944 00000003 00016d7c 00000003 00016d7c 00000003
00000000 
bfc0: 00000003 00016d7c bea5b354 00025990 00016d7c 00200200 00000000
00100100 
bfe0: 00000000 bea5b34c 0000d0a0 40119274 40000010 00000003 ffffffff
ffffffff 
Backtrace: 
[<c013ef94>] (strlen+0x0/0x2c) from [<c013b910>] (kobject_get_path
+0x2c/0xac)
[<c013b8e4>] (kobject_get_path+0x0/0xac) from [<c015b814>] (dev_uevent
+0x1e4/0x210)
 r8:c3802c60 r7:c391d000 r6:c3932e40 r5:00000003 r4:c0020258
[<c015b630>] (dev_uevent+0x0/0x210) from [<c013be28>]
(kobject_uevent_env+0x1e0/0x40c)
 r7:c3932e48 r6:c391d000 r5:00000003 r4:00000000
[<c013bc48>] (kobject_uevent_env+0x0/0x40c) from [<c013c068>]
(kobject_uevent+0x14/0x18)
[<c013c054>] (kobject_uevent+0x0/0x18) from [<c015b8bc>] (store_uevent
+0x7c/0x8c)
[<c015b840>] (store_uevent+0x0/0x8c) from [<c015b0a0>] (dev_attr_store
+0x28/0x2c)
 r5:00000003 r4:c39137c8
[<c015b078>] (dev_attr_store+0x0/0x2c) from [<c00cb83c>]
(sysfs_write_file+0x10c/0x144)
[<c00cb730>] (sysfs_write_file+0x0/0x144) from [<c008a804>] (vfs_write
+0xb4/0x160)
[<c008a750>] (vfs_write+0x0/0x160) from [<c008a984>] (sys_write
+0x4c/0x80)
 r8:c0024064 r7:00000004 r6:c38dee60 r5:00000000 r4:00000000
[<c008a938>] (sys_write+0x0/0x80) from [<c0023ec0>] (ret_fast_syscall
+0x0/0x2c)
 r6:bea5b354 r5:00016d7c r4:00000003
Code: e24cb004 e1a02000 ea000000 e2800001 (e5d03000) 
---[ end trace 0f9f3a3683bef81b ]---
Segmentation fault
failed.
Waiting for /dev to be fully populated...done.

Re: udev causes kernel oops when migrating from 2.6.30.4 to 2.6.31.2

[Kay Sievers]

On Wed, Oct 7, 2009 at 15:57, hoefle marco <marco.hoefle@nanotronic.ch> wrote:
> on your arm board (pxa270) we need to migrate to Kernel 2.6.31.x due to
> block device driver api changes.
> When compiling the new kernel with the same .config as with 2.6.30.4 we
> get a kernel oops (attached later).
> The udevd version is 125 which is that one used in debian. This version
> works well for kernel 2.6.30.1, 2.6.30.4 but not for 2.6.31.0, 2.6.31.1
> and 2.6.31.2.
> Do we need to change something in the kernel config?
> Has the according userspace API changed?
> Any ideas on how to debug this?

No, there was no change. It should work fine.

I would expect a kernel device driver problem. You use any drivers
which are not in the upstream git repo? If yes, any pointers to it?

Kay

Re: udev causes kernel oops when migrating from 2.6.30.4 to

[hoefle marco]

On Wed, 2009-10-07 at 16:32 +0200, Kay Sievers wrote:
> On Wed, Oct 7, 2009 at 15:57, hoefle marco
> <marco.hoefle@nanotronic.ch> wrote:
> > on your arm board (pxa270) we need to migrate to Kernel 2.6.31.x due
> to
> > block device driver api changes.
> > When compiling the new kernel with the same .config as with 2.6.30.4
> we
> > get a kernel oops (attached later).
> > The udevd version is 125 which is that one used in debian. This
> version
> > works well for kernel 2.6.30.1, 2.6.30.4 but not for 2.6.31.0,
> 2.6.31.1
> > and 2.6.31.2.
> > Do we need to change something in the kernel config?
> > Has the according userspace API changed?
> > Any ideas on how to debug this?
> 
> No, there was no change. It should work fine.
> 
> I would expect a kernel device driver problem. You use any drivers
> which are not in the upstream git repo? If yes, any pointers to it?
> 
> Kay

Hi Kay,
thanks for your quick response. I disabled all non upstream drivers (USB
Host controller driver, usb device controller, disk-on-chip).
So only drivers in the upstream kernel are used.

Marco

Re: udev causes kernel oops when migrating from 2.6.30.4 to 2.6.31.2

[Kay Sievers]

On Wed, Oct 7, 2009 at 16:59, hoefle marco <marco.hoefle@nanotronic.ch> wrote:
> On Wed, 2009-10-07 at 16:32 +0200, Kay Sievers wrote:
>> On Wed, Oct 7, 2009 at 15:57, hoefle marco
>> <marco.hoefle@nanotronic.ch> wrote:
>> > on your arm board (pxa270) we need to migrate to Kernel 2.6.31.x due
>> to
>> > block device driver api changes.
>> > When compiling the new kernel with the same .config as with 2.6.30.4
>> we
>> > get a kernel oops (attached later).
>> > The udevd version is 125 which is that one used in debian. This
>> version
>> > works well for kernel 2.6.30.1, 2.6.30.4 but not for 2.6.31.0,
>> 2.6.31.1
>> > and 2.6.31.2.
>> > Do we need to change something in the kernel config?
>> > Has the according userspace API changed?
>> > Any ideas on how to debug this?
>>
>> No, there was no change. It should work fine.
>>
>> I would expect a kernel device driver problem. You use any drivers
>> which are not in the upstream git repo? If yes, any pointers to it?

> thanks for your quick response. I disabled all non upstream drivers (USB
> Host controller driver, usb device controller, disk-on-chip).
> So only drivers in the upstream kernel are used.

Does the box allow you to boot with init=/bin/sh, or something
similar, mount /sys and run binaries?

Kay

Re: udev causes kernel oops when migrating from 2.6.30.4 to

[Greg KH]

On Wed, Oct 07, 2009 at 04:32:38PM +0200, Kay Sievers wrote:
> On Wed, Oct 7, 2009 at 15:57, hoefle marco <marco.hoefle@nanotronic.ch> wrote:
> > on your arm board (pxa270) we need to migrate to Kernel 2.6.31.x due to
> > block device driver api changes.
> > When compiling the new kernel with the same .config as with 2.6.30.4 we
> > get a kernel oops (attached later).
> > The udevd version is 125 which is that one used in debian. This version
> > works well for kernel 2.6.30.1, 2.6.30.4 but not for 2.6.31.0, 2.6.31.1
> > and 2.6.31.2.
> > Do we need to change something in the kernel config?
> > Has the according userspace API changed?
> > Any ideas on how to debug this?
> 
> No, there was no change. It should work fine.
> 
> I would expect a kernel device driver problem. You use any drivers
> which are not in the upstream git repo? If yes, any pointers to it?

It's a huge diff, will not fit on this list I think :(

Anyway, I just looked at it again.  Marco, there are numerous bugs in
that driver code.  NEVER have an empty release function for a device,
that is completly wrong, and is one reason why this is now failing.

Your driver also has devices in the __initdata section, which is being
thrown away by the kernel.  That's probably the biggest problem and I'm
amazed it even worked at all here.

thanks,

greg k-h

Re: udev causes kernel oops when migrating from 2.6.30.4 to

[hoefle marco]

On Wed, 2009-10-07 at 17:05 +0200, Kay Sievers wrote:
> On Wed, Oct 7, 2009 at 16:59, hoefle marco <marco.hoefle@nanotronic.ch> wrote:
> > On Wed, 2009-10-07 at 16:32 +0200, Kay Sievers wrote:
> >> On Wed, Oct 7, 2009 at 15:57, hoefle marco
> >> <marco.hoefle@nanotronic.ch> wrote:
> >> > on your arm board (pxa270) we need to migrate to Kernel 2.6.31.x due
> >> to
> >> > block device driver api changes.
> >> > When compiling the new kernel with the same .config as with 2.6.30.4
> >> we
> >> > get a kernel oops (attached later).
> >> > The udevd version is 125 which is that one used in debian. This
> >> version
> >> > works well for kernel 2.6.30.1, 2.6.30.4 but not for 2.6.31.0,
> >> 2.6.31.1
> >> > and 2.6.31.2.
> >> > Do we need to change something in the kernel config?
> >> > Has the according userspace API changed?
> >> > Any ideas on how to debug this?
> >>
> >> No, there was no change. It should work fine.
> >>
> >> I would expect a kernel device driver problem. You use any drivers
> >> which are not in the upstream git repo? If yes, any pointers to it?
> 
> > thanks for your quick response. I disabled all non upstream drivers (USB
> > Host controller driver, usb device controller, disk-on-chip).
> > So only drivers in the upstream kernel are used.
> 
> Does the box allow you to boot with init=/bin/sh, or something
> similar, mount /sys and run binaries?
> 
> Kay
> 

The failure was caused by:
static struct isp1760_platform_data isp1760_platform_data __initdata = {

__initdata causes a section mismatch and finally a kernel oops. So it
was driver related not as I thought an udev issue. Strange that this
behavior didn't show up in 2.6.30.4.

Cheers,
Marco