Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (rmmod BUG)

Jesper Dangaard Brouer
2008-11-19T12:10:49Z

[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]

On Fri, 2008-11-14 at 16:21 -0800, David Miller wrote:
> From: Jesper Dangaard Brouer <jdb@comx.dk>
> Date: Fri, 14 Nov 2008 19:49:22 +0100
>
> > On Fri, 2008-11-14 at 13:38 +0100, Jesper Dangaard Brouer wrote:
> > > On Thu, 2008-11-13 at 14:08 -0800, David Miller wrote:
> > > > I suspect that something might be changing np->num_ldg, but
> > > > anyways the following debugging patch should provide some
> > > > clues. Please reproduce this and send the logs it generates.
> > >
> > > Debugging the rmmod problem...
> > >
> > > I found a strange behavior, rmmod'ing the niu driver will only cause a
> > > kernel BUG, if the driver was loaded at boot time. If I remove the
> > > niu.ko driver from /lib/modules/2.6.28-rc4-davem/kernel/drivers/net/
> > > reboot the system. After that I can load and unload the niu.ko driver
> > > without problems... hmmm
> >
> > Perhaps this is a regression, as the problem is not in v2.6.27.
>
> This is what I started to suspect as well.
>
> > I'll start bisecting monday...
> >
> > I'm not sure its a NIU driver bug, as the number of changes to niu.c is
> > very small since v2.6.27. (git log v2.6.27.. drivers/net/niu.c)
>
> Ok, let me know what your bisect finds.

I have given up bisecting because during my bisect I have hit a kernel
that will not boot on my system (it hangs...)

I have attached the full bisect history document...

--
Med venlig hilsen / Best regards
Jesper Brouer
ComX Networks A/S
Linux Network developer
Cand. Scient Datalog / MSc.
Author of http://adsl-optimizer.dk
LinkedIn: http://www.linkedin.com/in/brouer

[-- Attachment #2: bisect_niu_rmmod.apt --]
[-- Type: text/plain, Size: 10091 bytes --]

~~ -*-text-*-

-------------------------------------------------------
Bisecting bug:
NIU driver rmmod MSI-X bug
-------------------------------------------------------
Jesper Dangaard Brouer (jdb@comx.dk)
-------------------------------------------------------
$LastChangedRevision: 772 $
$Date: 2008-11-19 13:08:13 +0100 (Wed, 19 Nov 2008) $
-------------------------------------------------------

git clone
~~~~~~~~~

+---------
cd /var/kernels/git/davem
git clone git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git net-2.6-bisect
+---------

Bug description
~~~~~~~~~~~~~~~

NIU driver rmmod kernel MSI-X bug.

I found a strange behavior, rmmod'ing the niu driver will only cause
a kernel BUG, if the driver was loaded at boot time. If I remove the
niu.ko driver from /lib/modules/2.6.28-rc4-davem/kernel/drivers/net/
reboot the system. After that I can load and unload the niu.ko
driver without problems... hmmm.

Reproduce / test
~~~~~~~~~~~~~~~~

1. Boot machine

2. rmmod niu

3. look at dmesg for kernel BUG output

Install trick
~~~~~~~~~~~~~

Installing kernel in a seperate directory.

+--------
export VER=`cat include/config/kernel.release`
echo $VER

export INSTALL_MOD_PATH=/var/kernels/git/install/
rm -rf $INSTALL_MOD_PATH/lib/modules/$VER/kernel/
make modules_install

export BOOT="$INSTALL_MOD_PATH/boot/"
[ -d $BOOT ] || mkdir $BOOT

cp -v arch/x86/boot/bzImage $BOOT/vmlinuz-$VER
cp -v arch/i386/boot/bzImage $BOOT/vmlinuz-$VER
cp -v System.map $BOOT/System.map-$VER
cp -v vmlinux $BOOT/vmlinux-$VER
+--------

* Push to test host
~~~~~~~~~~~~~~~~~

export KERNEL=2.6.27-davem
export KERNEL=2.6.28-rc2-davem
export KERNEL=2.6.28-rc4-davem

+--------
export HOST=ng
export KERNEL=$VER
pushd /var/kernels/git/install
rsync -e ssh -avz boot/vmlinuz-${KERNEL} root@${HOST}:/boot/
rsync -e ssh -avz boot/vmlinux-${KERNEL} root@${HOST}:/boot/
rsync -e ssh -avz --delete lib/modules/${KERNEL} root@${HOST}:/lib/modules/
popd
+--------

Create branch from tag:
~~~~~~~~~~~~~~~~~~~~~~~

Known good starting point.

+-----------
git branch tag_v2.6.27 v2.6.27
git checkout tag_v2.6.27
+-----------

Start bisect
~~~~~~~~~~~~

+--------
git checkout master
git bisect start
git bisect good v2.6.27
git bisect bad master
+--------

Create .config by <<<make oldconfig>>> or <<<make menuconfig>>>

+-------
bisect good
#Bisecting: 2157 revisions left to test after this
#[92b29b86fe2e183d44eb467e5e74a5f718ef2e43]
#Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
+-------

compile:
time make -j6 bzImage modules

Sort of "good" version, problem is that number of IRQs when down a
lot! This probably means bad performance as I cannot get enough IRQs
for the Sun NIC.

+--------
bisect good
#Bisecting: 1066 revisions left to test after this
#[ea541686d8454efac4f2b5c0767affb12d4b6a52]
#Merge branch 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds
+--------

compile:
time make -j6 bzImage modules

CRAP! - kernel (commit ea541686d8454efac4f2b5c0767affb12d4b6a52) will
not boot on my system :-(((

Stops with message:

+------
hpet0: 3 comparators, 64-bit 14.318180 Mhz counter
+------

<<Try to:>> removed "High Resolution Timer Support" in .config
CONFIG_HIGH_RES_TIMERS and CONFIG_SCHED_HRTICK.

Disable "HPET Timer Support" (under "Processor type and features").
Undefs CONFIG_HPET_TIMER and CONFIG_HPET_EMULATE_RTC.

Compiling ... installing ... It was not the problem... now I get a
"BUG: soft lockup" kstop EIP is a stop_cpu+0x37/0xb0.

* Parallel process#1:
~~~~~~~~~~~~~~~~~~~~

<<Try to:>> Pick a new commit point a make a new seperate branch and
try to see if we cab boot... random picked commit
9a1c3542768b5a58e45a9216921cd10a3bae1205

git checkout -b new_bisect_point01 9a1c3542768b5a58e45a9216921cd10a3bae1205

Compile on another tree davem/net-2.6-copy.

Kernel named "-test".

It can boot and unloading "niu" driver WORKS!

If I understand bisect it should be possible to call:

git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205

* Parallel process#2:
~~~~~~~~~~~~~~~~~~~~

<<Try to:>> use git-bisect skip

+------
git-bisect skip
Bisecting: 1066 revisions left to test after this
[969907a956752f88dde4aa23fa8c033b9a939aee] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6
+------

STILL HANGS on boot :-(((

* Back on track...
~~~~~~~~~~~~~~~~~~

+--------
git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205
Bisecting: 1000 revisions left to test after this
[1137fb670465b6b5d15b9db7d01707a5833ee3ae] arm ide breakage
+--------

compile... install...

BAD: Booting (now 2.6.28-rc1-bisect) and unloading "niu" causes the bug!

Mark bisecting as BAD.

+----------
git-bisect bad
Bisecting: 508 revisions left to test after this
[36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6
+----------

compile ... install ... (back on ver. 2.6.27-bisect)

GRRRR... Now I cannot boot again! :-(((

* Try to find a new place
~~~~~~~~~~~~~~~~~~~~~~~

Random pick: dbacefc9c4f6bd365243db379473ab7041656d90

+--------
cd /var/kernels/git/davem/net-2.6-copy/
git checkout -b new_bisect_point02 dbacefc9c4f6bd365243db379473ab7041656d90
+--------

VER=2.6.27-rc1-test

This version can boot and unloading niu works.

* Back on track(2) ... but not :-(
~~~~~~~~~~~~~~~~~~

git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90

+--------
git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90
Bisecting: 508 revisions left to test after this
[36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6
+---------

Hmmm... commit 36ec891895020f3bc9953c8b11d079c6d77d76bd was the same
as before... which could not boot...

* Try a bisect skip...
~~~~~~~~~~~~~~~~~~~~~

+---------
git bisect skip
Bisecting: 508 revisions left to test after this
[70740d6c93030b339b4ad17fd58ee135dfc13913] Merge branch 'drm-next' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6
+---------

compile ... install ... boot... CANNOT BOOT !!! :-(((

* Doing lucky guessing
~~~~~~~~~~~~~~~~~~~~~~

git-bisect good

+--------------
git-bisect good
Bisecting: 259 revisions left to test after this
[22484856402bfa1ff3defe47f6029ab0418240d9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/viro/bdev
+--------------

ARGH!!! -- cannot boot this kernel, it hangs :-(((

* Doing desperate guessing
~~~~~~~~~~~~~~~~~~~~~~~~~~

+--------
git-bisect good
Bisecting: 132 revisions left to test after this
[c3c9897c63ebb0b93b7f78724e38d6ee1da04041] Merge branch 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
+--------

compiling ... installing ...
... booting -- ARGH! cannot boot hang!

* Doing hopeless guessing
~~~~~~~~~~~~~~~~~~~~~~~~~

This is hopeless, I should give up!

+-----------------
git-bisect good
Bisecting: 65 revisions left to test after this
[969907a956752f88dde4aa23fa8c033b9a939aee] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6
+------------------

compiling ...
installing ...
still hangs after boot... :-(((

... GIVING UP!!!

* git bisect log
~~~~~~~~~~~~~~~~

+-------
git-bisect start
# good: [3fa8749e584b55f1180411ab1b51117190bac1e5] Linux 2.6.27
git-bisect good 3fa8749e584b55f1180411ab1b51117190bac1e5
# bad: [5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776] rtnetlink: propagate error from dev_change_flags in do_setlink()
git-bisect bad 5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776
# good: [29415c37f043d1d54dcf356601d738ff6633b72b] KVM: set debug registers after "schedulable" section
git-bisect good 29415c37f043d1d54dcf356601d738ff6633b72b
# good: [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect good 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
# skip: [ea541686d8454efac4f2b5c0767affb12d4b6a52] Merge branch 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds
git-bisect skip ea541686d8454efac4f2b5c0767affb12d4b6a52
# good: [9a1c3542768b5a58e45a9216921cd10a3bae1205] pass fmode_t to blkdev_put()
git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205
# bad: [1137fb670465b6b5d15b9db7d01707a5833ee3ae] arm ide breakage
git-bisect bad 1137fb670465b6b5d15b9db7d01707a5833ee3ae
# good: [dbacefc9c4f6bd365243db379473ab7041656d90] fs/buffer.c: uninline __remove_assoc_queue()
git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90
# skip: [36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6
git-bisect skip 36ec891895020f3bc9953c8b11d079c6d77d76bd
# good: [70740d6c93030b339b4ad17fd58ee135dfc13913] Merge branch 'drm-next' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6
git-bisect good 70740d6c93030b339b4ad17fd58ee135dfc13913
# good: [22484856402bfa1ff3defe47f6029ab0418240d9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/viro/bdev
git-bisect good 22484856402bfa1ff3defe47f6029ab0418240d9
# good: [c3c9897c63ebb0b93b7f78724e38d6ee1da04041] Merge branch 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect good c3c9897c63ebb0b93b7f78724e38d6ee1da04041
+-------

Compile:
~~~~~~~~

+-------
time make -j6 bzImage modules
+-------

NOTES:
~~~~~~

git log 29415c37f043d1d54dcf356601d738ff6633b72b..5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776

NR_IRQS changed ... could this be releated?

git show 7db282fa67b58daff8a57f9e1c93d4474b5908ff
git show 1b4897688011cd05e07f00dcfe6af3331eb36a3c
git show c78d0cf2925bffae8a6f00e7d9b8e971b0392edd