Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (rmmod BUG)

[Jesper Dangaard Brouer <jdb@comx.dk>]

[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]

On Fri, 2008-11-14 at 16:21 -0800, David Miller wrote:
> From: Jesper Dangaard Brouer <jdb@comx.dk>
> Date: Fri, 14 Nov 2008 19:49:22 +0100
> 
> > On Fri, 2008-11-14 at 13:38 +0100, Jesper Dangaard Brouer wrote:
> > > On Thu, 2008-11-13 at 14:08 -0800, David Miller wrote:
> > > > I suspect that something might be changing np->num_ldg, but
> > > > anyways the following debugging patch should provide some
> > > > clues.  Please reproduce this and send the logs it generates.
> > > 
> > > Debugging the rmmod problem...
> > > 
> > > I found a strange behavior, rmmod'ing the niu driver will only cause a
> > > kernel BUG, if the driver was loaded at boot time.  If I remove the
> > > niu.ko driver from /lib/modules/2.6.28-rc4-davem/kernel/drivers/net/
> > > reboot the system.  After that I can load and unload the niu.ko driver
> > > without problems... hmmm
> > 
> > Perhaps this is a regression, as the problem is not in v2.6.27.
> 
> This is what I started to suspect as well.
> 
> > I'll start bisecting monday...
> > 
> > I'm not sure its a NIU driver bug, as the number of changes to niu.c is
> > very small since v2.6.27. (git log v2.6.27.. drivers/net/niu.c)
> 
> Ok, let me know what your bisect finds.

I have given up bisecting because during my bisect I have hit a kernel
that will not boot on my system (it hangs...)

I have attached the full bisect history document...

-- 
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer

[-- Attachment #2: bisect_niu_rmmod.apt --]
[-- Type: text/plain, Size: 10091 bytes --]

~~ -*-text-*-

       -------------------------------------------------------
			    Bisecting bug:
		      NIU driver rmmod MSI-X bug
       -------------------------------------------------------
		 Jesper Dangaard Brouer (jdb@comx.dk)
       -------------------------------------------------------
	$LastChangedRevision: 772 $
	$Date: 2008-11-19 13:08:13 +0100 (Wed, 19 Nov 2008) $
       -------------------------------------------------------


git clone
~~~~~~~~~

+---------
 cd /var/kernels/git/davem
 git clone git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git net-2.6-bisect
+---------


Bug description
~~~~~~~~~~~~~~~

 NIU driver rmmod kernel MSI-X bug.

 I found a strange behavior, rmmod'ing the niu driver will only cause
 a kernel BUG, if the driver was loaded at boot time.  If I remove the
 niu.ko driver from /lib/modules/2.6.28-rc4-davem/kernel/drivers/net/
 reboot the system.  After that I can load and unload the niu.ko
 driver without problems... hmmm.

Reproduce / test
~~~~~~~~~~~~~~~~

 1. Boot machine

 2. rmmod niu

 3. look at dmesg for kernel BUG output


Install trick
~~~~~~~~~~~~~

 Installing kernel in a seperate directory.

+--------
 export VER=`cat include/config/kernel.release`
 echo $VER

 export INSTALL_MOD_PATH=/var/kernels/git/install/
 rm -rf $INSTALL_MOD_PATH/lib/modules/$VER/kernel/
 make modules_install

 export BOOT="$INSTALL_MOD_PATH/boot/"
 [ -d $BOOT ] ||  mkdir $BOOT

 cp -v arch/x86/boot/bzImage $BOOT/vmlinuz-$VER
 cp -v arch/i386/boot/bzImage $BOOT/vmlinuz-$VER
 cp -v System.map $BOOT/System.map-$VER
 cp -v vmlinux $BOOT/vmlinux-$VER
+--------

* Push to test host
  ~~~~~~~~~~~~~~~~~

 export KERNEL=2.6.27-davem
 export KERNEL=2.6.28-rc2-davem
 export KERNEL=2.6.28-rc4-davem

+--------
 export HOST=ng
 export KERNEL=$VER
 pushd /var/kernels/git/install
 rsync -e ssh -avz boot/vmlinuz-${KERNEL} root@${HOST}:/boot/
 rsync -e ssh -avz boot/vmlinux-${KERNEL} root@${HOST}:/boot/
 rsync -e ssh -avz --delete lib/modules/${KERNEL}  root@${HOST}:/lib/modules/
 popd
+--------


Create branch from tag:
~~~~~~~~~~~~~~~~~~~~~~~

 Known good starting point.

+-----------
 git branch   tag_v2.6.27 v2.6.27
 git checkout tag_v2.6.27
+-----------


Start bisect
~~~~~~~~~~~~

+--------
 git checkout master
 git bisect start
 git bisect good v2.6.27
 git bisect bad master
+--------

 Create .config by <<<make oldconfig>>> or <<<make menuconfig>>>

+-------
 bisect good
#Bisecting: 2157 revisions left to test after this
#[92b29b86fe2e183d44eb467e5e74a5f718ef2e43]
#Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
+-------

 compile:
  time make -j6 bzImage modules


 Sort of "good" version, problem is that number of IRQs when down a
 lot!  This probably means bad performance as I cannot get enough IRQs
 for the Sun NIC.

+--------
 bisect good
#Bisecting: 1066 revisions left to test after this
#[ea541686d8454efac4f2b5c0767affb12d4b6a52] 
#Merge branch 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds
+--------

 compile:
  time make -j6 bzImage modules


 CRAP! - kernel (commit ea541686d8454efac4f2b5c0767affb12d4b6a52) will
 not boot on my system :-(((

 Stops with message:

+------
 hpet0: 3 comparators, 64-bit 14.318180 Mhz counter
+------

 <<Try to:>> removed "High Resolution Timer Support" in .config
 CONFIG_HIGH_RES_TIMERS and CONFIG_SCHED_HRTICK.

 Disable "HPET Timer Support" (under "Processor type and features").
 Undefs CONFIG_HPET_TIMER and CONFIG_HPET_EMULATE_RTC.

 Compiling ... installing ... It was not the problem... now I get a
 "BUG: soft lockup" kstop EIP is a stop_cpu+0x37/0xb0.


* Parallel process#1:
~~~~~~~~~~~~~~~~~~~~

 <<Try to:>> Pick a new commit point a make a new seperate branch and
 try to see if we cab boot... random picked commit
 9a1c3542768b5a58e45a9216921cd10a3bae1205

 git checkout -b new_bisect_point01 9a1c3542768b5a58e45a9216921cd10a3bae1205

 Compile on another tree davem/net-2.6-copy.

 Kernel named "-test".

 It can boot and unloading "niu" driver WORKS!

 If I understand bisect it should be possible to call:

   git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205


* Parallel process#2:
~~~~~~~~~~~~~~~~~~~~

 <<Try to:>> use git-bisect skip

+------
git-bisect skip
Bisecting: 1066 revisions left to test after this
[969907a956752f88dde4aa23fa8c033b9a939aee] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6
+------

 STILL HANGS on boot :-(((


* Back on track...
~~~~~~~~~~~~~~~~~~

+--------
git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205
Bisecting: 1000 revisions left to test after this
[1137fb670465b6b5d15b9db7d01707a5833ee3ae] arm ide breakage
+--------

 compile... install...

 BAD: Booting (now 2.6.28-rc1-bisect) and unloading "niu" causes the bug!

 Mark bisecting as BAD.

+----------
git-bisect bad
Bisecting: 508 revisions left to test after this
[36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6
+----------

 compile ... install ... (back on ver. 2.6.27-bisect)

 GRRRR... Now I cannot boot again! :-(((

* Try to find a new place
  ~~~~~~~~~~~~~~~~~~~~~~~

 Random pick: dbacefc9c4f6bd365243db379473ab7041656d90

+--------
 cd /var/kernels/git/davem/net-2.6-copy/
 git checkout -b new_bisect_point02 dbacefc9c4f6bd365243db379473ab7041656d90
+--------

 VER=2.6.27-rc1-test

 This version can boot and unloading niu works.

* Back on track(2) ... but not :-(
~~~~~~~~~~~~~~~~~~

 git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90

+--------
git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90
Bisecting: 508 revisions left to test after this
[36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6
+---------

 Hmmm... commit 36ec891895020f3bc9953c8b11d079c6d77d76bd was the same
 as before... which could not boot...

* Try a bisect skip...
 ~~~~~~~~~~~~~~~~~~~~~

+---------
 git bisect skip
Bisecting: 508 revisions left to test after this
[70740d6c93030b339b4ad17fd58ee135dfc13913] Merge branch 'drm-next' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6
+---------

 compile ... install ... boot... CANNOT BOOT !!! :-(((


* Doing lucky guessing
~~~~~~~~~~~~~~~~~~~~~~

 git-bisect good

+--------------
git-bisect good
Bisecting: 259 revisions left to test after this
[22484856402bfa1ff3defe47f6029ab0418240d9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/viro/bdev
+--------------

 ARGH!!! -- cannot boot this kernel, it hangs :-(((

* Doing desperate guessing
~~~~~~~~~~~~~~~~~~~~~~~~~~

+--------
git-bisect good
Bisecting: 132 revisions left to test after this
[c3c9897c63ebb0b93b7f78724e38d6ee1da04041] Merge branch 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
+--------

 compiling ... installing ...
 ... booting -- ARGH! cannot boot hang!


* Doing hopeless guessing
~~~~~~~~~~~~~~~~~~~~~~~~~

 This is hopeless, I should give up!

+-----------------
git-bisect good
Bisecting: 65 revisions left to test after this
[969907a956752f88dde4aa23fa8c033b9a939aee] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6
+------------------

 compiling ...
 installing ...
 still hangs after boot... :-(((

 ... GIVING UP!!!


* git bisect log
~~~~~~~~~~~~~~~~

+-------
git-bisect start
# good: [3fa8749e584b55f1180411ab1b51117190bac1e5] Linux 2.6.27
git-bisect good 3fa8749e584b55f1180411ab1b51117190bac1e5
# bad: [5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776] rtnetlink: propagate error from dev_change_flags in do_setlink()
git-bisect bad 5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776
# good: [29415c37f043d1d54dcf356601d738ff6633b72b] KVM: set debug registers after "schedulable" section
git-bisect good 29415c37f043d1d54dcf356601d738ff6633b72b
# good: [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect good 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
# skip: [ea541686d8454efac4f2b5c0767affb12d4b6a52] Merge branch 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds
git-bisect skip ea541686d8454efac4f2b5c0767affb12d4b6a52
# good: [9a1c3542768b5a58e45a9216921cd10a3bae1205] pass fmode_t to blkdev_put()
git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205
# bad: [1137fb670465b6b5d15b9db7d01707a5833ee3ae] arm ide breakage
git-bisect bad 1137fb670465b6b5d15b9db7d01707a5833ee3ae
# good: [dbacefc9c4f6bd365243db379473ab7041656d90] fs/buffer.c: uninline __remove_assoc_queue()
git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90
# skip: [36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6
git-bisect skip 36ec891895020f3bc9953c8b11d079c6d77d76bd
# good: [70740d6c93030b339b4ad17fd58ee135dfc13913] Merge branch 'drm-next' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6
git-bisect good 70740d6c93030b339b4ad17fd58ee135dfc13913
# good: [22484856402bfa1ff3defe47f6029ab0418240d9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/viro/bdev
git-bisect good 22484856402bfa1ff3defe47f6029ab0418240d9
# good: [c3c9897c63ebb0b93b7f78724e38d6ee1da04041] Merge branch 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect good c3c9897c63ebb0b93b7f78724e38d6ee1da04041
+-------


Compile:
~~~~~~~~

+-------
 time make -j6 bzImage modules
+-------


NOTES:
~~~~~~

 git log 29415c37f043d1d54dcf356601d738ff6633b72b..5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776

 NR_IRQS changed ... could this be releated?

 git show 7db282fa67b58daff8a57f9e1c93d4474b5908ff
 git show 1b4897688011cd05e07f00dcfe6af3331eb36a3c
 git show c78d0cf2925bffae8a6f00e7d9b8e971b0392edd