Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Andrew Morton]

On Sat, 01 Dec 2007 11:30:01 +1300
Michael Cree <mcree@orcon.net.nz> wrote:

> Bob Tracy wrote:
> > Andrew Morton wrote:
> >> Could be something change in sysfs.  Please double-check the config
> >> options, make sure that something important didn't get disabled.
> >>
> >  Here's
> > hoping someone else is seeing this or can replicate it in the meantime.
> 
> Snap.
> 
> 2.6.24-rc2 works fine.   2.6.24-rc3 boots on Alpha but once /dev is 
> populated no partitions of the scsi sub-system are seen.  Looks like ide 
> sub-system similarly affected.

Rafael, I assume you have this regression in the list?

> Managed to get boot log.  Follows below (with output of various /proc info).
> 
> Cheerz
> Michael.
> 
> 
> Linux version 2.6.24-rc3 (mjc@alpha) (gcc version 4.1.3 20071019 
> (prerelease) (Debian 4.1.2-17)) #1 Mon Nov 26 19:28:58 NZDT 2007
> Booting on Tsunami variation Monet using machine vector Monet from SRM
> Major Options: EV67 LEGACY_START VERBOSE_MCHECK
> Command line: ro root=/dev/sda3 console=ttyS0
> memcluster 0, usage 1, start        0, end      215
> memcluster 1, usage 0, start      215, end   131062
> memcluster 2, usage 1, start   131062, end   131072
> freeing pages 215:384
> freeing pages 930:131062
> reserving pages 930:932
> 4096K Bcache detected; load hit latency 21 cycles, load miss latency 127 
> cycles
> Console graphics on hose 0
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 130167
> Kernel command line: ro root=/dev/sda3 console=ttyS0
> PID hash table entries: 4096 (order: 12, 32768 bytes)
> Using epoch = 2000
> Turning on RTC interrupts.
> Console: colour VGA+ 80x25
> console [ttyS0] enabled
> Dentry cache hash table entries: 131072 (order: 7, 1048576 bytes)
> Inode-cache hash table entries: 65536 (order: 6, 524288 bytes)
> Memory: 1030896k/1048496k available (2786k kernel code, 15216k reserved, 
> 370k data, 168k init)
> Mount-cache hash table entries: 512
> net_namespace: 120 bytes
> NET: Registered protocol family 16
> PCI: Bridge: 0001:01:08.0
>    IO window: 8000-8fff
>    MEM window: 09000000-090fffff
>    PREFETCH window: disabled.
> SMC37c669 Super I/O Controller found @ 0x3f0
> Linux Plug and Play Support v0.97 (c) Adam Belay
> SCSI subsystem initialized
> NET: Registered protocol family 2
> IP route cache hash table entries: 8192 (order: 3, 65536 bytes)
> TCP established hash table entries: 32768 (order: 6, 524288 bytes)
> TCP bind hash table entries: 32768 (order: 5, 262144 bytes)
> TCP: Hash tables configured (established 32768 bind 32768)
> TCP reno registered
> srm_env: version 0.0.6 loaded successfully
> io scheduler noop registered
> io scheduler cfq registered (default)
> tridentfb: Trident framebuffer 0.7.8-NEWAPI initializing
> isapnp: Scanning for PnP cards...
> isapnp: No Plug & Play device found
> rtc: SRM (post-2000) epoch (2000) detected
> Real Time Clock Driver v1.12ac
> Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
> serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
> Floppy drive(s): fd0 is 2.88M
> FDC 0 is a post-1991 82077
> Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
> CY82C693: IDE controller (0x1080:0xc693 rev 0x00) at  PCI slot 0000:00:07.1
> CY82C693: not 100% native mode: will probe irqs later
> CY82C693U driver v0.34 99-13-12 Andreas S. Krebs (akrebs@altavista.net)
>      ide0: BM-DMA at 0x8400-0x8407, BIOS settings: hda:pio, hdb:pio
> CY82C693: port 0x01f0 already claimed by ide0
> ALI15X3: IDE controller (0x10b9:0x5228 rev 0xc6) at  PCI slot 0001:02:09.1
> ALI15X3: 100% native mode on irq 28
>      ide1: BM-DMA at 0x200008410-0x200008417, BIOS settings: hdc:DMA, 
> hdd:DMA
>      ide2: BM-DMA at 0x200008418-0x20000841f, BIOS settings: hde:pio, 
> hdf:pio
> hdf: LITE-ON DVDRW SOHW-1653S, ATAPI CD/DVD-ROM drive
> hde: ST3200822A, ATA DISK drive
> ide2 at 0x200008438-0x20000843f,0x20000844e on irq 28
> hde: max request size: 512KiB
> hde: 390721968 sectors (200049 MB) w/8192KiB Cache, CHS=24321/255/63, 
> UDMA(100)
> hde: cache flushes supported
>   hde: hde1
> qla1280: QLA1040 found on PCI bus 1, dev 6
> scsi(0:0): Resetting SCSI BUS
> scsi0 : QLogic QLA1040 PCI to SCSI Host Adapter
>         Firmware version:  7.65.06, Driver version 3.26
> serio: i8042 KBD port at 0x60,0x64 irq 1
> serio: i8042 AUX port at 0x60,0x64 irq 12
> mice: PS/2 mouse device common for all mice
> scsi 0:0:1:0: Direct-Access     SEAGATE  ST336706LW       0109 PQ: 0 ANSI: 3
> scsi(0:0:1:0): Sync: period 10, offset 12, Wide
> input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
> atkbd.c: keyboard reset failed on isa0060/serio1
> TCP cubic registered
> Initializing XFRM netlink socket
> NET: Registered protocol family 1
> NET: Registered protocol family 17
> NET: Registered protocol family 15
> scsi: waiting for bus probes to complete ...
> sd 0:0:1:0: [sda] 71687370 512-byte hardware sectors (36704 MB)
> sd 0:0:1:0: [sda] Write Protect is off
> sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, supports 
> DPO and FUA
> sd 0:0:1:0: [sda] 71687370 512-byte hardware sectors (36704 MB)
> sd 0:0:1:0: [sda] Write Protect is off
> sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, supports 
> DPO and FUA
>   sda: sda1 sda2 sda3 sda4 sda5
> sd 0:0:1:0: [sda] Attached SCSI disk
> kjournald starting.  Commit interval 5 seconds
> EXT3-fs: mounted filesystem with ordered data mode.
> VFS: Mounted root (ext3 filesystem) readonly.
> Freeing unused kernel memory: 168k freed
> Mount failed for selinuxfs on /selinux:  No such file or directory
>   INIT: version 2.86 booting
> Starting the hotplug events dispatcher: udevd.
> Synthesizing the initial hotplug events...done.
> Waiting for /dev to be fully populated...scsi_id[321]: scsi_id: unable 
> to access '/block'
> end_request: I/O error, dev fd0, sector 0
> scsi_id[371]: scsi_id: unable to access '/block'
> usbcore: registered new interface driver usbfs
> usbcore: registered new interface driver hub
> Linux Tulip driver version 1.1.15 (Feb 27, 2007)
> tulip0:  EEPROM default media type Autosense.
> tulip0:  Index #0 - Media 10baseT (#0) described by a 21142 Serial PHY 
> (2) block.
> tulip0:  Index #1 - Media 10base2 (#1) described by a 21142 Serial PHY 
> (2) block.
> tulip0:  Index #2 - Media AUI (#2) described by a 21142 Serial PHY (2) 
> block.
> tulip0:  Index #3 - Media 10baseT-FDX (#4) described by a 21142 Serial 
> PHY (2) block.
> tulip0:  Index #4 - Media 100baseTx (#3) described by a 21143 SYM PHY 
> (4) block.
> tulip0:  Index #5 - Media 100baseTx-FDX (#5) described by a 21143 SYM 
> PHY (4) block.
> eth0: Digital DS21142/43 Tulip rev 65 at Port 0x200009400, 
> 08:00:2b:87:4c:b0, IRQ 45.
> Linux video capture interface: v2.00
> scsi_id[402]: scsi_id: unable to access '/block'

I guess this is where things go bad.

scsi_id is part of udev.  Perhaps some sysfs nodes aren't being created
correctly.

Random guess: what is your setting of CONFIG_SCSI_SCAN_ASYNC and what
happens if you invert it?


> input: PC Speaker as /devices/platform/pcspkr/input/input1
> scsi_id[422]: scsi_id: unable to access '/block'
> sata_uli 0001:02:09.0: version 1.3
> scsi1 : sata_uli
> scsi2 : sata_uli
> ata1: SATA max UDMA/133 cmd 0x200008420 ctl 0x200008440 bmdma 
> 0x200008400 irq 28
> ata2: SATA max UDMA/133 cmd 0x200008428 ctl 0x200008444 bmdma 
> 0x200008408 irq 28
> usbcore: registered new device driver usb
> hdf: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(66)
> Uniform CD-ROM driver Revision: 3.20
> scsi_id[469]: scsi_id: unable to access '/block'
> ata1: SATA link down (SStatus 0 SControl 300)
> ohci_hcd 0000:00:07.3: OHCI Host Controller
> ohci_hcd 0000:00:07.3: new USB bus registered, assigned bus number 1
> ohci_hcd 0000:00:07.3: irq 10, io mem 0x09050000
> scsi_id[490]: scsi_id: unable toivtv:  Start initialization, version 1.1.0
>   access '/block'
> scsi_id[510]: scsi_id: unable to access '/block'
> scsi_id[532]: scsi_id: unable to access '/block'
> scsi_id[552]: scsi_id: unable to access '/block'
> usb usb1: configuration #1 chosen from 1 choice
> hub 1-0:1.0: USB hub found
> hub 1-0:1.0: 2 ports detected
> scsi_id[600]: scsi_id: unable to access '/block'
> ivtv0: Initializing card #0
> ivtv0: Autodetected Hauppauge card (cx23415 based)
> ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
> ata2.00: ATA-7: ST3320620AS, 3.AAK, max UDMA/133
> ...

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Rafael J. Wysocki]

On Friday, 30 of November 2007, Andrew Morton wrote:
> On Sat, 01 Dec 2007 11:30:01 +1300
> Michael Cree <mcree@orcon.net.nz> wrote:
> 
> > Bob Tracy wrote:
> > > Andrew Morton wrote:
> > >> Could be something change in sysfs.  Please double-check the config
> > >> options, make sure that something important didn't get disabled.
> > >>
> > >  Here's
> > > hoping someone else is seeing this or can replicate it in the meantime.
> > 
> > Snap.
> > 
> > 2.6.24-rc2 works fine.   2.6.24-rc3 boots on Alpha but once /dev is 
> > populated no partitions of the scsi sub-system are seen.  Looks like ide 
> > sub-system similarly affected.
> 
> Rafael, I assume you have this regression in the list?

Yes, http://bugzilla.kernel.org/show_bug.cgi?id=9457

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Michael Cree]

On 1/12/2007, at 11:42 AM, Andrew Morton wrote:
> On Sat, 01 Dec 2007 11:30:01 +1300
> Michael Cree <mcree@orcon.net.nz> wrote:
>
>> Bob Tracy wrote:
>>> Andrew Morton wrote:
>>>> Could be something change in sysfs.  Please double-check the config
>>>> options, make sure that something important didn't get disabled.
>>>>
>>>  Here's
>>> hoping someone else is seeing this or can replicate it in the  
>>> meantime.
>>
>> Snap.
>>
>> 2.6.24-rc2 works fine.   2.6.24-rc3 boots on Alpha but once /dev is
>> populated no partitions of the scsi sub-system are seen.  Looks  
>> like ide
>> sub-system similarly affected.

[snip]

>> eth0: Digital DS21142/43 Tulip rev 65 at Port 0x200009400,
>> 08:00:2b:87:4c:b0, IRQ 45.
>> Linux video capture interface: v2.00
>> scsi_id[402]: scsi_id: unable to access '/block'
>
> I guess this is where things go bad.

Yes, that is what I thought too.

> scsi_id is part of udev.  Perhaps some sysfs nodes aren't being  
> created
> correctly.
>
> Random guess: what is your setting of CONFIG_SCSI_SCAN_ASYNC and what
> happens if you invert it?

Is set to Y.  Changed it to N and recompiled kernel and restarted.   
No change.  Same problems remain.

I now realise that not only SCSI drive device nodes are not appearing  
in /dev, but all disc nodes are not appearing.

In my case all of fd0, hde (IDE disc), hdf (CD/DVD), sda (SCSI disc),  
sdb (SATA disc), sdc (memory card reader), and their accompanying  
partition nodes, do not get made in /dev.

I'm not familiar with sysfs so don't know what what I should be  
looking for in particular, but I did have a quick noisy around /sys  
and noted that the above mentioned devices are all appearing at /sys/ 
block with what appeared to be sensible information in the  
subdirectories thereof.

Sorry, but it is unlikely that I'll be able to look further into this  
problem at the moment as I am running a conference this week and the  
storm is about to hit...

Cheers
Michael.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Michael Cree wrote:
> On 1/12/2007, at 11:42 AM, Andrew Morton wrote:
> > On Sat, 01 Dec 2007 11:30:01 +1300
> > Michael Cree <mcree@orcon.net.nz> wrote:
> >
> >> Bob Tracy wrote:
> >>>  Here's
> >>> hoping someone else is seeing this or can replicate it in the  
> >>> meantime.
> >>
> >> Snap.
> >>
> >> 2.6.24-rc2 works fine.   2.6.24-rc3 boots on Alpha but once /dev is
> >> populated no partitions of the scsi sub-system are seen.  Looks  
> >> like ide sub-system similarly affected.
> 
> [snip]
> 
> >> eth0: Digital DS21142/43 Tulip rev 65 at Port 0x200009400,
> >> 08:00:2b:87:4c:b0, IRQ 45.
> >> Linux video capture interface: v2.00
> >> scsi_id[402]: scsi_id: unable to access '/block'
> >
> > I guess this is where things go bad.
> 
> Yes, that is what I thought too.

Thanks for the confirmation of the error condition.  As best I can
recall, your boot log is substantially the same as what I saw.

Finally got back in town.  Starting the git-bisect process.  I've got
a relatively slow network connection, and the PWS 433au isn't exactly
what I would call "fast" by modern standards, so bear with me while I
get things set up and crank through this.  The clone of the 2.6 tree
will take several more hours to finish downloading.  I anticipate the
best pace I'll be able to manage after that is two iterations in a 24-
hour period.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Ingo Molnar]

* Bob Tracy <rct@gherkin.frus.com> wrote:

> Finally got back in town.  Starting the git-bisect process.  I've got 
> a relatively slow network connection, and the PWS 433au isn't exactly 
> what I would call "fast" by modern standards, so bear with me while I 
> get things set up and crank through this.  The clone of the 2.6 tree 
> will take several more hours to finish downloading.  I anticipate the 
> best pace I'll be able to manage after that is two iterations in a 24- 
> hour period.

once you are done with the download of the initial cloned git repository 
(which is 200MB+), all the bisection steps will be local and you'll be 
only limited by kernel rebuild speed and by bootup and testing speed, 
not by network bandwidth.

( once you have the cloned repository i'd suggest for you to keep it - 
  that way you can track susequent kernels via "git-pull" and it uses a 
  very network-efficient delta protocol. )

	Ingo

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Ingo Molnar wrote:
> once you are done with the download of the initial cloned git repository 
> (which is 200MB+), all the bisection steps will be local and you'll be 
> only limited by kernel rebuild speed and by bootup and testing speed, 
> not by network bandwidth.

ACK.  Have tested two kernels in the past 24 hours, and the third is
building as I type this.  The builds seem to be taking about 3 hours
each.  First two tests good, so the offending commit is somewhere in
the last 25% (roughly) of the changes between -rc2 and -rc3: git says
82 revisions left to test.  Might have this painted into a corner in
the next day or so.  I'll try to be quick about it, since -rc4 is out.

> ( once you have the cloned repository i'd suggest for you to keep it - 
>   that way you can track susequent kernels via "git-pull" and it uses a 
>   very network-efficient delta protocol. )

Will do...  I'm in the fortunate position of having enough disk space
on my Alpha that I can maintain multiple trees for this kind of effort.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Current progress: 11 revisions left to test.  The current partial
"git bisect log" is available per Ingo's suggestion on bugzilla.

http://bugzilla.kernel.org/show_bug.cgi?id=9457

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

OK.  Finally have this thing painted into a corner: git has identified
6f37ac793d6ba7b35d338f791974166f67fdd9ba as the first bad commit.

>From "git bisect log", this corresponds to 

# bad: [6f37ac793d6ba7b35d338f791974166f67fdd9ba] Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6

Here's the full log:

git-bisect start
# good: [9aae299f7fd1888ea3a195cfe0edef17bb647415] Linux 2.6.24-rc2
git-bisect good 9aae299f7fd1888ea3a195cfe0edef17bb647415
# bad: [f05092637dc0d9a3f2249c9b283b973e6e96b7d2] Linux 2.6.24-rc3
git-bisect bad f05092637dc0d9a3f2249c9b283b973e6e96b7d2
# good: [e6a5c27f3b0fef72e528fc35e343af4b2db790ff] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm
git-bisect good e6a5c27f3b0fef72e528fc35e343af4b2db790ff
# good: [42614fcde7bfdcbe43a7b17035c167dfebc354dd] vmstat: fix section mismatch warning
git-bisect good 42614fcde7bfdcbe43a7b17035c167dfebc354dd
# bad: [a052f4473603765eb6b4c19754689977601dc1d1] Merge git://git.kernel.org/pub/scm/linux/kernel/git/sam/x86
git-bisect bad a052f4473603765eb6b4c19754689977601dc1d1
# good: [d8e5219f9f5ca7518eb820db9f3d287a1d46fcf5] CRISv10 improve and bugfix fasttimer
git-bisect good d8e5219f9f5ca7518eb820db9f3d287a1d46fcf5
# good: [d90bf5a976793edfa88d3bb2393f0231eb8ce1e5] [NET]: rt_check_expire() can take a long time, add a cond_resched()
git-bisect good d90bf5a976793edfa88d3bb2393f0231eb8ce1e5
# good: [2a113281f5cd2febbab21a93c8943f8d3eece4d3] kconfig: use $K64BIT to set 64BIT with all*config targets
git-bisect good 2a113281f5cd2febbab21a93c8943f8d3eece4d3
# good: [2e2cd8bad6e03ceea73495ee6d557044213d95de] CRISv10 memset library add lineendings to asm
git-bisect good 2e2cd8bad6e03ceea73495ee6d557044213d95de
# bad: [6f37ac793d6ba7b35d338f791974166f67fdd9ba] Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6
git-bisect bad 6f37ac793d6ba7b35d338f791974166f67fdd9ba
# good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better
git-bisect good 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Andrew Morton]

On Thu, 6 Dec 2007 18:16:12 -0600 (CST)
rct@frus.com (Bob Tracy) wrote:

> OK.  Finally have this thing painted into a corner: git has identified
> 6f37ac793d6ba7b35d338f791974166f67fdd9ba as the first bad commit.
> 
> >From "git bisect log", this corresponds to 
> 
> # bad: [6f37ac793d6ba7b35d338f791974166f67fdd9ba] Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6
> 
> Here's the full log:
> 
> git-bisect start
> # good: [9aae299f7fd1888ea3a195cfe0edef17bb647415] Linux 2.6.24-rc2
> git-bisect good 9aae299f7fd1888ea3a195cfe0edef17bb647415
> # bad: [f05092637dc0d9a3f2249c9b283b973e6e96b7d2] Linux 2.6.24-rc3
> git-bisect bad f05092637dc0d9a3f2249c9b283b973e6e96b7d2
> # good: [e6a5c27f3b0fef72e528fc35e343af4b2db790ff] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm
> git-bisect good e6a5c27f3b0fef72e528fc35e343af4b2db790ff
> # good: [42614fcde7bfdcbe43a7b17035c167dfebc354dd] vmstat: fix section mismatch warning
> git-bisect good 42614fcde7bfdcbe43a7b17035c167dfebc354dd
> # bad: [a052f4473603765eb6b4c19754689977601dc1d1] Merge git://git.kernel.org/pub/scm/linux/kernel/git/sam/x86
> git-bisect bad a052f4473603765eb6b4c19754689977601dc1d1
> # good: [d8e5219f9f5ca7518eb820db9f3d287a1d46fcf5] CRISv10 improve and bugfix fasttimer
> git-bisect good d8e5219f9f5ca7518eb820db9f3d287a1d46fcf5
> # good: [d90bf5a976793edfa88d3bb2393f0231eb8ce1e5] [NET]: rt_check_expire() can take a long time, add a cond_resched()
> git-bisect good d90bf5a976793edfa88d3bb2393f0231eb8ce1e5
> # good: [2a113281f5cd2febbab21a93c8943f8d3eece4d3] kconfig: use $K64BIT to set 64BIT with all*config targets
> git-bisect good 2a113281f5cd2febbab21a93c8943f8d3eece4d3
> # good: [2e2cd8bad6e03ceea73495ee6d557044213d95de] CRISv10 memset library add lineendings to asm
> git-bisect good 2e2cd8bad6e03ceea73495ee6d557044213d95de
> # bad: [6f37ac793d6ba7b35d338f791974166f67fdd9ba] Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6
> git-bisect bad 6f37ac793d6ba7b35d338f791974166f67fdd9ba
> # good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better
> git-bisect good 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3

commit 6f37ac793d6ba7b35d338f791974166f67fdd9ba
Merge: 2f1f53b... d90bf5a...
Author: Linus Torvalds <torvalds@woody.linux-foundation.org>
Date:   Wed Nov 14 18:51:48 2007 -0800

    Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/n
    
    * 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6:
      [NET]: rt_check_expire() can take a long time, add a cond_resched()
      [ISDN] sc: Really, really fix warning
      [ISDN] sc: Fix sndpkt to have the correct number of arguments
      [TCP] FRTO: Clear frto_highmark only after process_frto that uses it
      [NET]: Remove notifier block from chain when register_netdevice_notifier f
      [FS_ENET]: Fix module build.
      [TCP]: Make sure write_queue_from does not begin with NULL ptr
      [TCP]: Fix size calculation in sk_stream_alloc_pskb
      [S2IO]: Fixed memory leak when MSI-X vector allocation fails
      [BONDING]: Fix resource use after free
      [SYSCTL]: Fix warning for token-ring from sysctl checker
      [NET] random : secure_tcp_sequence_number should not assume CONFIG_KTIME_S
      [IWLWIFI]: Not correctly dealing with hotunplug.
      [TCP] FRTO: Plug potential LOST-bit leak
      [TCP] FRTO: Limit snd_cwnd if TCP was application limited
      [E1000]: Fix schedule while atomic when called from mii-tool.
      [NETX]: Fix build failure added by 2.6.24 statistics cleanup.
      [EP93xx_ETH]: Build fix after 2.6.24 NAPI changes.
      [PKT_SCHED]: Check subqueue status before calling hard_start_xmit

I'm struggling to see how any of those could have broken block device
mounting on alpha.  Are you sure you bisected right?

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Andrew Morton wrote:
> commit 6f37ac793d6ba7b35d338f791974166f67fdd9ba
> Merge: 2f1f53b... d90bf5a...
> Author: Linus Torvalds <torvalds@woody.linux-foundation.org>
> Date:   Wed Nov 14 18:51:48 2007 -0800
> 
>     Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/n
>     
>     * 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6:
>       [NET]: rt_check_expire() can take a long time, add a cond_resched()
>       [ISDN] sc: Really, really fix warning
>       [ISDN] sc: Fix sndpkt to have the correct number of arguments
>       [TCP] FRTO: Clear frto_highmark only after process_frto that uses it
>       [NET]: Remove notifier block from chain when register_netdevice_notifier f
>       [FS_ENET]: Fix module build.
>       [TCP]: Make sure write_queue_from does not begin with NULL ptr
>       [TCP]: Fix size calculation in sk_stream_alloc_pskb
>       [S2IO]: Fixed memory leak when MSI-X vector allocation fails
>       [BONDING]: Fix resource use after free
>       [SYSCTL]: Fix warning for token-ring from sysctl checker
>       [NET] random : secure_tcp_sequence_number should not assume CONFIG_KTIME_S
>       [IWLWIFI]: Not correctly dealing with hotunplug.
>       [TCP] FRTO: Plug potential LOST-bit leak
>       [TCP] FRTO: Limit snd_cwnd if TCP was application limited
>       [E1000]: Fix schedule while atomic when called from mii-tool.
>       [NETX]: Fix build failure added by 2.6.24 statistics cleanup.
>       [EP93xx_ETH]: Build fix after 2.6.24 NAPI changes.
>       [PKT_SCHED]: Check subqueue status before calling hard_start_xmit
> 
> I'm struggling to see how any of those could have broken block device
> mounting on alpha.  Are you sure you bisected right?

Based on what's in that commit, it *does* appear something went wrong
with bisection.  If the implicated commit is the next one in time
sequence relative to

# good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better

then the test of whether I bisected correctly is as simple as applying
the commit and seeing if things break, because I'm running on the
kernel corresponding to 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 right
now.  Let me give that a try and I'll report back.  Worst case, I'll
have to start over and write off the past four days...

Sorry about this...

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Andrew Morton]

On Thu, 6 Dec 2007 23:07:08 -0600 (CST) rct@frus.com (Bob Tracy) wrote:

> Andrew Morton wrote:
> > commit 6f37ac793d6ba7b35d338f791974166f67fdd9ba
> > Merge: 2f1f53b... d90bf5a...
> > Author: Linus Torvalds <torvalds@woody.linux-foundation.org>
> > Date:   Wed Nov 14 18:51:48 2007 -0800
> > 
> >     Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/n
> >     
> >     * 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6:
> >       [NET]: rt_check_expire() can take a long time, add a cond_resched()
> >       [ISDN] sc: Really, really fix warning
> >       [ISDN] sc: Fix sndpkt to have the correct number of arguments
> >       [TCP] FRTO: Clear frto_highmark only after process_frto that uses it
> >       [NET]: Remove notifier block from chain when register_netdevice_notifier f
> >       [FS_ENET]: Fix module build.
> >       [TCP]: Make sure write_queue_from does not begin with NULL ptr
> >       [TCP]: Fix size calculation in sk_stream_alloc_pskb
> >       [S2IO]: Fixed memory leak when MSI-X vector allocation fails
> >       [BONDING]: Fix resource use after free
> >       [SYSCTL]: Fix warning for token-ring from sysctl checker
> >       [NET] random : secure_tcp_sequence_number should not assume CONFIG_KTIME_S
> >       [IWLWIFI]: Not correctly dealing with hotunplug.
> >       [TCP] FRTO: Plug potential LOST-bit leak
> >       [TCP] FRTO: Limit snd_cwnd if TCP was application limited
> >       [E1000]: Fix schedule while atomic when called from mii-tool.
> >       [NETX]: Fix build failure added by 2.6.24 statistics cleanup.
> >       [EP93xx_ETH]: Build fix after 2.6.24 NAPI changes.
> >       [PKT_SCHED]: Check subqueue status before calling hard_start_xmit
> > 
> > I'm struggling to see how any of those could have broken block device
> > mounting on alpha.  Are you sure you bisected right?
> 
> Based on what's in that commit, it *does* appear something went wrong
> with bisection.  If the implicated commit is the next one in time
> sequence relative to
> 
> # good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better
> 
> then the test of whether I bisected correctly is as simple as applying
> the commit and seeing if things break, because I'm running on the
> kernel corresponding to 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 right
> now.  Let me give that a try and I'll report back.  Worst case, I'll
> have to start over and write off the past four days...

Gad.  I trust the second time will be faster.

git-bisect _is_ very error prone.  I find one of the problems is that each
step is so far apart in time that you forget what you were doing.  Did I
remember to test that iteration?  Did I install the right kernel?  etc.

> Sorry about this...

Not appropriate ;)   Thanks for helping out.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Ingo Molnar]

* Andrew Morton <akpm@linux-foundation.org> wrote:

> > then the test of whether I bisected correctly is as simple as 
> > applying the commit and seeing if things break, because I'm running 
> > on the kernel corresponding to 
> > 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 right now.  Let me give 
> > that a try and I'll report back.  Worst case, I'll have to start 
> > over and write off the past four days...
> 
> Gad.  I trust the second time will be faster.
> 
> git-bisect _is_ very error prone.  I find one of the problems is that 
> each step is so far apart in time that you forget what you were doing.  
> Did I remember to test that iteration?  Did I install the right 
> kernel?  etc.

i have a fully automated bootup-hang bisection script. It is based on 
"git-bisect run". I run the script, it builds and boots kernels fully 
automatically, and when the bootup fails (the script notices that via 
the serial log, which it continuously watches - or via a timeout, if the 
system does not come up within 10 minutes it's a "bad" kernel), the 
script raises my attention via a beep and i power cycle the test box. 
(yeah, i should make use of a managed power outlet to 100% automate it) 

So i dont have to a single manual decision anytime during the bisection. 
But the scripts are very much tied to my ad-hoc test environment so it 
would not be of much general use.

	Ingo

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Andrew Morton wrote:
> On Thu, 6 Dec 2007 23:07:08 -0600 (CST) rct@frus.com (Bob Tracy) wrote:
> > Andrew Morton wrote:
> > > commit 6f37ac793d6ba7b35d338f791974166f67fdd9ba
> > > Merge: 2f1f53b... d90bf5a...
> > > Author: Linus Torvalds <torvalds@woody.linux-foundation.org>
> > > Date:   Wed Nov 14 18:51:48 2007 -0800
> > > 
> > >     Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/n
> > >     
> > >     * 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6:
> > >       (omitted for brevity)
> > > 
> > > I'm struggling to see how any of those could have broken block device
> > > mounting on alpha.  Are you sure you bisected right?
> > 
> > Based on what's in that commit, it *does* appear something went wrong
> > with bisection.  If the implicated commit is the next one in time
> > sequence relative to
> > 
> > # good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better
> > 
> > then the test of whether I bisected correctly is as simple as applying
> > the commit and seeing if things break, because I'm running on the
> > kernel corresponding to 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 right
> > now.  Let me give that a try and I'll report back.  Worst case, I'll
> > have to start over and write off the past four days...
> 
> Gad.  I trust the second time will be faster.
> 
> git-bisect _is_ very error prone.  I find one of the problems is that each
> step is so far apart in time that you forget what you were doing.  Did I
> remember to test that iteration?  Did I install the right kernel?  etc.
> 
> > Sorry about this...
> 
> Not appropriate ;)   Thanks for helping out.

Thanks for the kind words...  The above-mentioned test verified that the
bisection was/is correct: 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 works,
and 6f37ac793d6ba7b35d338f791974166f67fdd9ba doesn't.  Now I've got to
figure out why.

"git diff 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 6f37ac793d6ba7b35d338f791974166f67fdd9ba"
produced a relatively short patch (18,437 bytes).  The list of involved
files:

diff --git a/drivers/char/random.c b/drivers/char/random.c
diff --git a/drivers/isdn/sc/card.h b/drivers/isdn/sc/card.h
diff --git a/drivers/isdn/sc/packet.c b/drivers/isdn/sc/packet.c
diff --git a/drivers/isdn/sc/shmem.c b/drivers/isdn/sc/shmem.c
diff --git a/drivers/net/arm/ep93xx_eth.c b/drivers/net/arm/ep93xx_eth.c
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
diff --git a/drivers/net/e1000/e1000_main.c b/drivers/net/e1000/e1000_main.c
diff --git a/drivers/net/fs_enet/Kconfig b/drivers/net/fs_enet/Kconfig
diff --git a/drivers/net/fs_enet/Makefile b/drivers/net/fs_enet/Makefile
diff --git a/drivers/net/netx-eth.c b/drivers/net/netx-eth.c
diff --git a/drivers/net/s2io.c b/drivers/net/s2io.c
diff --git a/drivers/net/wireless/iwlwifi/iwl3945-base.c b/drivers/net/wireless/iwlwifi/iwl3945-base.c
diff --git a/include/net/sock.h b/include/net/sock.h
diff --git a/kernel/sysctl_check.c b/kernel/sysctl_check.c
diff --git a/net/core/dev.c b/net/core/dev.c
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c

Current state of the source tree is the 6f37ac... version, so I'll start
backing out the above diffs in related groups and continue until I've got
a working kernel.  For lack of an obvious target, I'll start with the
seemingly innocuous change to sysctl_check.c.  I'll report back when I've
got something.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

I wrote:
> "git diff 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 6f37ac793d6ba7b35d338f791974166f67fdd9ba"
> produced a relatively short patch (18,437 bytes).  The list of involved
> files:
> 
> (omitted)
>
> Current state of the source tree is the 6f37ac... version, so I'll start
> backing out the above diffs in related groups and continue until I've got
> a working kernel.  For lack of an obvious target, I'll start with the
> seemingly innocuous change to sysctl_check.c.  I'll report back when I've
> got something.

That was quick :-).  Backing out the sysctl_check.c diff gives me a
working kernel.  Beats the #$%@! out of me how/why, though.

Michael Cree: could you try backing out the diff below from your
2.6.24-rc3 tree and see if things are now working for you?

Here's "uname -a", just to confirm (maybe) I'm running on what I say
works:

Linux smirkin 2.6.24-rc2-g6f37ac79-dirty #2 Fri Dec 7 08:03:12 CST 2007 alpha

Here's the diff I backed out (patch -R).  It's short...

diff --git a/kernel/sysctl_check.c b/kernel/sysctl_check.c
index 5a2f2b2..4abc6d2 100644
--- a/kernel/sysctl_check.c
+++ b/kernel/sysctl_check.c
@@ -738,7 +738,7 @@ static struct trans_ctl_table trans_net_table[] = {
 	{ NET_ROSE,		"rose",		trans_net_rose_table },
 	{ NET_IPV6,		"ipv6",		trans_net_ipv6_table },
 	{ NET_X25,		"x25",		trans_net_x25_table },
-	{ NET_TR,		"tr",		trans_net_tr_table },
+	{ NET_TR,		"token-ring",	trans_net_tr_table },
 	{ NET_DECNET,		"decnet",	trans_net_decnet_table },
 	/*  NET_ECONET not used */
 	{ NET_SCTP,		"sctp",		trans_net_sctp_table },

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Ingo Molnar]

* Bob Tracy <rct@frus.com> wrote:

> > Current state of the source tree is the 6f37ac... version, so I'll 
> > start backing out the above diffs in related groups and continue 
> > until I've got a working kernel.  For lack of an obvious target, 
> > I'll start with the seemingly innocuous change to sysctl_check.c.  
> > I'll report back when I've got something.
> 
> That was quick :-).  Backing out the sysctl_check.c diff gives me a 
> working kernel.  Beats the #$%@! out of me how/why, though.
> 
> Michael Cree: could you try backing out the diff below from your 
> 2.6.24-rc3 tree and see if things are now working for you?
> 
> Here's "uname -a", just to confirm (maybe) I'm running on what I say 
> works:
> 
> Linux smirkin 2.6.24-rc2-g6f37ac79-dirty #2 Fri Dec 7 08:03:12 CST 2007 alpha
> 
> Here's the diff I backed out (patch -R).  It's short...
> 
> diff --git a/kernel/sysctl_check.c b/kernel/sysctl_check.c
> index 5a2f2b2..4abc6d2 100644
> --- a/kernel/sysctl_check.c
> +++ b/kernel/sysctl_check.c
> @@ -738,7 +738,7 @@ static struct trans_ctl_table trans_net_table[] = {
>  	{ NET_ROSE,		"rose",		trans_net_rose_table },
>  	{ NET_IPV6,		"ipv6",		trans_net_ipv6_table },
>  	{ NET_X25,		"x25",		trans_net_x25_table },
> -	{ NET_TR,		"tr",		trans_net_tr_table },
> +	{ NET_TR,		"token-ring",	trans_net_tr_table },
>  	{ NET_DECNET,		"decnet",	trans_net_decnet_table },
>  	/*  NET_ECONET not used */
>  	{ NET_SCTP,		"sctp",		trans_net_sctp_table },

reverting this makes the kernel image shorter by 8 bytes - so perhaps 
some alignment issue somewhere? Or something gets overflown? Does any of 
this get actually used by your bootup?

	Ingo

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Ingo Molnar wrote:
> 
> * Bob Tracy <rct@frus.com> wrote:
> 
> > > Current state of the source tree is the 6f37ac... version, so I'll 
> > > start backing out the above diffs in related groups and continue 
> > > until I've got a working kernel.  For lack of an obvious target, 
> > > I'll start with the seemingly innocuous change to sysctl_check.c.  
> > > I'll report back when I've got something.
> > 
> > That was quick :-).  Backing out the sysctl_check.c diff gives me a 
> > working kernel.  Beats the #$%@! out of me how/why, though.
> > 
> > Michael Cree: could you try backing out the diff below from your 
> > 2.6.24-rc3 tree and see if things are now working for you?
> > 
> > Here's "uname -a", just to confirm (maybe) I'm running on what I say 
> > works:
> > 
> > Linux smirkin 2.6.24-rc2-g6f37ac79-dirty #2 Fri Dec 7 08:03:12 CST 2007 alpha
> > 
> > Here's the diff I backed out (patch -R).  It's short...
> > 
> > diff --git a/kernel/sysctl_check.c b/kernel/sysctl_check.c
> > index 5a2f2b2..4abc6d2 100644
> > --- a/kernel/sysctl_check.c
> > +++ b/kernel/sysctl_check.c
> > @@ -738,7 +738,7 @@ static struct trans_ctl_table trans_net_table[] = {
> >  	{ NET_ROSE,		"rose",		trans_net_rose_table },
> >  	{ NET_IPV6,		"ipv6",		trans_net_ipv6_table },
> >  	{ NET_X25,		"x25",		trans_net_x25_table },
> > -	{ NET_TR,		"tr",		trans_net_tr_table },
> > +	{ NET_TR,		"token-ring",	trans_net_tr_table },
> >  	{ NET_DECNET,		"decnet",	trans_net_decnet_table },
> >  	/*  NET_ECONET not used */
> >  	{ NET_SCTP,		"sctp",		trans_net_sctp_table },
> 
> reverting this makes the kernel image shorter by 8 bytes - so perhaps 
> some alignment issue somewhere? Or something gets overflown? Does any of 
> this get actually used by your bootup?

Dunno...  The dmesg output is not terribly useful here, because most of
the "interesting" stuff concerning udev startup that appears on the
console never makes it into a log.  Note that, for the bad cases, I
don't see the same console output that Michael reported, although the
net effect is the same: the partitions don't get found, so I'm offered
the chance to enter my root password and do some poking around, and
when I do, none of the block devices are present under /dev.

I'm open to suggestions on how to take this analysis further.  Michael
indicated he's running a conference this week, so I don't know when he'll
be able to come up for air.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Ingo Molnar]

* Bob Tracy <rct@frus.com> wrote:

> Ingo Molnar wrote:
> > 
> > * Bob Tracy <rct@frus.com> wrote:
> > 
> > > > Current state of the source tree is the 6f37ac... version, so I'll 
> > > > start backing out the above diffs in related groups and continue 
> > > > until I've got a working kernel.  For lack of an obvious target, 
> > > > I'll start with the seemingly innocuous change to sysctl_check.c.  
> > > > I'll report back when I've got something.
> > > 
> > > That was quick :-).  Backing out the sysctl_check.c diff gives me a 
> > > working kernel.  Beats the #$%@! out of me how/why, though.
> > > 
> > > Michael Cree: could you try backing out the diff below from your 
> > > 2.6.24-rc3 tree and see if things are now working for you?
> > > 
> > > Here's "uname -a", just to confirm (maybe) I'm running on what I say 
> > > works:
> > > 
> > > Linux smirkin 2.6.24-rc2-g6f37ac79-dirty #2 Fri Dec 7 08:03:12 CST 2007 alpha
> > > 
> > > Here's the diff I backed out (patch -R).  It's short...
> > > 
> > > diff --git a/kernel/sysctl_check.c b/kernel/sysctl_check.c
> > > index 5a2f2b2..4abc6d2 100644
> > > --- a/kernel/sysctl_check.c
> > > +++ b/kernel/sysctl_check.c
> > > @@ -738,7 +738,7 @@ static struct trans_ctl_table trans_net_table[] = {
> > >  	{ NET_ROSE,		"rose",		trans_net_rose_table },
> > >  	{ NET_IPV6,		"ipv6",		trans_net_ipv6_table },
> > >  	{ NET_X25,		"x25",		trans_net_x25_table },
> > > -	{ NET_TR,		"tr",		trans_net_tr_table },
> > > +	{ NET_TR,		"token-ring",	trans_net_tr_table },
> > >  	{ NET_DECNET,		"decnet",	trans_net_decnet_table },
> > >  	/*  NET_ECONET not used */
> > >  	{ NET_SCTP,		"sctp",		trans_net_sctp_table },
> > 
> > reverting this makes the kernel image shorter by 8 bytes - so 
> > perhaps some alignment issue somewhere? Or something gets overflown? 
> > Does any of this get actually used by your bootup?
> 
> Dunno...  The dmesg output is not terribly useful here, because most 
> of the "interesting" stuff concerning udev startup that appears on the 
> console never makes it into a log.  Note that, for the bad cases, I 
> don't see the same console output that Michael reported, although the 
> net effect is the same: the partitions don't get found, so I'm offered 
> the chance to enter my root password and do some poking around, and 
> when I do, none of the block devices are present under /dev.
> 
> I'm open to suggestions on how to take this analysis further.  Michael 
> indicated he's running a conference this week, so I don't know when 
> he'll be able to come up for air.

i'm not sure how to do direct debugging on udev, so i can only guess 
about what effect on the kernel side could have caused this. One bad 
hack would be to "probe" udevd's behavior by changing the NET_TR entry 
in various ways:

  "tr" -> "token-ring"         # breaks
  "tr" -> "tr"                 # works
  "tr" -> "token-rin0"         # ?            (1)
  "tr" -> "TR"                 # ?            (2)

the question is, does tweak (1) and tweak (2) work or break?

but it would be a lot more effective i guess to get some udevd expert's 
attention on this ...

	Ingo

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Kay Sievers]

On Fri, 2007-12-07 at 19:06 +0100, Ingo Molnar wrote:
> * Bob Tracy <rct@frus.com> wrote:
> 
> > Ingo Molnar wrote:
> > > 
> > > * Bob Tracy <rct@frus.com> wrote:
> > > 
> > > > > Current state of the source tree is the 6f37ac... version, so I'll 
> > > > > start backing out the above diffs in related groups and continue 
> > > > > until I've got a working kernel.  For lack of an obvious target, 
> > > > > I'll start with the seemingly innocuous change to sysctl_check.c.  
> > > > > I'll report back when I've got something.
> > > > 
> > > > That was quick :-).  Backing out the sysctl_check.c diff gives me a 
> > > > working kernel.  Beats the #$%@! out of me how/why, though.
> > > > 
> > > > Michael Cree: could you try backing out the diff below from your 
> > > > 2.6.24-rc3 tree and see if things are now working for you?
> > > > 
> > > > Here's "uname -a", just to confirm (maybe) I'm running on what I say 
> > > > works:
> > > > 
> > > > Linux smirkin 2.6.24-rc2-g6f37ac79-dirty #2 Fri Dec 7 08:03:12 CST 2007 alpha
> > > > 
> > > > Here's the diff I backed out (patch -R).  It's short...
> > > > 
> > > > diff --git a/kernel/sysctl_check.c b/kernel/sysctl_check.c
> > > > index 5a2f2b2..4abc6d2 100644
> > > > --- a/kernel/sysctl_check.c
> > > > +++ b/kernel/sysctl_check.c
> > > > @@ -738,7 +738,7 @@ static struct trans_ctl_table trans_net_table[] = {
> > > >  	{ NET_ROSE,		"rose",		trans_net_rose_table },
> > > >  	{ NET_IPV6,		"ipv6",		trans_net_ipv6_table },
> > > >  	{ NET_X25,		"x25",		trans_net_x25_table },
> > > > -	{ NET_TR,		"tr",		trans_net_tr_table },
> > > > +	{ NET_TR,		"token-ring",	trans_net_tr_table },
> > > >  	{ NET_DECNET,		"decnet",	trans_net_decnet_table },
> > > >  	/*  NET_ECONET not used */
> > > >  	{ NET_SCTP,		"sctp",		trans_net_sctp_table },
> > > 
> > > reverting this makes the kernel image shorter by 8 bytes - so 
> > > perhaps some alignment issue somewhere? Or something gets overflown? 
> > > Does any of this get actually used by your bootup?
> > 
> > Dunno...  The dmesg output is not terribly useful here, because most 
> > of the "interesting" stuff concerning udev startup that appears on the 
> > console never makes it into a log.  Note that, for the bad cases, I 
> > don't see the same console output that Michael reported, although the 
> > net effect is the same: the partitions don't get found, so I'm offered 
> > the chance to enter my root password and do some poking around, and 
> > when I do, none of the block devices are present under /dev.
> > 
> > I'm open to suggestions on how to take this analysis further.  Michael 
> > indicated he's running a conference this week, so I don't know when 
> > he'll be able to come up for air.
> 
> i'm not sure how to do direct debugging on udev, so i can only guess 
> about what effect on the kernel side could have caused this. One bad 
> hack would be to "probe" udevd's behavior by changing the NET_TR entry 
> in various ways:
> 
>   "tr" -> "token-ring"         # breaks
>   "tr" -> "tr"                 # works
>   "tr" -> "token-rin0"         # ?            (1)
>   "tr" -> "TR"                 # ?            (2)
> 
> the question is, does tweak (1) and tweak (2) work or break?
> 
> but it would be a lot more effective i guess to get some udevd expert's 
> attention on this ...

Could we get the output of:
  ls -l /sys/block/sda/
and:
  grep . /sys/block/sda/*/dev
?

Kay

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Kay Sievers wrote:
> On Fri, 2007-12-07 at 19:06 +0100, Ingo Molnar wrote:
> > i'm not sure how to do direct debugging on udev, so i can only guess 
> > about what effect on the kernel side could have caused this. One bad 
> > hack would be to "probe" udevd's behavior by changing the NET_TR entry 
> > in various ways:
> > 
> >   "tr" -> "token-ring"         # breaks
> >   "tr" -> "tr"                 # works
> >   "tr" -> "token-rin0"         # ?            (1)
> >   "tr" -> "TR"                 # ?            (2)
> > 
> > the question is, does tweak (1) and tweak (2) work or break?
> > 
> > but it would be a lot more effective i guess to get some udevd expert's 
> > attention on this ...
> 
> Could we get the output of:
>   ls -l /sys/block/sda/
> and:
>   grep . /sys/block/sda/*/dev
> ?

Here are the requested items for the 2.6.24-rc2-g6f37ac79-dirty kernel
(the working one with the sysctl_check.c patch reverted):

smirkin:/# ls -l /sys/block/sda
total 0
-r--r--r-- 1 root root 8192 Dec  7 08:36 capability
-r--r--r-- 1 root root 8192 Dec  7 08:36 dev
lrwxrwxrwx 1 root root    0 Dec  7 08:36 device -> ../../devices/pci0000:00/0000:00:14.0/0000:01:09.0/host0/target0:0:0/0:0:0:0
drwxr-xr-x 2 root root    0 Dec  7 08:36 holders
drwxr-xr-x 3 root root    0 Dec  7 08:36 queue
-r--r--r-- 1 root root 8192 Dec  7 08:36 range
-r--r--r-- 1 root root 8192 Dec  7 08:36 removable
drwxr-xr-x 3 root root    0 Dec  7 08:36 sda1
drwxr-xr-x 3 root root    0 Dec  7 08:36 sda2
drwxr-xr-x 3 root root    0 Dec  7 08:36 sda3
drwxr-xr-x 3 root root    0 Dec  7 08:36 sda4
drwxr-xr-x 3 root root    0 Dec  7 08:36 sda5
drwxr-xr-x 3 root root    0 Dec  7 08:36 sda6
drwxr-xr-x 3 root root    0 Dec  7 08:36 sda7
-r--r--r-- 1 root root 8192 Dec  7 08:36 size
drwxr-xr-x 2 root root    0 Dec  7 08:36 slaves
-r--r--r-- 1 root root 8192 Dec  7 08:36 stat
lrwxrwxrwx 1 root root    0 Dec  7 08:36 subsystem -> ../../block
--w------- 1 root root 8192 Dec  7 08:36 uevent
smirkin:/# grep . /sys/block/sda/*/dev
/sys/block/sda/sda1/dev:8:1
/sys/block/sda/sda2/dev:8:2
/sys/block/sda/sda3/dev:8:3
/sys/block/sda/sda4/dev:8:4
/sys/block/sda/sda5/dev:8:5
/sys/block/sda/sda6/dev:8:6
/sys/block/sda/sda7/dev:8:7

Assuming /sys/block even exists for the non-working case, I'll forward
that info in a few hours when I can get home to reboot the machine.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Michael Cree]

Bob Tracy wrote:
> That was quick :-).  Backing out the sysctl_check.c diff gives me a
> working kernel.  Beats the #$%@! out of me how/why, though.
> 
> Michael Cree: could you try backing out the diff below from your
> 2.6.24-rc3 tree and see if things are now working for you?

Yes (conference is now over).  I backed out the sysctl_check patch from 
2.6.24-rc3 and, indeed, got a working kernel.

The working kernel (was probably 2.6.24-rc3 less sysctl_check patch, but 
might have been a 2.6.23 variant) has the following in /sys/block

alpha:~# ls -l /sys/block/
total 0
drwxr-xr-x  5 root root 0 2007-12-08 08:55 fd0
drwxr-xr-x  6 root root 0 2007-12-08 08:55 hde
drwxr-xr-x  5 root root 0 2007-12-08 08:55 hdf
drwxr-xr-x 10 root root 0 2007-12-08 08:55 sda
drwxr-xr-x  9 root root 0 2007-12-08 08:55 sdb
alpha:~# ls -l /sys/block/sda
total 0
-r--r--r-- 1 root root 8192 2007-12-08 08:55 capability
-r--r--r-- 1 root root 8192 2007-12-08 08:55 dev
lrwxrwxrwx 1 root root    0 2007-12-08 08:55 device -> 
../../devices/pci0001:01/0001:01:06.0/host0/target0:0:1/0:0:1:0
drwxr-xr-x 2 root root    0 2007-12-08 08:55 holders
drwxr-xr-x 3 root root    0 2007-12-08 08:55 queue
-r--r--r-- 1 root root 8192 2007-12-08 08:55 range
-r--r--r-- 1 root root 8192 2007-12-08 08:55 removable
drwxr-xr-x 3 root root    0 2007-12-08 08:55 sda1
drwxr-xr-x 3 root root    0 2007-12-08 08:55 sda2
drwxr-xr-x 3 root root    0 2007-12-08 08:55 sda3
drwxr-xr-x 3 root root    0 2007-12-08 08:55 sda4
drwxr-xr-x 3 root root    0 2007-12-08 08:55 sda5
-r--r--r-- 1 root root 8192 2007-12-08 08:55 size
drwxr-xr-x 2 root root    0 2007-12-08 08:55 slaves
-r--r--r-- 1 root root 8192 2007-12-08 08:55 stat
lrwxrwxrwx 1 root root    0 2007-12-08 08:55 subsystem -> ../../block
--w------- 1 root root 8192 2007-12-08 08:55 uevent
alpha:~# grep . /sys/block/sda/*/dev
/sys/block/sda/sda1/dev:8:1
/sys/block/sda/sda2/dev:8:2
/sys/block/sda/sda3/dev:8:3
/sys/block/sda/sda4/dev:8:4
/sys/block/sda/sda5/dev:8:5



The broken kernel (2.6.24-rc3) has the following in /sys/block

alpha:~# ls -l /sys/block/
total 0
drwxr-xr-x  5 root root 0 Dec  8 09:22 fd0
drwxr-xr-x  6 root root 0 Dec  8 09:22 hde
drwxr-xr-x  5 root root 0 Dec  8 09:23 hdf
drwxr-xr-x 10 root root 0 Dec  8 09:22 sda
drwxr-xr-x  9 root root 0 Dec  8 09:23 sdb
alpha:~# ls -l /sys/block/sda
total 0
-r--r--r-- 1 root root 8192 Dec  8 09:22 capability
-r--r--r-- 1 root root 8192 Dec  8 09:22 dev
lrwxrwxrwx 1 root root    0 Dec  8 09:23 device -> 
../../devices/pci0001:01/0001:01:06.0/host0/target0:0:1/0:0:1:0
drwxr-xr-x 2 root root    0 Dec  8 09:22 holders
drwxr-xr-x 3 root root    0 Dec  8 09:22 queue
-r--r--r-- 1 root root 8192 Dec  8 09:22 range
-r--r--r-- 1 root root 8192 Dec  8 09:22 removable
drwxr-xr-x 3 root root    0 Dec  8 09:22 sda1
drwxr-xr-x 3 root root    0 Dec  8 09:22 sda2
drwxr-xr-x 3 root root    0 Dec  8 09:22 sda3
drwxr-xr-x 3 root root    0 Dec  8 09:22 sda4
drwxr-xr-x 3 root root    0 Dec  8 09:22 sda5
-r--r--r-- 1 root root 8192 Dec  8 09:22 size
drwxr-xr-x 2 root root    0 Dec  8 09:22 slaves
-r--r--r-- 1 root root 8192 Dec  8 09:22 stat
lrwxrwxrwx 1 root root    0 Dec  8 09:22 subsystem -> ../../block
--w------- 1 root root 8192 Dec  8 09:22 uevent
alpha:~# grep . /sys/block/sda/*/dev
/sys/block/sda/sda1/dev:8:1
/sys/block/sda/sda2/dev:8:2
/sys/block/sda/sda3/dev:8:3
/sys/block/sda/sda4/dev:8:4
/sys/block/sda/sda5/dev:8:5

I failed to spot any difference (other than the formatting of the dates) 
between the two!

Cheers
Michael.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Kay Sievers]

On Sat, 2007-12-08 at 09:43 +1300, Michael Cree wrote:
> Bob Tracy wrote:
> > That was quick :-).  Backing out the sysctl_check.c diff gives me a
> > working kernel.  Beats the #$%@! out of me how/why, though.
> > 
> > Michael Cree: could you try backing out the diff below from your
> > 2.6.24-rc3 tree and see if things are now working for you?
> 
> Yes (conference is now over).  I backed out the sysctl_check patch from 
> 2.6.24-rc3 and, indeed, got a working kernel.
> 
> The working kernel (was probably 2.6.24-rc3 less sysctl_check patch, but 
> might have been a 2.6.23 variant) has the following in /sys/block

> alpha:~# grep . /sys/block/sda/*/dev
> /sys/block/sda/sda1/dev:8:1
> /sys/block/sda/sda2/dev:8:2
> /sys/block/sda/sda3/dev:8:3
> /sys/block/sda/sda4/dev:8:4
> /sys/block/sda/sda5/dev:8:5
> 
> I failed to spot any difference (other than the formatting of the dates) 
> between the two!

Yeah, that looks all fine.

What distro is that, and what's the udev version?

You are booting your kernel with an initramfs?

Is the udev daemon (still) running while it fails?

If you run /sbin/udevtrigger, do the nodes appear?

Kay

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Kay Sievers wrote:
> Yeah, that looks all fine.
> 
> What distro is that, and what's the udev version?

Mine is Debian Etch, normally with the latest released or -rcX kernel
from kernel.org.  Updates current as of about 18 hours ago.  Udev
package version is 0.105-4.  The RELEASE-NOTES file in /usr/share/doc/udev
says "udev 105".

> You are booting your kernel with an initramfs?

Not in my case: everything I need at boot time is built-in.

> Is the udev daemon (still) running while it fails?
> 
> If you run /sbin/udevtrigger, do the nodes appear?

I can answer the above later when I'm back in front of the machine, but
even in the "not good" case, I still see the following messages from
the /etc/rcS.d/S03udev file:

	Starting the hotplug events dispatcher udevd.
	Synthesizing the initial hotplug events.

This is where udevtrigger gets called, followed by the load_input_modules
and create_dev_makedev functions, then...

	Waiting for /dev to be fully populated.

which is where udevsettle gets called.

None of the above appear to be exiting abnormally for the bad case, but
I'll definitely take a closer look at what MAKEDEV (/dev/MAKEDEV -->
/sbin/MAKEDEV) is doing.  In particular, Debian MAKEDEV is looking at
/proc/devices to decide what to do, so maybe "cat /proc/devices" would
be useful to look at for the broken case.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Kay Sievers wrote:
> Is the udev daemon (still) running while it fails?

Yes.

> If you run /sbin/udevtrigger, do the nodes appear?

No.  Exit status is 0, and there are no errors.  Everything looks
fine under /sys/block, and there doesn't seem to be a problem with
/proc/devices either.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Kay Sievers wrote:
> Is the udev daemon (still) running while it fails?

Yes, and there's something else I forgot to mention that may be
significant...  For the bad case, in addition to udevd, "ps -ef"
shows a "sh -e /lib/udev/net.agent" running with a PPID of 1.  This
process doesn't exit until I reboot.  If this is normal under the
circumstances, please disregard.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Kay Sievers]

On Fri, 2007-12-07 at 23:05 -0600, Bob Tracy wrote:
> Kay Sievers wrote:
> > Is the udev daemon (still) running while it fails?
> 
> Yes, and there's something else I forgot to mention that may be
> significant...  For the bad case, in addition to udevd, "ps -ef"
> shows a "sh -e /lib/udev/net.agent" running with a PPID of 1.  This
> process doesn't exit until I reboot.  If this is normal under the
> circumstances, please disregard.

Does SysRq-T show where it hangs?

Kay

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Michael Cree]

Kay Sievers wrote:
> On Fri, 2007-12-07 at 23:05 -0600, Bob Tracy wrote:
>> Kay Sievers wrote:
>>> Is the udev daemon (still) running while it fails?
>> Yes, and there's something else I forgot to mention that may be
>> significant...  For the bad case, in addition to udevd, "ps -ef"
>> shows a "sh -e /lib/udev/net.agent" running with a PPID of 1.  This
>> process doesn't exit until I reboot.  If this is normal under the
>> circumstances, please disregard.
> 
> Does SysRq-T show where it hangs?

Ummm... No.  I didn't have the CONFIG_MAGIC_SYSRQ flag set, so I set it, 
and recompiled the kernel.  Guess what - now the system comes up 
normally without any problem.  The block devices appear in /dev.  To 
recap: without CONFIG_MAGIC_SYSRQ on the 2.6.24-rc3 kernel the missing 
block devices error in /dev occurs and the init scripts fall over on 
startup, and with CONFIG_MAGIC_SYSRQ the system comes up normally.

To answer the earlier questions about distro, and udev version, my 
system is similar to Bob's, except that I am running Debian 
testing/lenny which comes with udev version 114 (dpkg reports udev 
version 0.114-2).  I am running an EV67 variant CPU.

I do not run an initramfs - I have the necessary drivers for the various 
discs compiled into the kernel and use the root kernel option to point 
to the required root partition.

When running the broken kernel udev is running (according to 'ps') and 
executing /sbin/udevtrigger manually generates a number of errors of the 
form:

scsi_id[<pid>]: scsi_id: unable to access '/block'

The missing /dev/* entries do not appear.

Cheerz
Michael.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Michael Cree wrote:
> Kay Sievers wrote:
> > On Fri, 2007-12-07 at 23:05 -0600, Bob Tracy wrote:
> >> Kay Sievers wrote:
> >>> Is the udev daemon (still) running while it fails?
> >> Yes, and there's something else I forgot to mention that may be
> >> significant...  For the bad case, in addition to udevd, "ps -ef"
> >> shows a "sh -e /lib/udev/net.agent" running with a PPID of 1.  This
> >> process doesn't exit until I reboot.  If this is normal under the
> >> circumstances, please disregard.
> > 
> > Does SysRq-T show where it hangs?
> 
> Ummm... No.  I didn't have the CONFIG_MAGIC_SYSRQ flag set, so I set it, 
> and recompiled the kernel.  Guess what - now the system comes up 
> normally without any problem.  The block devices appear in /dev.  To 
> recap: without CONFIG_MAGIC_SYSRQ on the 2.6.24-rc3 kernel the missing 
> block devices error in /dev occurs and the init scripts fall over on 
> startup, and with CONFIG_MAGIC_SYSRQ the system comes up normally.

I *do* have CONFIG_MAGIC_SYSRQ set.  Anyone care to bet whether my
machine starts working again if I disable it?  Sheesh...  The "kernel
alignment issue" theory is making sense...  We change the size of an
initialized variable with the patch, and the problem shows up.  We
shift starting addresses a different way by tweaking kernel options,
and two wrongs make a right?  I've seen it happen, and tracking this
down isn't going to be easy.  Anyone want to wade through the different
System.map files and hazard a guess where we're leaving the rails?

Here's a very brief diff excerpt between the System.map files corresponding
to "sysctl_check patch reverted" (the -dirty version) and "with sysctl_check patch".
At least they agree up to line 10870 :-) ...

--- /boot/System.map-2.6.24-rc2-g6f37ac79-dirty 2007-12-07 08:03:50.000000000 -0
600
+++ System.map  2007-12-07 13:43:37.000000000 -0600
@@ -10868,9414 +10868,9414 @@
 fffffc0000684b00 R kallsyms_markers
 fffffc0000684d00 R kallsyms_token_table
 fffffc0000685100 R kallsyms_token_index
-fffffc00006f61e0 r __pci_fixup_PCI_VENDOR_ID_SERVERWORKSPCI_DEVICE_ID_SERVERWORKS_CSB5IDEquirk_svwks_csb5ide
-fffffc00006f61e0 R __start_pci_fixups_early
-fffffc00006f61f0 r __pci_fixup_PCI_VENDOR_ID_INTELPCI_DEVICE_ID_INTEL_82801CA_10quirk_ide_samemode
(...)
-fffffc0000716120 r __param_bic_scale
-fffffc0000716148 r __param_tcp_friendliness
-fffffc0000716170 R __end_rodata
-fffffc0000716170 R __stop___param
+fffffc00006f61f0 r __pci_fixup_PCI_VENDOR_ID_SERVERWORKSPCI_DEVICE_ID_SERVERWORKS_CSB5IDEquirk_svwks_csb5ide
+fffffc00006f61f0 R __start_pci_fixups_early
+fffffc00006f6200 r __pci_fixup_PCI_VENDOR_ID_INTELPCI_DEVICE_ID_INTEL_82801CA_10quirk_ide_samemode
(...)
+fffffc0000716130 r __param_bic_scale
+fffffc0000716158 r __param_tcp_friendliness
+fffffc0000716180 R __end_rodata
+fffffc0000716180 R __stop___param
 fffffc0000718000 A __init_begin
 fffffc0000718000 T _sinittext
 fffffc0000718000 t set_reset_devices

> When running the broken kernel udev is running (according to 'ps') and 
> executing /sbin/udevtrigger manually generates a number of errors of the 
> form:
> 
> scsi_id[<pid>]: scsi_id: unable to access '/block'
> 
> The missing /dev/* entries do not appear.

I don't get the errors that Michael is seeing, and udevtrigger seems to
be exiting without errors (return code 0).  The last part is the same:
the missing /dev/* entries do not appear.

--Bob T.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Ivan Kokshaysky]

On Sat, Dec 08, 2007 at 10:19:39PM -0600, Bob Tracy wrote:
> I *do* have CONFIG_MAGIC_SYSRQ set.  Anyone care to bet whether my
> machine starts working again if I disable it?  Sheesh...

Incredible...

Toggling CONFIG_MAGIC_SYSRQ works for me too, so I'm finally able
to reproduce the problem (which is the main positive result so far ;-)

There are lots of possible reasons why this happens, but at the
moment I honestly have no idea.
For now I have reassigned the bug #9457 to myself and will gradually hack
into udev...

Ivan.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Ivan Kokshaysky wrote:
> On Sat, Dec 08, 2007 at 10:19:39PM -0600, Bob Tracy wrote:
> > I *do* have CONFIG_MAGIC_SYSRQ set.  Anyone care to bet whether my
> > machine starts working again if I disable it?  Sheesh...
> 
> Incredible...
> 
> Toggling CONFIG_MAGIC_SYSRQ works for me too, so I'm finally able
> to reproduce the problem (which is the main positive result so far ;-)
> 
> There are lots of possible reasons why this happens, but at the
> moment I honestly have no idea.
> For now I have reassigned the bug #9457 to myself and will gradually hack
> into udev...

Thanks...  Let me know if there's anything useful I can do to help.

--Bob T.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Ivan Kokshaysky]

On Mon, Dec 10, 2007 at 09:08:53AM -0600, Bob Tracy wrote:
> Ivan Kokshaysky wrote:
> > For now I have reassigned the bug #9457 to myself and will gradually hack
> > into udev...
> 
> Thanks...  Let me know if there's anything useful I can do to help.

It turns out to be yet another strncpy() bug that indeed shows up only with
certain src/dst alignments and breaks kobject_get_path(). Ugh...

Hopefully I'll have a patch tomorrow.

Ivan.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

Kay Sievers wrote:
> On Fri, 2007-12-07 at 23:05 -0600, Bob Tracy wrote:
> > Kay Sievers wrote:
> > > Is the udev daemon (still) running while it fails?
> > 
> > Yes, and there's something else I forgot to mention that may be
> > significant...  For the bad case, in addition to udevd, "ps -ef"
> > shows a "sh -e /lib/udev/net.agent" running with a PPID of 1.  This
> > process doesn't exit until I reboot.  If this is normal under the
> > circumstances, please disregard.
> 
> Does SysRq-T show where it hangs?

A quick comparison of the trace sections for udevd and net.agent indicates
those traces are identical: none of the function names in the traces appear
to be what you might be looking for, i.e., the processes appear to have been
waiting for an event of some kind, and woke up long enough to process the
SysRq-T keyboard interrupt and the corresponding action.

Hmm...  Ok...  The state information itself is probably more useful in
this context.  Here's the info for net.agent:

net.agent      S      fffffc000032c37c   0    895    1
  fffffc0000743b10  0000000000000010  fffffc00004f3b5c  fffffc00007601a8
    0000000000000001  0000000000000074  fffffc0000747758  fffffc00230f0000
    0000000000000007  0000000000000007  fffffc00004f390c  0000000000000010
    fffffc00004e9eb8  fffffc00230f0000  0000000000000014  fffffc0023085140
    0000000000000001  0000000000000014  fffffc00001de000  0000000000000001
    fffffc000055dcfc  fffffc0023085140  fffffc00232788c0  0000000000000001

Addresses of presumed interest from System.map:

fffffc000032c000 t do_wait
fffffc0000740000 D init_thread_union
fffffc00004f3b40 t sysrq_handle_showstate
fffffc00007601a8 d sysrq_showstate_op
fffffc0000747758 D console_printk
fffffc002...  not in System.map (module?) -- I'll track this down later
   if needed.
fffffc00004f3850 T __handle_sysrq
fffffc00004e9850 t kbd_event
fffffc00001...  not in System.map (?? begins with fffffc0000300000 A swapper_pg_dir)
fffffc000055dc30 t input_pass_event

--Bob T.

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Ingo Molnar]

* Bob Tracy <rct@frus.com> wrote:

> > I'm struggling to see how any of those could have broken block 
> > device mounting on alpha.  Are you sure you bisected right?
> 
> Based on what's in that commit, it *does* appear something went wrong 
> with bisection.  If the implicated commit is the next one in time 
> sequence relative to
> 
> # good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better
> 
> then the test of whether I bisected correctly is as simple as applying 
> the commit and seeing if things break, because I'm running on the 
> kernel corresponding to 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 right 
> now.  Let me give that a try and I'll report back.  Worst case, I'll 
> have to start over and write off the past four days...

generally it's easier to just go "back in time" and re-try the last 
known "good" and last-known "bad" commit IDs to establish that they are 
indeed correctly identified. if they are not then step back one more in 
the bisection log. No need to spend another 4 days on this, if most of 
the bisection is OK. You can replay a corrected git bisection log 
quickly, by doing:

  git-bisect reset
  git-bisect < bisect.log

	Ingo

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Bob Tracy]

I wrote:
> If the implicated commit is the next one in time
> sequence relative to
> 
> # good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better
> 
> then the test of whether I bisected correctly is as simple as applying
> the commit and seeing if things break, because I'm running on the
> kernel corresponding to 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3 right
> now.  Let me give that a try and I'll report back.

Verified that 6f37ac793d6ba7b35d338f791974166f67fdd9ba is the next
commit after the "good" kernel I'm running now.  The build is running,
and I should have an answer for us in a few hours.

-- 
------------------------------------------------------------------------
Bob Tracy          |  "They couldn't hit an elephant at this dist- "
rct@frus.com       |   - Last words of Union General John Sedgwick,
                   |  Battle of Spotsylvania Court House, U.S. Civil War
------------------------------------------------------------------------

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Ingo Molnar]

* Andrew Morton <akpm@linux-foundation.org> wrote:

> > # bad: [6f37ac793d6ba7b35d338f791974166f67fdd9ba] Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6
> > git-bisect bad 6f37ac793d6ba7b35d338f791974166f67fdd9ba
> > # good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better
> > git-bisect good 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3

> I'm struggling to see how any of those could have broken block device 
> mounting on alpha.  Are you sure you bisected right?

the bisection log looks healthy so far - with nicely alternating 
good/bad bisection points. Barring the possibility that the bug is 
non-deterministic, i'd guess the bisection points are OK, at least 
judging from their statistical properties.

but ... i went over the diffs too, and i fail to see how they could 
affect the bootup path of an Alpha box, which i suspect has no 
networking dependency up to the failure point.

	Ingo

Re: [BUG] 2.6.23-rc3 can't see sd partitions on Alpha

[Rafael J. Wysocki]

On Friday, 7 of December 2007, Bob Tracy wrote:
> OK.  Finally have this thing painted into a corner: git has identified
> 6f37ac793d6ba7b35d338f791974166f67fdd9ba as the first bad commit.
> 
> From "git bisect log", this corresponds to 
> 
> # bad: [6f37ac793d6ba7b35d338f791974166f67fdd9ba] Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6

Something's gone wrong, as this commit doesn't modify code.

> Here's the full log:
> 
> git-bisect start
> # good: [9aae299f7fd1888ea3a195cfe0edef17bb647415] Linux 2.6.24-rc2
> git-bisect good 9aae299f7fd1888ea3a195cfe0edef17bb647415
> # bad: [f05092637dc0d9a3f2249c9b283b973e6e96b7d2] Linux 2.6.24-rc3
> git-bisect bad f05092637dc0d9a3f2249c9b283b973e6e96b7d2
> # good: [e6a5c27f3b0fef72e528fc35e343af4b2db790ff] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm
> git-bisect good e6a5c27f3b0fef72e528fc35e343af4b2db790ff
> # good: [42614fcde7bfdcbe43a7b17035c167dfebc354dd] vmstat: fix section mismatch warning
> git-bisect good 42614fcde7bfdcbe43a7b17035c167dfebc354dd
> # bad: [a052f4473603765eb6b4c19754689977601dc1d1] Merge git://git.kernel.org/pub/scm/linux/kernel/git/sam/x86
> git-bisect bad a052f4473603765eb6b4c19754689977601dc1d1
> # good: [d8e5219f9f5ca7518eb820db9f3d287a1d46fcf5] CRISv10 improve and bugfix fasttimer
> git-bisect good d8e5219f9f5ca7518eb820db9f3d287a1d46fcf5
> # good: [d90bf5a976793edfa88d3bb2393f0231eb8ce1e5] [NET]: rt_check_expire() can take a long time, add a cond_resched()
> git-bisect good d90bf5a976793edfa88d3bb2393f0231eb8ce1e5
> # good: [2a113281f5cd2febbab21a93c8943f8d3eece4d3] kconfig: use $K64BIT to set 64BIT with all*config targets
> git-bisect good 2a113281f5cd2febbab21a93c8943f8d3eece4d3
> # good: [2e2cd8bad6e03ceea73495ee6d557044213d95de] CRISv10 memset library add lineendings to asm
> git-bisect good 2e2cd8bad6e03ceea73495ee6d557044213d95de
> # bad: [6f37ac793d6ba7b35d338f791974166f67fdd9ba] Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6
> git-bisect bad 6f37ac793d6ba7b35d338f791974166f67fdd9ba
> # good: [2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3] CRISv10 fasttimer: Scrap INLINE and name timeval_cmp better
> git-bisect good 2f1f53bdc6531696934f6ee7bbdfa2ab4f4f62a3