Re: linux-3.4 broken on chardonnay and huxelrebe (Re: [linux-3.4 test] 61301: regressions - FAIL)

[Ian Campbell <ian.campbell@citrix.com>]

On Wed, 2015-09-09 at 10:44 +0100, Ian Campbell wrote:
> On Fri, 2015-09-04 at 08:58 +0000, osstest service owner wrote:
> > flight 61301 linux-3.4 real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/61301/
> > 
> > Regressions :-(
> > 
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  test-amd64-amd64-xl-qemut-win7-amd64  6 xen-boot          fail REGR.
> > vs. 30511
> 
> linux-3.4 seems to have trouble booting on chardonnay and huxelrebe,

I ran some adhoc tests on some jobs selected because I could see they
passed on these hosts (or their partners) on later Linux versions:

Tag	Revision					Flight	huxelrebe0	chardonnay1
										^test-amd64-amd64-libvirt
								^test-amd64-amd64-xl

v3.4	76e10d158efb6d4516018846f60c2ab5501900bc	61710	FAIL		PASS
v3.5	28a33cbc24e4256c143dce96c7d93bf423229f92	61713	PASS		FAIL
v3.6	a0d271cbfed1dd50278c6b06bead3d00ba0a88f9	61714	PASS		PASS
v3.7	29594404d7fe73cd80eaa4ee8c43dcc53970c60e	61715	PASS		PASS
v3.8	19f949f52599ba7c3f67a5897ac6be14bfcb1200	61716	PASS		FAIL
v3.9	c1be5a5b1b355d40e6cf79cc979eb66dafa24ad1	61717	PASS		PASS
v3.10	8bb495e3f02401ee6f76d1b1d77f3ac9f079e376	61703	PASS		PASS

(unfortunately due to a configuration issue I have no logs from these, just
the overall result :-(. Luckily I think for now they aren't necessary for
the analysis)

The Chardonnay case suggests that either something has been backported into
3.4.x which has broken things (current real flights, which reliably fail,
are running on 3.4.108) or that it is simply unreliable (or both). I think
I need to repeat things a few times to confirm.

The Huxelrebe case is apparently more obvious, I noticed that the Ethernet
device from [0] us a "Intel Corporation I210 Gigabit Network Connection"
which the Debian kernel detects (in [1]) as:
Aug 18 01:38:56.085002 [    2.386215] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
Aug 18 01:38:56.133067 [    2.393591] igb 0000:03:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 44:39:c4:6d:20:60
Aug 18 01:38:56.141009 [    2.401248] igb 0000:03:00.0: eth1: PBA No: 000300-000

But which 3.4 doesn't pick up at all in [2]. In the logs between 3.4 and
3.5 I see:

commit f96a8a0b78548c0ec06b0b4b438db6ee895d67e9
Author: Carolyn Wyborny <carolyn.wyborny@intel.com>
Date:   Fri Apr 6 23:25:19 2012 +0000

    igb: Add Support for new i210/i211 devices.
    
    This patch adds new initialization functions and device support
    for i210 and i211 devices.
    
    Signed-off-by: Carolyn Wyborny <carolyn.wyborny@intel.com>
    Tested-by: Jeff Pieper <jeffrey.e.pieper@intel.com>
    Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>

Which seems pretty straight forwards.

These systems actually have a pair of these devices (the other is
 0000:02:00.0/eth0) and another which is detected by Debian as:

Aug 18 01:38:56.149025 [    2.403785] e1000e 0000:00:19.0: eth2: (PCI Express:2.5GT/s:Width x1) 44:39:c4:39:a1:19
Aug 18 01:38:56.157078 [    2.403787] e1000e 0000:00:19.0: eth2: Intel(R) PRO/1000 Network Connection
Aug 18 01:38:56.165003 [    2.403862] e1000e 0000:00:19.0: eth2: MAC: 11, PHY: 12, PBA No: FFFFFF-0FF

But which linux-3.4 also doesn't see.

I've not confirmed whether f96a8a0b78 is the only required patch and it is
not entirely trivial so I don't know if it would be considered a valid
backport. We could perhaps consider arranging that these machines are only
used for newer kernels somehow?

Ian.

[0] http://logs.test-lab.xenproject.org/osstest/logs/60711/test-amd64-amd64-xl/huxelrebe1-output-lspci_-tv
[1] http://logs.test-lab.xenproject.org/osstest/logs/60711/test-amd64-amd64-xl/serial-huxelrebe1.log
[2] http://logs.test-lab.xenproject.org/osstest/logs/61301/test-amd64-amd64-xl/serial-huxelrebe0.log