3.0 -> 3.10 regression? poweroff instead of reboot?

3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

Hi,

after installing new server, we've noticed weird behaviour: when box is
about to reboot, it powers off instead. This was with 3.10.33, downgrading
to 3.0.101 seems to fix this problem. I didn't have time to play with it today
and box had to be shipped today, so it'll take few days before I'll have chance
for further research, but I wanted to ask whether somebody has met similar
problem and what I should check first?

The system is x86_64 centos 6, HW is supermicro x9sci-ln4f.

Any help appreciated

When I have chance to put my hands on this box again, I'll try
latest kernel and I can also do bisect, if needed.

BR

nik




-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 1880 bytes --]

(CC Tomas Winkler)

So finally I got some time to look deeper into this, and the problem
might not be regression after all.. It is caused by mei_me module,
blacklisting it "fixes it" for me.

So for me it's kinda of solution, although I guess MEI related people
might be interested in it..

If I could be of any help, please let me know - at least I've got hardware
on which I can reproduce the problem..

BR

nik


On Mon, Mar 10, 2014 at 03:55:30PM +0100, Nikola Ciprich wrote:
> Hi,
> 
> after installing new server, we've noticed weird behaviour: when box is
> about to reboot, it powers off instead. This was with 3.10.33, downgrading
> to 3.0.101 seems to fix this problem. I didn't have time to play with it today
> and box had to be shipped today, so it'll take few days before I'll have chance
> for further research, but I wanted to ask whether somebody has met similar
> problem and what I should check first?
> 
> The system is x86_64 centos 6, HW is supermicro x9sci-ln4f.
> 
> Any help appreciated
> 
> When I have chance to put my hands on this box again, I'll try
> latest kernel and I can also do bisect, if needed.
> 
> BR
> 
> nik
> 
> 
> 
> 
> -- 
> -------------------------------------
> Ing. Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28.rijna 168, 709 00 Ostrava
> 
> tel.:   +420 591 166 214
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: servis@linuxbox.cz
> -------------------------------------



-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

RE: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Winkler, Tomas]

> (CC Tomas Winkler)
> 
> So finally I got some time to look deeper into this, and the problem
> might not be regression after all.. It is caused by mei_me module,
> blacklisting it "fixes it" for me.
> 
> So for me it's kinda of solution, although I guess MEI related people
> might be interested in it..
> 
> If I could be of any help, please let me know - at least I've got hardware
> on which I can reproduce the problem..
> 
> BR
> 
> nik
> 
> 
> On Mon, Mar 10, 2014 at 03:55:30PM +0100, Nikola Ciprich wrote:
> > Hi,
> >
> > after installing new server, we've noticed weird behaviour: when box is
> > about to reboot, it powers off instead. This was with 3.10.33, downgrading
> > to 3.0.101 seems to fix this problem. I didn't have time to play with it today
> > and box had to be shipped today, so it'll take few days before I'll have chance
> > for further research, but I wanted to ask whether somebody has met similar
> > problem and what I should check first?
> >
> > The system is x86_64 centos 6, HW is supermicro x9sci-ln4f.
> >
> > Any help appreciated
> >
> > When I have chance to put my hands on this box again, I'll try
> > latest kernel and I can also do bisect, if needed.
> >
> > BR
> >
> > nik

If this is a server platform can you please try this patch https://lkml.org/lkml/2014/3/25/396

Thanks
Tomas

Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

Hi Tomas,

> If this is a server platform can you please try this patch https://lkml.org/lkml/2014/3/25/396

tried 3.10.35 with Your patch and now I can see following in dmesg:

[  +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
[  +0.007656] mei_me 0000:00:16.0: initialization failed.

now reboot works as expected. Once this hits ustream (if it hasn't already),
this could go to -stable, do You agree?

thanks!

nik



> 
> Thanks
> Tomas
> 
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

RE: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Winkler, Tomas]

> 
> Hi Tomas,
> 
> > If this is a server platform can you please try this patch
> https://lkml.org/lkml/2014/3/25/396
> 
> tried 3.10.35 with Your patch and now I can see following in dmesg:
> 
> [  +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
> [  +0.007656] mei_me 0000:00:16.0: initialization failed.
> 
> now reboot works as expected. Once this hits ustream (if it hasn't already),
> this could go to -stable, do You agree?
> 
> thanks!
> 
I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. 
Tomas

Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

Hi,

sure! count me in.

n.


On Mon, Mar 31, 2014 at 12:05:20PM +0000, Winkler, Tomas wrote:
> 
> > 
> > Hi Tomas,
> > 
> > > If this is a server platform can you please try this patch
> > https://lkml.org/lkml/2014/3/25/396
> > 
> > tried 3.10.35 with Your patch and now I can see following in dmesg:
> > 
> > [  +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
> > [  +0.007656] mei_me 0000:00:16.0: initialization failed.
> > 
> > now reboot works as expected. Once this hits ustream (if it hasn't already),
> > this could go to -stable, do You agree?
> > 
> > thanks!
> > 
> I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. 
> Tomas
> 
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

Hi Tomas,

bad news, we just had failed boot with 3.10.35 with your patch:

last line was:

mei_me 0000:00:16.0: wait hw ready failed. status = -110

so it can probably happen that boot hangs on mei_me init attempt..

colleague just told me it also happened to him with 3.13 kernel,
however I don't have more info on that case..

if I can provide more information, please let me know.

BR

nik

On Mon, Mar 31, 2014 at 12:05:20PM +0000, Winkler, Tomas wrote:
> 
> > 
> > Hi Tomas,
> > 
> > > If this is a server platform can you please try this patch
> > https://lkml.org/lkml/2014/3/25/396
> > 
> > tried 3.10.35 with Your patch and now I can see following in dmesg:
> > 
> > [  +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
> > [  +0.007656] mei_me 0000:00:16.0: initialization failed.
> > 
> > now reboot works as expected. Once this hits ustream (if it hasn't already),
> > this could go to -stable, do You agree?
> > 
> > thanks!
> > 
> I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. 
> Tomas
> 
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

RE: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Winkler, Tomas]

> 
> Hi Tomas,
> 
> bad news, we just had failed boot with 3.10.35 with your patch:
> 
> last line was:
> 
> mei_me 0000:00:16.0: wait hw ready failed. status = -110
> 
> so it can probably happen that boot hangs on mei_me init attempt..
> 
> colleague just told me it also happened to him with 3.13 kernel,
> however I don't have more info on that case..
> 
> if I can provide more information, please let me know.
>
I would need the devices ids of the MEI device on your platform (lspci  will do the job), is this same platform as you tested before? 
You should not get to requesting the HW readiness w/ my patch. 
Thanks
Tomas
 
-----------------------------

Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

Hello Tomas,
> I would need the devices ids of the MEI device on your platform (lspci  will do the job), is this same platform as you tested before? 
> You should not get to requesting the HW readiness w/ my patch. 

yes, it's the same box.

here are the IDs:

00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #2 (rev 04)

00:16.0 0780: 8086:1c3a (rev 04)
00:16.1 0780: 8086:1c3b (rev 04)

nik



> Thanks
> Tomas
>  
> -----------------------------
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

RE: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Winkler, Tomas]

> 
> Hello Tomas,
> > I would need the devices ids of the MEI device on your platform (lspci  will do
> the job), is this same platform as you tested before?
> > You should not get to requesting the HW readiness w/ my patch.
> 
> yes, it's the same box.
> 
> here are the IDs:
> 
> 00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series
> Chipset Family MEI Controller #1 (rev 04)
> 00:16.1 Communication controller: Intel Corporation 6 Series/C200 Series
> Chipset Family MEI Controller #2 (rev 04)
> 
> 00:16.0 0780: 8086:1c3a (rev 04)
> 00:16.1 0780: 8086:1c3b (rev 04)

Are use you have the patch applied, because the driver just should not load. 
If any case  you dump the firmware status registers"
At offset 0x40 and 0x48 in PCI config space 
sudo hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0
sudo hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0

Thanks
Tomas

Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

Hi Tomas,

yes, I just checked source rpm and it's applied..

here are the dumps:

[root@k310test ~]# hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0
0000048 a301 3900                              
000004c
[root@k310test ~]# hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0
0000040 0345 000f                              
0000044

BR

nik


> Are use you have the patch applied, because the driver just should not load. 
> If any case  you dump the firmware status registers"
> At offset 0x40 and 0x48 in PCI config space 
> sudo hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0
> sudo hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0
> 
> Thanks
> Tomas
> 
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

RE: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Winkler, Tomas]

> -----Original Message-----
> From: Nikola Ciprich [mailto:nikola.ciprich@linuxbox.cz]
> Sent: Monday, April 07, 2014 16:26
> To: Winkler, Tomas
> Cc: Greg KH (gregkh@linuxfoundation.org); linux-kernel@vger.kernel.org; Nikola
> Ciprich
> Subject: Re: 3.0 -> 3.10 regression? poweroff instead of reboot?
> 
> Hi Tomas,
> 
> yes, I just checked source rpm and it's applied..
> 
> here are the dumps:
> 
> [root@k310test ~]# hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0
> 0000048 a301 3900
> 000004c
> [root@k310test ~]# hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0
> 0000040 0345 000f
> 0000044

I'm not sure how this is possible that this is failing for you

if ((0xa301 & 0x600) == 0x200)
	goto no_mei; 
so this is okay, the driver should not bind

How the patch worked for you before this test?  

Sorry for asking but can you please double check your compilation setup. 
 
Thanks
Tomas

Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]

Hello Tomas,

> I'm not sure how this is possible that this is failing for you
> 
> if ((0xa301 & 0x600) == 0x200)
> 	goto no_mei; 
> so this is okay, the driver should not bind
> 
> How the patch worked for you before this test?  
> 
> Sorry for asking but can you please double check your compilation setup. 

You're right of course :-(

I tried rebuilding rpm again now, and patch spits following:

Hunk #1 succeeded at 105 (offset -2 lines).
patch unexpectedly ends in middle of line

it's bad rpm build didn't abort at this point, but apparently
this is patch's fault:

[root@dev6 linux-3.10]# cat /tmp/mei/1 | patch -sp1
patch unexpectedly ends in middle of line
[root@dev6 linux-3.10]# echo $?
0

:-(

well, this sucks, I really wouldn't expect that kind of bug in such widely
used program as patch :(

I'll report that.

(btw this error was caused by one whitespace character at last line of patch)

anyways, You can certainly forgot about that, it was my fault :(

sorry about that

nik


>  
> Thanks
> Tomas
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

RE: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Winkler, Tomas]

> -----Original Message-----
> From: Nikola Ciprich [mailto:nikola.ciprich@linuxbox.cz]
> Sent: Tuesday, April 08, 2014 09:50
> To: Winkler, Tomas
> Cc: Greg KH (gregkh@linuxfoundation.org); linux-kernel@vger.kernel.org; Nikola
> Ciprich
> Subject: Re: 3.0 -> 3.10 regression? poweroff instead of reboot?
> > Sorry for asking but can you please double check your compilation setup.
> 
> You're right of course :-(
> 
> I tried rebuilding rpm again now, and patch spits following:
> 
> Hunk #1 succeeded at 105 (offset -2 lines).
> patch unexpectedly ends in middle of line
> 
> it's bad rpm build didn't abort at this point, but apparently
> this is patch's fault:
> 
> [root@dev6 linux-3.10]# cat /tmp/mei/1 | patch -sp1
> patch unexpectedly ends in middle of line
> [root@dev6 linux-3.10]# echo $?
> 0
> 
> :-(
> 
> well, this sucks, I really wouldn't expect that kind of bug in such widely
> used program as patch :(
>
> I'll report that.


> (btw this error was caused by one whitespace character at last line of patch)

This is really strange, can it be that your mailer has introduced some blanks into the patch,
I've checked that patch applies to all relevant stable kernels.
 
> anyways, You can certainly forgot about that, it was my fault :(
> 
> sorry about that

No harm done, it's good this was sorted out.
Thanks
Tomas
 

Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

> This is really strange, can it be that your mailer has introduced some blanks into the patch,
> I've checked that patch applies to all relevant stable kernels.

I had it downloaded from link You've sent:
https://lkml.org/lkml/2014/3/25/396

using 

wget https://lkml.org/lkml/diff/2014/3/25/396/1

I get two whitespaces on the last line..

n.


>  
> > anyways, You can certainly forgot about that, it was my fault :(
> > 
> > sorry about that
> 
> No harm done, it's good this was sorted out.
> Thanks
> Tomas
>  
> 
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]