Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[jordan]

Hey list,

I've been meaning to post, but alas, I am working three jobs right now
(just took on more work) and haven't had a whole lot of free time to
investigate my boot problems on linux-rt (-rt9, -rt10, -rt11, -rt13).

On 3.12.8-rt11 - i was able to get the kernel to boot via reverting two
patches, from the split queue;

'timer-Raise-softirq-if-there-s-irq_work.patch'
'timers-do-not-raise-softirq-unconditionally.patch'

(i haven't tried reverting on -rt13, as i haven't had the time to do that).

however, while this was enough to boot - my system was unstable. thus i
downgraded my kernel back to trusty 3.12.5-rt7 - which works flawlessly for
me. Next, I saw the -rt13 release, thinking maybe the boot issue was
finally solved [according to the change log], but no - it fails just as
before... I tried to add debug to grub's commandline but in the end, it
wasn't really helpful. [the last thing shown is something about PnP and
then ACPI];

[    0.255329] pnp: PnP ACPI: found 10 devices
[    0.255330] ACPI: bus type PNP unregistered

(which i would normally see anyway.)

I might have some time over the weekend (hopefully), to take a closer look
- but i  thought i would at least notify the list ~ as i saw another user
have the same booting problems and he has the same CPU as i do; AMD
Phenom 965 X4 [not exactly exotic H/W and every RT release has worked
great on my setup, until 3.12.x-rt8. Now, nothing works properly.
[note: vanilla 3.12.x kernels boot just fine].

anyway, if anyone has some suggestions - that i could follow up on over the
weekend that would be fantastic.

cheers

Jordan

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[Sebastian Andrzej Siewior]

* jordan | 2014-02-05 20:54:59 [-0500]:

>Hey list,
Hi jordan,

>I've been meaning to post, but alas, I am working three jobs right now
>(just took on more work) and haven't had a whole lot of free time to
>investigate my boot problems on linux-rt (-rt9, -rt10, -rt11, -rt13).
>
>On 3.12.8-rt11 - i was able to get the kernel to boot via reverting two
>patches, from the split queue;
>
>'timer-Raise-softirq-if-there-s-irq_work.patch'
>'timers-do-not-raise-softirq-unconditionally.patch'

It was reported to the list that
timers-do-not-raise-softirq-unconditionally.patch sometimes stalled the
machine. The timer-Raise-softirq-if-there-s-irq_work.patch should have
fixes one common case and -rt13 should have coverted them all.

>(i haven't tried reverting on -rt13, as i haven't had the time to do that).

It should work on -rt13 without reverting them.

>however, while this was enough to boot - my system was unstable. thus i
>downgraded my kernel back to trusty 3.12.5-rt7 - which works flawlessly for
>me. Next, I saw the -rt13 release, thinking maybe the boot issue was
>finally solved [according to the change log], but no - it fails just as
>before... I tried to add debug to grub's commandline but in the end, it
>wasn't really helpful. [the last thing shown is something about PnP and
>then ACPI];
>
>[    0.255329] pnp: PnP ACPI: found 10 devices
>[    0.255330] ACPI: bus type PNP unregistered
>
>(which i would normally see anyway.)

this looks like a missing event (like timer interrupt or RCU wakeup and
"timers-do-not-raise-softi…" might be a source of this but then it
should be fixed).

>I might have some time over the weekend (hopefully), to take a closer look
>- but i  thought i would at least notify the list ~ as i saw another user
>have the same booting problems and he has the same CPU as i do; AMD
>Phenom 965 X4 [not exactly exotic H/W and every RT release has worked
>great on my setup, until 3.12.x-rt8. Now, nothing works properly.
>[note: vanilla 3.12.x kernels boot just fine].
>
>anyway, if anyone has some suggestions - that i could follow up on over the
>weekend that would be fantastic.

At [0] is my quilt queue containing tags for releases between the
working and non-working release. As mentioned before,
timers-do-not-raise-softirq-unconditionally.patch will cause stalls
which is finally fixed by
timer-rt-Always-raise-the-softirq-if-there-s-irq_wor.patch.

Looking at my queue between -rt8 and now there is nothing big except for
the "raise softirq unconditional" patch.
If you decide to bisect the queue to find the offending post -rt8 patch
that causes trouble, please note that take these patches
| timers-do-not-raise-softirq-unconditionally.patch
| timer-Raise-softirq-if-there-s-irq_work.patch
| timer-rt-Always-raise-the-softirq-if-there-s-irq_wor.patch
should be applied together since the later two are fixes for the former.

[0] git://git.kernel.org/pub/scm/linux/kernel/git/bigeasy/rt-devel.git

>cheers
>
>Jordan

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[jordan]

Hi,

sorry for the late reply. but i have an update / tested the latest patchset.

On Fri, Feb 7, 2014 at 10:22 AM, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> * jordan | 2014-02-05 20:54:59 [-0500]:
>
>>On 3.12.8-rt11 - i was able to get the kernel to boot via reverting two
>>patches, from the split queue;
>>
>>'timer-Raise-softirq-if-there-s-irq_work.patch'
>>'timers-do-not-raise-softirq-unconditionally.patch'
>
> It was reported to the list that
> timers-do-not-raise-softirq-unconditionally.patch sometimes stalled the
> machine. The timer-Raise-softirq-if-there-s-irq_work.patch should have
> fixes one common case and -rt13 should have coverted them all.

Well, i know for my processor [AMD Phenom 965 x4] -rt11 definitely
failed to boot [likewise so did -rt13]. Here is another user reporting
boot failure, with the same CPU [he was reporting on -rt11, IIRC];
https://aur.archlinux.org/packages/linux-rt/ ...vvd writes:  "I am
having the same problem as Ninez has. This does not boot. I am also
running an AMD Phenom II 965 x4. "... I'm going to ask him about
-rt15, if he has had a chance to try it out.

>>(i haven't tried reverting on -rt13, as i haven't had the time to do that).
>
> It should work on -rt13 without reverting them.

No. -rt13 fails just the same...And just now, I have tested -rt15 -
which also fails to boot.

I am going to revert the patches that you suggested [below] and see if
i can get the machine to boot. However, this won't solve the problem -
as i package my kernel for others to use.

>>however, while this was enough to boot - my system was unstable. thus i
>>downgraded my kernel back to trusty 3.12.5-rt7 - which works flawlessly for
>>me. Next, I saw the -rt13 release, thinking maybe the boot issue was
>>finally solved [according to the change log], but no - it fails just as
>>before... I tried to add debug to grub's commandline but in the end, it
>>wasn't really helpful. [the last thing shown is something about PnP and
>>then ACPI];
>>
>>[    0.255329] pnp: PnP ACPI: found 10 devices
>>[    0.255330] ACPI: bus type PNP unregistered
>>
>>(which i would normally see anyway.)
>
> this looks like a missing event (like timer interrupt or RCU wakeup and
> "timers-do-not-raise-softi..." might be a source of this but then it
> should be fixed).

It's not fixed in -rt15 [or -rt13] for me. It fails just the same.

> At [0] is my quilt queue containing tags for releases between the
> working and non-working release. As mentioned before,
> timers-do-not-raise-softirq-unconditionally.patch will cause stalls
> which is finally fixed by
> timer-rt-Always-raise-the-softirq-if-there-s-irq_wor.patch.
>
> Looking at my queue between -rt8 and now there is nothing big except for
> the "raise softirq unconditional" patch.
> If you decide to bisect the queue to find the offending post -rt8 patch
> that causes trouble, please note that take these patches
> | timers-do-not-raise-softirq-unconditionally.patch
> | timer-Raise-softirq-if-there-s-irq_work.patch
> | timer-rt-Always-raise-the-softirq-if-there-s-irq_wor.patch
> should be applied together since the later two are fixes for the former.
>
> [0] git://git.kernel.org/pub/scm/linux/kernel/git/bigeasy/rt-devel.git

you would be looking at differences between -rt7 and now, as -rt8
fails to boot. [-rt7 is stable, -rt8+ doesn't boot].

When i get home tonight, i will revert those three patches and see if
i get a working kernel, but again - because i package my kernel [for
other users], this is problematic. [ie: i can't update packages until
i am sure that linux-rt patchset will boot for everyone okay - right
now, i know it won't boot for some and if i revert those 3 patches,
while it may boot for me/possibly those who it doesn't boot for - it's
likely going to cause issues for other users... it's a bit of a
pickle! ;) ].

anyway, let me know if there is anything i can do - as is - i will
report back if reverting those three patches allows my machine to at
least boot.

> Sebastian

Jordan

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[Sebastian Andrzej Siewior]

On 02/11/2014 04:16 PM, jordan wrote:
> Hi,

Hi Jordan,

> you would be looking at differences between -rt7 and now, as -rt8
> fails to boot. [-rt7 is stable, -rt8+ doesn't boot].
> 
> When i get home tonight, i will revert those three patches and see if
> i get a working kernel, but again - because i package my kernel [for
> other users], this is problematic. [ie: i can't update packages until
> i am sure that linux-rt patchset will boot for everyone okay - right
> now, i know it won't boot for some and if i revert those 3 patches,
> while it may boot for me/possibly those who it doesn't boot for - it's
> likely going to cause issues for other users... it's a bit of a
> pickle! ;) ].
> 
> anyway, let me know if there is anything i can do - as is - i will
> report back if reverting those three patches allows my machine to at
> least boot.

Okay. v3.12.5-rt7 is something I released with -RT changes. v3.12.6-rt8
has zero -RT changes, only stable increment. That means, the -RT patch
should be exactly the same (except for localversion).
So whatever breaks booting v3.12.6-rt8 is something that came via
stable queue.

> 
> Jordan
> 
Sebastian

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[jordan]

On Tue, Feb 11, 2014 at 10:37 AM, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> On 02/11/2014 04:16 PM, jordan wrote:
>> Hi,
>
> Hi Jordan,

Hey Sebastian

> Okay. v3.12.5-rt7 is something I released with -RT changes. v3.12.6-rt8
> has zero -RT changes, only stable increment. That means, the -RT patch
> should be exactly the same (except for localversion).
> So whatever breaks booting v3.12.6-rt8 is something that came via
> stable queue.

Okay, so now I have a place to start looking. :) So, while
mainline/vanilla 3.12.6+ (including 3.12.10) boots just fine - there
appears to be some change on mainline post 3.12.6 that is causing a
boot failure [but only triggered] on -rt. [?]

I'll have a look and see if any obvious commits stands out, but i
guess i will likely have to do git-bisect to find the bad commit... I
don't know if i will have time tonight, but i should be able to find
some time over the next few days.

thanks

> Sebastian

Jordan

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[Kimmo Taskinen]

Hi,

Phenom has had boot problems with all patches after 3.12.5-rt7.

Today I noticed that an old Turion64 laptop didn't boot even with 
3.12.5-rt7. I tried 3.10.32-rt30 and that worked fine.

Cheers
Kimmo

On 11.02.2014 17:37, Sebastian Andrzej Siewior wrote:
> On 02/11/2014 04:16 PM, jordan wrote:
>> Hi,
> Hi Jordan,
>
>> you would be looking at differences between -rt7 and now, as -rt8
>> fails to boot. [-rt7 is stable, -rt8+ doesn't boot].
>>
>> When i get home tonight, i will revert those three patches and see if
>> i get a working kernel, but again - because i package my kernel [for
>> other users], this is problematic. [ie: i can't update packages until
>> i am sure that linux-rt patchset will boot for everyone okay - right
>> now, i know it won't boot for some and if i revert those 3 patches,
>> while it may boot for me/possibly those who it doesn't boot for - it's
>> likely going to cause issues for other users... it's a bit of a
>> pickle! ;) ].
>>
>> anyway, let me know if there is anything i can do - as is - i will
>> report back if reverting those three patches allows my machine to at
>> least boot.
> Okay. v3.12.5-rt7 is something I released with -RT changes. v3.12.6-rt8
> has zero -RT changes, only stable increment. That means, the -RT patch
> should be exactly the same (except for localversion).
> So whatever breaks booting v3.12.6-rt8 is something that came via
> stable queue.
>
>> Jordan
>>
> Sebastian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[Sebastian Andrzej Siewior]

On 03/02/2014 12:39 PM, Kimmo Taskinen wrote:
> Hi,

Hi,

> Phenom has had boot problems with all patches after 3.12.5-rt7.

The difference between 3.12.5-rt7 and v3.12.6-rt8 is just the stable
.6. I don't have anything close to the hardware to look at this myself.

> Today I noticed that an old Turion64 laptop didn't boot even with
> 3.12.5-rt7. I tried 3.10.32-rt30 and that worked fine.
> 
> Cheers
> Kimmo

Sebastian

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[Kimmo Taskinen]

On 07.03.2014 13:54, Sebastian Andrzej Siewior wrote:
> On 03/02/2014 12:39 PM, Kimmo Taskinen wrote:
>> Hi,
> Hi,
>
>> Phenom has had boot problems with all patches after 3.12.5-rt7.
> The difference between 3.12.5-rt7 and v3.12.6-rt8 is just the stable
> .6. I don't have anything close to the hardware to look at this myself.
>
>> Today I noticed that an old Turion64 laptop didn't boot even with
>> 3.12.5-rt7. I tried 3.10.32-rt30 and that worked fine.
>>
>> Cheers
>> Kimmo
> Sebastian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Hi,

I found out that the reason for the Turion boot problem is 
CONFIG_X86_POWERNOW_K8. As soon as I disabled it Turion booted fine.
And this test was done with linux-3.12.13-aufs-rt21.

However the Phenom (AMD Phenom II X4 910e) boot problem seems to be 
something else and it remains. I disabled CONFIG_CPU_FREQ completely but 
that didn't help.

Cheers
Kimmo

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[Kimmo Taskinen]

Hi,

Phenom has had boot problems with all patches after 3.12.5-rt7.

Today I noticed that an old Turion64 laptop didn't boot even with 
3.12.5-rt7. I tried 3.10.32-rt30 and that worked fine.

Cheers
Kimmo

On 11.02.2014 17:16, jordan wrote:
> Hi,
>
> sorry for the late reply. but i have an update / tested the latest patchset.
>
> On Fri, Feb 7, 2014 at 10:22 AM, Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> wrote:
>> * jordan | 2014-02-05 20:54:59 [-0500]:
>>
>>> On 3.12.8-rt11 - i was able to get the kernel to boot via reverting two
>>> patches, from the split queue;
>>>
>>> 'timer-Raise-softirq-if-there-s-irq_work.patch'
>>> 'timers-do-not-raise-softirq-unconditionally.patch'
>> It was reported to the list that
>> timers-do-not-raise-softirq-unconditionally.patch sometimes stalled the
>> machine. The timer-Raise-softirq-if-there-s-irq_work.patch should have
>> fixes one common case and -rt13 should have coverted them all.
> Well, i know for my processor [AMD Phenom 965 x4] -rt11 definitely
> failed to boot [likewise so did -rt13]. Here is another user reporting
> boot failure, with the same CPU [he was reporting on -rt11, IIRC];
> https://aur.archlinux.org/packages/linux-rt/ ...vvd writes:  "I am
> having the same problem as Ninez has. This does not boot. I am also
> running an AMD Phenom II 965 x4. "... I'm going to ask him about
> -rt15, if he has had a chance to try it out.
>
>>> (i haven't tried reverting on -rt13, as i haven't had the time to do that).
>> It should work on -rt13 without reverting them.
> No. -rt13 fails just the same...And just now, I have tested -rt15 -
> which also fails to boot.
>
> I am going to revert the patches that you suggested [below] and see if
> i can get the machine to boot. However, this won't solve the problem -
> as i package my kernel for others to use.
>
>>> however, while this was enough to boot - my system was unstable. thus i
>>> downgraded my kernel back to trusty 3.12.5-rt7 - which works flawlessly for
>>> me. Next, I saw the -rt13 release, thinking maybe the boot issue was
>>> finally solved [according to the change log], but no - it fails just as
>>> before... I tried to add debug to grub's commandline but in the end, it
>>> wasn't really helpful. [the last thing shown is something about PnP and
>>> then ACPI];
>>>
>>> [    0.255329] pnp: PnP ACPI: found 10 devices
>>> [    0.255330] ACPI: bus type PNP unregistered
>>>
>>> (which i would normally see anyway.)
>> this looks like a missing event (like timer interrupt or RCU wakeup and
>> "timers-do-not-raise-softi..." might be a source of this but then it
>> should be fixed).
> It's not fixed in -rt15 [or -rt13] for me. It fails just the same.
>
>> At [0] is my quilt queue containing tags for releases between the
>> working and non-working release. As mentioned before,
>> timers-do-not-raise-softirq-unconditionally.patch will cause stalls
>> which is finally fixed by
>> timer-rt-Always-raise-the-softirq-if-there-s-irq_wor.patch.
>>
>> Looking at my queue between -rt8 and now there is nothing big except for
>> the "raise softirq unconditional" patch.
>> If you decide to bisect the queue to find the offending post -rt8 patch
>> that causes trouble, please note that take these patches
>> | timers-do-not-raise-softirq-unconditionally.patch
>> | timer-Raise-softirq-if-there-s-irq_work.patch
>> | timer-rt-Always-raise-the-softirq-if-there-s-irq_wor.patch
>> should be applied together since the later two are fixes for the former.
>>
>> [0] git://git.kernel.org/pub/scm/linux/kernel/git/bigeasy/rt-devel.git
> you would be looking at differences between -rt7 and now, as -rt8
> fails to boot. [-rt7 is stable, -rt8+ doesn't boot].
>
> When i get home tonight, i will revert those three patches and see if
> i get a working kernel, but again - because i package my kernel [for
> other users], this is problematic. [ie: i can't update packages until
> i am sure that linux-rt patchset will boot for everyone okay - right
> now, i know it won't boot for some and if i revert those 3 patches,
> while it may boot for me/possibly those who it doesn't boot for - it's
> likely going to cause issues for other users... it's a bit of a
> pickle! ;) ].
>
> anyway, let me know if there is anything i can do - as is - i will
> report back if reverting those three patches allows my machine to at
> least boot.
>
>> Sebastian
> Jordan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux-rt fails to boot [any release past 3.12.5-rt7] on all of my machines

[Pavel Vasilyev]

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

02.03.2014 15:13, Kimmo Taskinen пишет:
> Hi,
> 
> Phenom has had boot problems with all patches after 3.12.5-rt7.
> 
> Today I noticed that an old Turion64 laptop didn't boot even with 3.12.5-rt7. I
> tried 3.10.32-rt30 and that worked fine.


I have yet three or four months ago that most x86 systems, almost all of the
patches to the kernel 3.12 do not work.

Partially or accidentally only work with debug information: if enable
CONFIG_DEBUG_KERNEL and DEBUG_LATENCYTOP,.... and other.


See mailing list history.

-- 

                                                         Pavel.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]