boot hang: async vs. kexec

boot hang: async vs. kexec

[Randy Dunlap]

I (try to) do daily build/boot testing.  The newly built kernel
is booted via kexec.  This was working until sometime between
2.6.28 and 2.6.29-rc1, so I bisected it.*

git bisect blames this commit:

96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634 is first bad commit
commit 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634
Author: Dave Kleikamp <shaggy@linux.vnet.ibm.com>
Date:   Thu Jan 8 09:46:31 2009 -0600

    async: Don't call async_synchronize_full_special() while holding sb_lock
    
    sync_filesystems() shouldn't be calling async_synchronize_full_special
    while holding a spinlock.  The second while loop in that function is the
    right place for this anyway.


The new/kexec-loaded kernel hangs during initcalls.  The last one that
I can see (via netconsole, might miss a few of the very last lines) is:

calling  net_ns_init+0x0/0x14d @ 1
net_namespace: 1008 bytes
initcall net_ns_init+0x0/0x14d returned 0 after 0 usecs



Any ideas/suggestions?
Thanks.



*caveat: This was all done with the "don't use gcc 4.1.[01]
because it miscompiles __weak" patch reverted.  Could that
be an issue/problem here?  (I'm using gcc 4.1.1.)

-- 
~Randy

Re: boot hang: async vs. kexec

[Dave Kleikamp]

On Thu, 2009-01-29 at 13:15 -0800, Randy Dunlap wrote:
> I (try to) do daily build/boot testing.  The newly built kernel
> is booted via kexec.  This was working until sometime between
> 2.6.28 and 2.6.29-rc1, so I bisected it.*
> 
> git bisect blames this commit:
> 
> 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634 is first bad commit
> commit 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634
> Author: Dave Kleikamp <shaggy@linux.vnet.ibm.com>
> Date:   Thu Jan 8 09:46:31 2009 -0600
> 
>     async: Don't call async_synchronize_full_special() while holding sb_lock
>     
>     sync_filesystems() shouldn't be calling async_synchronize_full_special
>     while holding a spinlock.  The second while loop in that function is the
>     right place for this anyway.
> 
> 
> The new/kexec-loaded kernel hangs during initcalls.  The last one that
> I can see (via netconsole, might miss a few of the very last lines) is:
> 
> calling  net_ns_init+0x0/0x14d @ 1
> net_namespace: 1008 bytes
> initcall net_ns_init+0x0/0x14d returned 0 after 0 usecs
> 
> 
> 
> Any ideas/suggestions?

I'm not sure about any limitations of git bisect, but it seems unlikely
to me that sync_filesystems() would be getting called this early.  Are
any filesystems even mounted at this point?

Does reverting that commit fix the problem?  (I would be surprised, but
stranger things have happened.)

> Thanks.
> 
> 
> 
> *caveat: This was all done with the "don't use gcc 4.1.[01]
> because it miscompiles __weak" patch reverted.  Could that
> be an issue/problem here?  (I'm using gcc 4.1.1.)

I have no idea.

Shaggy
-- 
David Kleikamp
IBM Linux Technology Center

Re: boot hang: async vs. kexec

[Randy Dunlap]

Dave Kleikamp wrote:
> On Thu, 2009-01-29 at 13:15 -0800, Randy Dunlap wrote:
>> I (try to) do daily build/boot testing.  The newly built kernel
>> is booted via kexec.  This was working until sometime between
>> 2.6.28 and 2.6.29-rc1, so I bisected it.*
>>
>> git bisect blames this commit:
>>
>> 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634 is first bad commit
>> commit 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634
>> Author: Dave Kleikamp <shaggy@linux.vnet.ibm.com>
>> Date:   Thu Jan 8 09:46:31 2009 -0600
>>
>>     async: Don't call async_synchronize_full_special() while holding sb_lock
>>     
>>     sync_filesystems() shouldn't be calling async_synchronize_full_special
>>     while holding a spinlock.  The second while loop in that function is the
>>     right place for this anyway.
>>
>>
>> The new/kexec-loaded kernel hangs during initcalls.  The last one that
>> I can see (via netconsole, might miss a few of the very last lines) is:
>>
>> calling  net_ns_init+0x0/0x14d @ 1
>> net_namespace: 1008 bytes
>> initcall net_ns_init+0x0/0x14d returned 0 after 0 usecs
>>
>>
>>
>> Any ideas/suggestions?
> 
> I'm not sure about any limitations of git bisect, but it seems unlikely
> to me that sync_filesystems() would be getting called this early.  Are
> any filesystems even mounted at this point?

I don't think so.

> Does reverting that commit fix the problem?  (I would be surprised, but
> stranger things have happened.)

I was also skeptical, and reverting it made no difference.


>> Thanks.
>>
>>
>>
>> *caveat: This was all done with the "don't use gcc 4.1.[01]
>> because it miscompiles __weak" patch reverted.  Could that
>> be an issue/problem here?  (I'm using gcc 4.1.1.)
> 
> I have no idea.

I am now using gcc 4.1.2 and seeing the same boot hang problem.

-- 
~Randy